# Stopword Filtering

## Overview

Stopwords are very common words (such as articles, prepositions, pronouns or conjunctions) that carry little semantic content of their own and are frequently removed prior to downstream processing such as indexing, classification, topic modeling, keyword extraction or similarity computation. Filtering stopwords can reduce noise, lower memory footprint and improve the signal-to-noise ratio of subsequent NLP components.

OpenNLP provides a small, dependency-free stopword filtering API in the `opennlp.tools.stopword` package. The central abstraction is the `StopwordFilter` interface, with a `Dictionary`-backed default implementation `DictionaryStopwordFilter`. Bundled stopword lists are available for eleven languages and can be loaded by ISO 639-1 code via the `StopwordLists` factory. Users may also load custom lists from any `InputStream`, mix them with the bundled defaults and add or remove individual entries at runtime.

OpenNLP further offers a tokenizer decorator (`StopwordFilteringTokenizer`) and an `ObjectStream` adapter (`StopwordFilterStream`) to plug stopword filtering into existing tokenization or training data pipelines, plus a command-line tool for ad-hoc filtering.

## Loading a bundled list

A bundled stopword list can be obtained from the `StopwordLists` factory by passing the desired ISO 639-1 language code. The returned filter is a `DictionaryStopwordFilter` that is case-insensitive by default.

    StopwordFilter filter = StopwordLists.forLanguage("en");

    boolean isStop = filter.isStopword("the");        // true
    String[] kept  = filter.filter(new String[] {
        "the", "quick", "brown", "fox"
    });
    // kept => { "quick", "brown", "fox" }

The set of supported languages can be queried at runtime:

    Set<String> supported = StopwordLists.supportedLanguages();

## Loading a custom list

Custom stopword lists can be loaded from any `InputStream` via `StopwordLists.load`. The method takes the input stream, a character set and a flag indicating whether matching should be case-sensitive.

    try (InputStream in = new FileInputStream("my-stopwords.txt")) {
      StopwordFilter filter = StopwordLists.load(
          in, StandardCharsets.UTF_8, false /* case-insensitive */);
    }

The expected file format is a plain UTF-8 text file with the following conventions:

- One stopword entry per line.

- Whitespace on a line separates the tokens of a multi-word entry, i.e. a line containing more than one token defines an n-gram stopword.

- Blank lines are ignored.

- Lines starting with `#` are treated as comments and ignored.

A small example file:

    # my custom English stopwords
    the
    a
    an

    # multi-word entries are defined by whitespace
    of the
    in spite of

## Extending or overriding defaults

A `DictionaryStopwordFilter` is immutable once constructed. To tailor a bundled list to a specific domain — for example to add project-specific noise terms or to retain a particular word that would otherwise be filtered — use the nested `DictionaryStopwordFilter.Builder`. The builder loads the bundled resource, layers user-supplied `add` / `remove` operations on top, and produces a fresh immutable filter from `build()`.

    InputStream bundled =
        StopwordLists.class.getResourceAsStream("/opennlp/tools/stopword/en.txt");

    StopwordFilter filter = DictionaryStopwordFilter.builder()
        .load(bundled, StandardCharsets.UTF_8)
        .add("foo")            // mark a new word as a stopword
        .remove("the")         // keep "the" in the output
        .build();

Bulk operations are also available via `addAll(Collection<String[]>)` and `removeAll(Collection<String[]>)`, which accept multi-token entries. `build()` applies all queued additions first, then all queued removals.

## Multi-word stopwords

Beyond single tokens, a `StopwordFilter` can match n-grams. The overloaded `isStopword(String...)` method accepts a sequence of tokens and returns `true` if the entire sequence is registered as a multi-word stopword entry.

    StopwordFilter filter = DictionaryStopwordFilter.builder()
        .add("of", "the")
        .add("in", "spite", "of")
        .build();

    filter.isStopword("of", "the");        // true
    filter.isStopword("in", "spite", "of"); // true
    filter.isStopword("of");                // false

Multi-word entries are matched as exact, contiguous sequences and are subject to the same case-sensitivity setting as the rest of the dictionary.

The `filter(String[])` method honors both 1-gram and n-gram entries. It performs a greedy, left-to-right window scan: at each position the longest registered window is tried first; if it matches, the entire window is dropped and scanning resumes after it. Otherwise the head token is kept and scanning advances by one. Tokens that are `null` are kept in place and never participate in a window match.

    StopwordFilter filter = DictionaryStopwordFilter.builder()
        .add("the")               // 1-gram
        .add("of", "the")         // 2-gram
        .add("in", "spite", "of") // 3-gram
        .build();

    String[] kept = filter.filter(new String[] {
        "the", "king", "of", "the", "hill",
        "won", "in", "spite", "of", "rain"
    });
    // kept => { "king", "hill", "won", "rain" }

When both a shorter and a longer registered entry could match at the same position, the longer match wins. If the longest window does not match, shorter windows down to length one are tried in turn.

## Decorating a Tokenizer

Any existing `Tokenizer` implementation can be wrapped in a `StopwordFilteringTokenizer` to transparently remove stopwords from its output. The decorator delegates tokenization to the wrapped instance and then runs the resulting token array through the provided `StopwordFilter`.

    StopwordFilter filter = StopwordLists.forLanguage("en");
    Tokenizer tokenizer = new StopwordFilteringTokenizer(
        SimpleTokenizer.INSTANCE, filter);

    String[] tokens = tokenizer.tokenize("The quick brown fox jumps over the lazy dog");
    // tokens => { "quick", "brown", "fox", "jumps", "lazy", "dog" }

The decorated tokenizer can be used anywhere a `Tokenizer` is expected, including in downstream OpenNLP pipelines.

## Streaming usage

For training-data and bulk-processing scenarios that already operate on `ObjectStream<String[]>` (for example tokenized sentences), the `StopwordFilterStream` adapter provides a drop-in filter that strips stopwords from each emitted token array.

    StopwordFilter filter = StopwordLists.forLanguage("en");

    try (ObjectStream<String[]> tokenStream = ...;
         ObjectStream<String[]> filtered = new StopwordFilterStream(tokenStream, filter)) {

      String[] tokens;
      while ((tokens = filtered.read()) != null) {
        // tokens no longer contains any stopwords
      }
    }

Empty arrays produced by removing every token of a short sentence are preserved so that the stream's record boundaries remain aligned with the source.

## Command-line usage

Stopword filtering is also exposed as a CLI tool. The `StopwordFilter` command reads whitespace-separated tokens from standard input and writes the non-stopword tokens to standard output. The single argument is either the ISO 639-1 code of a bundled list, or a path to a custom stopword list file (same format as the Java API: one entry per line, with `#` comments and blank lines ignored, loaded case-insensitively).

    $ echo "the quick brown fox" | opennlp StopwordFilter en
    quick brown fox

To use a custom list, pass its file path instead of a language code:

    $ echo "the quick brown fox" | opennlp StopwordFilter my-stopwords.txt

A bundled language code takes precedence over a file of the same name; to force loading a file whose name happens to be a language code, qualify it with a path, e.g. `./en`. The tool is intended for quick interactive checks and for use inside shell pipelines, for example chained behind a tokenizer:

    $ opennlp SimpleTokenizer < article.txt | opennlp StopwordFilter en > filtered.txt

## Bundled languages

The following stopword lists are bundled with OpenNLP and can be loaded by their ISO 639-1 code through `StopwordLists.forLanguage`:

ISO 639-1 code | Language
--- | ---
bg | Bulgarian
da | Danish
de | German
en | English
es | Spanish
fi | Finnish
fr | French
it | Italian
nl | Dutch
pt | Portuguese
ru | Russian

The bundled lists are derived from the snowball stopword files shipped with Apache Lucene and retain their original BSD license; see the `NOTICE` and `LICENSE` files for the attribution and license text.

## Thread-safety

`DictionaryStopwordFilter` is immutable once constructed and is therefore safe to share across threads without external synchronization. A filter returned by `StopwordLists.forLanguage(...)` or assembled via the `Builder` can be stored in a static field and accessed concurrently from any number of readers.

`StopwordFilteringTokenizer` and `StopwordFilterStream` are also stateless decorators with only `final` fields, so they inherit the thread-safety of the components they wrap. When paired with a `DictionaryStopwordFilter` and a thread-safe delegate tokenizer (e.g. `SimpleTokenizer.INSTANCE` or `WhitespaceTokenizer.INSTANCE`) the resulting pipeline is fully thread-safe.