# SpellChecker

## Overview

The OpenNLP SpellChecker is a native, language-agnostic spelling corrector built on the SymSpell (Symmetric Delete) algorithm. Given a (possibly misspelled) term it proposes corrections drawn from a frequency dictionary, ranked by edit distance and corpus frequency. It can also correct whole phrases, repairing wrongly inserted or omitted spaces (word splits and merges) with the help of a bigram language model.

SymSpell is fast because it pre-computes a deletes-only index: for every dictionary term it derives all strings obtained by deleting up to a small number of symbols and maps each such delete back to the originating terms. A query is answered by generating the deletes of the query and intersecting them with the index, which turns the costly fuzzy search into a handful of hash-map lookups. Candidates are then verified with an edit-distance metric. Because only deletes are generated (no insertions, substitutions or transpositions), candidate generation is orders of magnitude cheaper than a naïve search over the whole dictionary.

The SpellChecker ships as the optional `opennlp-spellcheck` extension module (package root `opennlp.spellcheck`). It is a good fit when you need to:

- clean up noisy user input (search queries, form fields, OCR output) before tokenization or classification;

- normalize a training or evaluation corpus as part of a preprocessing pipeline;

- offer interactive "did you mean" suggestions for a single term.

It is not a grammar checker and it does not require a trained statistical model – correction quality is determined entirely by the supplied frequency dictionary.

The algorithm and its compound-correction heuristic are an independent Java rewrite of the SymSpell reference implementation by Wolf Garbe (MIT). Attribution is recorded in the project NOTICE file.

## SpellChecker Tools

The extension provides two command line tools, dispatched through the `bin/spellcheck` launcher of the binary distribution: `SpellCheckModelBuilder` builds a binary model from plain-text frequency dictionaries, and `CorrectText` corrects text using such a model. Each tool prints its usage when invoked with the `help` parameter.

### Building a Model

A model is built from a unigram frequency list and, optionally, a bigram list (see for the file format). Usage of the tool:

    $ bin/spellcheck SpellCheckModelBuilder help
    Usage: opennlp-spellcheck SpellCheckModelBuilder -unigrams in [-bigrams in] -model out \
      [-maxEditDistance num] [-countThreshold num] [-distance damerau-osa|levenshtein] \
      [-corpusWordCount num] -lang lang [-prefixLength num] [-encoding charsetName]

    Arguments description:
    	-lang lang
    		The language tag of the model (e.g. en), stored as model.language.
    	-unigrams in
    		The unigram frequency dictionary file (word<TAB>count per line).
    	-bigrams in
    		An optional bigram frequency dictionary file (w1 w2<TAB>count per line).
    	-model out
    		The output file the binary SymSpell model is written to.
    	-maxEditDistance num
    		The largest precomputed dictionary edit distance. (default 2)
    	-prefixLength num
    		Number of leading symbols used for delete generation (must be > maxEditDistance). (default 7)
    	-countThreshold num
    		Minimum corpus count for a term to be indexed. (default 1)
    	-distance damerau-osa|levenshtein
    		The edit-distance metric used for verification. (default damerau-osa)
    	-corpusWordCount num
    		Corpus size N used for compound-correction scoring; 0 (default) derives N from the
    		dictionary's summed counts.
    	-encoding charsetName
    		encoding for reading and writing text, if absent the system default is used.

The following command builds an English model from a unigram and a bigram dictionary, using the default Damerau-OSA edit distance:

    $ bin/spellcheck SpellCheckModelBuilder -lang en \
      -unigrams en-unigrams.txt -bigrams en-bigrams.txt \
      -model en-spellcheck.bin -encoding UTF-8

### Correcting Text

Once a model has been built (or a packaged model is on the classpath), the `CorrectText` tool corrects text line by line. Usage of the tool:

    $ bin/spellcheck CorrectText help
    Usage: opennlp-spellcheck CorrectText [-model model] [-maxEditDistance num] [-lang lang] \
      [-suggest true|false] [-verbosity TOP|CLOSEST|ALL] [-compound true|false] \
      [-inputFile in] [-outputFile out] [-encoding charsetName]

    Arguments description:
    	-model model
    		A binary SymSpell model file. Either -model or -lang must be given.
    	-lang lang
    		Resolve a packaged spellcheck model for this language from the classpath.
    		Either -model or -lang must be given.
    	-inputFile in
    		The input text file (read line by line). If absent, standard input is used.
    	-outputFile out
    		The output text file. If absent, standard output is used.
    	-compound true|false
    		Correct each line as a whole phrase, repairing wrong word splits/merges. (default false)
    	-maxEditDistance num
    		The maximum edit distance considered per token. (default 2)
    	-suggest true|false
    		List the suggestions for each token (honoring -verbosity) instead of emitting
    		corrected text. (default false)
    	-verbosity TOP|CLOSEST|ALL
    		How many suggestions to list per token; only used with -suggest true.
    		TOP=best only, CLOSEST=all at the smallest distance, ALL=all within -maxEditDistance.
    		(default TOP)
    	-encoding charsetName
    		encoding for reading and writing text, if absent the system default is used.

By default each line is corrected token by token, reading from standard input and writing to standard output:

    $ echo "the qick brwn fox" | bin/spellcheck CorrectText -model en-spellcheck.bin
    the quick brown fox

With `-compound true` the whole line is corrected as a phrase, additionally repairing wrongly inserted or omitted spaces:

    $ echo "thequick brownfox" | bin/spellcheck CorrectText -model en-spellcheck.bin -compound true
    the quick brown fox

With `-suggest true` the tool lists the candidate suggestions for every token (honoring `-verbosity`) instead of emitting corrected text:

    $ echo "teh qick" | bin/spellcheck CorrectText -model en-spellcheck.bin -suggest true -verbosity CLOSEST
    teh => [the]
    qick => [quick, quack]

Instead of an explicit `-model` file you can resolve a packaged model from the classpath by language with `-lang` (see ):

    $ bin/spellcheck CorrectText -lang en -inputFile noisy.txt -outputFile clean.txt -encoding UTF-8

## SpellChecker API

The central abstraction is the `SpellChecker` interface; its default implementation is the `SymSpell` engine. The interface exposes two ways to correct input:

    public interface SpellChecker {

      // Suggestions for a single term, bounded by maxEditDistance.
      List<SuggestItem> lookup(String term, Verbosity verbosity, int maxEditDistance);

      // Convenience overload: Verbosity.TOP and the configured max edit distance.
      List<SuggestItem> lookup(String term);

      // Corrects a whole phrase, repairing word splits and merges.
      List<SuggestItem> lookupCompound(String input, int maxEditDistance);
    }

Each suggestion is an immutable `SuggestItem` record carrying the corrected term, the edit distance to the queried term and the corpus frequency of the suggestion. Its natural ordering sorts the best candidate first (ascending edit distance, then descending frequency), so `get(0)` is the top suggestion.

    public record SuggestItem(String term, int editDistance, long frequency)
        implements Comparable<SuggestItem> { }

The `Verbosity` enum controls how many suggestions a `lookup` call returns:

- `TOP` – the single best suggestion (cheapest to compute);

- `CLOSEST` – all suggestions sharing the smallest edit distance found;

- `ALL` – every suggestion within the requested maximum edit distance.

An engine is created from a `SymSpellConfig` and populated through `add` (unigrams) and `addBigram` (bigrams). In practice the engine is built from a dictionary or loaded from a model (see ), but it can also be filled programmatically:

    SymSpellConfig config = SymSpellConfig.builder()
        .maxDictionaryEditDistance(2)   // largest precomputed edit distance (default 2)
        .prefixLength(7)                // leading symbols used for deletes (default 7)
        .countThreshold(1)              // minimum count to index a term (default 1)
        .editDistance(DamerauOSADistance.INSTANCE) // verification metric (default)
        .build();

    SymSpell symSpell = new SymSpell(config);   // or: new SymSpell() for the defaults
    symSpell.add("quick", 1_000);
    symSpell.add("brown", 800);
    symSpell.addBigram("quick", "brown", 120);

Once populated, the engine is safe for concurrent `lookup` calls. Looking up a single term and reading the best suggestion:

    List<SuggestItem> suggestions = symSpell.lookup("qick", Verbosity.CLOSEST, 2);
    for (SuggestItem item : suggestions) {
      System.out.printf("%s (distance=%d, frequency=%d)%n",
          item.term(), item.editDistance(), item.frequency());
    }

Correcting a whole phrase, including wrongly split or merged words, with `lookupCompound` (which always returns a singleton list holding the best correction):

    String corrected = symSpell.lookupCompound("thequick brwn fox", 2).get(0).term();
    // -> "the quick brown fox"

### Choosing an Edit Distance

The verification metric is pluggable behind the `EditDistance` interface and selected through `SymSpellConfig`. Two implementations are provided:

- `DamerauOSADistance` (the default) – an Optimal String Alignment (restricted Damerau-Levenshtein) metric in which a transposition of two adjacent symbols counts as a single edit. It is Unicode-aware, operating on code points, and matches the variant used by the SymSpell reference implementation. Prefer it for natural-language text where transpositions ("teh" → "the") are common typos.

- `LevenshteinDistance` – a thin adapter over Apache Commons Text in which only insertions, deletions and substitutions count, so a transposition costs two edits. It operates on UTF-16 `char` units. Use it when you want plain Levenshtein semantics or to match another system's behavior.

Both expose a shared, stateless `INSTANCE` and are passed to the builder:

    SymSpellConfig config = SymSpellConfig.builder()
        .editDistance(LevenshteinDistance.INSTANCE)
        .build();

## Dictionary and Model Format

A SpellChecker is driven by plain-text frequency dictionaries, loaded into an engine by the `FrequencyDictionaryLoader`. Two UTF-8 (by default) line-based formats are supported. In both, columns are separated by a single TAB; blank lines, whitespace-only lines and lines starting with `#` (comments) are skipped.

The unigram dictionary lists one `word<TAB>count` per line, where `count` is the term's corpus frequency:

    the	23135851162
    of	13151942776
    and	12997637966
    quick	1885979
    brown	1019234
    fox	271289

The optional bigram dictionary lists one `word1 word2<TAB>count` per line (the two words separated by a space, then a TAB and the count). Bigrams improve compound correction:

    quick brown	12046
    brown fox	5634
    the quick	98765

A model is the built engine together with its source dictionaries and the metadata needed to identify it. The `SymSpellModels` facade builds, serializes and deserializes models, and writes the packaged form consumed by the model-resolver. The loaded model exposes a ready-to-query engine via `getSymSpell()`:

    InputStreamFactory unigrams = new MarkableFileInputStreamFactory(new File("en-unigrams.txt"));
    InputStreamFactory bigrams = new MarkableFileInputStreamFactory(new File("en-bigrams.txt"));

    SymSpellModel model = SymSpellModels.buildModel(
        "en", SymSpellConfig.defaultConfig(), StandardCharsets.UTF_8, unigrams, bigrams);

    // Persist the binary model.
    try (OutputStream out = Files.newOutputStream(Path.of("en-spellcheck.bin"))) {
      SymSpellModels.serialize(model, out);
    }

    // Reload it later and query the engine.
    try (InputStream in = Files.newInputStream(Path.of("en-spellcheck.bin"))) {
      SymSpellModel loaded = SymSpellModels.deserialize(in);
      SpellChecker checker = loaded.getSymSpell();
      List<SuggestItem> suggestions = checker.lookup("qick");
    }

Loading just the dictionaries into an existing engine – for example to add domain terms – can be done directly with the loader:

    SymSpell engine = new SymSpell(SymSpellConfig.defaultConfig());
    FrequencyDictionaryLoader loader = new FrequencyDictionaryLoader(StandardCharsets.UTF_8);
    loader.loadUnigrams(engine, unigrams);
    loader.loadBigrams(engine, bigrams);

## Multi-language Models

Production multi-language dictionaries are large and are therefore not bundled in the `opennlp-spellcheck` jar. Instead, each language ships as a separate Maven artifact named `opennlp-models-spellcheck-{lang}` (for example `opennlp-models-spellcheck-en`), packaged as a binary model plus a matching `model.properties` descriptor (`model.language`, `model.name`, `model.version`, `model.sha256`).

Add the model artifact for the language you need to your classpath:

    <dependency>
        <groupId>org.apache.opennlp</groupId>
        <artifactId>opennlp-models-spellcheck-en</artifactId>
        <version>CURRENT_OPENNLP_VERSION</version>
    </dependency>

The `SymSpellModelResolver` then discovers and loads the matching model using the standard OpenNLP model-resolver machinery (see ). It returns an `Optional` that is empty when no matching model jar is present:

    Optional<SymSpellModel> model = new SymSpellModelResolver().resolveByLanguage("en");
    SpellChecker checker = model.orElseThrow().getSymSpell();
    List<SuggestItem> suggestions = checker.lookup("qick");

On the command line the same resolution happens when you pass `-lang` instead of `-model` to `CorrectText`.

## Pipeline Integration

The SpellChecker integrates into OpenNLP preprocessing through the standard normalizer and stream abstractions, so correction can be slotted into an existing chain without bespoke glue code.

`SpellCheckingCharSequenceNormalizer` implements `opennlp.tools.util.normalizer.CharSequenceNormalizer` and can be composed with the other normalizers (URL, emoji, shrink, ...) inside an `AggregateCharSequenceNormalizer`. It works in one of two modes: `Mode.PER_TOKEN` (the default) corrects each whitespace-delimited token independently and preserves the line's whitespace and punctuation, while `Mode.COMPOUND` corrects the whole input as a phrase, repairing word splits and merges. It guards against over-correction (skipping short tokens, numbers and URL/email-like tokens, and never changing a word the dictionary already contains) and preserves the original casing pattern of each token.

    SymSpellModel model = SymSpellModels.deserialize(in);

    CharSequenceNormalizer normalizer = SpellCheckingCharSequenceNormalizer.builder(model)
        .mode(SpellCheckingCharSequenceNormalizer.Mode.PER_TOKEN)
        .maxEditDistance(2)
        .build();

    CharSequence clean = normalizer.normalize("the qick brwn fox"); // -> "the quick brown fox"

For stream-based preprocessing two `FilterObjectStream<String, String>` implementations wrap an existing `ObjectStream<String>` and correct each element as it is read:

- `SpellCorrectingObjectStream` corrects each line; in compound mode it may split or merge tokens. Use it for free text.

- `SpellCorrectingTokenStream` corrects tokenized data token-by-token and re-joins with the same delimiter. It is token-count preserving, so the corrected element stays aligned with any parallel annotation (tags, spans).

Dropping the corrector into a line-based reading chain:

    ObjectStream<String> lines =
        new PlainTextByLineStream(inputStreamFactory, StandardCharsets.UTF_8);
    ObjectStream<String> corrected = new SpellCorrectingObjectStream(lines, model);

    String line;
    while ((line = corrected.read()) != null) {
      // ... consume corrected lines ...
    }