# Tokenizer

## Tokenization

The OpenNLP Tokenizers segment an input character sequence into tokens. Tokens are usually words, punctuation, numbers, etc.

    Pierre Vinken, 61 years old, will join the board as a nonexecutive director Nov. 29.
    Mr. Vinken is chairman of Elsevier N.V., the Dutch publishing group.
    Rudolph Agnew, 55 years old and former chairman of Consolidated Gold Fields
        PLC, was named a director of this British industrial conglomerate.

The following result shows the individual tokens in a whitespace separated representation.

    Pierre Vinken , 61 years old , will join the board as a nonexecutive director Nov. 29 .
    Mr. Vinken is chairman of Elsevier N.V. , the Dutch publishing group .
    Rudolph Agnew , 55 years old and former chairman of Consolidated Gold Fields PLC ,
        was named a nonexecutive director of this British industrial conglomerate .
    A form of asbestos once used to make Kent cigarette filters has caused a high
        percentage of cancer deaths among a group of workers exposed to it more than 30 years ago ,
        researchers reported .

OpenNLP offers multiple tokenizer implementations:

- Whitespace Tokenizer - A whitespace tokenizer, non whitespace sequences are identified as tokens

- Simple Tokenizer - A character class tokenizer, sequences of the same character class are tokens

- Learnable Tokenizer - A maximum entropy tokenizer, detects token boundaries based on probability model

Most part-of-speech taggers, parsers and so on, work with text tokenized in this manner. It is important to ensure that your tokenizer produces tokens of the type expected by your later text processing components.

With OpenNLP (as with many systems), tokenization is a two-stage process: first, sentence boundaries are identified, then tokens within each sentence are identified.

### Tokenizer Tools

The easiest way to try out the tokenizers are the command line tools. The tools are only intended for demonstration and testing.

There are two tools, one for the Simple Tokenizer and one for the learnable tokenizer. A command line tool the for the Whitespace Tokenizer does not exist, because the whitespace separated output would be identical to the input.

The following command shows how to use the Simple Tokenizer Tool.

    $ opennlp SimpleTokenizer

To use the learnable tokenizer download the english token model from our website.

    $ opennlp TokenizerME en-token.bin

To test the tokenizer copy the sample from above to the console. The whitespace separated tokens will be written back to the console.

Usually the input is read from a file and written to a file.

    $ opennlp TokenizerME en-token.bin < article.txt > article-tokenized.txt

It can be done in the same way for the Simple Tokenizer.

Since most text comes truly raw and doesn't have sentence boundaries and such, its possible to create a pipe which first performs sentence boundary detection and tokenization. The following sample illustrates that.

    $ opennlp SentenceDetector sentdetect.model < article.txt | opennlp TokenizerME tokenize.model | more
    Loading model ... Loading model ... done
    done
    Showa Shell gained 20 to 1,570 and Mitsubishi Oil rose 50 to 1,500.
    Sumitomo Metal Mining fell five yen to 692 and Nippon Mining added 15 to 960 .
    Among other winners Wednesday was Nippon Shokubai , which was up 80 at 2,410 .
    Marubeni advanced 11 to 890 .
    London share prices were bolstered largely by continued gains on Wall Street and technical
        factors affecting demand for London 's blue-chip stocks .
    ...etc...

Of course this is all on the command line. Many people use the models directly in their Java code by creating SentenceDetector and Tokenizer objects and calling their methods as appropriate. The following section will explain how the Tokenizers can be used directly from java.

### Tokenizer API

The Tokenizers can be integrated into an application by the defined API. The shared instance of the WhitespaceTokenizer can be retrieved from a static field WhitespaceTokenizer.INSTANCE. The shared instance of the SimpleTokenizer can be retrieved in the same way from SimpleTokenizer.INSTANCE. To instantiate the TokenizerME (the learnable tokenizer) a Token Model must be created first. The following code sample shows how a model can be loaded.

    try (InputStream modelIn = new FileInputStream("en-token.bin")) {
      TokenizerModel model = new TokenizerModel(modelIn);
    }

After the model is loaded the TokenizerME can be instantiated.

    Tokenizer tokenizer = new TokenizerME(model);

The tokenizer offers two tokenize methods, both expect an input String object which contains the untokenized text. If possible it should be a sentence, but depending on the training of the learnable tokenizer this is not required. The first returns an array of Strings, where each String is one token.

    String tokens[] = tokenizer.tokenize("An input sample sentence.");

The output will be an array with these tokens.

    "An", "input", "sample", "sentence", "."

The second method, tokenizePos returns an array of Spans, each Span contain the begin and end character offsets of the token in the input String.

    Span tokenSpans[] = tokenizer.tokenizePos("An input sample sentence.");

The tokenSpans array now contain 5 elements. To get the text for one span call Span.getCoveredText which takes a span and the input text. The TokenizerME is able to output the probabilities for the detected tokens. The getTokenProbabilities method must be called directly after one of the tokenize methods was called.

    TokenizerME tokenizer = ...

    String tokens[] = tokenizer.tokenize(...);
    double tokenProbs[] = tokenizer.getTokenProbabilities();

The tokenProbs array now contains one double value per token, the value is between 0 and 1, where 1 is the highest possible probability and 0 the lowest possible probability.

## Tokenizer Training

### Training Tool

OpenNLP has a command line tool which is used to train the models available from the model download page on various corpora. The data can be converted to the OpenNLP Tokenizer training format or used directly. The OpenNLP format contains one sentence per line. Tokens are either separated by a whitespace or by a special <SPLIT> tag. Tokens are split automaticaly on whitespace and at least one <SPLIT> tag must be present in the training text. The following sample shows the sample from above in the correct format.

    Pierre Vinken<SPLIT>, 61 years old<SPLIT>, will join the board as a nonexecutive director Nov. 29<SPLIT>.
    Mr. Vinken is chairman of Elsevier N.V.<SPLIT>, the Dutch publishing group<SPLIT>.
    Rudolph Agnew<SPLIT>, 55 years old and former chairman of Consolidated Gold Fields PLC<SPLIT>,
        was named a nonexecutive director of this British industrial conglomerate<SPLIT>.

Usage of the tool:

    $ opennlp TokenizerTrainer
    Usage: opennlp TokenizerTrainer[.namefinder|.conllx|.pos] [-abbDict path] \
                    [-alphaNumOpt isAlphaNumOpt] [-params paramsFile] [-iterations num] \
                    [-cutoff num] -model modelFile -lang language -data sampleData \
                    [-encoding charsetName]

    Arguments description:
            -abbDict path
                    abbreviation dictionary in XML format.
            -alphaNumOpt isAlphaNumOpt
                    Optimization flag to skip alpha numeric tokens for further tokenization
            -params paramsFile
                    training parameters file.
            -iterations num
                    number of training iterations, ignored if -params is used.
            -cutoff num
                    minimal number of times a feature must be seen, ignored if -params is used.
            -model modelFile
                    output model file.
            -lang language
                    language which is being processed.
            -data sampleData
                    data to be used, usually a file name.
            -encoding charsetName
                    encoding for reading and writing text, if absent the system default is used.

To train the english tokenizer use the following command:

    $ opennlp TokenizerTrainer -model en-token.bin -alphaNumOpt -lang en -data en-token.train -encoding UTF-8

    Indexing events using cutoff of 5

    	Computing event counts...  done. 262271 events
    	Indexing...  done.
    Sorting and merging events... done. Reduced 262271 events to 59060.
    Done indexing.
    Incorporating indexed data for training...
    done.
    	Number of Event Tokens: 59060
    	    Number of Outcomes: 2
    	  Number of Predicates: 15695
    ...done.
    Computing model parameters...
    Performing 100 iterations.
      1:  .. loglikelihood=-181792.40419263614	0.9614292087192255
      2:  .. loglikelihood=-34208.094253153664	0.9629238459456059
      3:  .. loglikelihood=-18784.123872910015	0.9729211388220581
      4:  .. loglikelihood=-13246.88162585859	0.9856103038460219
      5:  .. loglikelihood=-10209.262670265718	0.9894422181636552

     ...<skipping a bunch of iterations>...

     95:  .. loglikelihood=-769.2107474529454	0.999511955191386
     96:  .. loglikelihood=-763.8891914534009	0.999511955191386
     97:  .. loglikelihood=-758.6685383254891	0.9995157680414533
     98:  .. loglikelihood=-753.5458314695236	0.9995157680414533
     99:  .. loglikelihood=-748.5182305519613	0.9995157680414533
    100:  .. loglikelihood=-743.5830058068038	0.9995157680414533
    Wrote tokenizer model.
    Path: en-token.bin

### Training API

The Tokenizer offers an API to train a new tokenization model. Basically three steps are necessary to train it:

- The application must open a sample data stream

- Call the TokenizerME.train method

- Save the TokenizerModel to a file or directly use it

The following sample code illustrates these steps:

    ObjectStream<String> lineStream = new PlainTextByLineStream(new FileInputStream("en-sent.train"),
        StandardCharsets.UTF_8);
    ObjectStream<TokenSample> sampleStream = new TokenSampleStream(lineStream);

    TokenizerModel model;

    try {
      model = TokenizerME.train("en", sampleStream, true, TrainingParameters.defaultParams());
    }
    finally {
      sampleStream.close();
    }

    OutputStream modelOut = null;
    try {
      modelOut = new BufferedOutputStream(new FileOutputStream(modelFile));
      model.serialize(modelOut);
    } finally {
      if (modelOut != null)
         modelOut.close();
    }

## Detokenizing

Detokenizing is simple the opposite of tokenization, the original non-tokenized string should be constructed out of a token sequence. The OpenNLP implementation was created to undo the tokenization of training data for the tokenizer. It can also be used to undo the tokenization of such a trained tokenizer. The implementation is strictly rule based and defines how tokens should be attached to a sentence wise character sequence.

The rule dictionary assign to every token an operation which describes how it should be attached to one continuous character sequence.

The following rules can be assigned to a token:

- MERGE_TO_LEFT - Merges the token to the left side.

- MERGE_TO_RIGHT - Merges the token to the right side.

- RIGHT_LEFT_MATCHING - Merges the token to the right side on first occurrence and to the left side on second occurrence.

The following sample will illustrate how the detokenizer with a small rule dictionary (illustration format, not the xml data format):

    . MERGE_TO_LEFT
    " RIGHT_LEFT_MATCHING

The dictionary should be used to de-tokenize the following whitespace tokenized sentence:

    He said " This is a test " .

The tokens would get these tags based on the dictionary:

    He -> NO_OPERATION
    said -> NO_OPERATION
    " -> MERGE_TO_RIGHT
    This -> NO_OPERATION
    is -> NO_OPERATION
    a -> NO_OPERATION
    test -> NO_OPERATION
    " -> MERGE_TO_LEFT
    . -> MERGE_TO_LEFT

That will result in the following character sequence:

    He said "This is a test".

### Detokenizing API

The Detokenizer can be used to detokenize the tokens to String. To instantiate the Detokenizer (a rule based detokenizer) a DetokenizationDictionary (the rule of dictionary) must be created first. The following code sample shows how a rule dictionary can be loaded.

    InputStream dictIn = new FileInputStream("latin-detokenizer.xml");
    DetokenizationDictionary dict = new DetokenizationDictionary(dictIn);

After the rule dictionary is loadeed the DictionaryDetokenizer can be instantiated.

    Detokenizer detokenizer = new DictionaryDetokenizer(dict);

The detokenizer offers two detokenize methods,the first detokenize the input tokens into a String.

    String[] tokens = new String[]{"A", "co", "-", "worker", "helped", "."};
    String sentence = detokenizer.detokenize(tokens, null);
    Assert.assertEquals("A co-worker helped.", sentence);

Tokens which are connected without a space in-between can be separated by a split marker.

    String sentence = detokenizer.detokenize(tokens, "<SPLIT>");
    Assert.assertEquals("A co<SPLIT>-<SPLIT>worker helped<SPLIT>.", sentence);

The API also offers a method which simply returns operations array in the input tokens array.

    DetokenizationOperation[] operations = detokenizer.detokenize(tokens);
    for (DetokenizationOperation operation : operations) {
      System.out.println(operation);
    }

Output:

    NO_OPERATION
    NO_OPERATION
    MERGE_BOTH
    NO_OPERATION
    NO_OPERATION
    MERGE_TO_LEFT

### Detokenizer Dictionary

Detokenization Dictionary is the rule dictionary about detokenizer. tokens - an array of tokens that should be detokenized according to an operation. operations - an array of operations which specifies which operation should be used for the provided tokens. The following code sample shows how a rule dictionary can be created.

    String[] tokens = new String[]{".", "!", "(", ")", "\"", "-"};
    Operation[] operations = new Operation[]{
        Operation.MOVE_LEFT,
        Operation.MOVE_LEFT,
        Operation.MOVE_RIGHT,
        Operation.MOVE_LEFT,
        Operation.RIGHT_LEFT_MATCHING,
        Operation.MOVE_BOTH};
    DetokenizationDictionary dict = new DetokenizationDictionary(tokens, operations);