# Chunker

## Chunking

Text chunking consists of dividing a text in syntactically correlated parts of words, like noun groups, verb groups, but does not specify their internal structure, nor their role in the main sentence.

### Chunker Tool

The easiest way to try out the Chunker is the command line tool. The tool is only intended for demonstration and testing.

Download the English maxent chunker model from the website and start the Chunker Tool with this command:

    $ opennlp ChunkerME en-chunker.bin

The Chunker now reads a pos tagged sentence per line from stdin. Copy these two sentences to the console:

    Rockwell_NNP International_NNP Corp._NNP 's_POS Tulsa_NNP unit_NN said_VBD it_PRP signed_VBD
        a_DT tentative_JJ agreement_NN extending_VBG its_PRP$ contract_NN with_IN Boeing_NNP Co._NNP
        to_TO provide_VB structural_JJ parts_NNS for_IN Boeing_NNP 's_POS 747_CD jetliners_NNS ._.
    Rockwell_NNP said_VBD the_DT agreement_NN calls_VBZ for_IN it_PRP to_TO supply_VB 200_CD
        additional_JJ so-called_JJ shipsets_NNS for_IN the_DT planes_NNS ._.

The Chunker will now echo the sentences grouped tokens to the console:

    [NP Rockwell_NNP International_NNP Corp._NNP ] [NP 's_POS Tulsa_NNP unit_NN ] [VP said_VBD ]
        [NP it_PRP ] [VP signed_VBD ] [NP a_DT tentative_JJ agreement_NN ] [VP extending_VBG ]
        [NP its_PRP$ contract_NN ] [PP with_IN ] [NP Boeing_NNP Co._NNP ] [VP to_TO provide_VB ]
        [NP structural_JJ parts_NNS ] [PP for_IN ] [NP Boeing_NNP ] [NP 's_POS 747_CD jetliners_NNS ] ._.
    [NP Rockwell_NNP ] [VP said_VBD ] [NP the_DT agreement_NN ] [VP calls_VBZ ] [SBAR for_IN ]
        [NP it_PRP ] [VP to_TO supply_VB ] [NP 200_CD additional_JJ so-called_JJ shipsets_NNS ]
        [PP for_IN ] [NP the_DT planes_NNS ] ._.

The tag set used by the English pos model is the Penn Treebank tag set .

### Chunking API

The Chunker can be embedded into an application via its API. First the chunker model must be loaded into memory from disk or another source. In the sample below it is loaded from disk.

    InputStream modelIn = null;
    ChunkerModel model = null;

    try (modelIn = new FileInputStream("en-chunker.bin")){
      model = new ChunkerModel(modelIn);
    }

After the model is loaded a Chunker can be instantiated.

    ChunkerME chunker = new ChunkerME(model);

The Chunker instance is now ready to tag data. It expects a tokenized sentence as input, which is represented as a String array, each String object in the array is one token, and the POS tags associated with each token. As of OpenNLP 3.0.0 ChunkerME is thread safe and a single instance can be shared across threads; the legacy ThreadSafeChunkerME wrapper is retained for backward compatibility and is now deprecated.

The following code shows how to determine the most likely chunk tag sequence for a sentence.

    String[] sent = new String[] { "Rockwell", "International", "Corp.", "'s",
        "Tulsa", "unit", "said", "it", "signed", "a", "tentative", "agreement",
        "extending", "its", "contract", "with", "Boeing", "Co.", "to",
        "provide", "structural", "parts", "for", "Boeing", "'s", "747",
        "jetliners", "." };

    String[] pos = new String[] { "NNP", "NNP", "NNP", "POS", "NNP", "NN",
        "VBD", "PRP", "VBD", "DT", "JJ", "NN", "VBG", "PRP$", "NN", "IN",
        "NNP", "NNP", "TO", "VB", "JJ", "NNS", "IN", "NNP", "POS", "CD", "NNS",
        "." };

    String[] tag = chunker.chunk(sent, pos);

The tags array contains one chunk tag for each token in the input array. The corresponding tag can be found at the same index as the token has in the input array. The confidence scores for the returned tags can be easily retrieved from a ChunkerME with the following method call:

    double[] probs = chunker.probs();

The call to probs is stateful and will always return the probabilities of the last tagged sentence. When the chunker is shared across threads, the "last tagged sentence" is tracked per thread, so each thread must call chunk before calling probs on that same thread. The probs method should only be called when the chunk method was called before on the current thread, otherwise the behavior is undefined.

Some applications need to retrieve the n-best chunk tag sequences and not only the best sequence. The topKSequences method is capable of returning the top sequences. It can be called in a similar way as chunk.

    Sequence[] topSequences = chunk.topKSequences(sent, pos);

Each Sequence object contains one sequence. The sequence can be retrieved via Sequence.getOutcomes() which returns a tags array and Sequence.getProbs() returns the probability array for this sequence.

## Chunker Training

The pre-trained models might not be available for a desired language, can not detect important entities or the performance is not good enough outside the news domain.

These are the typical reason to do custom training of the chunker on a new corpus or on a corpus which is extended by private training data taken from the data which should be analyzed.

The training data can be converted to the OpenNLP chunker training format, which is based on CoNLL2000. Other formats may also be available. The training data consist of three columns separated one single space. Each word has been put on a separate line and there is an empty line after each sentence. The first column contains the current word, the second its part-of-speech tag and the third its chunk tag. The chunk tags contain the name of the chunk type, for example I-NP for noun phrase words and I-VP for verb phrase words. Most chunk types have two types of chunk tags, B-CHUNK for the first word of the chunk and I-CHUNK for each other word in the chunk. Here is an example of the file format:

Sample sentence of the training data:

    He        PRP  B-NP
    reckons   VBZ  B-VP
    the       DT   B-NP
    current   JJ   I-NP
    account   NN   I-NP
    deficit   NN   I-NP
    will      MD   B-VP
    narrow    VB   I-VP
    to        TO   B-PP
    only      RB   B-NP
    #         #    I-NP
    1.8       CD   I-NP
    billion   CD   I-NP
    in        IN   B-PP
    September NNP  B-NP
    .         .    O

For improved visualization the example above uses tabs instead of a single space as column separator.

### Training Tool

OpenNLP has a command line tool which is used to train the models available from the model download page on various corpora.

Usage of the tool:

    $ opennlp ChunkerTrainerME
    Usage: opennlp ChunkerTrainerME[.ad] [-params paramsFile] [-iterations num] [-cutoff num] \
                   -model modelFile -lang language -data sampleData [-encoding charsetName]

    Arguments description:
            -params paramsFile
                    training parameters file.
            -iterations num
                    number of training iterations, ignored if -params is used.
            -cutoff num
                    minimal number of times a feature must be seen, ignored if -params is used.
            -model modelFile
                    output model file.
            -lang language
                    language which is being processed.
            -data sampleData
                    data to be used, usually a file name.
            -encoding charsetName
                    encoding for reading and writing text, if absent the system default is used.

Its now assumed that the English chunker model should be trained from a file called en-chunker.train which is encoded as UTF-8. The following command will train the name finder and write the model to en-chunker.bin:

    $ opennlp ChunkerTrainerME -model en-chunker.bin -lang en -data en-chunker.train -encoding UTF-8

Additionally its possible to specify the number of iterations, the cutoff and to overwrite all types in the training data with a single type.

### Training API

The Chunker offers an API to train a new chunker model. The following sample code illustrates how to do it:

    ObjectStream<String> lineStream =
        new PlainTextByLineStream(new MarkableFileInputStreamFactory(new File("en-chunker.train")), StandardCharsets.UTF_8);

    ChunkerModel model;
    try(ObjectStream<ChunkSample> sampleStream = new ChunkSampleStream(lineStream)) {
      	model = ChunkerME.train("eng", sampleStream, TrainingParameters.defaultParams(), new ChunkerFactory());
    }

    try (OutputStream modelOut = new BufferedOutputStream(new FileOutputStream(modelFile))) {
      	model.serialize(modelOut);
    }

## Chunker Evaluation

The built-in evaluation can measure the chunker performance. The performance is either measured on a test dataset or via cross validation.

### Chunker Evaluation Tool

The following command shows how the tool can be run:

    $ opennlp ChunkerEvaluator
    Usage: opennlp ChunkerEvaluator[.ad] -model model [-misclassified true|false] \
                   [-detailedF true|false] -lang language -data sampleData [-encoding charsetName]

A sample of the command considering you have a data sample named en-chunker.eval and you trained a model called en-chunker.bin:

    $ opennlp ChunkerEvaluator -model en-chunker.bin -data en-chunker.eval -encoding UTF-8

and here is a sample output:

    Precision: 0.9255923572240226
    Recall: 0.9220610430991112
    F-Measure: 0.9238233255623465

You can also use the tool to perform 10-fold cross validation of the Chunker. The following command shows how the tool can be run:

    $ opennlp ChunkerCrossValidator
    Usage: opennlp ChunkerCrossValidator[.ad] [-params paramsFile] [-iterations num] [-cutoff num] \
                   [-misclassified true|false] [-folds num] [-detailedF true|false] \
                   -lang language -data sampleData [-encoding charsetName]

    Arguments description:
            -params paramsFile
                    training parameters file.
            -iterations num
                    number of training iterations, ignored if -params is used.
            -cutoff num
                    minimal number of times a feature must be seen, ignored if -params is used.
            -misclassified true|false
                    if true will print false negatives and false positives.
            -folds num
                    number of folds, default is 10.
            -detailedF true|false
                    if true will print detailed FMeasure results.
            -lang language
                    language which is being processed.
            -data sampleData
                    data to be used, usually a file name.
            -encoding charsetName
                    encoding for reading and writing text, if absent the system default is used.

It is not necessary to pass a model. The tool will automatically split the data to train and evaluate:

    $ opennlp ChunkerCrossValidator -lang pt -data en-chunker.cross -encoding UTF-8