# Document Categorizer

## Classifying

The OpenNLP Document Categorizer can classify text into pre-defined categories. It is based on maximum entropy framework. For someone interested in Gross Margin, the sample text given below could be classified as GMDecrease

    Major acquisitions that have a lower gross margin than the existing network
    also had a negative impact on the overall gross margin, but it should improve
    following the implementation of its integration strategies.

and the text below could be classified as GMIncrease

    The upward movement of gross margin resulted from amounts pursuant to
    adjustments to obligations towards dealers.

To be able to classify a text, the document categorizer needs a model. The classifications are requirements-specific and hence there is no pre-built model for document categorizer under OpenNLP project.

### Document Categorizer Tool

The easiest way to try out the document categorizer is the command line tool. The tool is only intended for demonstration and testing. The following command shows how to use the document categorizer tool.

    $ opennlp Doccat model

The input is read from standard input and output is written to standard output, unless they are redirected or piped. As with most components in OpenNLP, document categorizer expects input which is segmented into sentences.

### Document Categorizer API

To perform classification you will need a maxent model - these are encapsulated in the DoccatModel class of OpenNLP tools.

First you need to grab the bytes from the serialized model on an InputStream - we'll leave it you to do that, since you were the one who serialized it to begin with. Now for the easy part:

    InputStream is = ...
    DoccatModel m = new DoccatModel(is);

With the DoccatModel in hand we are just about there:

    String inputText = ...
    DocumentCategorizerME myCategorizer = new DocumentCategorizerME(m);
    double[] outcomes = myCategorizer.categorize(inputText);
    String category = myCategorizer.getBestCategory(outcomes);

## Training

The Document Categorizer can be trained on annotated training material. The data can be in OpenNLP Document Categorizer training format. This is one document per line, containing category and text separated by a whitespace. Other formats can also be available. The following sample shows the sample from above in the required format. Here GMDecrease and GMIncrease are the categories.

    GMDecrease Major acquisitions that have a lower gross margin than the existing network also \
               had a negative impact on the overall gross margin, but it should improve following \
               the implementation of its integration strategies .
    GMIncrease The upward movement of gross margin resulted from amounts pursuant to adjustments \
               to obligations towards dealers .

Note: The line breaks marked with a backslash are just inserted for formatting purposes and must not be included in the training data.

### Training Tool

The following command will train the document categorizer and write the model to en-doccat.bin:

    $ opennlp DoccatTrainer -model en-doccat.bin -lang en -data en-doccat.train -encoding UTF-8

Additionally it is possible to specify the number of iterations, and the cutoff.

### Training API

So, naturally you will need some access to many pre-classified events to train your model. The class opennlp.tools.doccat.DocumentSample encapsulates a text document and its classification. DocumentSample has two constructors. Each take the text's category as one argument. The other argument can either be raw text, or an array of tokens. By default, the raw text will be split into tokens by whitespace. So, let's say your training data was contained in a text file, where the format is as described above. Then you might want to write something like this to create a collection of DocumentSamples:

    DoccatModel model = null;
    InputStream dataIn = null;

    try (dataIn = new FileInputStream("en-sentiment.train")) {
      ObjectStream<String> lineStream =
    		new PlainTextByLineStream(dataIn, StandardCharsets.UTF_8);
      ObjectStream<DocumentSample> sampleStream = new DocumentSampleStream(lineStream);

      model = DocumentCategorizerME.train("en", sampleStream);
    }

Now might be a good time to cruise over to Hulu or something, because this could take a while if you've got a large training set. You may see a lot of output as well. Once you're done, you can pretty quickly step to classification directly, but first we'll cover serialization. Feel free to skim.

    try (OutputStream modelOut = new BufferedOutputStream(new FileOutputStream(modelFile))) {
      model.serialize(modelOut);
    }