# UIMA Integration

The UIMA Integration module wraps the OpenNLP components as UIMA Analysis Engines. These annotators can be used in any UIMA pipeline to automatically annotate text with sentences, tokens, named entities, part-of-speech tags, chunks, and parse trees. The module is located in the `opennlp-uima` artifact.

## Adding the Dependency

To use the OpenNLP UIMA annotators, add the following dependency to your project:

    <dependency>
      <groupId>org.apache.opennlp</groupId>
      <artifactId>opennlp-uima</artifactId>
      <version>${opennlp.version}</version>
    </dependency>

This module depends on Apache UIMA and the OpenNLP runtime. The UIMA framework dependency is included transitively.

## Type System

The module ships with a default type system defined in `TypeSystem.xml` inside the descriptors directory. This type system defines the following annotation types:

- `opennlp.uima.Sentence` - Sentence boundary annotations

- `opennlp.uima.Token` - Token annotations with a `pos` feature for part-of-speech tags

- `opennlp.uima.Chunk` - Chunk annotations with a `chunkType` feature (e.g. NP, VP)

- `opennlp.uima.Person | Organization | Location` - Named entity types

- `opennlp.uima.Date | Time | Money | Percentage` - Additional named entity types

- `opennlp.uima.Parse` - Parse tree node annotations with `parseType`, `children`, and `prob` features

The default type system can be replaced with a custom type system. To do so, update the type references in the analysis engine descriptors to point to your custom types and import your custom type system instead of `TypeSystem.xml`.

## Descriptor Structure

Each OpenNLP UIMA annotator is configured through a UIMA analysis engine descriptor XML file. It specifies:

- The annotator implementation class

- Configuration parameters (e.g. which type system types to use)

- An external resource dependency for the OpenNLP model file

- A reference to the type system

Models are loaded through the UIMA external resource mechanism. Each ML-based annotator declares a dependency on a model resource with the key `opennlp.uima.ModelName`. The model file is bound to this key through the resource manager configuration. For example, to configure the sentence detector model:

    <externalResourceDependencies>
      <externalResourceDependency>
        <key>opennlp.uima.ModelName</key>
        <interfaceName>opennlp.uima.sentdetect.SentenceModelResource</interfaceName>
      </externalResourceDependency>
    </externalResourceDependencies>

    <resourceManagerConfiguration>
      <externalResources>
        <externalResource>
          <name>SentenceModel</name>
          <fileResourceSpecifier>
            <fileUrl>file:en-sent.bin</fileUrl>
          </fileResourceSpecifier>
          <implementationName>opennlp.uima.sentdetect.SentenceModelResourceImpl</implementationName>
        </externalResource>
      </externalResources>
      <externalResourceBindings>
        <externalResourceBinding>
          <key>opennlp.uima.ModelName</key>
          <resourceName>SentenceModel</resourceName>
        </externalResourceBinding>
      </externalResourceBindings>
    </resourceManagerConfiguration>

## Sentence Detector

The `opennlp.uima.sentdetect.SentenceDetector` annotator detects sentence boundaries and creates sentence annotations in the CAS.

Configuration Parameters:

- `opennlp.uima.SentenceType` (mandatory) - The full name of the sentence annotation type. Default: `opennlp.uima.Sentence`

- `opennlp.uima.ContainerType` (optional) - If set, sentence detection is restricted to within annotations of this type. Useful for detecting sentences only inside specific regions of a document (e.g. paragraphs).

- `opennlp.uima.ProbabilityFeature` (optional) - Feature name for storing the detection confidence score on each sentence annotation.

Model Resource Interface: `opennlp.uima.sentdetect.SentenceModelResource`

Example Descriptor: See `descriptors/SentenceDetector.xml` in the opennlp-uima module.

## Tokenizer

Three tokenizer implementations are available as UIMA annotators. All tokenizers require sentence annotations to already be present in the CAS.

### Learnable Tokenizer

The `opennlp.uima.tokenize.Tokenizer` annotator uses a maximum entropy model to identify token boundaries.

Configuration Parameters:

- `opennlp.uima.SentenceType` (mandatory) - The sentence annotation type. Default: `opennlp.uima.Sentence`

- `opennlp.uima.TokenType` (mandatory) - The token annotation type. Default: `opennlp.uima.Token`

- `opennlp.uima.tokenizer.IsAlphaNumericOptimization` (optional) - If set, enables an optimization that treats purely alphanumeric sequences as single tokens without consulting the model.

- `opennlp.uima.ProbabilityFeature` (optional) - Feature name for storing token probability scores.

Model Resource Interface: `opennlp.uima.tokenize.TokenizerModelResource`

### Simple Tokenizer

The `opennlp.uima.tokenize.SimpleTokenizer` annotator is a rule-based tokenizer that splits text by character class boundaries. It requires no model.

Configuration Parameters:

- `opennlp.uima.SentenceType` (mandatory) - The sentence annotation type.

- `opennlp.uima.TokenType` (mandatory) - The token annotation type.

### Whitespace Tokenizer

The `opennlp.uima.tokenize.WhitespaceTokenizer` annotator splits text at whitespace boundaries. It requires no model.

Configuration Parameters:

- `opennlp.uima.SentenceType` (mandatory) - The sentence annotation type.

- `opennlp.uima.TokenType` (mandatory) - The token annotation type.

## Name Finder

Two named entity recognition annotators are provided: a machine learning-based annotator and a dictionary-based annotator. Both require sentence and token annotations to already be present in the CAS.

### Learnable Name Finder

The `opennlp.uima.namefind.NameFinder` annotator uses a maximum entropy model to detect named entities such as person names, organizations, and locations.

Configuration Parameters:

- `opennlp.uima.SentenceType` (mandatory) - The sentence annotation type.

- `opennlp.uima.TokenType` (mandatory) - The token annotation type.

- `opennlp.uima.NameType` (mandatory) - The annotation type for detected entities (e.g. `opennlp.uima.Person`).

- `opennlp.uima.ProbabilityFeature` (optional) - Feature name for storing entity probability scores.

- `opennlp.uima.BeamSize` (optional) - Beam size for the beam search.

- `opennlp.uima.DocumentConfidenceType` (optional) - Annotation type for storing document-level confidence information.

Model Resource Interface: `opennlp.uima.namefind.TokenNameFinderModelResource`

To detect multiple entity types, configure one Name Finder annotator per entity type, each with its own model. The provided descriptors include pre-configured annotators for person, organization, location, date, time, money, and percentage entities.

### Dictionary Name Finder

The `opennlp.uima.namefind.DictionaryNameFinder` annotator performs dictionary-based named entity recognition. It matches token sequences against entries in an OpenNLP dictionary file. No machine learning model is required.

Configuration Parameters:

- `opennlp.uima.SentenceType` (mandatory) - The sentence annotation type.

- `opennlp.uima.TokenType` (mandatory) - The token annotation type.

- `opennlp.uima.NameType` (mandatory) - The annotation type for detected entities.

- `opennlp.uima.Dictionary` (mandatory) - External resource key for the OpenNLP dictionary file to use for matching.

## POS Tagger

The `opennlp.uima.postag.POSTagger` annotator assigns part-of-speech tags to tokens. It requires sentence and token annotations to already be present in the CAS.

Configuration Parameters:

- `opennlp.uima.SentenceType` (mandatory) - The sentence annotation type. Default: `opennlp.uima.Sentence`

- `opennlp.uima.TokenType` (mandatory) - The token annotation type. Default: `opennlp.uima.Token`

- `opennlp.uima.POSFeature` (mandatory) - The feature name on the token type where the POS tag will be stored. Default: `pos`

- `opennlp.uima.ProbabilityFeature` (optional) - Feature name for storing tagging probability scores.

- `opennlp.uima.BeamSize` (optional) - Beam size for the beam search.

- `opennlp.uima.DictionaryName` (optional) - External resource key for a tag dictionary that constrains possible tags for known words.

Model Resource Interface: `opennlp.uima.postag.POSModelResource`

## Chunker

The `opennlp.uima.chunker.Chunker` annotator identifies non-recursive syntactic phrases (chunks) such as noun phrases (NP) and verb phrases (VP). It requires sentence and token annotations with POS tags to already be present in the CAS.

Configuration Parameters:

- `opennlp.uima.SentenceType` (mandatory) - The sentence annotation type. Default: `opennlp.uima.Sentence`

- `opennlp.uima.TokenType` (mandatory) - The token annotation type. Default: `opennlp.uima.Token`

- `opennlp.uima.POSFeature` (mandatory) - The feature name for reading POS tags from tokens. Default: `pos`

- `opennlp.uima.ChunkType` (mandatory) - The annotation type for chunk annotations. Default: `opennlp.uima.Chunk`

- `opennlp.uima.ChunkTagFeature` (mandatory) - The feature name on the chunk type where the chunk tag (e.g. NP, VP) will be stored. Default: `chunkType`

- `opennlp.uima.BeamSize` (optional) - Beam size for the beam search.

Model Resource Interface: `opennlp.uima.chunker.ChunkerModelResource`

## Parser

The `opennlp.uima.parser.Parser` annotator performs full syntactic parsing and creates a hierarchical parse tree structure in the CAS. It requires sentence and token annotations to already be present in the CAS.

Configuration Parameters:

- `opennlp.uima.SentenceType` (mandatory) - The sentence annotation type. Default: `opennlp.uima.Sentence`

- `opennlp.uima.TokenType` (mandatory) - The token annotation type. Default: `opennlp.uima.Token`

- `opennlp.uima.ParseType` (mandatory) - The annotation type for parse tree nodes. Default: `opennlp.uima.Parse`

- `opennlp.uima.TypeFeature` (mandatory) - The feature name for storing the parse node type (e.g. S, NP, VP). Default: `parseType`

- `opennlp.uima.ChildrenFeature` (mandatory) - The feature name for storing references to child parse nodes. Default: `children`

- `opennlp.uima.ProbabilityFeature` (optional) - Feature name for storing parse probability scores. Default: `prob`

- `opennlp.uima.BeamSize` (optional) - Beam size for the beam search.

Model Resource Interface: `opennlp.uima.parser.ParserModelResource`

## Document Categorizer

The `opennlp.uima.doccat.DocumentCategorizer` annotator classifies document text into categories using a trained document categorization model.

Configuration Parameters:

- `opennlp.uima.doccat.CategoryType` (mandatory) - The annotation type for the category result.

- `opennlp.uima.doccat.CategoryFeature` (mandatory) - The feature name on the category type where the classification result is stored.

Model Resource Interface: `opennlp.uima.doccat.DoccatModelResource`

## Language Detector

The `opennlp.uima.doccat.LanguageDetector` annotator identifies the language of the document text and sets the CAS document language accordingly.

Configuration Parameters:

- `opennlp.uima.SentenceType` (mandatory) - The sentence annotation type. Default: `opennlp.uima.Sentence`

Model Resource Interface: `opennlp.uima.doccat.DoccatModelResource`

Example Descriptor: See `descriptors/LanguageDetector.xml` in the opennlp-uima module.

## Normalizer

The `opennlp.uima.normalizer.Normalizer` annotator extracts structured data from named entity annotations. It can convert the covered text of a named entity into typed values (e.g. parsing a money amount into a numeric value) and optionally look up normalized forms in a dictionary.

Configuration Parameters:

- `opennlp.uima.NameType` (mandatory) - The named entity annotation type to normalize.

- `opennlp.uima.normalizer.StructureFeature` (mandatory) - The feature name where the normalized value is stored.

- `opennlp.uima.Dictionary` (optional) - External resource key for a dictionary used to look up normalized forms.

The normalizer supports the following target feature types: `String`, `Byte`, `Short`, `Integer`, `Long`, `Float`, and `Double`. Number parsing is locale-aware and uses the CAS document language.

## Building an Aggregate Pipeline

The annotators are designed to be composed into an aggregate analysis engine where each annotator builds on the annotations produced by earlier ones. The standard processing order is:

- Sentence Detector (produces sentence annotations)

- Tokenizer (produces token annotations within sentences)

- Name Finders (produce entity annotations from tokens)

- POS Tagger (adds POS tags to tokens)

- Chunker (produces chunk annotations from POS-tagged tokens)

- Parser (produces parse tree from tokens within sentences)

The module includes a pre-configured aggregate descriptor `descriptors/OpenNlpTextAnalyzer.xml` that chains sentence detection, tokenization, multiple name finders (person, organization, location, date, time, money, percentage), POS tagging, chunking, and parsing in the correct order.

This aggregate descriptor demonstrates how to bind models for all annotators in one place using the resource manager configuration. Each annotator's model key follows the pattern `AnnotatorKey/opennlp.uima.ModelName`, for example:

    <externalResourceBinding>
      <key>SentenceDetector/opennlp.uima.ModelName</key>
      <resourceName>SentenceModel</resourceName>
    </externalResourceBinding>
    <externalResourceBinding>
      <key>Tokenizer/opennlp.uima.ModelName</key>
      <resourceName>TokenModel</resourceName>
    </externalResourceBinding>

Below is a complete example showing how to create and run an aggregate pipeline programmatically using the UIMA framework APIs:

    // Load the aggregate analysis engine descriptor
    XMLInputSource in = new XMLInputSource("descriptors/OpenNlpTextAnalyzer.xml");
    ResourceSpecifier specifier = UIMAFramework.getXMLParser()
        .parseResourceSpecifier(in);

    // Create the analysis engine
    AnalysisEngine ae = UIMAFramework.produceAnalysisEngine(specifier);

    // Create a CAS and set the document text
    CAS cas = ae.newCAS();
    cas.setDocumentText("Pierre Vinken, 61 years old, will join the board "
        + "as a nonexecutive director Nov. 29. Mr. Vinken is chairman "
        + "of Elsevier N.V., the Dutch publishing group.");
    cas.setDocumentLanguage("en");

    // Run the pipeline
    ae.process(cas);

    // Iterate over detected sentences
    Type sentenceType = cas.getTypeSystem().getType("opennlp.uima.Sentence");
    for (AnnotationFS sentence : cas.getAnnotationIndex(sentenceType)) {
      System.out.println("Sentence: " + sentence.getCoveredText());
    }

    // Iterate over detected tokens
    Type tokenType = cas.getTypeSystem().getType("opennlp.uima.Token");
    Feature posFeature = tokenType.getFeatureByBaseName("pos");
    for (AnnotationFS token : cas.getAnnotationIndex(tokenType)) {
      System.out.println("Token: " + token.getCoveredText()
          + " POS: " + token.getStringValue(posFeature));
    }

    // Iterate over detected person names
    Type personType = cas.getTypeSystem().getType("opennlp.uima.Person");
    for (AnnotationFS person : cas.getAnnotationIndex(personType)) {
      System.out.println("Person: " + person.getCoveredText());
    }

    // Clean up
    cas.release();
    ae.destroy();

## Using Custom Type Systems

The default type system can be replaced with your own custom types. This is useful when integrating OpenNLP annotators into an existing UIMA pipeline that already defines its own type system.

To use custom types:

- Create your own type system descriptor with the annotation types you need.

- Update the annotator descriptor to import your custom type system instead of `TypeSystem.xml`.

- Set the configuration parameters (e.g. `opennlp.uima.SentenceType`, `opennlp.uima.TokenType`) to reference your custom type names.

For example, if your type system defines sentences as `my.types.Sentence` and tokens as `my.types.Token`, update the descriptor:

    <configurationParameterSettings>
      <nameValuePair>
        <name>opennlp.uima.SentenceType</name>
        <value>
          <string>my.types.Sentence</string>
        </value>
      </nameValuePair>
      <nameValuePair>
        <name>opennlp.uima.TokenType</name>
        <value>
          <string>my.types.Token</string>
        </value>
      </nameValuePair>
    </configurationParameterSettings>

## Running the PEAR Sample in CVD

The CAS Visual Debugger (CVD) is shipped as part of the UIMA distribution and is a tool which can run the OpenNLP UIMA Annotators and display their analysis results. The source distribution comes with a script which can create a sample UIMA application. This includes the sentence detector, tokenizer, POS tagger, chunker, and name finders for English. This sample application is packaged in the PEAR format and must be installed with the PEAR installer before it can be run by CVD. Please consult the UIMA documentation for further information about the PEAR installer.

After the PEAR is installed, start the CAS Visual Debugger shipped with the UIMA framework and click on Tools -> Load AE. Then select the `opennlp.uima.OpenNlpTextAnalyzer_pear.xml` file in the file dialog. Now enter some text and start the analysis engine with "Run -> Run OpenNLPTextAnalyzer". Afterwards the results will be displayed. You should see sentences, tokens, chunks, POS tags, and possibly some named entities. Remember the input text must be written in English.