Uses of Interface opennlp.tools.tokenize.Tokenizer (Apache OpenNLP Tools 1.9.5 API)
Date: 2026-06-18

## Uses of Interface
opennlp.tools.tokenize.Tokenizer

- Packages that use Tokenizer
  Package | Description
  --- | ---
  opennlp.tools.cmdline.parser |
  opennlp.tools.formats.brat |
  opennlp.tools.formats.muc |
  opennlp.tools.tokenize | Contains classes related to finding token or words in a string.
  opennlp.tools.util.featuregen | This package contains classes for generating sequence features.

- - ### Uses of Tokenizer in opennlp.tools.cmdline.parser

    Methods in opennlp.tools.cmdline.parser with parameters of type Tokenizer
    Modifier and Type | Method and Description
    --- | ---
    `static Parse[]` | ParserTool.`parseLine(String line, Parser parser, Tokenizer tokenizer, int numParses)`

  - ### Uses of Tokenizer in opennlp.tools.formats.brat

    Constructors in opennlp.tools.formats.brat with parameters of type Tokenizer
    Constructor and Description
    `BratDocumentParser(SentenceDetector sentenceDetector, Tokenizer tokenizer)`
    `BratDocumentParser(SentenceDetector sentenceDetector, Tokenizer tokenizer, Set<String> nameTypes)`
    `BratNameSampleStream(SentenceDetector sentDetector, Tokenizer tokenizer, ObjectStream<BratDocument> samples)` Creates a new `BratNameSampleStream`.
    `BratNameSampleStream(SentenceDetector sentDetector, Tokenizer tokenizer, ObjectStream<BratDocument> samples, Set<String> nameTypes)` Creates a new `BratNameSampleStream`.

  - ### Uses of Tokenizer in opennlp.tools.formats.muc

    Constructors in opennlp.tools.formats.muc with parameters of type Tokenizer
    Constructor and Description
    `MucNameContentHandler(Tokenizer tokenizer, List<NameSample> storedSamples)`
    `MucNameSampleStream(Tokenizer tokenizer, ObjectStream<String> samples)`

  - ### Uses of Tokenizer in opennlp.tools.tokenize

    Classes in opennlp.tools.tokenize that implement Tokenizer
    Modifier and Type | Class and Description
    --- | ---
    `class` | `SimpleTokenizer` Performs tokenization using character classes.
    `class` | `TokenizerME` A Tokenizer for converting raw text into separated tokens.
    `class` | `WhitespaceTokenizer` This tokenizer uses white spaces to tokenize the input text.

    Constructors in opennlp.tools.tokenize with parameters of type Tokenizer
    Constructor and Description
    `TokenizerEvaluator(Tokenizer tokenizer, TokenizerEvaluationMonitor... listeners)` Initializes the current instance with the given `Tokenizer`.
    `TokenizerStream(Tokenizer tokenizer, ObjectStream<String> input)`

  - ### Uses of Tokenizer in opennlp.tools.util.featuregen

    Constructors in opennlp.tools.util.featuregen with parameters of type Tokenizer
    Constructor and Description
    `TokenPatternFeatureGenerator(Tokenizer supportTokenizer)` Initializes a new instance.

Copyright © 2026 The Apache Software Foundation. All rights reserved.