Uses of Package opennlp.tools.tokenize (Apache OpenNLP :: Core :: Runtime 3.0.0-M4 API)
Description: use: package: opennlp.tools.tokenize

# Uses of Package
opennlp.tools.tokenize

Packages that use opennlp.tools.tokenize
Package
Description
opennlp.tools.tokenize

- Classes in opennlp.tools.tokenize used by opennlp.tools.tokenize
  Class
  Description
  BPEModel
  The `BPEModel` stores learned BPE merge operations and can be serialized and deserialized for reuse.
  BPETokenizer.SymbolPair
  Represents a pair of adjacent symbols in BPE.
  BPETokenizerFactory
  A `BaseToolFactory` for BPE tokenization that manages the BPE merge rules artifact and its serialization within a `BPEModel`.
  DetokenizationDictionary
  DetokenizationDictionary.Operation
  SimpleTokenizer
  A basic `Tokenizer` implementation which performs tokenization using character classes.
  TokenizerFactory
  The factory that provides `Tokenizer` default implementation and resources.
  TokenizerModel
  The `TokenizerModel` is the model used by a learnable `Tokenizer`.