opennlp.tools.tokenize (Apache OpenNLP :: API 3.0.0-M4 API)
Description: declaration: package: opennlp.tools.tokenize

# Package opennlp.tools.tokenize

package opennlp.tools.tokenize

- Class
  Description
  BertTokenizer
  A `Tokenizer` implementation of the full BERT tokenization pipeline: basic tokenization (text normalization) followed by wordpiece tokenization.
  Detokenizer
  A `Detokenizer` merges tokens back to their detokenized representation.
  Detokenizer.DetokenizationOperation
  This enum contains an operation for every token to merge the tokens together to their detokenized form.
  TokenContextGenerator
  Interface for context generators required for `tokenizer implementations`.
  Tokenizer
  The interface for tokenizers, which segment a string into its tokens.
  TokenizerEvaluationMonitor
  A marker interface for evaluating `tokenizers`.
  TokenSample
  A `TokenSample` is text with token spans.
  WhitespaceTokenizer
  A basic `Tokenizer` implementation which performs tokenization using white spaces.
  WordpieceTokenizer
  A `Tokenizer` implementation which performs tokenization using word pieces.