SimplePatternTokenizer (Lucene 10.5.1 common API)
Description: declaration: package: org.apache.lucene.analysis.pattern, class: SimplePatternTokenizer

Package org.apache.lucene.analysis.pattern

# Class SimplePatternTokenizer

java.lang.Object
org.apache.lucene.util.AttributeSource
org.apache.lucene.analysis.TokenStream
org.apache.lucene.analysis.Tokenizer
org.apache.lucene.analysis.pattern.SimplePatternTokenizer

All Implemented Interfaces:
  `Closeable`, `AutoCloseable`

public final class SimplePatternTokenizer extends Tokenizer
This tokenizer uses a Lucene `RegExp` or (expert usage) a pre-built determinized `Automaton`, to locate tokens. The regexp syntax is more limited than `PatternTokenizer`, but the tokenization is quite a bit faster. The provided regex should match valid token characters (not token separator characters, like `String.split`). The matching is greedy: the longest match at a given start point will be the next token. Empty string tokens are never produced.

WARNING: This API is experimental and might change in incompatible ways in the next release.

- ## Nested Class Summary

  ## Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

  `AttributeSource.State`

- ## Field Summary

  ### Fields inherited from class org.apache.lucene.analysis.Tokenizer

  `input`

  ### Fields inherited from class org.apache.lucene.analysis.TokenStream

  `DEFAULT_TOKEN_ATTRIBUTE_FACTORY`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `SimplePatternTokenizer(String regexp)`
  See `RegExp` for the accepted syntax.
  `SimplePatternTokenizer(AttributeFactory factory, String regexp, int determinizeWorkLimit)`
  See `RegExp` for the accepted syntax.
  `SimplePatternTokenizer(AttributeFactory factory, Automaton dfa)`
  Runs a pre-built automaton.
  `SimplePatternTokenizer(Automaton dfa)`
  Runs a pre-built automaton.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `void`
  `end()`
  `boolean`
  `incrementToken()`
  `void`
  `reset()`

  ### Methods inherited from class org.apache.lucene.analysis.Tokenizer

  `close, correctOffset, setReader, setReaderTestPoint`

  ### Methods inherited from class org.apache.lucene.util.AttributeSource

  `addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString`

  ### Methods inherited from class java.lang.Object

  `clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

- ## Constructor Details

  - ### SimplePatternTokenizer

    public SimplePatternTokenizer(String regexp)
    See `RegExp` for the accepted syntax.

  - ### SimplePatternTokenizer

    public SimplePatternTokenizer(Automaton dfa)
    Runs a pre-built automaton.

  - ### SimplePatternTokenizer

    public SimplePatternTokenizer(AttributeFactory factory, String regexp, int determinizeWorkLimit)
    See `RegExp` for the accepted syntax.

  - ### SimplePatternTokenizer

    public SimplePatternTokenizer(AttributeFactory factory, Automaton dfa)
    Runs a pre-built automaton.

- ## Method Details

  - ### incrementToken

    public boolean incrementToken() throws IOException
    Specified by:
      `incrementToken` in class `TokenStream`
    Throws:
      `IOException`

  - ### end

    public void end() throws IOException
    Overrides:
      `end` in class `TokenStream`
    Throws:
      `IOException`

  - ### reset

    public void reset() throws IOException
    Overrides:
      `reset` in class `Tokenizer`
    Throws:
      `IOException`