PatternTokenizer (Lucene 10.5.1 common API)
Description: declaration: package: org.apache.lucene.analysis.pattern, class: PatternTokenizer

Package org.apache.lucene.analysis.pattern

# Class PatternTokenizer

java.lang.Object
org.apache.lucene.util.AttributeSource
org.apache.lucene.analysis.TokenStream
org.apache.lucene.analysis.Tokenizer
org.apache.lucene.analysis.pattern.PatternTokenizer

All Implemented Interfaces:
  `Closeable`, `AutoCloseable`

public final class PatternTokenizer extends Tokenizer
This tokenizer uses regex pattern matching to construct distinct tokens for the input stream. It takes two arguments: "pattern" and "group".

- "pattern" is the regular expression.
- "group" says which group to extract into tokens.

group=-1 (the default) is equivalent to "split". In this case, the tokens will be equivalent to the output from (without empty tokens): `String.split(java.lang.String)`

Using group >= 0 selects the matching group as the token. For example, if you have:

      pattern = \'([^\']+)\'
      group = 0
      input = aaa 'bbb' 'ccc'

the output will be two tokens: 'bbb' and 'ccc' (including the ' marks). With the same input but using group=1, the output would be: bbb and ccc (no ' marks)

NOTE: This Tokenizer does not output tokens that are of zero length.

See Also:
  - `Pattern`

- ## Nested Class Summary

  ## Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

  `AttributeSource.State`

- ## Field Summary

  ### Fields inherited from class org.apache.lucene.analysis.Tokenizer

  `input`

  ### Fields inherited from class org.apache.lucene.analysis.TokenStream

  `DEFAULT_TOKEN_ATTRIBUTE_FACTORY`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `PatternTokenizer(Pattern pattern, int group)`
  creates a new PatternTokenizer returning tokens from group (-1 for split functionality)
  `PatternTokenizer(AttributeFactory factory, Pattern pattern, int group)`
  creates a new PatternTokenizer returning tokens from group (-1 for split functionality)

- ## Method Summary

  Modifier and Type
  Method
  Description
  `void`
  `close()`
  `void`
  `end()`
  `boolean`
  `incrementToken()`
  `void`
  `reset()`

  ### Methods inherited from class org.apache.lucene.analysis.Tokenizer

  `correctOffset, setReader, setReaderTestPoint`

  ### Methods inherited from class org.apache.lucene.util.AttributeSource

  `addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString`

  ### Methods inherited from class java.lang.Object

  `clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

- ## Constructor Details

  - ### PatternTokenizer

    public PatternTokenizer(Pattern pattern, int group)
    creates a new PatternTokenizer returning tokens from group (-1 for split functionality)

  - ### PatternTokenizer

    public PatternTokenizer(AttributeFactory factory, Pattern pattern, int group)
    creates a new PatternTokenizer returning tokens from group (-1 for split functionality)

- ## Method Details

  - ### incrementToken

    public boolean incrementToken()
    Specified by:
      `incrementToken` in class `TokenStream`

  - ### end

    public void end() throws IOException
    Overrides:
      `end` in class `TokenStream`
    Throws:
      `IOException`

  - ### close

    public void close() throws IOException
    Specified by:
      `close` in interface `AutoCloseable`
    Specified by:
      `close` in interface `Closeable`
    Overrides:
      `close` in class `Tokenizer`
    Throws:
      `IOException`

  - ### reset

    public void reset() throws IOException
    Overrides:
      `reset` in class `Tokenizer`
    Throws:
      `IOException`