Tokenizer (Lucene 10.5.1 core API)
Description: declaration: package: org.apache.lucene.analysis, class: Tokenizer

Package org.apache.lucene.analysis

# Class Tokenizer

java.lang.Object
org.apache.lucene.util.AttributeSource
org.apache.lucene.analysis.TokenStream
org.apache.lucene.analysis.Tokenizer

All Implemented Interfaces:
  `Closeable`, `AutoCloseable`

Direct Known Subclasses:
  `StandardTokenizer`

public abstract class Tokenizer extends TokenStream
A Tokenizer is a TokenStream whose input is a Reader.

This is an abstract class; subclasses must override `TokenStream.incrementToken()`

NOTE: Subclasses overriding `TokenStream.incrementToken()` must call `AttributeSource.clearAttributes()` before setting attributes.

- ## Nested Class Summary

  ## Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

  `AttributeSource.State`

- ## Field Summary

  Fields
  Modifier and Type
  Field
  Description
  `protected Reader`
  `input`
  The text source for this Tokenizer.

  ### Fields inherited from class org.apache.lucene.analysis.TokenStream

  `DEFAULT_TOKEN_ATTRIBUTE_FACTORY`

- ## Constructor Summary

  Constructors
  Modifier
  Constructor
  Description
  `protected`
  `Tokenizer()`
  Construct a tokenizer with no input, awaiting a call to `setReader(java.io.Reader)` to provide input.
  `protected`
  `Tokenizer(AttributeFactory factory)`
  Construct a tokenizer with no input, awaiting a call to `setReader(java.io.Reader)` to provide input.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `void`
  `close()`
  Releases resources associated with this stream.
  `protected final int`
  `correctOffset(int currentOff)`
  Return the corrected offset.
  `void`
  `reset()`
  This method is called by a consumer before it begins consumption using `TokenStream.incrementToken()`.
  `final void`
  `setReader(Reader input)`
  Expert: Set a new reader on the Tokenizer.
  `protected void`
  `setReaderTestPoint()`

  ### Methods inherited from class org.apache.lucene.analysis.TokenStream

  `end, incrementToken`

  ### Methods inherited from class org.apache.lucene.util.AttributeSource

  `addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString`

  ### Methods inherited from class java.lang.Object

  `clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

- ## Field Details

  - ### input

    protected Reader input
    The text source for this Tokenizer.

- ## Constructor Details

  - ### Tokenizer

    protected Tokenizer()
    Construct a tokenizer with no input, awaiting a call to `setReader(java.io.Reader)` to provide input.

  - ### Tokenizer

    protected Tokenizer(AttributeFactory factory)
    Construct a tokenizer with no input, awaiting a call to `setReader(java.io.Reader)` to provide input.
    Parameters:
      `factory` - attribute factory.

- ## Method Details

  - ### close

    public void close() throws IOException
    Releases resources associated with this stream.

    If you override this method, always call `super.close()`, otherwise some internal state will not be correctly reset (e.g., `Tokenizer` will throw `IllegalStateException` on reuse).

    NOTE: The default implementation closes the input Reader, so be sure to call `super.close()` when overriding this method.

    Specified by:
      `close` in interface `AutoCloseable`
    Specified by:
      `close` in interface `Closeable`
    Overrides:
      `close` in class `TokenStream`
    Throws:
      `IOException`

  - ### correctOffset

    protected final int correctOffset(int currentOff)
    Return the corrected offset. If `input` is a `CharFilter` subclass this method calls `CharFilter.correctOffset(int)`, else returns `currentOff`.
    Parameters:
      `currentOff` - offset as seen in the output
    Returns:
      corrected offset based on the input
    See Also:
      - `CharFilter.correctOffset(int)`

  - ### setReader

    public final void setReader(Reader input)
    Expert: Set a new reader on the Tokenizer. Typically, an analyzer (in its tokenStream method) will use this to re-use a previously created tokenizer.

  - ### reset

    public void reset() throws IOException
    Description copied from class: `TokenStream`
    This method is called by a consumer before it begins consumption using `TokenStream.incrementToken()`.

    Resets this stream to a clean state. Stateful implementations must implement this method so that they can be reused, just as if they had been created fresh.

    If you override this method, always call `super.reset()`, otherwise some internal state will not be correctly reset (e.g., `Tokenizer` will throw `IllegalStateException` on further usage).

    Overrides:
      `reset` in class `TokenStream`
    Throws:
      `IOException`

  - ### setReaderTestPoint

    protected void setReaderTestPoint()
    NOTE: This API is for internal purposes only and might change in incompatible ways in the next release.