ICUTokenizer (Lucene 10.5.1 icu API)
Description: declaration: package: org.apache.lucene.analysis.icu.segmentation, class: ICUTokenizer

Package org.apache.lucene.analysis.icu.segmentation

# Class ICUTokenizer

java.lang.Object
org.apache.lucene.util.AttributeSource
org.apache.lucene.analysis.TokenStream
org.apache.lucene.analysis.Tokenizer
org.apache.lucene.analysis.icu.segmentation.ICUTokenizer

All Implemented Interfaces:
  `Closeable`, `AutoCloseable`

public final class ICUTokenizer extends Tokenizer
Breaks text into words according to UAX #29: Unicode Text Segmentation (http://www.unicode.org/reports/tr29/)

Words are broken across script boundaries, then segmented according to the BreakIterator and typing provided by the `ICUTokenizerConfig`

See Also:
  - `ICUTokenizerConfig`
WARNING: This API is experimental and might change in incompatible ways in the next release.

- ## Nested Class Summary

  ## Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

  `AttributeSource.State`

- ## Field Summary

  ### Fields inherited from class org.apache.lucene.analysis.Tokenizer

  `input`

  ### Fields inherited from class org.apache.lucene.analysis.TokenStream

  `DEFAULT_TOKEN_ATTRIBUTE_FACTORY`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `ICUTokenizer()`
  Construct a new ICUTokenizer that breaks text into words from the given Reader.
  `ICUTokenizer(ICUTokenizerConfig config)`
  Construct a new ICUTokenizer that breaks text into words from the given Reader, using a tailored BreakIterator configuration.
  `ICUTokenizer(AttributeFactory factory, ICUTokenizerConfig config)`
  Construct a new ICUTokenizer that breaks text into words from the given Reader, using a tailored BreakIterator configuration.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `void`
  `end()`
  `boolean`
  `incrementToken()`
  `void`
  `reset()`

  ### Methods inherited from class org.apache.lucene.analysis.Tokenizer

  `close, correctOffset, setReader, setReaderTestPoint`

  ### Methods inherited from class org.apache.lucene.util.AttributeSource

  `addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString`

  ### Methods inherited from class java.lang.Object

  `clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

- ## Constructor Details

  - ### ICUTokenizer

    public ICUTokenizer()
    Construct a new ICUTokenizer that breaks text into words from the given Reader.

    The default script-specific handling is used.

    The default attribute factory is used.

    See Also:
      - `DefaultICUTokenizerConfig`

  - ### ICUTokenizer

    public ICUTokenizer(ICUTokenizerConfig config)
    Construct a new ICUTokenizer that breaks text into words from the given Reader, using a tailored BreakIterator configuration.

    The default attribute factory is used.

    Parameters:
      `config` - Tailored BreakIterator configuration

  - ### ICUTokenizer

    public ICUTokenizer(AttributeFactory factory, ICUTokenizerConfig config)
    Construct a new ICUTokenizer that breaks text into words from the given Reader, using a tailored BreakIterator configuration.
    Parameters:
      `factory` - AttributeFactory to use
      `config` - Tailored BreakIterator configuration

- ## Method Details

  - ### incrementToken

    public boolean incrementToken() throws IOException
    Specified by:
      `incrementToken` in class `TokenStream`
    Throws:
      `IOException`

  - ### reset

    public void reset() throws IOException
    Overrides:
      `reset` in class `Tokenizer`
    Throws:
      `IOException`

  - ### end

    public void end() throws IOException
    Overrides:
      `end` in class `TokenStream`
    Throws:
      `IOException`