TruncateTokenFilter (Lucene 10.5.1 common API)
Description: declaration: package: org.apache.lucene.analysis.miscellaneous, class: TruncateTokenFilter

Package org.apache.lucene.analysis.miscellaneous

# Class TruncateTokenFilter

java.lang.Object
org.apache.lucene.util.AttributeSource
org.apache.lucene.analysis.TokenStream
org.apache.lucene.analysis.TokenFilter
org.apache.lucene.analysis.miscellaneous.TruncateTokenFilter

All Implemented Interfaces:
  `Closeable`, `AutoCloseable`, `Unwrappable<TokenStream>`

public final class TruncateTokenFilter extends TokenFilter
A token filter for truncating the terms into a specific length (number of codepoints). Fixed prefix truncation, as a stemming method, produces good results on Turkish language. It is reported that F5, using first 5 characters, produced best results in Information Retrieval on Turkish Texts

Since Lucene 10.5, the filter is able to correctly handle codepoints and truncates after the given number of codepoints, no longer producing incomplete surrogate pairs. Use the modern factory method `truncateAfterCodePoints(TokenStream, int)` to enable this mode. Legacy behaviour is still available with `truncateAfterChars(TokenStream, int)`

- ## Nested Class Summary

  ## Nested classes/interfaces inherited from class org.apache.lucene.util.AttributeSource

  `AttributeSource.State`

- ## Field Summary

  ### Fields inherited from class org.apache.lucene.analysis.TokenFilter

  `input`

  ### Fields inherited from class org.apache.lucene.analysis.TokenStream

  `DEFAULT_TOKEN_ATTRIBUTE_FACTORY`

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `TruncateTokenFilter(TokenStream input, int nChars)`
  Deprecated.
  This constructor is deprecated, use `truncateAfterChars(TokenStream, int)` for backwards compatibility, or `truncateAfterCodePoints(TokenStream, int)` to be unicode conformant.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `final boolean`
  `incrementToken()`
  `static TruncateTokenFilter`
  `truncateAfterChars(TokenStream input, int nChars)`
  Returns a filter with a prefix of `nChars` Java Characters.
  `static TruncateTokenFilter`
  `truncateAfterCodePoints(TokenStream input, int nCodePoints)`
  Returns a filter with a prefix of `nCodePoints`.

  ### Methods inherited from class org.apache.lucene.analysis.TokenFilter

  `close, end, reset, unwrap`

  ### Methods inherited from class org.apache.lucene.util.AttributeSource

  `addAttribute, addAttributeImpl, captureState, clearAttributes, cloneAttributes, copyTo, endAttributes, equals, getAttribute, getAttributeClassesIterator, getAttributeFactory, getAttributeImplsIterator, hasAttribute, hasAttributes, hashCode, reflectAsString, reflectWith, removeAllAttributes, restoreState, toString`

  ### Methods inherited from class java.lang.Object

  `clone, finalize, getClass, notify, notifyAll, wait, wait, wait`

- ## Constructor Details

  - ### TruncateTokenFilter

    @Deprecated public TruncateTokenFilter(TokenStream input, int nChars)
    Deprecated.
    This constructor is deprecated, use `truncateAfterChars(TokenStream, int)` for backwards compatibility, or `truncateAfterCodePoints(TokenStream, int)` to be unicode conformant.
    Instantiates filter with a prefix of `nChars` Java Characters. This may split surrogate pairs.

- ## Method Details

  - ### truncateAfterCodePoints

    public static TruncateTokenFilter truncateAfterCodePoints(TokenStream input, int nCodePoints)
    Returns a filter with a prefix of `nCodePoints`.

  - ### truncateAfterChars

    public static TruncateTokenFilter truncateAfterChars(TokenStream input, int nChars)
    Returns a filter with a prefix of `nChars` Java Characters. This may split surrogate pairs.

  - ### incrementToken

    public final boolean incrementToken() throws IOException
    Specified by:
      `incrementToken` in class `TokenStream`
    Throws:
      `IOException`