Package opennlp.tools.tokenize.uax29

Class WordTokenizer

java.lang.Object
opennlp.tools.tokenize.uax29.WordTokenizer
All Implemented Interfaces:
opennlp.tools.tokenize.Tokenizer
public final class WordTokenizer extends Object implements opennlp.tools.tokenize.Tokenizer
A word tokenizer built on the Unicode Text Segmentation algorithm (UAX #29). It finds segments with WordSegmenter, keeps the ones that are words (letters, digits, ideographs, kana, Hangul, Southeast-Asian script, or emoji), drops whitespace and punctuation, and classifies each kept token with a WordType. Emoji here means any Extended_Pictographic code point, so symbol-like characters such as the copyright, trademark, and double-exclamation signs are kept (typed WordType.EMOJI) rather than dropped as punctuation.

A token longer than maxTokenLength is emitted as consecutive pieces, never splitting a surrogate pair. The tokenizer reports offset Spans, so the original text and its character offsets are preserved for downstream normalization.

It implements Tokenizer: tokenize(String) returns the token strings and tokenizePos(String) their offsets. tokenizeTyped(CharSequence) additionally carries each token's WordType, and tokenize(CharSequence, TokenHandler) streams tokens with no per-token allocation. Instances are immutable and thread-safe.

  • Field Details

    • DEFAULT_MAX_TOKEN_LENGTH

      public static final int DEFAULT_MAX_TOKEN_LENGTH
      The default maximum token length: 255.
      See Also:

  • Constructor Details

    • WordTokenizer

      public WordTokenizer()
      Instantiates a tokenizer with the default maximum token length.

    • WordTokenizer

      public WordTokenizer(int maxTokenLength)
      Instantiates a tokenizer with the given maximum token length.
      Parameters:
      maxTokenLength - The maximum number of characters in a token; longer tokens are chopped into consecutive pieces. Must be at least 1.
      Throws:
      IllegalArgumentException - Thrown if maxTokenLength is equal to or less than 0.

  • Method Details

    • tokenize

      public void tokenize(CharSequence text, WordTokenizer.TokenHandler handler)
      Streams the word tokens of text to handler in order, allocating nothing.
      Parameters:
      text - The text to tokenize.
      handler - The receiver of the tokens.
      Throws:
      IllegalArgumentException - Thrown if text or handler is null.

    • tokenize

      public String[] tokenize(String text)
      Returns the word tokens of text as strings, in order.
      Specified by:
      tokenize in interface opennlp.tools.tokenize.Tokenizer
      Parameters:
      text - The text to tokenize; an empty text yields no tokens.
      Returns:
      The token strings.
      Throws:
      IllegalArgumentException - Thrown if text is null.

    • tokenizePos

      public opennlp.tools.util.Span[] tokenizePos(String text)
      Returns the offset spans of the word tokens of text, in order.
      Specified by:
      tokenizePos in interface opennlp.tools.tokenize.Tokenizer
      Parameters:
      text - The text to tokenize; an empty text yields no spans.
      Returns:
      The token spans.
      Throws:
      IllegalArgumentException - Thrown if text is null.

    • tokenizeSpans

      public List<opennlp.tools.util.Span> tokenizeSpans(CharSequence text)
      Computes the offset spans of the word tokens in text, in order.
      Parameters:
      text - The text to tokenize; an empty text yields no spans.
      Returns:
      The word-token spans.
      Throws:
      IllegalArgumentException - Thrown if text is null.

    • tokenizeTyped

      public List<WordToken> tokenizeTyped(CharSequence text)
      Returns the word tokens in text, each carrying its WordType, in order.
      Parameters:
      text - The text to tokenize; an empty text yields no tokens.
      Returns:
      The typed word tokens.
      Throws:
      IllegalArgumentException - Thrown if text is null.