Package opennlp.tools.tokenize.uax29

Class WordSegmenter

java.lang.Object
opennlp.tools.tokenize.uax29.WordSegmenter
public final class WordSegmenter extends Object
Finds word boundaries in text using the Unicode Text Segmentation algorithm (UAX #29), rules WB1 through WB999.

The implementation is a single forward cursor pass with O(1) WordBreakProperty lookups and no regular expression. It decodes each code point once, keeps only a constant amount of state, and allocates nothing per character. It implements the "ignore" semantics of WB4 (a base character absorbs following Extend, Format, and ZWJ), the look-ahead rules WB6/WB7/WB7b/WB12, the Hebrew quote rules WB7a-WB7c, the emoji zero-width-joiner rule WB3c, and regional-indicator pairing WB15/WB16. The look-ahead for the WB6/WB7b/WB12 rules is resolved lazily and only at mid-word punctuation, so the common case never scans ahead.

forEachSegment(CharSequence, SegmentConsumer) streams the segments with no allocation; boundaries(CharSequence) returns every boundary offset (always including 0 and the text length); segments(CharSequence) returns the spans between them.

  • Method Details

    • forEachSegment

      public static void forEachSegment(CharSequence text, WordSegmenter.SegmentConsumer consumer)
      Streams the word segments of text to consumer in order, allocating nothing. Each segment is delivered as the half-open character range [start, end); the segments are contiguous and together cover the whole text.
      Parameters:
      text - The text to segment; may be empty, in which case no segment is delivered.
      consumer - The receiver of the segment ranges.

    • boundaries

      public static int[] boundaries(CharSequence text)
      Returns the word boundary character offsets in text, in ascending order, including the boundaries at 0 and text.length().
      Parameters:
      text - The text to segment.
      Returns:
      The boundary offsets; for empty text, [0].

    • segments

      public static List<opennlp.tools.util.Span> segments(CharSequence text)
      Returns the word segments of text as spans between consecutive boundaries.
      Parameters:
      text - The text to segment.
      Returns:
      The segment spans, in order.