Package opennlp.tools.tokenize.uax29

Class WordBreakProperty

java.lang.Object
opennlp.tools.tokenize.uax29.WordBreakProperty
public final class WordBreakProperty extends Object
Looks up the Unicode Word_Break property of a code point.

The data is loaded once from the WordBreakProperty.txt resource of the Unicode Character Database (parsed with simple cursor scanning, no regular expression). Lookup is O(1) for the Basic Multilingual Plane (a direct array index) and O(log n) for supplementary code points (a binary search over a small sorted range table), so it imposes no per-character allocation on the word boundary algorithm.

  • Method Details

    • of

      public static WordBreak of(int codePoint)
      Returns the WordBreak value of a code point.
      Parameters:
      codePoint - The code point. Values outside [0, U+10FFFF] return WordBreak.OTHER.
      Returns:
      the WordBreak value of a code point

    • ordinalOf

      public static int ordinalOf(int codePoint)
      Returns the ordinal of a code point's WordBreak value. This is the allocation-free form of of(int) for hot loops that work with ordinals.
      Parameters:
      codePoint - The code point. Values outside [0, U+10FFFF] return the ordinal of WordBreak.OTHER.
      Returns:
      the ordinal of a code point's WordBreak value