Package opennlp.tools.tokenize.uax29

package opennlp.tools.tokenize.uax29

Related Packages

Package	Description
opennlp.tools.tokenize
opennlp.tools.tokenize.lang

Class	Description
ExtendedPictographic	Checks the Unicode Extended_Pictographic property of a code point.
WordBreak	The Unicode Word_Break property values, used by the UAX #29 word boundary algorithm.
WordBreakProperty	Looks up the Unicode Word_Break property of a code point.
WordSegmenter	Finds word boundaries in text using the Unicode Text Segmentation algorithm (UAX #29), rules WB1 through WB999.
WordSegmenter.SegmentConsumer	Receives each word segment as the half-open character range [start, end).
WordToken	A word token produced by WordTokenizer: the character Span into the source text together with its WordType.
WordTokenizer	A word tokenizer built on the Unicode Text Segmentation algorithm (UAX #29).
WordTokenizer.TokenHandler	Receives each word token as a character range and its type, with no allocation.
WordType	The category of a word token.