Package opennlp.tools.tokenize

Contains classes related to finding token or words in a string. All tokenizer implement the Tokenizer interface. Currently, there is the learnable TokenizerME, the WhitespaceTokenizer and the SimpleTokenizer which is a character class tokenizer.

Interface Summary

Interface	Description
Detokenizer	A Detokenizer merges tokens back to their detokenized representation.
TokenContextGenerator	Interface for context generators required for TokenizerME.
Tokenizer	The interface for tokenizers, which segment a string into its tokens.
TokenizerEvaluationMonitor	A marker interface for evaluating tokenizers.

Class Summary

Class	Description
DefaultTokenContextGenerator	A default TokenContextGenerator which produces events for maxent decisions for tokenization.
DetokenizationDictionary
DetokenizerEvaluator	The DetokenizerEvaluator measures the performance of the given Detokenizer with the provided reference samples.
DictionaryDetokenizer	A rule based detokenizer.
SimpleTokenizer	A basic Tokenizer implementation which performs tokenization using character classes.
TokenizerCrossValidator	A cross validator for tokenizers.
TokenizerEvaluator	The TokenizerEvaluator measures the performance of the given Tokenizer with the provided reference samples.
TokenizerFactory	The factory that provides Tokenizer default implementation and resources.
TokenizerME	A Tokenizer for converting raw text into separated tokens.
TokenizerModel	The TokenizerModel is the model used by a learnable Tokenizer.
TokenizerStream	The TokenizerStream uses a Tokenizer to tokenize the input string and output samples.
TokenSample	A TokenSample is text with token spans.
TokenSampleStream	This class is a stream filter which reads in string encoded samples and creates samples out of them.
TokSpanEventStream	This class reads the samples via an Iterator and converts the samples into events which can be used by the maxent library for training.
WhitespaceTokenizer	A basic Tokenizer implementation which performs tokenization using white spaces.
WhitespaceTokenStream	This stream formats ObjectStream of samples into whitespace separated token strings.
WordpieceTokenizer	A Tokenizer implementation which performs tokenization using word pieces.

Enum Summary

Enum	Description
DetokenizationDictionary.Operation
Detokenizer.DetokenizationOperation	This enum contains an operation for every token to merge the tokens together to their detokenized form.