Package opennlp.tools.tokenize

package opennlp.tools.tokenize

Related Packages

Package	Description
opennlp.tools.tokenize.lang

Class	Description
DefaultTokenContextGenerator	A default TokenContextGenerator which produces events for maxent decisions for tokenization.
DetokenizationDictionary
DetokenizationDictionary.Operation
DictionaryDetokenizer	A rule based detokenizer.
SimpleTokenizer	A basic Tokenizer implementation which performs tokenization using character classes.
ThreadSafeTokenizerME	A thread-safe version of TokenizerME.
TokenizerCrossValidator	A cross validator for tokenizers.
TokenizerEvaluator	The TokenizerEvaluator measures the performance of the given Tokenizer with the provided reference samples.
TokenizerFactory	The factory that provides Tokenizer default implementation and resources.
TokenizerME	A Tokenizer for converting raw text into separated tokens.
TokenizerModel	The TokenizerModel is the model used by a learnable Tokenizer.
TokenizerStream	The TokenizerStream uses a Tokenizer to tokenize the input string and output samples.
TokenSampleStream	This class is a stream filter which reads in string encoded samples and creates samples out of them.
TokSpanEventStream	This class reads the samples via an Iterator and converts the samples into events which can be used by the maxent library for training.
WhitespaceTokenStream	This stream formats ObjectStream of samples into whitespace separated token strings.