Package opennlp.tools.tokenize

package opennlp.tools.tokenize

Class	Description
Detokenizer	A Detokenizer merges tokens back to their detokenized representation.
Detokenizer.DetokenizationOperation	This enum contains an operation for every token to merge the tokens together to their detokenized form.
TokenContextGenerator	Interface for context generators required for tokenizer implementations.
Tokenizer	The interface for tokenizers, which segment a string into its tokens.
TokenizerEvaluationMonitor	A marker interface for evaluating tokenizers.
TokenSample	A TokenSample is text with token spans.
WhitespaceTokenizer	A basic Tokenizer implementation which performs tokenization using white spaces.
WordpieceTokenizer	A Tokenizer implementation which performs tokenization using word pieces.