Uses of Package
opennlp.tools.tokenize

Packages that use opennlp.tools.tokenize

Package	Description
opennlp.tools.doccat	Package for classifying a document into a category.
opennlp.tools.formats	Experimental package related to converting various corpora to OpenNLP Format.
opennlp.tools.formats.ad
opennlp.tools.formats.brat
opennlp.tools.formats.convert
opennlp.tools.formats.letsmt
opennlp.tools.formats.muc
opennlp.tools.sentdetect	Package related to identifying sentece boundries.
opennlp.tools.tokenize	Contains classes related to finding token or words in a string.
opennlp.tools.tokenize.lang
opennlp.tools.util.featuregen	This package contains classes for generating sequence features.

Classes in opennlp.tools.tokenize used by opennlp.tools.doccat

Class and Description
Tokenizer The interface for tokenizers, which segment a string into its tokens.

Classes in opennlp.tools.tokenize used by opennlp.tools.formats

Class and Description
Detokenizer A Detokenizer merges tokens back to their untokenized representation.
Tokenizer The interface for tokenizers, which segment a string into its tokens.
TokenSample A TokenSample is text with token spans.

Classes in opennlp.tools.tokenize used by opennlp.tools.formats.ad

Class and Description
TokenSample A TokenSample is text with token spans.

Classes in opennlp.tools.tokenize used by opennlp.tools.formats.brat

Class and Description
Tokenizer The interface for tokenizers, which segment a string into its tokens.
TokenizerModel The TokenizerModel is the model used by a learnable Tokenizer.

Classes in opennlp.tools.tokenize used by opennlp.tools.formats.convert

Class and Description
Detokenizer A Detokenizer merges tokens back to their untokenized representation.
TokenSample A TokenSample is text with token spans.

Classes in opennlp.tools.tokenize used by opennlp.tools.formats.letsmt

Class and Description
Detokenizer A Detokenizer merges tokens back to their untokenized representation.

Classes in opennlp.tools.tokenize used by opennlp.tools.formats.muc

Class and Description
Tokenizer The interface for tokenizers, which segment a string into its tokens.

Classes in opennlp.tools.tokenize used by opennlp.tools.sentdetect

Class and Description
Detokenizer A Detokenizer merges tokens back to their untokenized representation.

Classes in opennlp.tools.tokenize used by opennlp.tools.tokenize

Class and Description
DetokenizationDictionary
DetokenizationDictionary.Operation
Detokenizer A Detokenizer merges tokens back to their untokenized representation.
Detokenizer.DetokenizationOperation This enum contains an operation for every token to merge the tokens together to their detokenized form.
SimpleTokenizer Performs tokenization using character classes.
TokenContextGenerator Interface for TokenizerME context generators.
Tokenizer The interface for tokenizers, which segment a string into its tokens.
TokenizerEvaluationMonitor
TokenizerFactory The factory that provides Tokenizer default implementations and resources.
TokenizerModel The TokenizerModel is the model used by a learnable Tokenizer.
TokenSample A TokenSample is text with token spans.
WhitespaceTokenizer This tokenizer uses white spaces to tokenize the input text.

Classes in opennlp.tools.tokenize used by opennlp.tools.tokenize.lang

Class and Description
TokenContextGenerator Interface for TokenizerME context generators.

Classes in opennlp.tools.tokenize used by opennlp.tools.util.featuregen

Class and Description
Tokenizer The interface for tokenizers, which segment a string into its tokens.