Uses of Interface
opennlp.tools.tokenize.Tokenizer

Packages that use Tokenizer

Package	Description
opennlp.tools.formats	Experimental package related to converting various corpora to OpenNLP Format.
opennlp.tools.formats.brat
opennlp.tools.formats.muc
opennlp.tools.tokenize	Contains classes related to finding token or words in a string.
opennlp.tools.util.featuregen	This package contains classes for generating sequence features.

Uses of Tokenizer in opennlp.tools.formats

Constructors in opennlp.tools.formats with parameters of type Tokenizer

Constructor and Description
LeipzigDoccatSampleStream(String language, int sentencesPerDocument, Tokenizer tokenizer, InputStreamFactory in) Creates a new LeipzigDoccatSampleStream with the specified parameters.

Uses of Tokenizer in opennlp.tools.formats.brat

Constructors in opennlp.tools.formats.brat with parameters of type Tokenizer

Constructor and Description
BratNameSampleStream(SentenceDetector sentDetector, Tokenizer tokenizer, ObjectStream<BratDocument> samples)

Uses of Tokenizer in opennlp.tools.formats.muc

Constructors in opennlp.tools.formats.muc with parameters of type Tokenizer

Constructor and Description
MucNameContentHandler(Tokenizer tokenizer, List<NameSample> storedSamples)
MucNameSampleStream(Tokenizer tokenizer, ObjectStream<String> samples)

Uses of Tokenizer in opennlp.tools.tokenize

Classes in opennlp.tools.tokenize that implement Tokenizer

Modifier and Type	Class and Description
class	SimpleTokenizer Performs tokenization using character classes.
class	TokenizerME A Tokenizer for converting raw text into separated tokens.
class	WhitespaceTokenizer This tokenizer uses white spaces to tokenize the input text.

Constructors in opennlp.tools.tokenize with parameters of type Tokenizer

Constructor and Description
TokenizerEvaluator(Tokenizer tokenizer, TokenizerEvaluationMonitor... listeners) Initializes the current instance with the given Tokenizer.
TokenizerStream(Tokenizer tokenizer, ObjectStream<String> input)

Uses of Tokenizer in opennlp.tools.util.featuregen

Constructors in opennlp.tools.util.featuregen with parameters of type Tokenizer

Constructor and Description
TokenPatternFeatureGenerator(Tokenizer supportTokenizer) Initializes a new instance.