Uses of Interface
opennlp.tools.tokenize.Tokenizer

Packages that use Tokenizer

Package	Description
opennlp.tools.doccat	Package for classifying a document into a category.
opennlp.tools.formats.brat
opennlp.tools.formats.muc
opennlp.tools.tokenize	Contains classes related to finding token or words in a string.
opennlp.tools.util.featuregen	This package contains classes for generating sequence features.

Uses of Tokenizer in opennlp.tools.doccat

Methods in opennlp.tools.doccat that return Tokenizer

Modifier and Type	Method and Description
Tokenizer	DoccatFactory.getTokenizer()

Methods in opennlp.tools.doccat with parameters of type Tokenizer

Modifier and Type	Method and Description
static DoccatFactory	DoccatFactory.create(String subclassName, Tokenizer tokenizer, FeatureGenerator[] featureGenerators)
protected void	DoccatFactory.init(Tokenizer tokenizer, FeatureGenerator[] featureGenerators)
void	DoccatFactory.setTokenizer(Tokenizer tokenizer)

Constructors in opennlp.tools.doccat with parameters of type Tokenizer

Constructor and Description
DoccatFactory(Tokenizer tokenizer, FeatureGenerator[] featureGenerators) Creates a DoccatFactory.

Uses of Tokenizer in opennlp.tools.formats.brat

Constructors in opennlp.tools.formats.brat with parameters of type Tokenizer

Constructor and Description
BratNameSampleStream(SentenceDetector sentDetector, Tokenizer tokenizer, ObjectStream<BratDocument> samples)

Uses of Tokenizer in opennlp.tools.formats.muc

Constructors in opennlp.tools.formats.muc with parameters of type Tokenizer

Constructor and Description
MucNameContentHandler(Tokenizer tokenizer, List<NameSample> storedSamples)
MucNameSampleStream(Tokenizer tokenizer, ObjectStream<String> samples)

Uses of Tokenizer in opennlp.tools.tokenize

Classes in opennlp.tools.tokenize that implement Tokenizer

Modifier and Type	Class and Description
class	SimpleTokenizer Performs tokenization using character classes.
class	TokenizerME A Tokenizer for converting raw text into separated tokens.
class	WhitespaceTokenizer This tokenizer uses white spaces to tokenize the input text.

Constructors in opennlp.tools.tokenize with parameters of type Tokenizer

Constructor and Description
TokenizerEvaluator(Tokenizer tokenizer, TokenizerEvaluationMonitor... listeners) Initializes the current instance with the given Tokenizer.
TokenizerStream(Tokenizer tokenizer, ObjectStream<String> input)

Uses of Tokenizer in opennlp.tools.util.featuregen

Constructors in opennlp.tools.util.featuregen with parameters of type Tokenizer

Constructor and Description
TokenPatternFeatureGenerator(Tokenizer supportTokenizer) Initializes a new instance.