Package opennlp.tools.tokenize


package opennlp.tools.tokenize
Contains classes related to finding token or words in a string. All tokenizer implement the Tokenizer interface. Currently, there is the learnable TokenizerME, the WhitespaceTokenizer and the SimpleTokenizer which is a character class tokenizer.