Package opennlp.tools.tokenize

Class SimpleTokenizer

java.lang.Object

opennlp.tools.tokenize.SimpleTokenizer

All Implemented Interfaces:

opennlp.tools.tokenize.Tokenizer

public class SimpleTokenizer
extends Object

A basic Tokenizer implementation which performs tokenization using character classes.

To obtain an instance of this tokenizer use the static final INSTANCE field.

Field Summary

Fields

Modifier and Type	Field	Description
static final SimpleTokenizer	INSTANCE	Use this static reference to retrieve an instance of the SimpleTokenizer.

Method Summary

Modifier and Type	Method	Description
void	setKeepNewLines(boolean arg0)
String[]	tokenize(String arg0)
opennlp.tools.util.Span[]	tokenizePos(String s)

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

INSTANCE

public static final SimpleTokenizer INSTANCE

Use this static reference to retrieve an instance of the SimpleTokenizer.

Method Details

tokenizePos

public opennlp.tools.util.Span[] tokenizePos(String s)

tokenize

public String[] tokenize(String arg0)

Specified by:

tokenize in interface opennlp.tools.tokenize.Tokenizer

setKeepNewLines

public void setKeepNewLines(boolean arg0)