Package opennlp.tools.tokenize

Klasse WhitespaceTokenizer

java.lang.Object

opennlp.tools.tokenize.WhitespaceTokenizer

Alle implementierten Schnittstellen:

Tokenizer

public class WhitespaceTokenizer
extends Object

A basic Tokenizer implementation which performs tokenization using white spaces.

To obtain an instance of this tokenizer use the static final INSTANCE field.

Feldübersicht

Felder

Modifizierer und Typ	Feld	Beschreibung
static final WhitespaceTokenizer	INSTANCE	Use this static reference to retrieve an instance of the WhitespaceTokenizer.

Methodenübersicht

Modifizierer und Typ	Methode	Beschreibung
void	setKeepNewLines(boolean keepNewLines)	Switches whether to keep new lines or not.
String[]	tokenize(String s)	Splits a string into its atomic parts.
Span[]	tokenizePos(String d)	Finds the boundaries of atomic parts in a string.

Von Klasse geerbte Methoden java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Felddetails

INSTANCE

public static final WhitespaceTokenizer INSTANCE

Use this static reference to retrieve an instance of the WhitespaceTokenizer.

Methodendetails

tokenizePos

public Span[] tokenizePos(String d)

Beschreibung aus Schnittstelle kopiert: Tokenizer

Finds the boundaries of atomic parts in a string.

Parameter:

d - The string to be tokenized.

Gibt zurück:

The spans (offsets into s) for each token as the individuals array elements.

tokenize

public String[] tokenize(String s)

Beschreibung aus Schnittstelle kopiert: Tokenizer

Splits a string into its atomic parts.

Angegeben von:

tokenize in Schnittstelle Tokenizer

Parameter:

s - The string to be tokenized.

Gibt zurück:

The String[] with the individual tokens as the array elements.

setKeepNewLines

public void setKeepNewLines(boolean keepNewLines)

Switches whether to keep new lines or not.

Parameter:

keepNewLines - True if new lines are kept, false otherwise.