opennlp.tools.tokenize

Class TokenSample

java.lang.Object

opennlp.tools.tokenize.TokenSample

public class TokenSample
extends Object

A TokenSample is text with token spans.

Field Summary

Fields

Modifier and Type	Field and Description
static String	DEFAULT_SEPARATOR_CHARS

Constructor Summary

Constructors

Constructor and Description
TokenSample(Detokenizer detokenizer, String[] tokens)
TokenSample(String text, Span[] tokenSpans) Initializes the current instance.

Method Summary

Methods

Modifier and Type	Method and Description
boolean	equals(Object obj)
String	getText() Retrieves the text.
Span[]	getTokenSpans() Retrieves the token spans.
static TokenSample	parse(String sampleString, String separatorChars)
String	toString()

Methods inherited from class java.lang.Object

getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

DEFAULT_SEPARATOR_CHARS

public static final String DEFAULT_SEPARATOR_CHARS

See Also:

Constant Field Values

Constructor Detail

TokenSample

public TokenSample(String text,
           Span[] tokenSpans)

Initializes the current instance.

Parameters:

text - the text which contains the tokens.

tokenSpans - the spans which mark the begin and end of the tokens.

TokenSample

public TokenSample(Detokenizer detokenizer,
           String[] tokens)

Method Detail

getText

public String getText()

Retrieves the text.

getTokenSpans

public Span[] getTokenSpans()

Retrieves the token spans.

toString

public String toString()

Overrides:

toString in class Object

parse

public static TokenSample parse(String sampleString,
                String separatorChars)

equals

public boolean equals(Object obj)

Overrides:

equals in class Object