Package opennlp.tools.tokenize

Class TokenSample

java.lang.Object

opennlp.tools.tokenize.TokenSample

All Implemented Interfaces:

Serializable, Sample

public class TokenSample
extends Object
implements Sample

A TokenSample is text with token spans.

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field	Description
static String	DEFAULT_SEPARATOR_CHARS

Constructor Summary

Constructors

Constructor	Description
TokenSample(String text, Span[] tokenSpans)	Initializes a instance.
TokenSample(Detokenizer detokenizer, String[] tokens)	Initializes a instance via a Detokenizer.

Method Summary

Modifier and Type	Method	Description
boolean	equals(Object obj)
String	getText()
Span[]	getTokenSpans()
int	hashCode()
static TokenSample	parse(String sampleString, String separatorChars)	Parses a string sample.
String	toString()

Methods inherited from class java.lang.Object

getClass, notify, notifyAll, wait, wait, wait

Field Detail

DEFAULT_SEPARATOR_CHARS

public static final String DEFAULT_SEPARATOR_CHARS

See Also:

Constant Field Values

Constructor Detail

TokenSample

public TokenSample(String text,
                   Span[] tokenSpans)

Initializes a instance.

Parameters:

text - The text which contains the tokens. Must not be null.

tokenSpans - The spans which mark the start and end of the tokens. Must not be null.

TokenSample

public TokenSample(Detokenizer detokenizer,
                   String[] tokens)

Initializes a instance via a Detokenizer.

Parameters:

detokenizer - The text which contains the tokens. Must not be null.

tokens - The tokens to be processed. Must not be null.

Method Detail

getText

public String getText()

Returns:

Retrieves the text.

getTokenSpans

public Span[] getTokenSpans()

Returns:

Retrieves the token spans.

toString

public String toString()

Overrides:

toString in class Object

parse

public static TokenSample parse(String sampleString,
                                String separatorChars)

Parses a string sample.

Parameters:

sampleString - The sample to be parsed. Must not be null.

separatorChars - The characters to be considered separators. See DEFAULT_SEPARATOR_CHARS. Must not be null.

Returns:

A valid TokenSample instance.

hashCode

public int hashCode()

Overrides:

hashCode in class Object

equals

public boolean equals(Object obj)

Overrides:

equals in class Object