public class TokenSample extends Object implements Serializable
TokenSample
is text with token spans.Modifier and Type | Field and Description |
---|---|
static String |
DEFAULT_SEPARATOR_CHARS |
Constructor and Description |
---|
TokenSample(Detokenizer detokenizer,
String[] tokens) |
TokenSample(String text,
Span[] tokenSpans)
Initializes the current instance.
|
Modifier and Type | Method and Description |
---|---|
boolean |
equals(Object obj) |
String |
getText()
Retrieves the text.
|
Span[] |
getTokenSpans()
Retrieves the token spans.
|
int |
hashCode() |
static TokenSample |
parse(String sampleString,
String separatorChars) |
String |
toString() |
public static final String DEFAULT_SEPARATOR_CHARS
public TokenSample(String text, Span[] tokenSpans)
text
- the text which contains the tokens.tokenSpans
- the spans which mark the begin and end of the tokens.public TokenSample(Detokenizer detokenizer, String[] tokens)
public String getText()
public Span[] getTokenSpans()
public static TokenSample parse(String sampleString, String separatorChars)
Copyright © 2018 The Apache Software Foundation. All rights reserved.