Package opennlp.tools.tokenize

Class TokenSample

java.lang.Object
opennlp.tools.tokenize.TokenSample
All Implemented Interfaces:
Serializable, Sample
public class TokenSample extends Object implements Sample
A TokenSample is text with token spans.
See Also:

  • Field Details

  • Constructor Details

    • TokenSample

      public TokenSample(String text, Span[] tokenSpans)
      Initializes a instance.
      Parameters:
      text - The text which contains the tokens. Must not be null.
      tokenSpans - The spans which mark the start and end of the tokens. Must not be null.

    • TokenSample

      public TokenSample(Detokenizer detokenizer, String[] tokens)
      Initializes a instance via a Detokenizer.
      Parameters:
      detokenizer - The text which contains the tokens. Must not be null.
      tokens - The tokens to be processed. Must not be null.

  • Method Details

    • getText

      public String getText()
      Returns:
      Retrieves the text.

    • getTokenSpans

      public Span[] getTokenSpans()
      Returns:
      Retrieves the token spans.

    • toString

      public String toString()
      Overrides:
      toString in class Object

    • parse

      public static TokenSample parse(String sampleString, String separatorChars)
      Parses a string sample.
      Parameters:
      sampleString - The sample to be parsed. Must not be null.
      separatorChars - The characters to be considered separators. See DEFAULT_SEPARATOR_CHARS. Must not be null.
      Returns:
      A valid TokenSample instance.

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class Object

    • equals

      public boolean equals(Object obj)
      Overrides:
      equals in class Object