Package opennlp.tools.util.normalizer

Class Term

java.lang.Object
opennlp.tools.util.normalizer.Term
public final class Term extends Object
One token as a stack of normalization layers. The original() form is the source of truth; the other layers are derived, increasingly aggressive Dimensions tuned for matching. The dimensions configured on the producing TermAnalyzer are computed eagerly and cached; any other dimension is computed on first request, applied on top of the configured form, and then cached.

Because the original is always retained, aggressive folding is safe: a match on a derived layer can always be reported in original coordinates through span(). Querying a configured layer, or peeling the last-applied one, is O(1); adding an unconfigured dimension costs one transform on first touch and is O(1) thereafter.

Instances are created by TermAnalyzer and are thread-safe as long as the analyzer's configured transforms are (see TermAnalyzer for the stemmer caveat). Concurrent first requests for the same unconfigured dimension may run its transform more than once, but every thread observes the same cached value.

  • Method Summary

    Modifier and Type
    Method
    Description
    String
    at(Dimension dimension)
    Returns the token at dimension.
    String
    normalized()
    Returns the token at the analyzer's final configured dimension; equal to original() when no dimensions were configured.
    String
    original()
    Returns the original token text.
    String
    peel()
    Returns the token at the dimension just below the final configured one.
    opennlp.tools.util.Span
    span()
    Returns the source span of this token, or null if it was supplied as a pre-tokenized string.

    Methods inherited from class java.lang.Object

    equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Method Details

    • span

      public opennlp.tools.util.Span span()
      Returns the source span of this token, or null if it was supplied as a pre-tokenized string. The span indexes into the text passed to TermAnalyzer.analyze(CharSequence).
      Returns:
      the source span of this token, or null if it was supplied as a pre-tokenized string

    • original

      public String original()
      Returns the original token text.
      Returns:
      the original token text

    • normalized

      public String normalized()
      Returns the token at the analyzer's final configured dimension; equal to original() when no dimensions were configured.
      Returns:
      the token at the analyzer's final configured dimension; equal to original() when no dimensions were configured

    • at

      public String at(Dimension dimension)
      Returns the token at dimension. Configured dimensions are cached; an unconfigured dimension is computed by applying its transform to normalized() and then cached.

      Note: an unconfigured dimension is applied on top of normalized() (the most aggressive configured layer), not spliced into pipeline order. Because the transforms do not commute (see Dimension), requesting a dimension that ranks earlier than the configured ones can differ from having configured it. For example, asking for Dimension.CASE_FOLD on an analyzer configured only through Dimension.ACCENT_FOLD case-folds the already accent-folded text, which is not the same as case-folding first. Configure the dimension on the analyzer when pipeline order matters.

      Parameters:
      dimension - The dimension to project to. Must not be null.
      Returns:
      The token at that dimension.
      Throws:
      IllegalArgumentException - if dimension is null.
      IllegalStateException - if the dimension needs an engine or tag that was not configured (see Dimension.STEM and Dimension.LEMMA).

    • peel

      public String peel()
      Returns the token at the dimension just below the final configured one. This is the last-applied layer removed (for example the form before stemming when Dimension.STEM is the final dimension); equal to original() when at most one dimension is configured.
      Returns:
      the token at the dimension just below the final configured one