Package opennlp.tools.util.normalizer

Class TermAnalyzer

java.lang.Object
opennlp.tools.util.normalizer.TermAnalyzer
public final class TermAnalyzer extends Object
Builds Terms by segmenting text and applying a configured stack of normalization Dimensions to each token. The analyzer is the configuration; each Term is the layered result for one token, with the configured dimensions computed eagerly and any other dimension computed lazily on first request.

Segmentation uses the Unicode UAX #29 word tokenizer, so the input does not need to be pre-tokenized. The character-level dimensions (Dimension.NFC through Dimension.CONFUSABLE_FOLD) have built-in defaults; Dimension.STEM and Dimension.LEMMA are enabled by supplying a Stemmer or Lemmatizer.

An instance is immutable and is thread-safe when its configured transforms are. The built-in character normalizers are stateless and the Snowball stemmers are thread-safe. When Dimension.STEM is enabled through TermAnalyzer.Builder.stem(StemmerFactory), stemming is routed through a CachingStemmer: the analyzer is safe to share across threads even when the factory mints stateful stemmers, and repeated words resolve from a bounded per-thread cache instead of being re-stemmed. A raw Stemmer passed to TermAnalyzer.Builder.stem(Stemmer) is only safe when that stemmer is itself thread-safe or the analyzer is confined to one thread.

  • Method Details

    • builder

      public static TermAnalyzer.Builder builder()
      Returns a new TermAnalyzer.Builder. The builder starts with no dimensions enabled and the default UAX #29 word tokenizer; enable dimensions and set a stemmer, lemmatizer, or tokenizer on it, then call TermAnalyzer.Builder.build().
      Returns:
      a new TermAnalyzer.Builder

    • analyze

      public List<Term> analyze(CharSequence text)
      Segments text with the UAX #29 word tokenizer and returns one Term per word token, in order. The terms carry no part-of-speech tag, so Dimension.LEMMA cannot be computed from this entry point: if a lemmatizer is configured, this method throws -- use analyze(String[], String[]) when lemmas are needed.
      Parameters:
      text - The text to analyze. Must not be null.
      Returns:
      The terms.
      Throws:
      IllegalArgumentException - if text is null.
      IllegalStateException - if Dimension.LEMMA is configured, because no part-of-speech tags are available from raw text.

    • analyze

      public List<Term> analyze(String[] tokens, String[] tags)
      Returns one Term per supplied token, attaching the matching part-of-speech tag so that Dimension.LEMMA can be computed. The terms have no source span.
      Parameters:
      tokens - The tokens. Must not be null or contain null elements.
      tags - The part-of-speech tag for each token; must be the same length as tokens and must not be null. A null tag is only acceptable when Dimension.LEMMA is not computed for that token.
      Returns:
      The terms.
      Throws:
      IllegalArgumentException - if tokens or tags is null, if they differ in length, or if tokens contains a null element.

    • dimensions

      public List<Dimension> dimensions()
      Returns the configured dimensions that are computed eagerly, in pipeline order. The list never includes Dimension.ORIGINAL, which is always present.
      Returns:
      the configured dimensions that are computed eagerly, in pipeline order