Package opennlp.tools.util.normalizer

Enum Class Dimension

java.lang.Object
java.lang.Enum<Dimension>
opennlp.tools.util.normalizer.Dimension
All Implemented Interfaces:
Serializable, Comparable<Dimension>, Constable
public enum Dimension extends Enum<Dimension>
A layer of the Term normalization stack, in increasing order of aggressiveness. The declaration order is the canonical application order, because the transforms do not commute. Each character-level dimension carries its default CharSequenceNormalizer, resolved lazily on first request.

ORIGINAL is the source token and is always present. STEM and LEMMA are token-level and have no default normalizer; they require a Stemmer or Lemmatizer on the analyzer (LEMMA also a part-of-speech tag).

  • Nested Class Summary

    Nested classes/interfaces inherited from class java.lang.Enum

    Enum.EnumDesc<E extends Enum<E>>

  • Enum Constant Summary

    Enum Constants
    Enum Constant
    Description
    ACCENT_FOLD
    Diacritic and accent folding; lossy, script gated, and language-wrong for some languages.
    CASE_FOLD
    Case folding; lossy and locale sensitive.
    CONFUSABLE_FOLD
    Confusable (homoglyph) skeleton folding per UTS #39; lossy, for matching only.
    DASH
    Unicode dashes folded to the ASCII hyphen-minus.
    EMOJI_FOLD
    Emoji folded to ASCII emoticons where a mapping exists; lossy, for matching only.
    FULL_CASE_FOLD
    Unicode full case folding (UTS #21); lossy and expanding (sharp s to ss, the ligatures).
    LEMMA
    Lemmatization through a configured Lemmatizer.
    NFC
    Unicode canonical composition (NFC); lossless under canonical equivalence.
    NFKC
    Unicode compatibility composition (NFKC); lossy (for example superscripts to digits).
    ORIGINAL
    The original token text, the source of truth.
    STEM
    Stemming through a configured Stemmer.
    WHITESPACE
    Unicode whitespace folded to ASCII spaces.

  • Method Summary

    Modifier and Type
    Method
    Description
    opennlp.tools.util.normalizer.CharSequenceNormalizer
    defaultNormalizer()
    Returns the default character-level normalizer for this dimension, or null for ORIGINAL, STEM, and LEMMA.
    static Dimension
    valueOf(String name)
    Returns the enum constant of this class with the specified name.
    static Dimension[]
    values()
    Returns an array containing the constants of this enum class, in the order they are declared.

    Methods inherited from class java.lang.Enum

    compareTo, describeConstable, equals, getDeclaringClass, hashCode, name, ordinal, toString, valueOf

    Methods inherited from class java.lang.Object

    getClass, notify, notifyAll, wait, wait, wait

  • Enum Constant Details

    • ORIGINAL

      public static final Dimension ORIGINAL
      The original token text, the source of truth.

    • NFC

      public static final Dimension NFC
      Unicode canonical composition (NFC); lossless under canonical equivalence.

    • NFKC

      public static final Dimension NFKC
      Unicode compatibility composition (NFKC); lossy (for example superscripts to digits).

    • WHITESPACE

      public static final Dimension WHITESPACE
      Unicode whitespace folded to ASCII spaces.

    • DASH

      public static final Dimension DASH
      Unicode dashes folded to the ASCII hyphen-minus.

    • CASE_FOLD

      public static final Dimension CASE_FOLD
      Case folding; lossy and locale sensitive.

    • FULL_CASE_FOLD

      public static final Dimension FULL_CASE_FOLD
      Unicode full case folding (UTS #21); lossy and expanding (sharp s to ss, the ligatures).

    • ACCENT_FOLD

      public static final Dimension ACCENT_FOLD
      Diacritic and accent folding; lossy, script gated, and language-wrong for some languages.

    • EMOJI_FOLD

      public static final Dimension EMOJI_FOLD
      Emoji folded to ASCII emoticons where a mapping exists; lossy, for matching only.

    • CONFUSABLE_FOLD

      public static final Dimension CONFUSABLE_FOLD
      Confusable (homoglyph) skeleton folding per UTS #39; lossy, for matching only.

    • STEM

      public static final Dimension STEM
      Stemming through a configured Stemmer.

    • LEMMA

      public static final Dimension LEMMA
      Lemmatization through a configured Lemmatizer.

  • Method Details

    • values

      public static Dimension[] values()
      Returns an array containing the constants of this enum class, in the order they are declared.
      Returns:
      an array containing the constants of this enum class, in the order they are declared

    • valueOf

      public static Dimension valueOf(String name)
      Returns the enum constant of this class with the specified name. The string must match exactly an identifier used to declare an enum constant in this class. (Extraneous whitespace characters are not permitted.)
      Parameters:
      name - the name of the enum constant to be returned.
      Returns:
      the enum constant with the specified name
      Throws:
      IllegalArgumentException - if this enum class has no constant with the specified name
      NullPointerException - if the argument is null

    • defaultNormalizer

      public opennlp.tools.util.normalizer.CharSequenceNormalizer defaultNormalizer()
      Returns the default character-level normalizer for this dimension, or null for ORIGINAL, STEM, and LEMMA. The normalizer is resolved lazily, so it is not initialized until first requested.
      Returns:
      the default character-level normalizer for this dimension, or null for ORIGINAL, STEM, and LEMMA