Package opennlp.tools.util.normalizer

Class FullCaseFoldCharSequenceNormalizer

java.lang.Object
opennlp.tools.util.normalizer.FullCaseFoldCharSequenceNormalizer
All Implemented Interfaces:
Serializable, opennlp.tools.util.normalizer.CharSequenceNormalizer, opennlp.tools.util.normalizer.OffsetAwareNormalizer
public final class FullCaseFoldCharSequenceNormalizer extends Object implements opennlp.tools.util.normalizer.OffsetAwareNormalizer
A CharSequenceNormalizer that applies Unicode full case folding for case-insensitive matching, as defined by the Default Case Algorithms in Section 3.13 of the Unicode Standard, using the bundled CaseFolding.txt data of the Unicode Character Database.

Unlike CaseFoldCharSequenceNormalizer, which lower cases with Locale.ROOT, this applies the full case foldings (the C common and F full status mappings), including the expanding folds that plain lower casing does not perform: the sharp s (U+00DF) to ss, the Latin ligatures (for example U+FB00 to ff), and the Greek and Armenian multi-character folds. It is therefore an expanding, offset-changing transform, so it is offset-aware: normalizeAligned(CharSequence) reports the Alignment from the folded text back to the input. A single cursor pass with no regular expression.

The S simple and T Turkic status mappings are excluded, so the Turkish and Azerbaijani dotless-i rule is not applied here. Input is expected in NFC: the fold matches precomposed code points, so decomposed sequences pass through unchanged.

See Also:

  • Method Details

    • getInstance

      public static FullCaseFoldCharSequenceNormalizer getInstance()
      Returns the shared, stateless instance.
      Returns:
      the shared, stateless instance

    • normalize

      public CharSequence normalize(CharSequence text)
      Specified by:
      normalize in interface opennlp.tools.util.normalizer.CharSequenceNormalizer
      Throws:
      IllegalArgumentException - if text is null.

    • normalizeAligned

      public opennlp.tools.util.normalizer.AlignedText normalizeAligned(CharSequence text)
      Specified by:
      normalizeAligned in interface opennlp.tools.util.normalizer.OffsetAwareNormalizer
      Throws:
      IllegalArgumentException - if text is null.