Package opennlp.tools.util.normalizer

Class AccentFoldCharSequenceNormalizer

java.lang.Object
opennlp.tools.util.normalizer.AccentFoldCharSequenceNormalizer
All Implemented Interfaces:
Serializable, opennlp.tools.util.normalizer.CharSequenceNormalizer
public class AccentFoldCharSequenceNormalizer extends Object implements opennlp.tools.util.normalizer.CharSequenceNormalizer
A CharSequenceNormalizer that folds diacritics for matching, the multilingual-safe counterpart to a Latin-only ASCII folding filter.

Folding decomposes the text (NFD) and drops nonspacing combining marks, but only for base characters whose script is in foldScripts (Latin, Greek, and Cyrillic by default). Marks on other scripts are left untouched, because there they are essential orthography rather than decoration: stripping an Indic vowel sign or a virama, an Arabic harakat, a Hebrew point, or a Thai vowel changes the word. This script gating is the key correctness rule; never strip all nonspacing marks globally.

Many "accented" Latin letters are atomic and do not decompose (o with stroke, the ae/oe ligatures, eszett, thorn, and so on). When foldStrokeLetters is enabled (the default) these are mapped to an ASCII approximation. Folding is a recall optimization, not a linguistically correct transform, so it is intended for a matching token rather than for display or language-specific analysis.

Scanning is a single cursor pass over the decomposed text; no regular expression is used, and no global \p{Mn} strip is performed.

See Also:

  • Constructor Details

    • AccentFoldCharSequenceNormalizer

      public AccentFoldCharSequenceNormalizer(Set<Character.UnicodeScript> foldScripts, boolean foldStrokeLetters)
      Creates an accent-folding normalizer.
      Parameters:
      foldScripts - The scripts whose base characters' diacritics are folded; marks on every other script are preserved.
      foldStrokeLetters - Whether atomic Latin letters such as the stroke letters and ligatures are mapped to an ASCII approximation.

  • Method Details

    • getInstance

      public static AccentFoldCharSequenceNormalizer getInstance()
      Returns the shared instance with the safe defaults: Latin, Greek, and Cyrillic plus the stroke-letter map.
      Returns:
      the shared instance with the safe defaults: Latin, Greek, and Cyrillic plus the stroke-letter map

    • normalize

      public CharSequence normalize(CharSequence text)
      Specified by:
      normalize in interface opennlp.tools.util.normalizer.CharSequenceNormalizer