Package opennlp.tools.util.normalizer

Class InvisibleCharSequenceNormalizer

java.lang.Object
opennlp.tools.util.normalizer.InvisibleCharSequenceNormalizer
All Implemented Interfaces:
Serializable, opennlp.tools.util.normalizer.CharSequenceNormalizer, opennlp.tools.util.normalizer.OffsetAwareNormalizer
public class InvisibleCharSequenceNormalizer extends Object implements opennlp.tools.util.normalizer.OffsetAwareNormalizer
A CharSequenceNormalizer that removes invisible format and bidirectional control characters that add no textual content and are a common source of noise and spoofing (the byte-order mark, zero width space, word joiner, bidi marks/embeddings/overrides/isolates, the invisible math operators, soft hyphen, and the Arabic letter mark).

Membership is an O(1) CharClass lookup and removal is a single cursor pass with no regular expression. The zero width joiner (U+200D) and non-joiner (U+200C) are deliberately kept, because they carry meaning in Persian, Indic scripts, and emoji sequences; so are variation selectors. Use this only for a matching/search form, not for display.

See Also:

  • Constructor Details

    • InvisibleCharSequenceNormalizer

      public InvisibleCharSequenceNormalizer()

  • Method Details

    • getInstance

      public static InvisibleCharSequenceNormalizer getInstance()
      Returns the shared, stateless instance.
      Returns:
      the shared, stateless instance

    • normalize

      public CharSequence normalize(CharSequence text)
      Specified by:
      normalize in interface opennlp.tools.util.normalizer.CharSequenceNormalizer

    • normalizeAligned

      public opennlp.tools.util.normalizer.AlignedText normalizeAligned(CharSequence text)
      Specified by:
      normalizeAligned in interface opennlp.tools.util.normalizer.OffsetAwareNormalizer