Package opennlp.tools.util.normalizer

Class SocialMediaCharSequenceNormalizer

java.lang.Object
opennlp.tools.util.normalizer.SocialMediaCharSequenceNormalizer
All Implemented Interfaces:
Serializable, opennlp.tools.util.normalizer.CharSequenceNormalizer
public class SocialMediaCharSequenceNormalizer extends Object implements opennlp.tools.util.normalizer.CharSequenceNormalizer
A CharSequenceNormalizer implementation that normalizes social media text: hashtags, user handles, retweet markers, emoticons, and repeated laughter. Every encounter will be replaced by a whitespace.

Normalization runs in four passes: a # or @ together with the following non-whitespace run becomes one space; each sequence of case-insensitive rt units, each followed by a space or colon, becomes one space when it starts on a word boundary; each emoticon (eyes :, ; or x, an optional - nose, and a mouth out of ( ) d o p, case-insensitive) becomes one space, including inside words. Finally, repeated laughter shrinks to two of its repeating units ("hahaha" to "haha"), comparing ASCII case-insensitively.

See Also:

  • Constructor Details

    • SocialMediaCharSequenceNormalizer

      public SocialMediaCharSequenceNormalizer()

  • Method Details

    • getInstance

      public static SocialMediaCharSequenceNormalizer getInstance()
      Returns the shared, stateless instance.
      Returns:
      the shared, stateless instance

    • normalize

      public CharSequence normalize(CharSequence text)
      Specified by:
      normalize in interface opennlp.tools.util.normalizer.CharSequenceNormalizer