Package opennlp.tools.util.normalizer

Class TextNormalizer.Builder

java.lang.Object
opennlp.tools.util.normalizer.TextNormalizer.Builder
Enclosing class:
TextNormalizer
public static final class TextNormalizer.Builder extends Object
A fluent builder that appends normalization steps in order and composes them into one CharSequenceNormalizer via build().

  • Method Details

    • nfc

      public TextNormalizer.Builder nfc()
      Returns this builder with NFC canonical composition appended.
      Returns:
      this builder with NFC canonical composition appended

    • nfkc

      public TextNormalizer.Builder nfkc()
      Returns this builder with NFKC compatibility composition appended.
      Returns:
      this builder with NFKC compatibility composition appended

    • stripInvisible

      public TextNormalizer.Builder stripInvisible()
      Returns this builder with invisible/bidi control stripping appended.
      Returns:
      this builder with invisible/bidi control stripping appended

    • whitespace

      public TextNormalizer.Builder whitespace()
      Returns this builder with Unicode whitespace collapsing appended.
      Returns:
      this builder with Unicode whitespace collapsing appended

    • whitespacePreservingLineBreaks

      public TextNormalizer.Builder whitespacePreservingLineBreaks()
      Returns this builder with whitespace collapsing that preserves line and paragraph breaks appended. Horizontal runs collapse to a single space, but a run containing a line break collapses to a single newline, so paragraph structure survives.
      Returns:
      this builder with whitespace collapsing that preserves line and paragraph breaks appended

    • quotes

      public TextNormalizer.Builder quotes()
      Returns this builder with quotation-mark folding appended.
      Returns:
      this builder with quotation-mark folding appended

    • dashes

      public TextNormalizer.Builder dashes()
      Returns this builder with dash folding appended.
      Returns:
      this builder with dash folding appended

    • digits

      public TextNormalizer.Builder digits()
      Returns this builder with decimal-digit folding appended.
      Returns:
      this builder with decimal-digit folding appended

    • ellipsis

      public TextNormalizer.Builder ellipsis()
      Returns this builder with ellipsis expansion appended.
      Returns:
      this builder with ellipsis expansion appended

    • bullets

      public TextNormalizer.Builder bullets()
      Returns this builder with list-bullet replacement appended.
      Returns:
      this builder with list-bullet replacement appended

    • caseFold

      public TextNormalizer.Builder caseFold()
      Returns this builder with case folding appended.
      Returns:
      this builder with case folding appended

    • fullCaseFold

      public TextNormalizer.Builder fullCaseFold()
      Returns this builder with Unicode full case folding appended.

      Applies the full case foldings of CaseFolding.txt including the expanding folds (sharp s to ss, the ligatures). It expects composed (NFC) input, since it maps per code point. Prefer caseFold() when the plain lower-case fold is sufficient.

      Returns:
      this builder with Unicode full case folding appended

    • accentFold

      public TextNormalizer.Builder accentFold()
      Returns this builder with script-gated diacritic folding appended.
      Returns:
      this builder with script-gated diacritic folding appended

    • emojiToEmoticon

      public TextNormalizer.Builder emojiToEmoticon()
      Returns this builder with emoji-to-emoticon folding appended.

      Folds pictographs with a bundled ASCII emoticon mapping (for example U+1F642 to :)) and copies everything else through. It is offset-aware, so it composes into buildAligned().

      Returns:
      this builder with emoji-to-emoticon folding appended

    • emoticonToEmoji

      public TextNormalizer.Builder emoticonToEmoji()
      Returns this builder with emoticon-to-emoji folding appended.

      Folds a whitespace-delimited ASCII emoticon to its pictograph (for example :-) to U+1F642); an emoticon sequence embedded in other text, such as the :/ in https://, is left alone. Apply before tokenization so emoticons survive as single emoji tokens. It is offset-aware, so it composes into buildAligned().

      Returns:
      this builder with emoticon-to-emoji folding appended

    • with

      public TextNormalizer.Builder with(opennlp.tools.util.normalizer.CharSequenceNormalizer custom)
      Appends a custom normalizer.
      Parameters:
      custom - The normalizer to append. Must not be null.
      Returns:
      This builder.
      Throws:
      IllegalArgumentException - if custom is null.

    • build

      public opennlp.tools.util.normalizer.CharSequenceNormalizer build()
      Returns the composed normalizer for the steps added so far.
      Returns:
      the composed normalizer for the steps added so far

    • buildAligned

      public opennlp.tools.util.normalizer.OffsetAwareNormalizer buildAligned()
      Returns an offset-aware composition of the steps added so far.

      Every step must be an OffsetAwareNormalizer. NFC, NFKC, accent folding, confusable folding, and case folding are rejected because they delegate to Normalizer or JDK case mapping and cannot report their per-character edits. The returned normalizer's OffsetAwareNormalizer.normalizeAligned(CharSequence) maps a span found in the fully normalized text back to the original input through every stage.

      Returns:
      an offset-aware composition of the steps added so far
      Throws:
      IllegalStateException - if any step cannot report an alignment; the message names the offending step