Package opennlp.tools.util.normalizer

Class CharClass

java.lang.Object
opennlp.tools.util.normalizer.CharClass
public final class CharClass extends Object
A configurable class of Unicode code points and the cursor based operations over it.

A CharClass pairs a CodePointSet of member code points with a single canonical ASCII replacement code point. Whitespace and dashes are the two built-in presets (whitespace(), dashes()); any other class is one more configured instance with no new engine code.

Every operation is a single forward cursor pass over the text: no regular expression and no per-call allocation beyond the result.

Instances are immutable and thread-safe.

  • Method Details

    • of

      public static CharClass of(CodePointSet members, int replacement)
      Creates a class from a member set and a replacement code point.
      Parameters:
      members - The member code points.
      replacement - The canonical code point used by normalize(CharSequence) and collapse(CharSequence).
      Returns:
      The class.
      Throws:
      IllegalArgumentException - Thrown if members is null or replacement is not a valid code point.

    • whitespace

      public static CharClass whitespace()
      Returns the whitespace preset: the Unicode White_Space set, replacement U+0020.
      Returns:
      the whitespace preset: the Unicode White_Space set, replacement U+0020

    • dashes

      public static CharClass dashes()
      Returns the dash preset: the Unicode Dash set excluding the mathematical minus signs, replacement U+002D.
      Returns:
      the dash preset: the Unicode Dash set excluding the mathematical minus signs, replacement U+002D

    • withAdditional

      public CharClass withAdditional(CodePointSet extra)
      Returns a copy of this class whose member set is extended with extra (for example, user-defined code points loaded from CodePointSet.fromFile(java.nio.file.Path, java.lang.String)).
      Parameters:
      extra - The additional member code points.
      Returns:
      A new CharClass; this instance is unchanged.
      Throws:
      IllegalArgumentException - Thrown if extra is null.

    • members

      public CodePointSet members()
      Returns the member code points of this class.
      Returns:
      the member code points of this class

    • replacement

      public int replacement()
      Returns the canonical replacement code point.
      Returns:
      the canonical replacement code point

    • contains

      public boolean contains(int codePoint)
      Tests membership.
      Parameters:
      codePoint - The code point to test.
      Returns:
      true if the code point is a member of this class.

    • splitSpans

      public List<Span> splitSpans(CharSequence text)
      Splits text into the maximal runs of non-member code points, as character spans into the original text. Runs of members are delimiters and produce no empty spans.
      Parameters:
      text - The text to split.
      Returns:
      The token spans, in order.
      Throws:
      IllegalArgumentException - Thrown if text is null.

    • split

      public String[] split(CharSequence text)
      Splits text into the maximal runs of non-member code points.
      Parameters:
      text - The text to split.
      Returns:
      The tokens, in order, with no empty entries.
      Throws:
      IllegalArgumentException - Thrown if text is null.

    • normalize

      public String normalize(CharSequence text)
      Replaces each member code point with the replacement, one for one.

      When no code point of text is a member, the text is returned unchanged (as its string form) without copying.

      Parameters:
      text - The text to normalize.
      Returns:
      The normalized text.
      Throws:
      IllegalArgumentException - Thrown if text is null.

    • collapse

      public String collapse(CharSequence text)
      Collapses each maximal run of member code points to a single replacement.

      Edges are not trimmed: a leading or trailing run becomes a single replacement, and a string made up entirely of members collapses to one replacement (for whitespace, a single space, not the empty string). Use trim(CharSequence) to drop edge members, or collapse and then trim to do both.

      When no code point of text is a member, the text is returned unchanged (as its string form) without copying.

      Parameters:
      text - The text to collapse.
      Returns:
      The collapsed text.
      Throws:
      IllegalArgumentException - Thrown if text is null.

    • collapsePreserving

      public String collapsePreserving(CharSequence text, CodePointSet keep, int keepReplacement)
      Collapses runs of members like collapse(CharSequence), but emits keepReplacement instead of the usual replacement for any run that contains a code point in keep. The whitespace "squish" that preserves a line break uses this with the line-break code points as keep and '\n' as keepReplacement.
      Parameters:
      text - The text to collapse.
      keep - The member code points whose presence in a run preserves structure.
      keepReplacement - The replacement emitted for a run that contains a keep member.
      Returns:
      The collapsed text.
      Throws:
      IllegalArgumentException - Thrown if text or keep is null, or keepReplacement is not a valid code point.

    • trim

      public String trim(CharSequence text)
      Removes leading and trailing member code points.
      Parameters:
      text - The text to trim.
      Returns:
      The trimmed text.
      Throws:
      IllegalArgumentException - Thrown if text is null.

    • removeAll

      public String removeAll(CharSequence text)
      Removes every member code point.

      When no code point of text is a member, the text is returned unchanged (as its string form) without copying.

      Parameters:
      text - The text to filter.
      Returns:
      The text with all members removed.
      Throws:
      IllegalArgumentException - Thrown if text is null.

    • normalizeAligned

      public AlignedText normalizeAligned(CharSequence text)
      Like normalize(CharSequence) but also produces the Alignment back to the original text.
      Parameters:
      text - The text to normalize.
      Returns:
      The normalized text and its alignment.
      Throws:
      IllegalArgumentException - Thrown if text is null.

    • collapseAligned

      public AlignedText collapseAligned(CharSequence text)
      Like collapse(CharSequence) but also produces the Alignment back to the original text. Each collapsed run maps to the run's whole original extent.
      Parameters:
      text - The text to collapse.
      Returns:
      The collapsed text and its alignment.
      Throws:
      IllegalArgumentException - Thrown if text is null.

    • collapsePreservingAligned

      public AlignedText collapsePreservingAligned(CharSequence text, CodePointSet keep, int keepReplacement)
      Like collapsePreserving(CharSequence, CodePointSet, int) but also produces the Alignment back to the original text.
      Parameters:
      text - The text to collapse.
      keep - The member code points whose presence in a run preserves structure.
      keepReplacement - The replacement emitted for a run that contains a keep member.
      Returns:
      The collapsed text and its alignment.
      Throws:
      IllegalArgumentException - Thrown if text or keep is null, or keepReplacement is not a valid code point.

    • trimAligned

      public AlignedText trimAligned(CharSequence text)
      Like trim(CharSequence) but also produces the Alignment back to the original text. The trimmed leading and trailing members appear as deletions, so a span never reports through them.
      Parameters:
      text - The text to trim.
      Returns:
      The trimmed text and its alignment.
      Throws:
      IllegalArgumentException - Thrown if text is null.

    • removeAllAligned

      public AlignedText removeAllAligned(CharSequence text)
      Like removeAll(CharSequence) but also produces the Alignment back to the original text. Every removed member appears as a deletion, so a span never reports through one.
      Parameters:
      text - The text to filter.
      Returns:
      The filtered text and its alignment.
      Throws:
      IllegalArgumentException - Thrown if text is null.

    • substitute

      public static String substitute(CharSequence text, IntFunction<String> substitution)
      Applies a per-code-point substitution: each code point for which substitution returns a non-null string is replaced by that string, and the rest are copied through. This is the shared cursor pass behind the expanding folds, with no regular expression.

      When substitution returns null for every code point of text, the text is returned unchanged (as its string form) without copying. The mapper is still applied exactly once per code point.

      Parameters:
      text - The text to transform.
      substitution - The replacement for a code point, or null to copy it through.
      Returns:
      The transformed text.
      Throws:
      IllegalArgumentException - Thrown if text or substitution is null.

    • substituteAligned

      public static AlignedText substituteAligned(CharSequence text, IntFunction<String> substitution)
      Like substitute(CharSequence, IntFunction) but also produces the Alignment back to the original text. Each replaced code point maps to its replacement string as one block.
      Parameters:
      text - The text to transform.
      substitution - The replacement for a code point, or null to copy it through.
      Returns:
      The transformed text and its alignment.
      Throws:
      IllegalArgumentException - Thrown if text or substitution is null.