Package opennlp.tools.util.normalizer

Class UnicodeDash

java.lang.Object
opennlp.tools.util.normalizer.UnicodeDash
public final class UnicodeDash extends Object
Reference data for Unicode dashes, plus O(1) membership lookups.

This is a static, immutable table of every code point that carries the Unicode Dash property (Unicode Character Database, PropList.txt). The set is broader than the Pd (dash punctuation) general category: it also includes the swung dash (Po) and the mathematical minus signs (Sm). Java offers no Dash predicate and \p{Pd} would miss the Sm and Po members, which is why the set is kept here explicitly.

Two distinctions matter for normalization:

  • The three mathematical minus signs (U+207B, U+208B, U+2212, all category Sm) are excluded from defaultDashCodePoints() because flattening them to U+002D can change mathematical meaning. They remain available through codePoints() for callers that opt in.
  • U+00AD SOFT HYPHEN is deliberately absent: it is a format character (White_Space=no, Dash=no), an invisible line-break hint, and must not be turned into a visible hyphen.

  • Field Details

    • HYPHEN_MINUS

      public static final int HYPHEN_MINUS
      The canonical ASCII dash that dashes are normalized to: U+002D HYPHEN-MINUS.
      See Also:

  • Method Details

    • isDash

      public static boolean isDash(int codePoint)
      Tests whether a code point carries the Unicode Dash property.
      Parameters:
      codePoint - The code point to test. Out-of-range values return false.
      Returns:
      true if the code point is one of the Unicode dash characters.

    • byCodePoint

      public static Optional<UnicodeDash.DashCharacter> byCodePoint(int codePoint)
      Looks up the reference entry for a dash code point.
      Parameters:
      codePoint - The code point.
      Returns:
      The UnicodeDash.DashCharacter, or Optional.empty() if it is not a dash.

    • all

      public static List<UnicodeDash.DashCharacter> all()
      Returns all Unicode dash characters, in ascending code point order.
      Returns:
      all Unicode dash characters, in ascending code point order

    • mathematical

      public static List<UnicodeDash.DashCharacter> mathematical()
      Returns the mathematical minus signs, excluded from the default normalization set.
      Returns:
      the mathematical minus signs, excluded from the default normalization set

    • codePoints

      public static int[] codePoints()
      Returns all dash code points, in ascending order, including the mathematical minus signs.
      Returns:
      all dash code points, in ascending order, including the mathematical minus signs

    • defaultDashCodePoints

      public static int[] defaultDashCodePoints()
      Returns the dash code points used for normalization by default, in ascending order.

      This is every dash except the mathematical minus signs, so flattening to HYPHEN_MINUS does not silently rewrite mathematics.

      Returns:
      the dash code points used for normalization by default, in ascending order