Package opennlp.tools.util.normalizer

Class CodePointSet

java.lang.Object
opennlp.tools.util.normalizer.CodePointSet
public final class CodePointSet extends Object
An immutable set of Unicode code points with O(1) membership.

Backed by a BitSet keyed directly by code point, so contains(int) is a single array-word read with no boxing, hashing, or branching beyond a range check. Memory is bounded by the largest member code point (the whole of Unicode would cost about 136 KiB, and the standard whitespace and dash sets are entirely or almost entirely in the Basic Multilingual Plane, so a few kilobytes in practice).

This type carries no opinion about what the code points mean. It is the explicit, standards-sourced data layer that CharClass and the reference tables (UnicodeWhitespace, UnicodeDash) are built from, and that users extend or override through fromFile(Path, String).

  • Method Summary

    Modifier and Type
    Method
    Description
    boolean
    contains(int codePoint)
    Tests membership.
    boolean
    equals(Object o)
     
    static CodePointSet
    fromFile(Path definitions, String section)
    Loads the code points declared under one section of a user definitions file.
    int
    hashCode()
     
    boolean
    isEmpty()
    Returns whether this set is empty.
    static CodePointSet
    of(int... codePoints)
    Creates a set from explicit code points.
    static CodePointSet
    ofRange(int firstInclusive, int lastInclusive)
    Creates a set covering an inclusive code point range.
    int
    size()
    Returns the number of code points in this set.
    int[]
    toArray()
    Returns the member code points, in ascending order.
    CodePointSet
    union(CodePointSet other)
    Returns a new set containing every code point in this set or other.

    Methods inherited from class java.lang.Object

    getClass, notify, notifyAll, toString, wait, wait, wait

  • Method Details

    • of

      public static CodePointSet of(int... codePoints)
      Creates a set from explicit code points.
      Parameters:
      codePoints - The code points to include.
      Returns:
      The set.
      Throws:
      IllegalArgumentException - Thrown if any value is not a valid Unicode code point (outside [0, U+10FFFF]).

    • ofRange

      public static CodePointSet ofRange(int firstInclusive, int lastInclusive)
      Creates a set covering an inclusive code point range.
      Parameters:
      firstInclusive - The first code point in the range.
      lastInclusive - The last code point in the range.
      Returns:
      The set.
      Throws:
      IllegalArgumentException - Thrown if either bound is invalid or firstInclusive is greater than lastInclusive.

    • fromFile

      public static CodePointSet fromFile(Path definitions, String section) throws IOException
      Loads the code points declared under one section of a user definitions file.

      The format is line oriented and parsed with simple cursor scanning, not a regular expression: a [name] line opens a section; a # begins a comment that runs to end of line; each remaining line is a single hex code point (U+00A0, 0x00A0, or 00A0) or an inclusive range (U+2000-U+200A). Section names match case insensitively. Only entries under the requested section are returned, so one file can carry, for example, both [whitespace] and [dash] sections.

      Parameters:
      definitions - The file to read (UTF-8).
      section - The section whose entries should be loaded.
      Returns:
      The code points declared under section, or an empty set if the section is absent.
      Throws:
      IOException - Thrown if the file cannot be read.
      IllegalArgumentException - Thrown if a line is malformed, naming the offending line.

    • contains

      public boolean contains(int codePoint)
      Tests membership.
      Parameters:
      codePoint - The code point to test. Out-of-range values return false.
      Returns:
      true if the code point is in this set.

    • union

      public CodePointSet union(CodePointSet other)
      Returns a new set containing every code point in this set or other.
      Parameters:
      other - The set to union with.
      Returns:
      The union, a new set; neither input is modified.

    • size

      public int size()
      Returns the number of code points in this set.
      Returns:
      the number of code points in this set

    • isEmpty

      public boolean isEmpty()
      Returns whether this set is empty.
      Returns:
      whether this set is empty

    • toArray

      public int[] toArray()
      Returns the member code points, in ascending order.
      Returns:
      the member code points, in ascending order

    • equals

      public boolean equals(Object o)
      Overrides:
      equals in class Object

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class Object