Package opennlp.tools.util

Class StringUtil

java.lang.Object
opennlp.tools.util.StringUtil
public class StringUtil extends Object

  • Field Details

    • ASCII_DIGIT_STRINGS

      public static final List<String> ASCII_DIGIT_STRINGS
      The ten ASCII digit strings "0" to "9", indexed by digit value. Precomputed so code folding digits to ASCII does not allocate a new single-character string per digit; the list is immutable and safe to share.

  • Constructor Details

    • StringUtil

      public StringUtil()

  • Method Details

    • isWhitespace

      public static boolean isWhitespace(char charCode)
      Determines if the specified Character is a whitespace under the active WhitespaceMode: the Unicode White_Space property by default, or, when the "opennlp.whitespace.mode" system property selects WhitespaceMode.LEGACY, the union of Character.isWhitespace(int) and the Unicode Zs category (Character.SPACE_SEPARATOR) that OpenNLP 1.x and 2.x used.

      Tokenization, corpus format parsing, and feature generation all resolve whitespace through this method, so they share one WhitespaceMode for the life of the process; use isUnicodeWhitespace(char) directly where the Unicode definition should apply unconditionally, such as user-text normalization with no trained-model dependency.

      Parameters:
      charCode - The character to check.
      Returns:
      true if charCode represents a white space under the active WhitespaceMode, false otherwise.

    • isWhitespace

      public static boolean isWhitespace(int charCode)
      Determines if the specified code point is a whitespace under the active WhitespaceMode; see isWhitespace(char) for details.
      Parameters:
      charCode - An int representation of a character to check.
      Returns:
      true if charCode represents a white space under the active WhitespaceMode, false otherwise.

    • isUnicodeWhitespace

      public static boolean isUnicodeWhitespace(char charCode)
      Determines if the specified Character is a whitespace under the Unicode White_Space property; delegates to UnicodeWhitespace.isWhitespace(int). This is also what isWhitespace(char) resolves to under the default WhitespaceMode.
      Parameters:
      charCode - The character to check.
      Returns:
      true if charCode has the White_Space property, false otherwise.

    • isUnicodeWhitespace

      public static boolean isUnicodeWhitespace(int charCode)
      Determines if the specified code point is a whitespace under the Unicode White_Space property; delegates to UnicodeWhitespace.isWhitespace(int). This is also what isWhitespace(int) resolves to under the default WhitespaceMode.
      Parameters:
      charCode - An int representation of a character to check.
      Returns:
      true if charCode has the White_Space property, false otherwise.

    • splitOnUnicodeWhitespace

      public static String[] splitOnUnicodeWhitespace(CharSequence input)
      Splits input on runs of Unicode White_Space. Leading and trailing runs are ignored, so whitespace-only input yields an empty array. This is a code-point scan, not a regular expression.
      Parameters:
      input - The text to split. Must not be null.
      Returns:
      The non-whitespace terms in order.
      Throws:
      IllegalArgumentException - If input is null.

    • trimUnicodeWhitespace

      public static String trimUnicodeWhitespace(CharSequence input)
      Trims leading and trailing runs of Unicode White_Space, the same set splitOnUnicodeWhitespace(CharSequence) breaks terms on.
      Parameters:
      input - The text to trim. Must not be null.
      Returns:
      The trimmed string; may be empty when input is whitespace-only.
      Throws:
      IllegalArgumentException - If input is null.

    • isUnicodeBlank

      public static boolean isUnicodeBlank(CharSequence input)
      true when input is null, empty, or consists only of Unicode White_Space code points.
      Parameters:
      input - The text to test; null is treated as blank.
      Returns:
      true if there is no non-whitespace code point.

    • toLowerCase

      public static String toLowerCase(CharSequence string)
      Converts a CharSequence to lower case, independent of the current Locale via Character.toLowerCase(int) which uses mapping information from the UnicodeData file.
      Parameters:
      string - The CharSequence to transform.
      Returns:
      The lower-cased String.

    • toLowerCaseCharBuffer

      public static CharBuffer toLowerCaseCharBuffer(CharSequence sequence)

    • toUpperCase

      public static String toUpperCase(CharSequence string)
      Converts a CharSequence to upper case, independent of the current Locale via Character.toUpperCase(char) which uses mapping information from the UnicodeData file.
      Parameters:
      string - The CharSequence to transform.
      Returns:
      The upper-cased String

    • isEmpty

      public static boolean isEmpty(CharSequence theString)
      Returns:
      true if CharSequence.length() is 0 or null, otherwise false
      Since:
      1.5.1

    • levenshteinDistance

      public static int[][] levenshteinDistance(String wordForm, String lemma)
      Computes the Levenshtein distance of two strings in a matrix.

      Based on this pseudo-code which in turn is based on the paper Wagner, Robert A.; Fischer, Michael J. (1974), "The String-to-String Correction Problem", Journal of the ACM 21 (1): 168-173

      Parameters:
      wordForm - The form as input.
      lemma - The target lemma.
      Returns:
      A 2-dimensional Levenshtein distance matrix.

    • computeShortestEditScript

      public static void computeShortestEditScript(String wordForm, String lemma, int[][] distance, StringBuffer permutations)
      Computes the Shortest Edit Script (SES) to convert a word into its lemma. This is based on Chrupala's PhD thesis (2008).
      Parameters:
      wordForm - The token.
      lemma - The target lemma.
      distance - A 2-dimensional Levenshtein distance matrix.
      permutations - The number of permutations.

    • decodeShortestEditScript

      public static String decodeShortestEditScript(String wordForm, String permutations)
      Reads the predicted Shortest Edit Script (SES) by a lemmatizer model and applies the permutations to obtain the lemma from the wordForm.
      Parameters:
      wordForm - The wordForm as input.
      permutations - The permutations predicted by the lemmatizer model.
      Returns:
      The decoded lemma.

    • getShortestEditScript

      public static String getShortestEditScript(String wordForm, String lemma)
      Parameters:
      wordForm - The word as input.
      lemma - The target lemma.
      Returns:
      Retrieves the Shortest Edit Script (SES) required to go from a word to a lemma.