Package opennlp.tools.util.normalizer
package opennlp.tools.util.normalizer
-
ClassDescriptionA
CharSequenceNormalizerthat folds diacritics for matching, the multilingual-safe counterpart to a Latin-only ASCII folding filter.ACharSequenceNormalizerimplementation that aggregates the functionality of other normalizers.ACharSequenceNormalizerthat replaces unambiguous list-bullet characters with a space, so a bullet acts as a token separator rather than sticking to the following word.ACharSequenceNormalizerthat lower cases text for case-insensitive matching.Computes the Unicode confusable skeleton of text, following the skeleton algorithm defined in UTS #39 (Unicode Security Mechanisms).ACharSequenceNormalizerthat reduces text to its Unicode confusable skeleton (UTS #39).ACharSequenceNormalizerthat maps every Unicode dash to an ASCII hyphen-minus (U+002D), reusing the cursor basedCharClass.dashes()engine.ACharSequenceNormalizerthat maps Unicode decimal digits to their ASCII equivalents, so for example Arabic-Indic, Devanagari, or fullwidth digits all become0-9.A layer of theTermnormalization stack, in increasing order of aggressiveness.ACharSequenceNormalizerthat expands the ellipsis and leader characters to ASCII dots: the horizontal ellipsis (U+2026) to"..."and the two-dot leader (U+2025) to"..".The annotation record of one emoji: an extensible per-symbol store of attribute values, each carrying its own provenance.One attribute value with its provenance, mirroring one row of the bundled data file.The joined-facts layer of the emoji annotation record store: a hook through which facts that live in user-installed data, typically a gazetteer (coordinates, containment chain, GeoNames or Who's On First identifiers), are resolved at run time and merged into anEmojiAnnotation.The bundled-facts layer of the emoji annotation record store: license-clean, provenance-tagged attributes intrinsic to a pictograph (name, coarse sentiment, entity type, document category), loaded once from the project-authoredemoji-annotations.txtresource.Assembles oneEmojiAnnotationper symbol from bundled facts (EmojiAnnotations), derived facts (EmojiFlags), and optional joined facts (EmojiAnnotationJoin).The document-category hint of an annotated emoji, one constant per group header of the upstream Unicodeemoji-test.txt(Emoji Keyboard/Display Test Data for UTS #51).Deprecated, for removal: This API element is subject to removal in a future version.Replaces every supplementary-plane code point with a space, not only emoji.The coarse entity type of an annotated emoji, a small project-authored taxonomy assigned from the subgroup headers of the upstream Unicodeemoji-test.txt(Emoji Keyboard/Display Test Data for UTS #51): for example theface-smilingandface-concernedsubgroups map toEmojiEntityType.FACEandtransport-groundmaps toEmojiEntityType.VEHICLE.The derived-facts layer of the emoji annotation record store: decodes flag emoji to ISO 3166 codes purely from the code point sequence, with no data file.ACharSequenceNormalizerthat folds emoji to ASCII emoticons, using the bundledemoji-emoticons.txtmapping (for example U+1F642 SLIGHTLY SMILING FACE to:)).ACharSequenceNormalizerthat folds ASCII emoticons to emoji, using the bundledemoji-emoticons.txtmapping (for example:-)to U+1F642 SLIGHTLY SMILING FACE).ACharSequenceNormalizerthat applies Unicode full case folding for case-insensitive matching, as defined by the Default Case Algorithms in Section 3.13 of the Unicode Standard, using the bundledCaseFolding.txtdata of the Unicode Character Database.ACharSequenceNormalizerthat transliterates German umlauts and the eszett the way German conventionally expands them (DIN 5007-2): a-umlaut toae, o-umlaut tooe, u-umlaut toue, and eszett toss, with the capital umlauts and the capital eszett (U+1E9E) expanded likewise.ACharSequenceNormalizerthat removes invisible format and bidirectional control characters that add no textual content and are a common source of noise and spoofing (the byte-order mark, zero width space, word joiner, bidi marks/embeddings/overrides/isolates, the invisible math operators, soft hyphen, and the Arabic letter mark).AnOffsetAwareNormalizerthat collapses runs of Unicode whitespace likeWhitespaceCharSequenceNormalizer, but keeps line and paragraph structure: any whitespace run that contains a line break collapses to a single newline (U+000A) instead of a space, and leading and trailing whitespace is trimmed.ACharSequenceNormalizerthat applies Unicode Normalization Form C (canonical composition, UAX #15).ACharSequenceNormalizerthat applies Unicode Normalization Form KC (compatibility composition, UAX #15).Per-language normalization settings, mirroring how OpenNLP already selects a Snowball stemmer by language.A registry ofNormalizationProfiles by language, with detection-based fallback.ACharSequenceNormalizerimplementation that normalizes text in terms of numbers: every maximal run of ASCII digits (0to9), that is the longest unbroken stretch of consecutive digits, is replaced by a single space.ACharSequenceNormalizerthat folds typographic quotation marks to their ASCII forms: the single quotes and apostrophes to'and the double quotes to".ACharSequenceNormalizerimplementation that shrinks repeated whitespace and repeated characters in text, in three steps: Each run of two or more ASCII whitespace characters (tab, line feed, vertical tab, form feed, carriage return, or space) collapses to a single space.ACharSequenceNormalizerimplementation that normalizes social media text: hashtags, user handles, retweet markers, emoticons, and repeated laughter.One token as a stack of normalization layers.A builder forTermAnalyzer.Entry point for composing normalization steps into a singleCharSequenceNormalizer.A fluent builder that appends normalization steps in order and composes them into oneCharSequenceNormalizerviaTextNormalizer.Builder.build().ACharSequenceNormalizerimplementation that normalizes text in terms of URls and email addresses.ACharSequenceNormalizerthat collapses each run of Unicode whitespace to a single ASCII space and trims the edges, reusing the cursor basedCharClass.whitespace()engine.