Package opennlp.tools.util.normalizer

Class TermAnalyzer.Builder

java.lang.Object

opennlp.tools.util.normalizer.TermAnalyzer.Builder

Enclosing class:

TermAnalyzer

public static final class TermAnalyzer.Builder
extends Object

A builder for TermAnalyzer.

Method Summary

Modifier and Type	Method	Description
TermAnalyzer.Builder	accentFold()	Enables Dimension.ACCENT_FOLD.
TermAnalyzer.Builder	accentFold(Set<Character.UnicodeScript> foldScripts, boolean foldStrokeLetters)	Enables Dimension.ACCENT_FOLD restricted to a specific set of scripts, instead of the default Latin/Greek/Cyrillic.
TermAnalyzer	build()	Returns a new TermAnalyzer with this configuration.
TermAnalyzer.Builder	caseFold()	Enables Dimension.CASE_FOLD.
TermAnalyzer.Builder	caseFold(Locale locale)	Enables Dimension.CASE_FOLD using the given locale's case rules (for example Turkish dotted/dotless i), instead of the default Locale.ROOT.
TermAnalyzer.Builder	confusableFold()	Enables Dimension.CONFUSABLE_FOLD.
TermAnalyzer.Builder	dash()	Enables Dimension.DASH.
TermAnalyzer.Builder	dash(opennlp.tools.util.normalizer.CharSequenceNormalizer normalizer)	Enables Dimension.DASH with a specific normalizer (a custom dash set or target).
TermAnalyzer.Builder	emojiFold()	Enables Dimension.EMOJI_FOLD, folding emoji to ASCII emoticons.
TermAnalyzer.Builder	fullCaseFold()	Enables Dimension.FULL_CASE_FOLD, Unicode full case folding.
TermAnalyzer.Builder	lemmatize(opennlp.tools.lemmatizer.Lemmatizer value)	Enables Dimension.LEMMA through the given lemmatizer.
TermAnalyzer.Builder	maxTokenLength(int maxTokenLength)	Sets the maximum token length of the tokenizer used by TermAnalyzer.analyze(CharSequence).
TermAnalyzer.Builder	nfc()	Enables Dimension.NFC.
TermAnalyzer.Builder	nfkc()	Enables Dimension.NFKC.
TermAnalyzer.Builder	stem(opennlp.tools.stemmer.Stemmer value)	Enables Dimension.STEM through the given stemmer.
TermAnalyzer.Builder	stem(opennlp.tools.stemmer.StemmerFactory factory)	Enables Dimension.STEM through a StemmerFactory.
TermAnalyzer.Builder	stem(opennlp.tools.stemmer.StemmerFactory factory, int cacheSize)	Enables Dimension.STEM through a StemmerFactory with an explicit stem cache capacity, otherwise identical to stem(StemmerFactory).
TermAnalyzer.Builder	tokenizer(WordTokenizer value)	Sets the tokenizer used by TermAnalyzer.analyze(CharSequence).
TermAnalyzer.Builder	transform(Dimension dimension, opennlp.tools.util.normalizer.CharSequenceNormalizer normalizer)	Enables a character-level dimension with a specific normalizer, overriding its default (for example a locale-specific case fold for a language profile).
TermAnalyzer.Builder	whitespace()	Enables Dimension.WHITESPACE.
TermAnalyzer.Builder	whitespace(opennlp.tools.util.normalizer.CharSequenceNormalizer normalizer)	Enables Dimension.WHITESPACE with a specific normalizer, choosing the fold target and behavior.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Details

nfc

public TermAnalyzer.Builder nfc()

Enables Dimension.NFC.

Returns:

this builder

nfkc

public TermAnalyzer.Builder nfkc()

Enables Dimension.NFKC.

Returns:

this builder

whitespace

public TermAnalyzer.Builder whitespace()

Enables Dimension.WHITESPACE.

Returns:

this builder

whitespace

public TermAnalyzer.Builder whitespace(opennlp.tools.util.normalizer.CharSequenceNormalizer normalizer)

Enables Dimension.WHITESPACE with a specific normalizer, choosing the fold target and behavior. For a custom class and target use a CharClass method reference, for example whitespace(CharClass.of(members, replacement)::collapse).

Parameters:

normalizer - The whitespace normalizer to use. Must not be null.

Returns:

this builder

Throws:

IllegalArgumentException - if normalizer is null.

dash

public TermAnalyzer.Builder dash()

Enables Dimension.DASH.

Returns:

this builder

dash

public TermAnalyzer.Builder dash(opennlp.tools.util.normalizer.CharSequenceNormalizer normalizer)

Enables Dimension.DASH with a specific normalizer (a custom dash set or target).

Parameters:

normalizer - The dash normalizer to use. Must not be null.

Returns:

this builder

Throws:

IllegalArgumentException - if normalizer is null.

caseFold

public TermAnalyzer.Builder caseFold()

Enables Dimension.CASE_FOLD.

Returns:

this builder

Throws:

IllegalStateException - if Dimension.FULL_CASE_FOLD is already configured.

caseFold

public TermAnalyzer.Builder caseFold(Locale locale)

Enables Dimension.CASE_FOLD using the given locale's case rules (for example Turkish dotted/dotless i), instead of the default Locale.ROOT.

Parameters:

locale - The locale whose case rules to apply. Must not be null.

Returns:

this builder

Throws:

IllegalArgumentException - if locale is null.

IllegalStateException - if Dimension.FULL_CASE_FOLD is already configured.

fullCaseFold

public TermAnalyzer.Builder fullCaseFold()

Enables Dimension.FULL_CASE_FOLD, Unicode full case folding. Unlike caseFold() it also applies the expanding folds (the sharp s to ss, the Latin ligatures), so use one or the other rather than both.

Returns:

this builder

Throws:

IllegalStateException - if Dimension.CASE_FOLD is already configured.

accentFold

public TermAnalyzer.Builder accentFold()

Enables Dimension.ACCENT_FOLD.

Returns:

this builder

accentFold

public TermAnalyzer.Builder accentFold(Set<Character.UnicodeScript> foldScripts,
 boolean foldStrokeLetters)

Enables Dimension.ACCENT_FOLD restricted to a specific set of scripts, instead of the default Latin/Greek/Cyrillic.

Parameters:

foldScripts - The scripts whose diacritics to fold. Must not be null or contain null elements.

foldStrokeLetters - Whether to also fold stroke letters such as o-slash and l-stroke.

Returns:

this builder

Throws:

IllegalArgumentException - if foldScripts is null or contains a null element.

emojiFold

public TermAnalyzer.Builder emojiFold()

Enables Dimension.EMOJI_FOLD, folding emoji to ASCII emoticons. Only this direction exists as a per-token layer: the reverse (emoticon to emoji) is a pre-tokenization transform, because a tokenizer splits an emoticon such as :-) into punctuation, so it can never arrive as a single token; see EmoticonToEmojiCharSequenceNormalizer.

Returns:

this builder

confusableFold

public TermAnalyzer.Builder confusableFold()

Enables Dimension.CONFUSABLE_FOLD.

Returns:

this builder

transform

public TermAnalyzer.Builder transform(Dimension dimension,
 opennlp.tools.util.normalizer.CharSequenceNormalizer normalizer)

Enables a character-level dimension with a specific normalizer, overriding its default (for example a locale-specific case fold for a language profile).

Parameters:

dimension - The character-level dimension to enable. Must not be null.

normalizer - The normalizer to use for it. Must not be null.

Returns:

this builder

Throws:

IllegalArgumentException - if dimension or normalizer is null, or if dimension is Dimension.ORIGINAL, Dimension.STEM, or Dimension.LEMMA.

IllegalStateException - if dimension is Dimension.CASE_FOLD or Dimension.FULL_CASE_FOLD and the other one is already configured.

stem

public TermAnalyzer.Builder stem(opennlp.tools.stemmer.Stemmer value)

Enables Dimension.STEM through the given stemmer.

Parameters:

value - The stemmer. Must not be null.

Returns:

this builder

Throws:

IllegalArgumentException - if value is null.

stem

public TermAnalyzer.Builder stem(opennlp.tools.stemmer.StemmerFactory factory)

Enables Dimension.STEM through a StemmerFactory. The analyzer receives a CachingStemmer with the default cache capacity: it can be shared across threads, and repeated words resolve from a bounded per-thread cache instead of being re-stemmed. The cache is keyed to the thread, so the memoization pays off on threads reused across many tokens (a fixed platform-thread pool); on a virtual-thread-per-task executor every task starts with an empty cache.

Parameters:

factory - The stemmer factory. Must not be null.

Returns:

this builder

Throws:

IllegalArgumentException - if factory is null.

stem

public TermAnalyzer.Builder stem(opennlp.tools.stemmer.StemmerFactory factory,
 int cacheSize)

Enables Dimension.STEM through a StemmerFactory with an explicit stem cache capacity, otherwise identical to stem(StemmerFactory).

Parameters:

factory - The stemmer factory. Must not be null.

cacheSize - The maximum number of word-to-stem entries kept per thread; must be positive.

Returns:

this builder

Throws:

IllegalArgumentException - if factory is null or cacheSize is not positive.

lemmatize

public TermAnalyzer.Builder lemmatize(opennlp.tools.lemmatizer.Lemmatizer value)

Enables Dimension.LEMMA through the given lemmatizer.

Parameters:

value - The lemmatizer. Must not be null.

Returns:

this builder

Throws:

IllegalArgumentException - if value is null.

tokenizer

public TermAnalyzer.Builder tokenizer(WordTokenizer value)

Sets the tokenizer used by TermAnalyzer.analyze(CharSequence).

Parameters:

value - The tokenizer. Must not be null.

Returns:

this builder

Throws:

IllegalArgumentException - if value is null.

maxTokenLength

public TermAnalyzer.Builder maxTokenLength(int maxTokenLength)

Sets the maximum token length of the tokenizer used by TermAnalyzer.analyze(CharSequence). Convenience for tokenizer(new WordTokenizer(maxTokenLength)).

Parameters:

maxTokenLength - The maximum number of characters in a token. Must be at least 1.

Returns:

this builder

Throws:

IllegalArgumentException - if maxTokenLength is less than 1.

build

public TermAnalyzer build()

Returns a new TermAnalyzer with this configuration.

Returns:

a new TermAnalyzer with this configuration