Package opennlp.spellcheck.normalizer

Class SpellCheckingCharSequenceNormalizer

java.lang.Object
opennlp.spellcheck.normalizer.SpellCheckingCharSequenceNormalizer
All Implemented Interfaces:
Serializable, opennlp.tools.util.normalizer.CharSequenceNormalizer
public class SpellCheckingCharSequenceNormalizer extends Object implements opennlp.tools.util.normalizer.CharSequenceNormalizer
A CharSequenceNormalizer that corrects spelling in text using a SpellChecker (typically a SymSpell engine).

The normalizer works in one of two modes:

  • PER_TOKEN (default) – the input is split into whitespace-delimited tokens (whitespace being the Unicode White_Space set, StringUtil.isUnicodeWhitespace(int)) and each token is corrected independently with SpellChecker.lookup(java.lang.String, opennlp.spellcheck.Verbosity, int). The original whitespace runs between tokens are preserved verbatim, so the shape of the line is kept. Tokens the dictionary already contains (best suggestion at edit distance 0) are left untouched.
  • COMPOUND – the whole input is passed to SpellChecker.lookupCompound(java.lang.String, int), which additionally repairs wrongly inserted or omitted spaces (word splits and merges). This collapses runs of whitespace to single spaces, as the compound corrector re-tokenizes the input.

Several guards, all configurable through the SpellCheckingCharSequenceNormalizer.Builder, keep the corrector from "fixing" tokens that should be left as they are: tokens shorter than minTokenLength are skipped, numeric tokens are skipped when skipNumbers is set (on by default), URL- and email-like tokens are skipped when skipUrls is set (on by default), and a token whose lower-cased form is already in the dictionary is never changed.

Casing. Lookups are performed on the lower-cased token and the original casing pattern is re-applied to the correction: an all-upper token yields an all-upper correction, a leading-capital token yields a leading-capital correction, otherwise the suggestion's own casing is used. When no correction applies, the original token is emitted unchanged.

This normalizer composes cleanly inside an AggregateCharSequenceNormalizer; place it after noise-removing normalizers (URL, emoji, shrink) so it sees clean tokens.

Serialization. The settings fields are serialized, but the backing SpellChecker is held in a transient field and is null after Java deserialization; re-attach a checker with withSpellChecker(SpellChecker) to obtain a working copy with the same settings. Calling normalize(java.lang.CharSequence) on an instance with no checker throws IllegalStateException.

See Also: