All Classes and Interfaces
Class
Description
Abstract class which contains code to tag and chunk parses for bottom up parsing and
leaves implementation of advancing parses and completing parses to extend class.
Abstract class containing many of the methods used to generate contexts for parsing.
A base
ObjectStream implementation for events.Abstract class extended by parser event streams which perform tagging and chunking.
A
CharSequenceNormalizer that folds diacritics for matching, the
multilingual-safe counterpart to a Latin-only ASCII folding filter.The
AdditionalContextFeatureGenerator generates the context from the passed
in additional context.A
CharSequenceNormalizer implementation that aggregates the
functionality of other normalizers.The
AggregatedFeatureGenerator aggregates a set of
feature generators and calls them
to generate the features.A
GeneratorFactory that produces AggregatedFeatureGenerator instances
when AggregatedFeatureGeneratorFactory.create() is called.Internal class used by Snowball stemmers
Class for storing the Ancora Spanish head rules associated with parsing.
This class implements the stemming algorithm defined by a snowball script.
Generates predictive contexts for deciding how constituents should be attached.
The
Attributes class stores name value pairs.Generates a feature for each word in a document.
This is a common base model which can be used by the components' specific
model classes.
Base class for all tool
factories.Adds bigram features based on tokens and token classes.
A
GeneratorFactory that produces BigramNameFeatureGenerator instances
when BigramNameFeatureGeneratorFactory.create() is called.The default
SequenceCodec implementation according to the BILOU scheme.A
SequenceValidator implementation for the BilouCodec.The default
SequenceCodec implementation according to the BIO scheme:
B: 'beginning' of a NE
I: 'inside', the word is inside a NE
O: 'outside', the word is a regular word outside a NE
See also the paper by Roth D. and Ratinov L.:
Design Challenges and Misconceptions in Named Entity Recognition.The
BPEModel stores learned BPE merge operations and can be
serialized and deserialized for reuse.A
Tokenizer implementation that performs subword tokenization
using Byte Pair Encoding (BPE).Represents a pair of adjacent symbols in BPE.
A
BaseToolFactory for BPE tokenization that manages
the BPE merge rules artifact and its serialization within a
BPEModel.Learns BPE merge operations from a training corpus and
produces a
BPEModel.Generates Brown cluster features for token bigrams.
Class to load a Brown cluster document in the format:
word\tword_class\tprob.A
GeneratorFactory that produces BrownBigramFeatureGenerator instances
when BrownClusterBigramFeatureGeneratorFactory.create() is called.A
GeneratorFactory that produces BrownTokenClassFeatureGenerator instances
when BrownClusterTokenClassFeatureGeneratorFactory.create() is called.A
GeneratorFactory that produces BrownTokenFeatureGenerator instances
when BrownClusterTokenFeatureGeneratorFactory.create() is called.Obtain the paths listed in the pathLengths array from the Brown class.
Generates
BrownCluster features for current token and token class.Generates
BrownCluster features for current token.Generates predictive contexts for deciding how constituents should be combined.
Creates the features or contexts for the building phase of parsing.
A
CharSequenceNormalizer that replaces unambiguous list-bullet characters with a space,
so a bullet acts as a token separator rather than sticking to the following word.An
ArtifactSerializer implementation for binary data, kept in byte[].Caches features of the aggregated
generators.A
GeneratorFactory that produces CachedFeatureGenerator instances
when CachedFeatureGeneratorFactory.create() is called.A
Stemmer that memoizes word-to-stem mappings in a bounded per-thread LRU cache.A
CharSequenceNormalizer that lower cases text for case-insensitive matching.This class implements the stemming algorithm defined by a snowball script.
The
CharacterNgramFeatureGenerator uses character ngrams to
generate features about each token.A
GeneratorFactory that produces CharacterNgramFeatureGenerator instances
when CharacterNgramFeatureGeneratorFactory.create() is called.Generates predictive context for deciding when a constituent is complete.
Generates predictive context for deciding when a constituent is complete.
A
StringInterner implementation by Aleksey Shipilëv with relaxed canonical requirements.A
StringInterner implementation based on ConcurrentHashMap by Aleksey Shipilëv.Creates predictive context for the pre-chunking phases of parsing.
Cross validator for
Chunker.The
ChunkerEvaluator measures the performance of the given Chunker with the provided
reference samples.Class for creating an event stream out of data files for training a
Chunker.The class represents a maximum-entropy-based
Chunker.The
ChunkerModel is the model used by a learnable Chunker.An
ArtifactSerializer implementation for models.A
SequenceStream implementation encapsulating samples.Parses the conll 2000 shared task shallow parser training data.
An
ObjectStream implementation that works on a
Collection of CollectionObjectStream as source for elements.A configurable
context generator for a POSTagger.Computes the Unicode confusable skeleton of text, following the skeleton algorithm
defined in UTS #39 (Unicode Security
Mechanisms).
A
CharSequenceNormalizer that reduces text to its Unicode confusable
skeleton (UTS #39).Holds feature information about a specific
Parse node.Provides access to training and test partitions for n-fold cross validation.
The
CrossValidationPartitioner.TrainingSampleStream which iterates over
all training elements.This class implements the stemming algorithm defined by a snowball script.
A
CharSequenceNormalizer that maps every Unicode dash to an ASCII hyphen-minus
(U+002D), reusing the cursor based CharClass.dashes() engine.Features based on chunking model described in Fei Sha and Fernando Pereira.
The default chunker
SequenceValidator implementation.Default implementation of the
EndOfSentenceScanner.A context generator for
language detector.Simple feature generator for learning statistical lemmatizers.
The default lemmatizer
SequenceValidator implementation.A
NameContextGenerator implementation for determining contextual features
for a tag-chunk style named-entity recognizer.A default
context generator for a POSTagger.The default POS tagger
SequenceValidator implementation.Generate event contexts for maxent decisions for sentence detection.
A default
TokenContextGenerator which produces events for maxent decisions
for tokenization.A
GeneratorFactory that produces OutcomePriorFeatureGenerator instances
when DefinitionFeatureGeneratorFactory.create() is called.An iterable and serializable dictionary implementation.
A rule based detokenizer.
A persistor used by for reading and writing
dictionaries
of all kinds.The
DictionaryFeatureGenerator uses a DictionaryNameFinder
to generate features for detected names based on the InSpanGenerator.A
GeneratorFactory that produces DictionaryFeatureGenerator instances
when DictionaryFeatureGeneratorFactory.create() is called.A
Lemmatizer implementation that works by simple dictionary lookup into
a Map built from a file containing, for each line:This is a
Dictionary based name finder.An
ArtifactSerializer implementation for dictionaries.An immutable, thread-safe
StopwordFilter backed by an OpenNLP
Dictionary.Fluent builder for
DictionaryStopwordFilter.A
CharSequenceNormalizer that maps Unicode decimal digits to their ASCII equivalents,
so for example Arabic-Indic, Devanagari, or fullwidth digits all become 0-9.A layer of the
Term normalization stack, in increasing order of aggressiveness.Cross validator for
DocumentCategorizer.The factory that provides Doccat default implementations and resources.
A model for document categorization
This feature generator creates document begin features.
A
GeneratorFactory that produces DocumentBeginFeatureGenerator instances
when DocumentBeginFeatureGeneratorFactory.create() is called.The
DocumentCategorizerEvaluator measures the performance of
the given DocumentCategorizer with the provided reference
samples.Iterator-like class for modeling document classification events.
A Max-Ent based implementation of
DocumentCategorizer.Reads in string encoded training samples, parses them and
outputs
DocumentSample objects.This class facilitates the downloading of pretrained OpenNLP models.
This class implements the stemming algorithm defined by a snowball script.
A
CharSequenceNormalizer that expands the ellipsis and leader characters to ASCII dots:
the horizontal ellipsis (U+2026) to "..." and the two-dot leader
(U+2025) to "..".The annotation record of one emoji: an extensible per-symbol store of attribute values, each
carrying its own provenance.
One attribute value with its provenance, mirroring one row of the bundled data file.
Generates features from the emoji annotation layer for the token at the given index: the
project-authored coarse sentiment score (
emojiSentiment=2), the entity type
(emojiType=HEART), the document category (emojiCategory=SMILEYS_AND_EMOTION),
and for flags the ISO 3166 region (emojiRegion=DE).A
GeneratorFactory that produces EmojiAnnotationFeatureGenerator instances
when EmojiAnnotationFeatureGeneratorFactory.create() is called, for use in a feature generation descriptor:The joined-facts layer of the emoji annotation record store: a hook through which facts that
live in user-installed data, typically a gazetteer (coordinates, containment chain, GeoNames or
Who's On First identifiers), are resolved at run time and merged into an
EmojiAnnotation.The bundled-facts layer of the emoji annotation record store: license-clean, provenance-tagged
attributes intrinsic to a pictograph (name, coarse sentiment, entity type, document category),
loaded once from the project-authored
emoji-annotations.txt resource.Assembles one
EmojiAnnotation per symbol from bundled facts
(EmojiAnnotations), derived facts (EmojiFlags), and optional joined facts
(EmojiAnnotationJoin).The document-category hint of an annotated emoji, one constant per group header of the upstream
Unicode
emoji-test.txt (Emoji Keyboard/Display Test Data for
UTS #51).Deprecated, for removal: This API element is subject to removal in a future version.
Replaces every supplementary-plane code point with a space, not only emoji.
The coarse entity type of an annotated emoji, a small project-authored taxonomy assigned from
the subgroup headers of the upstream Unicode
emoji-test.txt (Emoji Keyboard/Display Test
Data for UTS #51): for example the
face-smiling and face-concerned subgroups map to EmojiEntityType.FACE and
transport-ground maps to EmojiEntityType.VEHICLE.Generates document features from the emoji annotation layer: for every annotated token the
project-authored coarse sentiment score (
emojiSentiment=2), the entity type
(emojiType=HEART), the document category hint
(emojiCategory=SMILEYS_AND_EMOTION), and for flags the ISO 3166 region
(emojiRegion=DE).The derived-facts layer of the emoji annotation record store: decodes flag emoji to ISO 3166
codes purely from the code point sequence, with no data file.
A
SentimentContextGenerator that adds emoji annotation features to the default
token context: for every annotated token the project-authored coarse sentiment score
(emojiSentiment=2), the entity type (emojiType=HEART), the category
(emojiCategory=SMILEYS_AND_EMOTION), and for flags the region (emojiRegion=DE).A
SentimentFactory whose context generator adds emoji annotation features (see
EmojiSentimentContextGenerator).A
CharSequenceNormalizer that folds emoji to ASCII emoticons, using the bundled
emoji-emoticons.txt mapping (for example U+1F642 SLIGHTLY SMILING FACE to :)).A
CharSequenceNormalizer that folds ASCII emoticons to emoji, using the bundled
emoji-emoticons.txt mapping (for example :-) to U+1F642 SLIGHTLY SMILING FACE).ObjectStream to clean up empty lines for empty line separated document streams.- Skips empty line at training data start
- Transforms multiple empty lines in a row into one
- Replaces white space lines with empty lines
- TODO: Terminates last document with empty line if it is missing
This stream should be used by the components that mark empty lines to mark document boundaries.
This class implements the stemming algorithm defined by a snowball script.
Generates a
EntityLinker instances via a properties file configuration.An
Entry is a StringList which can
optionally be mapped to attributes.An abstract base class for evaluators.
Checks the Unicode
Extended_Pictographic property of a code point.This class provide common utilities for feature generation.
Abstract base class for filtering
streams.This class implements the stemming algorithm defined by a snowball script.
This class implements the stemming algorithm defined by a snowball script.
A
CharSequenceNormalizer that applies Unicode full case folding for case-insensitive
matching, as defined by the Default Case Algorithms in
Section 3.13 of the
Unicode Standard, using the bundled CaseFolding.txt data of the Unicode Character
Database.Creates a set of feature generators based on a provided XML descriptor.
An
ArtifactSerializer implementation for models.This class implements the stemming algorithm defined by a snowball script.
A
CharSequenceNormalizer that transliterates German umlauts and the eszett the way German
conventionally expands them (DIN 5007-2): a-umlaut to ae, o-umlaut to oe,
u-umlaut to ue, and eszett to ss, with the capital umlauts and the capital eszett
(U+1E9E) expanded likewise.GloVe is an unsupervised learning algorithm for obtaining vector representations for words.
This class implements the stemming algorithm defined by a snowball script.
Class for storing the English
HeadRules associated with parsing.A
StringInterner implementation based on HashMap by Aleksey Shipilëv.This class implements the stemming algorithm defined by a snowball script.
This classes indexes
string lists.This class implements the stemming algorithm defined by a snowball script.
Generates features if the tokens are recognized by the provided
TokenNameFinder.A
CharSequenceNormalizer that removes invisible format and bidirectional control
characters that add no textual content and are a common source of noise and spoofing (the
byte-order mark, zero width space, word joiner, bidi marks/embeddings/overrides/isolates, the
invisible math operators, soft hyphen, and the Arabic letter mark).This class implements the stemming algorithm defined by a snowball script.
This class implements the stemming algorithm defined by a snowball script.
A
StringInterner implementation based on String.intern().Cross validator for
LanguageDetector.The
LanguageDetectorEvaluator measures the performance of
the given LanguageDetector with the provided reference
LanguageSamples.Iterator-like class for modeling an event stream of
samples.Default factory used by
LanguageDetector.Implements a learnable
LanguageDetector.The
LanguageDetectorModel is the model used by a learnable LanguageDetector.This class reads in string encoded
training samples, parses them
and outputs LanguageSample objects.Stores a per-thread "last result" value for APIs that expose probabilities or other data from
the most recent decode call (e.g.
Class for creating an event stream out of data files for training a probabilistic
Lemmatizer.A
SequenceStream implementation encapsulating samples.Reads data for training and testing the
Lemmatizer.The
LemmatizerEvaluator measures the performance of
the given Lemmatizer with the provided reference
samples.The factory that provides
Lemmatizer default implementation and
resources.A probabilistic
Lemmatizer implementation.The
LemmatizerModel is the model used by a learnable Lemmatizer.An
OffsetAwareNormalizer that collapses runs of Unicode whitespace like
WhitespaceCharSequenceNormalizer, but keeps line and paragraph structure: any whitespace
run that contains a line break collapses to a single newline (U+000A) instead of a space,
and leading and trailing whitespace is trimmed.This class serves as an adapter for a
Logger used within a PrintStream.Utility class for handling of
models.This is a non-thread safe mutable int.
Class for creating an event stream out of data files for training an
TokenNameFinder.A maximum-entropy-based
name finder implementation.The default name finder
SequenceValidator implementation.The
NameSampleDataStream class converts tagged tokens
provided by a DataStream to NameSample objects.A
SequenceStream implementation encapsulating samples.A
stream which removes name samples
which do not have a certain type.The Newline
SentenceDetector assumes that sentences are line delimited and
recognizes one sentence per non-empty line.A
CharSequenceNormalizer that applies Unicode Normalization Form C (canonical
composition, UAX #15).A
CharSequenceNormalizer that applies Unicode Normalization Form KC (compatibility
composition, UAX #15).The
NGramCharModel can be used to create character ngrams.Generates ngram features for a document.
Generates an nGram, via an optional separator, and returns the grams as a list
of strings
A
LanguageModel based on a NGramModel using Stupid Backoff to get
the probabilities of the ngrams.The
NGramModel can be used to crate ngrams and character ngrams.Utility class for ngrams.
A no-op
StringInterner implementation, which does not intern.Per-language normalization settings, mirroring how OpenNLP already selects a Snowball stemmer by
language.
A registry of
NormalizationProfiles by language, with detection-based fallback.This class implements the stemming algorithm defined by a snowball script.
A
CharSequenceNormalizer implementation that normalizes text in terms of numbers:
every maximal run of ASCII digits (0 to 9), that is the longest unbroken
stretch of consecutive digits, is replaced by a single space.The definition feature maps the underlying distribution of outcomes.
Routes mutable per-invocation state either to a single owner-thread field or to
ThreadLocal storage
once multiple threads use the same enclosing instance.A shift reduce style
Parser implementation
based on Adwait Ratnaparkhi's 1998 thesis.A built-attach
Parser implementation.The parser chunker
SequenceValidator implementation.Wrapper class for one of four
shift-reduce parser event streams.Wrapper class for one of four
built-attach parser event streams.This is the default
ParserModel implementation.This class implements the stemming algorithm defined by a snowball script.
A
Stemmer, implementing the
Porter Stemming AlgorithmA thread-safe factory for
PorterStemmer instances.This class implements the stemming algorithm defined by a snowball script.
Provides a means of determining which tags are valid for a particular word
based on a
TagDictionary read from a file.The
POSEvaluator measures the performance of the given POSTagger
with the provided reference samples.The
POSModel is the model used by a learnable POSTagger.An
ArtifactSerializer implementation for models.Reads the
samples from an Iterator
and converts those samples into events which
can be used by the maxent library for training.A
SequenceStream implementation encapsulating samples.Defines the format for part-of-speech tagging, i.e.
A mapping implementation for converting between different POS tag formats.
The factory that provides
POSTagger default implementations and resources.A POS tagging driven feature generator.
A
GeneratorFactory that produces PosTaggerFeatureGenerator instances
when PosTaggerFeatureGeneratorFactory.create() is called.A
part-of-speech tagger implementation that uses maximum entropy.Adds the token POS tag as feature.
A
GeneratorFactory that produces POSTaggerNameFeatureGenerator instances
when POSTaggerNameFeatureGeneratorFactory.create() is called.A feature generator implementation that generates prefix-based features.
A
GeneratorFactory that produces PrefixFeatureGenerator instances
when PrefixFeatureGeneratorFactory.create() is called.This
AdaptiveFeatureGenerator generates features indicating the
outcome associated with a previously occurring word.A
GeneratorFactory that produces PreviousMapFeatureGenerator instances
when PreviousMapFeatureGeneratorFactory.create() is called.This
AdaptiveFeatureGenerator generates features indicating the
outcome associated with two previously occurring words.A data container encapsulating language detection results.
A
CharSequenceNormalizer that folds typographic quotation marks to their ASCII forms:
the single quotes and apostrophes to ' and the double quotes to ".A
TokenNameFinder implementation based on a series of regular expressions.Returns a
RegexNameFinder based on a selection of
defaults or a configuration and a selection of defaults.Enumeration of typical regex expressions available in OpenNLP.
This class implements the stemming algorithm defined by a snowball script.
This class implements the stemming algorithm defined by a snowball script.
A cross validator for
sentence detectors.Creates contexts/features for end-of-sentence detection in Thai text.
The
SentenceDetectorEvaluator measures the performance of
the given SentenceDetector with the provided reference
SentenceSamples.The factory that provides
SentenceDetector default implementations and
resourcesA sentence detector for splitting up raw text into sentences.
This feature generator creates sentence begin and end features.
A
GeneratorFactory that produces SentenceFeatureGenerator instances
when SentenceFeatureGeneratorFactory.create() is called.The
SentenceModel is the model used by a learnable
SentenceDetector.This class is a stream filter which reads a sentence by line samples from
an
ObjectStream and converts them into SentenceSample objects.Class for using a Context Generator for Sentiment Analysis.
Class for performing cross validation on the Sentiment Analysis Parser.
The
SentimentEvaluator measures the performance of
the given SentimentME with the provided reference
SentimentSamples.Class for creating events for Sentiment Analysis that is later sent to
MaxEnt.
Class for creating sentiment factories for training.
A
SentimentDetector implementation for creating and using
maximum-entropy-based Sentiment Analysis models.Class for the basis of the Sentiment Analysis model.
Class for converting Strings through Data Stream to
SentimentSample using
tokenised text.Class for creating a type filter.
A
Stemmer that is safe to share across threads by routing each call to a per-thread
delegate minted from a StemmerFactory.A
CharSequenceNormalizer implementation that shrinks repeated whitespace and repeated
characters in text, in three steps:
Each run of two or more ASCII whitespace characters (tab, line feed, vertical tab,
form feed, carriage return, or space) collapses to a single space.A basic
Tokenizer implementation which performs tokenization
using character classes.Base class for a snowball stemmer
A
Stemmer backed by the generated Snowball engines.A thread-safe factory that captures a Snowball stemmer configuration for APIs that accept a
StemmerFactory.A
CharSequenceNormalizer implementation that normalizes social media text: hashtags,
user handles, retweet markers, emoticons, and repeated laughter.This class implements the stemming algorithm defined by a snowball script.
A
Tokenizer decorator which delegates tokenization to a wrapped
Tokenizer and then removes any tokens identified as stopwords by
the supplied StopwordFilter.A
FilterObjectStream which removes stopwords from each
String[] sample produced by an underlying
ObjectStream<String[]>.Static factory for
StopwordFilter instances backed by bundled
language-specific stopword resources or caller-supplied input streams.Provides string interning utility methods.
A
StringList is an immutable list of Strings.Recognizes predefined patterns in strings.
A feature generator implementation that generates suffix-based features.
A
GeneratorFactory that produces SuffixFeatureGenerator instances
when SuffixFeatureGeneratorFactory.create() is called.This class implements the stemming algorithm defined by a snowball script.
One token as a stack of normalization layers.
A builder for
TermAnalyzer.Entry point for composing normalization steps into a single
CharSequenceNormalizer.A fluent builder that appends normalization steps in order and composes them into one
CharSequenceNormalizer via TextNormalizer.Builder.build().Deprecated.
As of OPENNLP-1816,
ChunkerME is
itself thread-safe.Deprecated.
As of OPENNLP-1816,
LanguageDetectorME is itself thread-safe.Deprecated.
As of OPENNLP-1816,
LemmatizerME is
itself thread-safe.Deprecated.
As of OPENNLP-1816,
NameFinderME is
itself thread-safe.Deprecated.
As of OPENNLP-1816,
POSTaggerME is
itself thread-safe.Deprecated.
As of OPENNLP-1816,
SentenceDetectorME is itself thread-safe.Deprecated.
As of OPENNLP-1816,
TokenizerME is
itself thread-safe.Generates features for the class of a token.
A
GeneratorFactory that produces TokenClassFeatureGenerator instances
when TokenClassFeatureGeneratorFactory.create() is called.Generates a feature which contains a token itself.
A
GeneratorFactory that produces TokenFeatureGenerator instances
when TokenFeatureGeneratorFactory.create() is called.A cross validator for
tokenizers.The
TokenizerEvaluator measures the performance of
the given Tokenizer with the provided reference
samples.The factory that provides
Tokenizer default implementation and
resources.A
Tokenizer for converting raw text into separated tokens.The
TokenizerModel is the model used
by a learnable Tokenizer.Cross validator for
TokenNameFinder.The
TokenNameFinderEvaluator measures the performance
of the given TokenNameFinder with the provided
reference samples.The factory that provides
TokenNameFinder default implementations and
resources.The
TokenNameFinderModel is the model used by a learnable TokenNameFinder.Partitions tokens into sub-tokens based on character classes and generates
class features for each of the sub-tokens and combinations of those sub-tokens.
A
GeneratorFactory instantiates TokenPatternFeatureGenerator instances
when TokenPatternFeatureGeneratorFactory.create() is called.Class which produces an Iterator<TokenSample> from a file of space delimited token.
This class is a
stream filter which reads in string encoded
samples and creates samples out of them.This class reads the
samples via an Iterator
and converts the samples into events which
can be used by the maxent library for training.A factory to initialize
Trainer instances depending on a trainer type
configured via Parameters.Adds trigram features based on tokens and token classes.
A
GeneratorFactory that produces TrigramNameFeatureGenerator instances
when TrigramNameFeatureGeneratorFactory.create() is called.This class implements the stemming algorithm defined by a snowball script.
An
InputStream which cannot be closed.A
CharSequenceNormalizer implementation that normalizes text
in terms of URls and email addresses.The
Version class represents the OpenNLP Tools library version.A
CharSequenceNormalizer that collapses each run of Unicode whitespace to a single ASCII
space and trims the edges, reusing the cursor based CharClass.whitespace() engine.This stream formats
ObjectStream of samples into whitespace
separated token strings.Generates previous (left-sided) and next (right-sided) features for a
given
AdaptiveFeatureGenerator.A
GeneratorFactory that produces WindowFeatureGenerator instances
when WindowFeatureGeneratorFactory.create() is called.The Unicode
Word_Break property values, used by the UAX #29 word boundary algorithm.Looks up the Unicode
Word_Break property of a code point.An
AdaptiveFeatureGenerator implementation of a word cluster feature generator.Defines a word cluster
GeneratorFactory; it reads an element containing
'w2vwordcluster' as a tag name.Finds word boundaries in text using the Unicode Text Segmentation algorithm
(UAX #29), rules WB1 through WB999.
Receives each word segment as the half-open character range
[start, end).A
stream filter which reads a sentence per line that contains
words and tags in word_tag format and outputs a POSSample objects.A word token produced by
WordTokenizer: the character Span into the source text
together with its WordType.A word tokenizer built on the Unicode Text Segmentation algorithm (UAX #29).
Receives each word token as a character range and its type, with no allocation.
The category of a word token.