public class TokenizerME extends Object
This tokenizer needs a statistical model to tokenize a text which reproduces
the tokenization observed in the training data used to create the model.
The TokenizerModel
class encapsulates the model and provides
methods to create it from the binary representation.
A tokenizer instance is not thread safe. For each thread one tokenizer
must be instantiated which can share one TokenizerModel
instance
to safe memory.
To train a new model {train(ObjectStream, TokenizerFactory, TrainingParameters)
method
can be used.
Sample usage:
InputStream modelIn;
...
TokenizerModel model = TokenizerModel(modelIn);
Tokenizer tokenizer = new TokenizerME(model);
String tokens[] = tokenizer.tokenize("A sentence to be tokenized.");
Tokenizer
,
TokenizerModel
,
TokenSample
Modifier and Type | Field and Description |
---|---|
static Pattern |
alphaNumeric
Deprecated.
As of release 1.5.2, replaced by
Factory.getAlphanumeric(String) |
static String |
NO_SPLIT
Constant indicates no token split.
|
static String |
SPLIT
Constant indicates a token split.
|
Constructor and Description |
---|
TokenizerME(TokenizerModel model) |
TokenizerME(TokenizerModel model,
Factory factory)
Deprecated.
use
TokenizerFactory to extend the Tokenizer
functionality |
Modifier and Type | Method and Description |
---|---|
double[] |
getTokenProbabilities()
Returns the probabilities associated with the most recent
calls to
AbstractTokenizer.tokenize(String) or tokenizePos(String) . |
String[] |
tokenize(String s)
Splits a string into its atomic parts
|
Span[] |
tokenizePos(String d)
Tokenizes the string.
|
static TokenizerModel |
train(ObjectStream<TokenSample> samples,
TokenizerFactory factory,
TrainingParameters mlParams)
Trains a model for the
TokenizerME . |
boolean |
useAlphaNumericOptimization()
Returns the value of the alpha-numeric optimization flag.
|
public static final String SPLIT
public static final String NO_SPLIT
@Deprecated public static final Pattern alphaNumeric
Factory.getAlphanumeric(String)
public TokenizerME(TokenizerModel model)
public TokenizerME(TokenizerModel model, Factory factory)
TokenizerFactory
to extend the Tokenizer
functionalitypublic double[] getTokenProbabilities()
AbstractTokenizer.tokenize(String)
or tokenizePos(String)
.public Span[] tokenizePos(String d)
d
- The string to be tokenized.public static TokenizerModel train(ObjectStream<TokenSample> samples, TokenizerFactory factory, TrainingParameters mlParams) throws IOException
TokenizerME
.samples
- the samples used for the training.factory
- a TokenizerFactory
to get resources frommlParams
- the machine learning train parametersTokenizerModel
IOException
- it throws an IOException
if an IOException
is
thrown during IO operations on a temp file which is created
during training. Or if reading from the ObjectStream
fails.public boolean useAlphaNumericOptimization()
Copyright © 2018 The Apache Software Foundation. All rights reserved.