Class ThreadSafeTokenizerME

java.lang.Object

opennlp.tools.tokenize.ThreadSafeTokenizerME

All Implemented Interfaces:

AutoCloseable, opennlp.tools.ml.Probabilistic, opennlp.tools.tokenize.Tokenizer

@ThreadSafe
public class ThreadSafeTokenizerME
extends Object
implements opennlp.tools.tokenize.Tokenizer, opennlp.tools.ml.Probabilistic, AutoCloseable

A thread-safe version of TokenizerME. Using it is completely transparent. You can use it in a single-threaded context as well, it only incurs a minimal overhead.

Note:
This implementation uses a ThreadLocal. Although the implementation is lightweight because the model is not duplicated, if you have many long-running threads, you may run into memory problems.

Be careful when using this in a Jakarta EE application, for example.

The user is responsible for clearing the ThreadLocal via calling close().

See Also:

• Probabilistic
• Tokenizer
• TokenizerME

Constructor Summary

Constructors

Constructor	Description
ThreadSafeTokenizerME(String language)	Initializes a ThreadSafeTokenizerME by downloading a default model for a given language.
ThreadSafeTokenizerME(TokenizerModel model)	Initializes a ThreadSafeTokenizerME with the specified model.
ThreadSafeTokenizerME(TokenizerModel model, Dictionary abbDict)	Instantiates a ThreadSafeTokenizerME with an existing TokenizerModel.

Method Summary

Modifier and Type	Method	Description
void	close()
double[]	getProbabilities()	Deprecated, for removal: This API element is subject to removal in a future version. Use probs() instead.
double[]	probs()
String[]	tokenize(String s)
opennlp.tools.util.Span[]	tokenizePos(String s)

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

ThreadSafeTokenizerME

public ThreadSafeTokenizerME(String language)
                      throws IOException

Initializes a ThreadSafeTokenizerME by downloading a default model for a given language.

Parameters:

language - An ISO conform language code.

Throws:

IOException - Thrown if the model could not be downloaded or saved.

ThreadSafeTokenizerME

public ThreadSafeTokenizerME(TokenizerModel model)

Initializes a ThreadSafeTokenizerME with the specified model.

Parameters:

model - A valid TokenizerModel.

ThreadSafeTokenizerME

public ThreadSafeTokenizerME(TokenizerModel model,
 Dictionary abbDict)

Instantiates a ThreadSafeTokenizerME with an existing TokenizerModel.

Parameters:

model - The TokenizerModel to be used.

abbDict - The Dictionary to be used. It must fit the language of the model.

Method Details

tokenize

public String[] tokenize(String s)

Specified by:

tokenize in interface opennlp.tools.tokenize.Tokenizer

tokenizePos

public opennlp.tools.util.Span[] tokenizePos(String s)

Specified by:

tokenizePos in interface opennlp.tools.tokenize.Tokenizer

probs

public double[] probs()

Specified by:

probs in interface opennlp.tools.ml.Probabilistic

getProbabilities

@Deprecated(forRemoval=true,
            since="2.5.5")
public double[] getProbabilities()

Deprecated, for removal: This API element is subject to removal in a future version.

Use probs() instead.

close

public void close()

Specified by:

close in interface AutoCloseable