Package opennlp.tools.tokenize

Class ThreadSafeTokenizerME

java.lang.Object

opennlp.tools.tokenize.ThreadSafeTokenizerME

All Implemented Interfaces:

AutoCloseable, Probabilistic, Tokenizer

@ThreadSafe
public class ThreadSafeTokenizerME
extends Object
implements Tokenizer, Probabilistic, AutoCloseable

A thread-safe version of TokenizerME. Using it is completely transparent. You can use it in a single-threaded context as well, it only incurs a minimal overhead.

Note:
This implementation uses a ThreadLocal. Although the implementation is lightweight because the model is not duplicated, if you have many long-running threads, you may run into memory problems.

Be careful when using this in a Jakarta EE application, for example.

The user is responsible for clearing the ThreadLocal via calling close().

See Also:

• Probabilistic
• Tokenizer
• TokenizerME

Constructor Summary

Constructors

Constructor	Description
ThreadSafeTokenizerME(String language)	Initializes a ThreadSafeTokenizerME by downloading a default model for a given language.
ThreadSafeTokenizerME(TokenizerModel model)	Initializes a ThreadSafeTokenizerME with the specified model.
ThreadSafeTokenizerME(TokenizerModel model, Dictionary abbDict)	Instantiates a ThreadSafeTokenizerME with an existing TokenizerModel.

Method Summary

Modifier and Type	Method	Description
void	close()
double[]	getProbabilities()	Deprecated, for removal: This API element is subject to removal in a future version. Use probs() instead.
double[]	probs()	Retrieves the probabilities of the last decoded sequence.
String[]	tokenize(String s)	Splits a string into its atomic parts.
Span[]	tokenizePos(String s)	Finds the boundaries of atomic parts in a string.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

ThreadSafeTokenizerME

public ThreadSafeTokenizerME(String language)
                      throws IOException

Initializes a ThreadSafeTokenizerME by downloading a default model for a given language.

Parameters:

language - An ISO conform language code.

Throws:

IOException - Thrown if the model could not be downloaded or saved.

ThreadSafeTokenizerME

public ThreadSafeTokenizerME(TokenizerModel model)

Initializes a ThreadSafeTokenizerME with the specified model.

Parameters:

model - A valid TokenizerModel.

ThreadSafeTokenizerME

public ThreadSafeTokenizerME(TokenizerModel model,
 Dictionary abbDict)

Instantiates a ThreadSafeTokenizerME with an existing TokenizerModel.

Parameters:

model - The TokenizerModel to be used.

abbDict - The Dictionary to be used. It must fit the language of the model.

Method Details

tokenize

public String[] tokenize(String s)

Description copied from interface: Tokenizer

Splits a string into its atomic parts.

Specified by:

tokenize in interface Tokenizer

Parameters:

s - The string to be tokenized.

Returns:

The String[] with the individual tokens as the array elements.

tokenizePos

public Span[] tokenizePos(String s)

Description copied from interface: Tokenizer

Finds the boundaries of atomic parts in a string.

Specified by:

tokenizePos in interface Tokenizer

Parameters:

s - The string to be tokenized.

Returns:

The spans (offsets into s) for each token as the individuals array elements.

probs

public double[] probs()

Description copied from interface: Probabilistic

Retrieves the probabilities of the last decoded sequence.

Specified by:

probs in interface Probabilistic

Returns:

An array with the same number of probabilities as tokens were sent to the computational method when it was last called.

getProbabilities

@Deprecated(forRemoval=true,
            since="2.5.5")
public double[] getProbabilities()

Deprecated, for removal: This API element is subject to removal in a future version.

Use probs() instead.

close

public void close()

Specified by:

close in interface AutoCloseable