Package opennlp.tools.tokenize

Class ThreadSafeTokenizerME

java.lang.Object

opennlp.tools.tokenize.ThreadSafeTokenizerME

All Implemented Interfaces:

AutoCloseable, Tokenizer

@ThreadSafe
public class ThreadSafeTokenizerME
extends Object
implements Tokenizer, AutoCloseable

A thread-safe version of TokenizerME. Using it is completely transparent. You can use it in a single-threaded context as well, it only incurs a minimal overhead.

See Also:

• Tokenizer
• TokenizerME

Constructor Summary

Constructors

Constructor	Description
ThreadSafeTokenizerME(String language)	Initializes a ThreadSafeTokenizerME by downloading a default model for a given language.
ThreadSafeTokenizerME(TokenizerModel model)	Initializes a ThreadSafeTokenizerME with the specified model.
ThreadSafeTokenizerME(TokenizerModel model, Dictionary abbDict)	Instantiates a ThreadSafeTokenizerME with an existing TokenizerModel.

Method Summary

Modifier and Type	Method	Description
void	close()
double[]	getProbabilities()
String[]	tokenize(String s)	Splits a string into its atomic parts.
Span[]	tokenizePos(String s)	Finds the boundaries of atomic parts in a string.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

ThreadSafeTokenizerME

public ThreadSafeTokenizerME(String language)
                      throws IOException

Initializes a ThreadSafeTokenizerME by downloading a default model for a given language.

Parameters:

language - An ISO conform language code.

Throws:

IOException - Thrown if the model could not be downloaded or saved.

ThreadSafeTokenizerME

public ThreadSafeTokenizerME(TokenizerModel model)

Initializes a ThreadSafeTokenizerME with the specified model.

Parameters:

model - A valid TokenizerModel.

ThreadSafeTokenizerME

public ThreadSafeTokenizerME(TokenizerModel model,
 Dictionary abbDict)

Instantiates a ThreadSafeTokenizerME with an existing TokenizerModel.

Parameters:

model - The TokenizerModel to be used.

abbDict - The Dictionary to be used. It must fit the language of the model.

Method Details

tokenize

public String[] tokenize(String s)

Description copied from interface: Tokenizer

Splits a string into its atomic parts.

Specified by:

tokenize in interface Tokenizer

Parameters:

s - The string to be tokenized.

Returns:

The String[] with the individual tokens as the array elements.

tokenizePos

public Span[] tokenizePos(String s)

Description copied from interface: Tokenizer

Finds the boundaries of atomic parts in a string.

Specified by:

tokenizePos in interface Tokenizer

Parameters:

s - The string to be tokenized.

Returns:

The spans (offsets into s) for each token as the individuals array elements.

getProbabilities

public double[] getProbabilities()

close

public void close()

Specified by:

close in interface AutoCloseable