Package opennlp.tools.tokenize

Klasse ThreadSafeTokenizerME

java.lang.Object

opennlp.tools.tokenize.ThreadSafeTokenizerME

Alle implementierten Schnittstellen:

AutoCloseable, Tokenizer

@ThreadSafe
public class ThreadSafeTokenizerME
extends Object
implements Tokenizer, AutoCloseable

A thread-safe version of TokenizerME. Using it is completely transparent. You can use it in a single-threaded context as well, it only incurs a minimal overhead.

Siehe auch:

• Tokenizer
• TokenizerME

Konstruktorübersicht

Konstruktoren

Konstruktor	Beschreibung
ThreadSafeTokenizerME(String language)	Initializes a ThreadSafeTokenizerME by downloading a default model for a given language.
ThreadSafeTokenizerME(TokenizerModel model)	Initializes a ThreadSafeTokenizerME with the specified model.
ThreadSafeTokenizerME(TokenizerModel model, Dictionary abbDict)	Instantiates a ThreadSafeTokenizerME with an existing TokenizerModel.

Methodenübersicht

Modifizierer und Typ	Methode	Beschreibung
void	close()
double[]	getProbabilities()
String[]	tokenize(String s)	Splits a string into its atomic parts.
Span[]	tokenizePos(String s)	Finds the boundaries of atomic parts in a string.

Von Klasse geerbte Methoden java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Konstruktordetails

ThreadSafeTokenizerME

public ThreadSafeTokenizerME(String language)
                      throws IOException

Initializes a ThreadSafeTokenizerME by downloading a default model for a given language.

Parameter:

language - An ISO conform language code.

Löst aus:

IOException - Thrown if the model could not be downloaded or saved.

ThreadSafeTokenizerME

public ThreadSafeTokenizerME(TokenizerModel model)

Initializes a ThreadSafeTokenizerME with the specified model.

Parameter:

model - A valid TokenizerModel.

ThreadSafeTokenizerME

public ThreadSafeTokenizerME(TokenizerModel model,
 Dictionary abbDict)

Instantiates a ThreadSafeTokenizerME with an existing TokenizerModel.

Parameter:

model - The TokenizerModel to be used.

abbDict - The Dictionary to be used. It must fit the language of the model.

Methodendetails

tokenize

public String[] tokenize(String s)

Beschreibung aus Schnittstelle kopiert: Tokenizer

Splits a string into its atomic parts.

Angegeben von:

tokenize in Schnittstelle Tokenizer

Parameter:

s - The string to be tokenized.

Gibt zurück:

The String[] with the individual tokens as the array elements.

tokenizePos

public Span[] tokenizePos(String s)

Beschreibung aus Schnittstelle kopiert: Tokenizer

Finds the boundaries of atomic parts in a string.

Angegeben von:

tokenizePos in Schnittstelle Tokenizer

Parameter:

s - The string to be tokenized.

Gibt zurück:

The spans (offsets into s) for each token as the individuals array elements.

getProbabilities

public double[] getProbabilities()

close

public void close()

Angegeben von:

close in Schnittstelle AutoCloseable