Class TokenizerModel

java.lang.Object

opennlp.tools.util.model.BaseModel

opennlp.tools.tokenize.TokenizerModel

All Implemented Interfaces:

Serializable, opennlp.tools.util.model.ArtifactProvider

public final class TokenizerModel
extends BaseModel

The TokenizerModel is the model used by a learnable Tokenizer.

See Also:

• TokenizerME
• TokenizerFactory
• Serialized Form

Field Summary

Fields inherited from class BaseModel

TRAINING_CUTOFF_PROPERTY, TRAINING_EVENTHASH_PROPERTY, TRAINING_ITERATIONS_PROPERTY

Constructor Summary

Constructors

Constructor	Description
TokenizerModel(File modelFile)	Initializes a TokenizerModel instance via a valid File.
TokenizerModel(InputStream in)	Initializes a TokenizerModel instance via a valid InputStream.
TokenizerModel(URL modelURL)	Initializes a TokenizerModel instance via a valid URL.
TokenizerModel(Path modelPath)	Initializes a TokenizerModel instance via a valid Path.
TokenizerModel(opennlp.tools.ml.model.MaxentModel tokenizerModel, Map<String,String> manifestInfoEntries, TokenizerFactory tokenizerFactory)	Initializes a TokenizerModel instance via a MaxentModel and related resources.

Method Summary

Modifier and Type	Method	Description
boolean	equals(Object obj)
Dictionary	getAbbreviations()
TokenizerFactory	getFactory()
opennlp.tools.ml.model.MaxentModel	getMaxentModel()
int	hashCode()
boolean	useAlphaNumericOptimization()

Methods inherited from class BaseModel

getArtifact, getLanguage, getManifestProperty, getVersion, isLoadedFromSerialized, serialize, serialize, serialize

Methods inherited from class Object

getClass, notify, notifyAll, toString, wait, wait, wait

Constructor Details

TokenizerModel

public TokenizerModel(opennlp.tools.ml.model.MaxentModel tokenizerModel,
 Map<String,String> manifestInfoEntries,
 TokenizerFactory tokenizerFactory)

Initializes a TokenizerModel instance via a MaxentModel and related resources.

Parameters:

tokenizerModel - The model to be used.

manifestInfoEntries - Additional information kept in the manifest.

tokenizerFactory - The TokenizerFactory to be used internally.

TokenizerModel

public TokenizerModel(InputStream in)
               throws IOException

Initializes a TokenizerModel instance via a valid InputStream.

Parameters:

in - The InputStream used for loading the model.

Throws:

IOException - Thrown if IO errors occurred during initialization.

TokenizerModel

public TokenizerModel(File modelFile)
               throws IOException

Initializes a TokenizerModel instance via a valid File.

Parameters:

modelFile - The File used for loading the model.

Throws:

IOException - Thrown if IO errors occurred during initialization.

TokenizerModel

public TokenizerModel(Path modelPath)
               throws IOException

Initializes a TokenizerModel instance via a valid Path.

Parameters:

modelPath - The Path used for loading the model.

Throws:

IOException - Thrown if IO errors occurred during initialization.

TokenizerModel

public TokenizerModel(URL modelURL)
               throws IOException

Initializes a TokenizerModel instance via a valid URL.

Parameters:

modelURL - The URL used for loading the model.

Throws:

IOException - Thrown if IO errors occurred during initialization.

Method Details

getFactory

public TokenizerFactory getFactory()

Returns:

Retrieves the active TokenizerFactory.

getMaxentModel

public opennlp.tools.ml.model.MaxentModel getMaxentModel()

Returns:

Retrieves the model as MaxentModel instance.

getAbbreviations

public Dictionary getAbbreviations()

Returns:

Retrieves the active abbreviation Dictionary.

useAlphaNumericOptimization

public boolean useAlphaNumericOptimization()

Returns:

true if alphanumeric optimization is active, false otherwise.

hashCode

public int hashCode()

Overrides:

hashCode in class Object

equals

public boolean equals(Object obj)

Overrides:

equals in class Object