opennlp.tools.tokenize
Class TokenizerModel

java.lang.Object
  opennlp.tools.util.model.BaseModel
      opennlp.tools.tokenize.TokenizerModel

All Implemented Interfaces:

ArtifactProvider

public final class TokenizerModel
extends BaseModel

The TokenizerModel is the model used by a learnable Tokenizer.

See Also:

TokenizerME

Field Summary

Fields inherited from class opennlp.tools.util.model.BaseModel

TRAINING_CUTOFF_PROPERTY, TRAINING_EVENTHASH_PROPERTY, TRAINING_ITERATIONS_PROPERTY

Constructor Summary

TokenizerModel(opennlp.model.AbstractModel tokenizerModel, Map<String,String> manifestInfoEntries, TokenizerFactory tokenizerFactory)
Initializes the current instance.

TokenizerModel(File modelFile)

TokenizerModel(InputStream in)
Initializes the current instance.

TokenizerModel(String language, opennlp.model.AbstractModel tokenizerMaxentModel, boolean useAlphaNumericOptimization)
Deprecated. Use TokenizerModel#TokenizerModel(String, AbstractModel, Map, TokenizerFactory) instead and pass in a TokenizerFactory.

TokenizerModel(String language, opennlp.model.AbstractModel tokenizerMaxentModel, boolean useAlphaNumericOptimization, Map<String,String> manifestInfoEntries)
Deprecated. Use TokenizerModel#TokenizerModel(String, AbstractModel, Map, TokenizerFactory) instead and pass in a TokenizerFactory.

TokenizerModel(String language, opennlp.model.AbstractModel tokenizerMaxentModel, Dictionary abbreviations, boolean useAlphaNumericOptimization, Map<String,String> manifestInfoEntries)
Deprecated. Use TokenizerModel#TokenizerModel(String, AbstractModel, Map, TokenizerFactory) instead and pass in a TokenizerFactory.

TokenizerModel(URL modelURL)

Method Summary

Dictionary	getAbbreviations()
TokenizerFactory	getFactory()
opennlp.model.AbstractModel	getMaxentModel()
static void	main(String[] args)
boolean	useAlphaNumericOptimization()

Methods inherited from class opennlp.tools.util.model.BaseModel

getArtifact, getLanguage, getManifestProperty, getVersion, isLoadedFromSerialized, serialize

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TokenizerModel

public TokenizerModel(opennlp.model.AbstractModel tokenizerModel,
                      Map<String,String> manifestInfoEntries,
                      TokenizerFactory tokenizerFactory)

Initializes the current instance.

Parameters:

tokenizerModel - the model

manifestInfoEntries - the manifest

tokenizerFactory - the factory

TokenizerModel

public TokenizerModel(String language,
                      opennlp.model.AbstractModel tokenizerMaxentModel,
                      Dictionary abbreviations,
                      boolean useAlphaNumericOptimization,
                      Map<String,String> manifestInfoEntries)

Deprecated. Use TokenizerModel#TokenizerModel(String, AbstractModel, Map, TokenizerFactory) instead and pass in a TokenizerFactory.

Initializes the current instance.

Parameters:

tokenizerMaxentModel -

useAlphaNumericOptimization -

TokenizerModel

public TokenizerModel(String language,
                      opennlp.model.AbstractModel tokenizerMaxentModel,
                      boolean useAlphaNumericOptimization,
                      Map<String,String> manifestInfoEntries)

Deprecated. Use TokenizerModel#TokenizerModel(String, AbstractModel, Map, TokenizerFactory) instead and pass in a TokenizerFactory.

Initializes the current instance.

Parameters:

language -

tokenizerMaxentModel -

useAlphaNumericOptimization -

manifestInfoEntries -

TokenizerModel

public TokenizerModel(String language,
                      opennlp.model.AbstractModel tokenizerMaxentModel,
                      boolean useAlphaNumericOptimization)

Deprecated. Use TokenizerModel#TokenizerModel(String, AbstractModel, Map, TokenizerFactory) instead and pass in a TokenizerFactory.

Initializes the current instance.

Parameters:

language -

tokenizerMaxentModel -

useAlphaNumericOptimization -

TokenizerModel

public TokenizerModel(InputStream in)
               throws IOException,
                      InvalidFormatException

Initializes the current instance.

Parameters:

in -

Throws:

IOException

InvalidFormatException

TokenizerModel

public TokenizerModel(File modelFile)
               throws IOException,
                      InvalidFormatException

Throws:

IOException

InvalidFormatException

TokenizerModel

public TokenizerModel(URL modelURL)
               throws IOException,
                      InvalidFormatException

Throws:

IOException

InvalidFormatException

Method Detail

getFactory

public TokenizerFactory getFactory()

getMaxentModel

public opennlp.model.AbstractModel getMaxentModel()

getAbbreviations

public Dictionary getAbbreviations()

useAlphaNumericOptimization

public boolean useAlphaNumericOptimization()

main

public static void main(String[] args)
                 throws IOException

Throws:

IOException