Package opennlp.tools.tokenize

Class BPEModel

java.lang.Object

opennlp.tools.util.model.BaseModel

opennlp.tools.tokenize.BPEModel

All Implemented Interfaces:

Serializable, opennlp.tools.util.model.ArtifactProvider

public final class BPEModel
extends BaseModel

The BPEModel stores learned BPE merge operations and can be serialized and deserialized for reuse.

A model is created by the BPETokenizerTrainer and contains an ordered list of BPETokenizer.SymbolPair merge operations that define the BPE vocabulary. The model is persisted as a standard OpenNLP ZIP package with a bpe.merges artifact containing the merge rules.

Usage:

 // Create via training
 BPETokenizerTrainer trainer = new BPETokenizerTrainer();
 BPEModel model = trainer.train(corpus, 10000, "en");

 // Save to disk
 model.serialize(Path.of("bpe-en.bin"));

 // Load from disk
 BPEModel loaded = new BPEModel(Path.of("bpe-en.bin"));

 // Use for tokenization
 BPETokenizer tokenizer = new BPETokenizer(loaded);
 

See Also:

• BPETokenizer
• BPETokenizerTrainer
• BPETokenizerFactory
• Serialized Form

Field Summary

Fields inherited from class opennlp.tools.util.model.BaseModel

TRAINING_CUTOFF_PROPERTY, TRAINING_EVENTHASH_PROPERTY, TRAINING_ITERATIONS_PROPERTY

Constructor Summary

Constructors

Constructor	Description
BPEModel(File modelFile)	Initializes a BPEModel from a File.
BPEModel(InputStream in)	Initializes a BPEModel from an InputStream.
BPEModel(URL modelURL)	Initializes a BPEModel from a URL.
BPEModel(Path modelPath)	Initializes a BPEModel from a Path.
BPEModel(List<BPETokenizer.SymbolPair> merges, Map<String,String> manifestInfoEntries, BPETokenizerFactory factory)	Creates a BPEModel from trained merge rules.

Method Summary

Modifier and Type	Method	Description
BPETokenizerFactory	getFactory()
List<BPETokenizer.SymbolPair>	getMerges()

Methods inherited from class opennlp.tools.util.model.BaseModel

getArtifact, getLanguage, getManifestProperty, getVersion, isLoadedFromSerialized, serialize, serialize, serialize

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

BPEModel

public BPEModel(List<BPETokenizer.SymbolPair> merges,
 Map<String,String> manifestInfoEntries,
 BPETokenizerFactory factory)

Creates a BPEModel from trained merge rules.

Parameters:

merges - The ordered list of merge operations. Must not be null.

manifestInfoEntries - Additional manifest info.

factory - The BPETokenizerFactory.

BPEModel

public BPEModel(InputStream in)
         throws IOException

Initializes a BPEModel from an InputStream.

Parameters:

in - The InputStream for loading the model.

Throws:

IOException - Thrown if IO errors occurred.

BPEModel

public BPEModel(File modelFile)
         throws IOException

Initializes a BPEModel from a File.

Parameters:

modelFile - The File for loading the model.

Throws:

IOException - Thrown if IO errors occurred.

BPEModel

public BPEModel(Path modelPath)
         throws IOException

Initializes a BPEModel from a Path.

Parameters:

modelPath - The Path for loading the model.

Throws:

IOException - Thrown if IO errors occurred.

BPEModel

public BPEModel(URL modelURL)
         throws IOException

Initializes a BPEModel from a URL.

Parameters:

modelURL - The URL for loading the model.

Throws:

IOException - Thrown if IO errors occurred.

Method Details

getFactory

public BPETokenizerFactory getFactory()

Returns:

The active BPETokenizerFactory.

getMerges

public List<BPETokenizer.SymbolPair> getMerges()

Returns:

An unmodifiable, ordered list of BPE merge operations stored in this model.