Package opennlp.tools.tokenize

Class BPETokenizerTrainer

java.lang.Object

opennlp.tools.tokenize.BPETokenizerTrainer

All Implemented Interfaces:

opennlp.tools.commons.Trainer<opennlp.tools.util.Parameters>

public final class BPETokenizerTrainer
extends Object
implements opennlp.tools.commons.Trainer<opennlp.tools.util.Parameters>

Learns BPE merge operations from a training corpus and produces a BPEModel.

Implements the BPE learning algorithm from Sennrich et al. (2016):

1. Build a vocabulary of character-level symbol sequences from the corpus, where each word is split into individual characters with an end-of-word marker.
2. Count all adjacent symbol pairs across the vocabulary, weighted by word frequency.
3. Merge the most frequent pair into a single new symbol.
4. Repeat until the desired number of merges (numMerges) is reached.

The number of merges controls the granularity of the resulting vocabulary: fewer merges produce finer-grained (more character-level) tokens, while more merges produce coarser (more word-level) tokens. A typical value ranges from a few thousand to tens of thousands, depending on the corpus size and language.

Usage:

 List<String> corpus = List.of(
     "the cat sat on the mat",
     "the dog sat on the log"
 );

 BPETokenizerTrainer trainer = new BPETokenizerTrainer();
 BPEModel model = trainer.train(corpus, 10000, "en");

 // Persist the model
 model.serialize(Path.of("bpe-en.bin"));

 // Use it for tokenization
 BPETokenizer tokenizer = new BPETokenizer(model);
 String[] tokens = tokenizer.tokenize("the cat");
 

For reference see:

• Sennrich, R., Haddow, B., & Birch, A. (2016). Neural Machine Translation of Rare Words with Subword Units. https://arxiv.org/abs/1508.07909

See Also:

• BPETokenizer
• BPEModel

Constructor Summary

Constructors

Constructor	Description
BPETokenizerTrainer()	Creates a new BPETokenizerTrainer.

Method Summary

Modifier and Type	Method	Description
void	init(opennlp.tools.util.Parameters trainParams, Map<String,String> reportMap)
void	init(opennlp.tools.util.Parameters trainParams, Map<String,String> reportMap, opennlp.tools.util.TrainingConfiguration config)
BPEModel	train(Iterable<String> corpus, int numMerges, String languageCode)	Learns BPE merge operations from a training corpus and returns a BPEModel.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

BPETokenizerTrainer

public BPETokenizerTrainer()

Creates a new BPETokenizerTrainer.

Method Details

init

public void init(opennlp.tools.util.Parameters trainParams,
 Map<String,String> reportMap)

Specified by:

init in interface opennlp.tools.commons.Trainer<opennlp.tools.util.Parameters>

init

public void init(opennlp.tools.util.Parameters trainParams,
 Map<String,String> reportMap,
 opennlp.tools.util.TrainingConfiguration config)

Specified by:

init in interface opennlp.tools.commons.Trainer<opennlp.tools.util.Parameters>

train

public BPEModel train(Iterable<String> corpus,
 int numMerges,
 String languageCode)

Learns BPE merge operations from a training corpus and returns a BPEModel.

Parameters:

corpus - An iterable of text strings (e.g., sentences or documents). Must not be null.

numMerges - The number of merge operations to learn. Must be positive.

languageCode - The ISO language code (e.g., "en", "de"). Must not be null.

Returns:

A trained BPEModel containing the learned merge operations.

Throws:

IllegalArgumentException - if numMerges is not positive, or if corpus or languageCode is null.