Package opennlp.tools.ml.model

Class ModelParameterChunker

java.lang.Object

opennlp.tools.ml.model.ModelParameterChunker

public final class ModelParameterChunker
extends Object

A helper class that handles Strings with more than 64k (65535 bytes) in length. This is achieved via the signature SIGNATURE_CHUNKED_PARAMS at the beginning of the String instance to be written to a DataOutputStream.

Background: In OpenNLP, for large(r) corpora, we train models whose (UTF String) parameters will exceed the MAX_CHUNK_SIZE_BYTES bytes limit set in DataOutputStream. For writing and reading those models, we have to chunk up those string instances in 64kB blocks and recombine them correctly upon reading a (binary) model file.

The problem was raised in ticket OPENNLP-1366.

Solution strategy:

• If writing parameters to a DataOutputStream blows up with a UTFDataFormatException a large String instance is chunked up and written as appropriate blocks.
• To indicate that chunking was conducted, we start with the SIGNATURE_CHUNKED_PARAMS indicator, directly followed by the number of chunks used. This way, when reading in chunked model parameters, recombination is achieved transparently.

Note: Both, existing (binary) model files and newly trained models which don't require the chunking technique, will be supported like in previous OpenNLP versions.

Author:

Martin Wiesner, Mark Struberg

Field Summary

Fields

Modifier and Type	Field	Description
static String	SIGNATURE_CHUNKED_PARAMS

Method Summary

Modifier and Type	Method	Description
static String	readUTF(DataInputStream dis)	Reads model parameters from dis.
static void	writeUTF(DataOutputStream dos, String s)	Writes the model parameter s to dos.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SIGNATURE_CHUNKED_PARAMS

public static final String SIGNATURE_CHUNKED_PARAMS

See Also:

Constant Field Values

Method Detail

readUTF

public static String readUTF(DataInputStream dis)
                      throws IOException

Reads model parameters from dis. In case the stream start with SIGNATURE_CHUNKED_PARAMS, the number of chunks is detected and the original large parameter string is reconstructed from several chunks.

Parameters:

dis - The stream which will be used to read the model parameter from.

Throws:

IOException

writeUTF

public static void writeUTF(DataOutputStream dos,
                            String s)
                     throws IOException

Writes the model parameter s to dos. In case s does exceed MAX_CHUNK_SIZE_BYTES in length, the chunking mechanism is used; otherwise the parameter is written 'as is'.

Parameters:

dos - The DataOutputStream stream which will be used to persist the model.

s - The input string that is checked for length and chunked if MAX_CHUNK_SIZE_BYTES is exceeded.

Throws:

IOException