Package opennlp.tools.tokenize

Class TokenSampleStream

java.lang.Object

opennlp.tools.util.FilterObjectStream<String,TokenSample>

opennlp.tools.tokenize.TokenSampleStream

All Implemented Interfaces:

AutoCloseable, ObjectStream<TokenSample>

public class TokenSampleStream
extends FilterObjectStream<String,TokenSample>

This class is a stream filter which reads in string encoded samples and creates samples out of them. The input string sample is tokenized if a whitespace or the special separator chars occur.

Sample:
"token1 token2 token3<SPLIT>token4"
The tokens token1 and token2 are separated by a whitespace, token3 and token4 are separated by the special character sequence. In this case, the default split sequence applies.

Note: The sequence must be unique in the input string and is not escaped.

Constructor Summary

Constructors

Constructor	Description
TokenSampleStream(ObjectStream<String> sentences)	Initializes a instance.
TokenSampleStream(ObjectStream<String> samples, String separatorChars)	Initializes a instance.

Method Summary

Modifier and Type	Method	Description
TokenSample	read()	Returns the next ObjectStream object.

Methods inherited from class opennlp.tools.util.FilterObjectStream

close, reset

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

TokenSampleStream

public TokenSampleStream(ObjectStream<String> samples,
 String separatorChars)

Initializes a instance.

Parameters:

samples - A plain text line stream. Must not be null.

separatorChars - The characters to be considered separators. See TokenSample.DEFAULT_SEPARATOR_CHARS. Must not be null.

TokenSampleStream

public TokenSampleStream(ObjectStream<String> sentences)

Initializes a instance.

Parameters:

sentences - A plain text line stream. Must not be null.

Method Details

read

public TokenSample read()
                 throws IOException

Description copied from interface: ObjectStream

Returns the next ObjectStream object. Calling this method repeatedly until it returns null will return each object from the underlying source exactly once.

Returns:

The next object or null to signal that the stream is exhausted.

Throws:

IOException - Thrown if there is an error during reading.