Package opennlp.tools.tokenize

Class TokenizerStream

java.lang.Object

opennlp.tools.tokenize.TokenizerStream

All Implemented Interfaces:

AutoCloseable, ObjectStream<TokenSample>

public class TokenizerStream
extends Object
implements ObjectStream<TokenSample>

The TokenizerStream uses a Tokenizer to tokenize the input string and output samples.

Constructor Summary

Constructors

Constructor	Description
TokenizerStream(Tokenizer tokenizer, ObjectStream<String> input)	Initializes a instance.

Method Summary

Modifier and Type	Method	Description
void	close()	Closes the ObjectStream and releases all allocated resources.
TokenSample	read()	Returns the next ObjectStream object.
void	reset()	Repositions the stream at the beginning and the previously seen object sequence will be repeated exactly.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

TokenizerStream

public TokenizerStream(Tokenizer tokenizer,
 ObjectStream<String> input)

Initializes a instance.

Parameters:

tokenizer - A working Tokenizer instance.

input - A plain text line stream.

Method Details

read

public TokenSample read()
                 throws IOException

Description copied from interface: ObjectStream

Returns the next ObjectStream object. Calling this method repeatedly until it returns null will return each object from the underlying source exactly once.

Specified by:

read in interface ObjectStream<TokenSample>

Returns:

The next object or null to signal that the stream is exhausted.

Throws:

IOException - Thrown if there is an error during reading.

close

public void close()
           throws IOException

Description copied from interface: ObjectStream

Closes the ObjectStream and releases all allocated resources. After close was called, it's not allowed to call ObjectStream.read() or ObjectStream.reset().

Specified by:

close in interface AutoCloseable

Specified by:

close in interface ObjectStream<TokenSample>

Throws:

IOException - Thrown if there is an error during closing the stream.

reset

public void reset()
           throws IOException,
UnsupportedOperationException

Description copied from interface: ObjectStream

Repositions the stream at the beginning and the previously seen object sequence will be repeated exactly. This method can be used to re-read the stream if multiple passes over the objects are required.

The implementation of this method is optional.

Specified by:

reset in interface ObjectStream<TokenSample>

Throws:

IOException - Thrown if there is an error during resetting the stream.

UnsupportedOperationException - Thrown if the reset() is not supported. By default, this is the case.