Package opennlp.tools.tokenize

Interface Detokenizer

All Known Implementing Classes:

DictionaryDetokenizer

public interface Detokenizer

A Detokenizer merges tokens back to their detokenized representation.

Nested Class Summary

Nested Classes

Modifier and Type	Interface	Description
static enum	Detokenizer.DetokenizationOperation	This enum contains an operation for every token to merge the tokens together to their detokenized form.

Method Summary

Modifier and Type	Method	Description
Detokenizer.DetokenizationOperation[]	detokenize(String[] tokens)	Detokenizes the collection of tokens.
String	detokenize(String[] tokens, String splitMarker)	Detokenizes the input tokens into a String.

Method Details

detokenize

Detokenizer.DetokenizationOperation[] detokenize(String[] tokens)

Detokenizes the collection of tokens.

Parameters:

tokens - The elements which should be detokenized.

Returns:

The merge operations to handle given tokens.

detokenize

String detokenize(String[] tokens,
 String splitMarker)

Detokenizes the input tokens into a String. Tokens which are connected without a whitespace character in between can be separated by a given splitMarker.

Parameters:

tokens - The elements which should be concatenated.

splitMarker - The split marker or null.

Returns:

The concatenated tokens as a single string.