opennlp.tools.tokenize

Interface Detokenizer

All Known Implementing Classes:

DictionaryDetokenizer

public interface Detokenizer

A Detokenizer merges tokens back to their untokenized representation.

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static class	Detokenizer.DetokenizationOperation This enum contains an operation for every token to merge the tokens together to their detokenized form.

Method Summary

Methods

Modifier and Type	Method and Description
Detokenizer.DetokenizationOperation[]	detokenize(String[] tokens) Detokenize the input tokens.
String	detokenize(String[] tokens, String splitMarker) Detokenize the input tokens into a String.

Method Detail

detokenize

Detokenizer.DetokenizationOperation[] detokenize(String[] tokens)

Detokenize the input tokens.

Parameters:

tokens - the tokens to detokenize.

Returns:

the merge operations to detokenize the input tokens.

detokenize

String detokenize(String[] tokens,
                String splitMarker)

Detokenize the input tokens into a String. Tokens which are connected without a space inbetween can be separated by a split marker.

Parameters:

tokens - the token which should be concatenated

splitMarker - the split marker or null

Returns:

the concatenated tokens