opennlp.tools.tokenize
Interface Detokenizer

All Known Implementing Classes:

DictionaryDetokenizer

public interface Detokenizer

A Detokenizer merges tokens back to their untokenized representation.

Nested Class Summary

static class

Detokenizer.DetokenizationOperation This enum contains an operation for every token to merge the tokens together to their detokenized form.

Method Summary

Detokenizer.DetokenizationOperation[]	detokenize(String[] tokens) Detokenize the input tokens.
String	detokenize(String[] tokens, String splitMarker) Detokenize the input tokens into a String.

Method Detail

detokenize

Detokenizer.DetokenizationOperation[] detokenize(String[] tokens)

Detokenize the input tokens.

Parameters:

tokens - the tokens to detokenize.

Returns:

the merge operations to detokenize the input tokens.

detokenize

String detokenize(String[] tokens,
                  String splitMarker)

Detokenize the input tokens into a String. Tokens which are connected without a space inbetween can be separated by a split marker.

Parameters:

tokens -

splitMarker - the split marker or null

Returns: