opennlp.tools.tokenize

Class DictionaryDetokenizer

java.lang.Object

opennlp.tools.tokenize.DictionaryDetokenizer

All Implemented Interfaces:

Detokenizer

public class DictionaryDetokenizer
extends Object
implements Detokenizer

A rule based detokenizer. Simple rules which indicate in which direction a token should be moved are looked up in a DetokenizationDictionary object.

See Also:

Detokenizer, DetokenizationDictionary

Nested Class Summary

Nested classes/interfaces inherited from interface opennlp.tools.tokenize.Detokenizer

Detokenizer.DetokenizationOperation

Constructor Summary

Constructors

Constructor and Description
DictionaryDetokenizer(DetokenizationDictionary dict)

Method Summary

Modifier and Type	Method and Description
Detokenizer.DetokenizationOperation[]	detokenize(String[] tokens) Detokenize the input tokens.
String	detokenize(String[] tokens, String splitMarker) Detokenize the input tokens into a String.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DictionaryDetokenizer

public DictionaryDetokenizer(DetokenizationDictionary dict)

Method Detail

detokenize

public Detokenizer.DetokenizationOperation[] detokenize(String[] tokens)

Description copied from interface: Detokenizer

Detokenize the input tokens.

Specified by:

detokenize in interface Detokenizer

Parameters:

tokens - the tokens to detokenize.

Returns:

the merge operations to detokenize the input tokens.

detokenize

public String detokenize(String[] tokens,
                         String splitMarker)

Description copied from interface: Detokenizer

Detokenize the input tokens into a String. Tokens which are connected without a space inbetween can be separated by a split marker.

Specified by:

detokenize in interface Detokenizer

Parameters:

tokens - the token which should be concatenated

splitMarker - the split marker or null

Returns:

the concatenated tokens