Package opennlp.tools.tokenize

Class DictionaryDetokenizer

java.lang.Object

opennlp.tools.tokenize.DictionaryDetokenizer

All Implemented Interfaces:

Detokenizer

public class DictionaryDetokenizer
extends Object
implements Detokenizer

A rule based detokenizer. Simple rules which indicate in which direction a token should be moved are looked up in a dictionary.

See Also:

• Detokenizer
• DetokenizationDictionary

Nested Class Summary

Nested classes/interfaces inherited from interface opennlp.tools.tokenize.Detokenizer

Detokenizer.DetokenizationOperation

Constructor Summary

Constructors

Constructor	Description
DictionaryDetokenizer(DetokenizationDictionary dict)	Initializes a DictionaryDetokenizer instance.

Method Summary

Modifier and Type	Method	Description
Detokenizer.DetokenizationOperation[]	detokenize(String[] tokens)	Detokenizes the collection of tokens.
String	detokenize(String[] tokens, String splitMarker)	Detokenizes the input tokens into a String.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

DictionaryDetokenizer

public DictionaryDetokenizer(DetokenizationDictionary dict)

Initializes a DictionaryDetokenizer instance.

Parameters:

dict - The DetokenizationDictionary to be used.

Method Details

detokenize

public Detokenizer.DetokenizationOperation[] detokenize(String[] tokens)

Description copied from interface: Detokenizer

Detokenizes the collection of tokens.

Specified by:

detokenize in interface Detokenizer

Parameters:

tokens - The elements which should be detokenized.

Returns:

The merge operations to handle given tokens.

detokenize

public String detokenize(String[] tokens,
 String splitMarker)

Description copied from interface: Detokenizer

Detokenizes the input tokens into a String. Tokens which are connected without a whitespace character in between can be separated by a given splitMarker.

Specified by:

detokenize in interface Detokenizer

Parameters:

tokens - The elements which should be concatenated.

splitMarker - The split marker or null.

Returns:

The concatenated tokens as a single string.