Class DictionaryDetokenizer

java.lang.Object

opennlp.tools.tokenize.DictionaryDetokenizer

All Implemented Interfaces:

opennlp.tools.tokenize.Detokenizer

public class DictionaryDetokenizer
extends Object
implements opennlp.tools.tokenize.Detokenizer

A rule based detokenizer. Simple rules which indicate in which direction a token should be moved are looked up in a dictionary.

See Also:

• Detokenizer
• DetokenizationDictionary

Nested Class Summary

Nested classes/interfaces inherited from interface opennlp.tools.tokenize.Detokenizer

opennlp.tools.tokenize.Detokenizer.DetokenizationOperation

Constructor Summary

Constructors

Constructor	Description
DictionaryDetokenizer(DetokenizationDictionary dict)	Initializes a DictionaryDetokenizer instance.

Method Summary

Modifier and Type	Method	Description
opennlp.tools.tokenize.Detokenizer.DetokenizationOperation[]	detokenize(String[] tokens)
String	detokenize(String[] tokens, String splitMarker)

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

DictionaryDetokenizer

public DictionaryDetokenizer(DetokenizationDictionary dict)

Initializes a DictionaryDetokenizer instance.

Parameters:

dict - The DetokenizationDictionary to be used.

Method Details

detokenize

public opennlp.tools.tokenize.Detokenizer.DetokenizationOperation[] detokenize(String[] tokens)

Specified by:

detokenize in interface opennlp.tools.tokenize.Detokenizer

detokenize

public String detokenize(String[] tokens,
 String splitMarker)

Specified by:

detokenize in interface opennlp.tools.tokenize.Detokenizer