opennlp.tools.tokenize
Class DictionaryDetokenizer
java.lang.Object
opennlp.tools.tokenize.DictionaryDetokenizer
- All Implemented Interfaces:
- Detokenizer
public class DictionaryDetokenizer
- extends Object
- implements Detokenizer
A rule based detokenizer. Simple rules which indicate in which direction a token should be
moved are looked up in a DetokenizationDictionary
object.
- See Also:
Detokenizer
,
DetokenizationDictionary
DictionaryDetokenizer
public DictionaryDetokenizer(DetokenizationDictionary dict)
detokenize
public Detokenizer.DetokenizationOperation[] detokenize(String[] tokens)
- Description copied from interface:
Detokenizer
- Detokenize the input tokens.
- Specified by:
detokenize
in interface Detokenizer
- Parameters:
tokens
- the tokens to detokenize.
- Returns:
- the merge operations to detokenize the input tokens.
detokenize
public String detokenize(String[] tokens,
String splitMarker)
- Description copied from interface:
Detokenizer
- Detokenize the input tokens into a String. Tokens which
are connected without a space inbetween can be separated by
a split marker.
- Specified by:
detokenize
in interface Detokenizer
splitMarker
- the split marker or null
- Returns:
Copyright © 2013 The Apache Software Foundation. All Rights Reserved.