opennlp.tools.tokenize
Class DictionaryDetokenizer

java.lang.Object
  extended by opennlp.tools.tokenize.DictionaryDetokenizer
All Implemented Interfaces:
Detokenizer

public class DictionaryDetokenizer
extends Object
implements Detokenizer

A rule based detokenizer. Simple rules which indicate in which direction a token should be moved are looked up in a DetokenizationDictionary object.

See Also:
Detokenizer, DetokenizationDictionary

Nested Class Summary
 
Nested classes/interfaces inherited from interface opennlp.tools.tokenize.Detokenizer
Detokenizer.DetokenizationOperation
 
Constructor Summary
DictionaryDetokenizer(DetokenizationDictionary dict)
           
 
Method Summary
 Detokenizer.DetokenizationOperation[] detokenize(String[] tokens)
          Detokenize the input tokens.
 String detokenize(String[] tokens, String splitMarker)
          Detokenize the input tokens into a String.
 
Methods inherited from class java.lang.Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

DictionaryDetokenizer

public DictionaryDetokenizer(DetokenizationDictionary dict)
Method Detail

detokenize

public Detokenizer.DetokenizationOperation[] detokenize(String[] tokens)
Description copied from interface: Detokenizer
Detokenize the input tokens.

Specified by:
detokenize in interface Detokenizer
Parameters:
tokens - the tokens to detokenize.
Returns:
the merge operations to detokenize the input tokens.

detokenize

public String detokenize(String[] tokens,
                         String splitMarker)
Description copied from interface: Detokenizer
Detokenize the input tokens into a String. Tokens which are connected without a space inbetween can be separated by a split marker.

Specified by:
detokenize in interface Detokenizer
splitMarker - the split marker or null
Returns:


Copyright © 2013 The Apache Software Foundation. All Rights Reserved.