public interface Detokenizer
Modifier and Type | Interface and Description |
---|---|
static class |
Detokenizer.DetokenizationOperation
This enum contains an operation for every token to merge the
tokens together to their detokenized form.
|
Modifier and Type | Method and Description |
---|---|
Detokenizer.DetokenizationOperation[] |
detokenize(String[] tokens)
Detokenize the input tokens.
|
String |
detokenize(String[] tokens,
String splitMarker)
Detokenize the input tokens into a String.
|
Detokenizer.DetokenizationOperation[] detokenize(String[] tokens)
tokens
- the tokens to detokenize.String detokenize(String[] tokens, String splitMarker)
tokens
- the token which should be concatenatedsplitMarker
- the split marker or nullCopyright © 2020 The Apache Software Foundation. All rights reserved.