public interface Detokenizer
|Modifier and Type||Interface and Description|
This enum contains an operation for every token to merge the tokens together to their detokenized form.
Detokenizer.DetokenizationOperation detokenize(String tokens)
tokens- the tokens to detokenize.
tokens- the token which should be concatenated
splitMarker- the split marker or null
Copyright © 2017 The Apache Software Foundation. All rights reserved.