Interface Detokenizer

  • All Known Implementing Classes:
    DictionaryDetokenizer

    public interface Detokenizer
    A Detokenizer merges tokens back to their untokenized representation.
    • Method Detail

      • detokenize

        Detokenizer.DetokenizationOperation[] detokenize​(String[] tokens)
        Detokenize the input tokens.
        Parameters:
        tokens - the tokens to detokenize.
        Returns:
        the merge operations to detokenize the input tokens.
      • detokenize

        String detokenize​(String[] tokens,
                          String splitMarker)
        Detokenize the input tokens into a String. Tokens which are connected without a space inbetween can be separated by a split marker.
        Parameters:
        tokens - the token which should be concatenated
        splitMarker - the split marker or null
        Returns:
        the concatenated tokens