Interface LanguageModel

public interface LanguageModel
A language model can calculate the probability p (between 0 and 1) of a certain sequence of tokens, given its underlying vocabulary.

  • Method Summary

    Modifier and Type
    Method
    Description
    double
    calculateProbability(String... tokens)
    Calculate the probability of a series of tokens (e.g. a sentence), given a vocabulary.
    String[]
    predictNextTokens(String... tokens)
    Predict the most probable output sequence of tokens, given an input sequence of tokens.

  • Method Details

    • calculateProbability

      double calculateProbability(String... tokens)
      Calculate the probability of a series of tokens (e.g. a sentence), given a vocabulary.
      Parameters:
      tokens - the text tokens to calculate the probability for.
      Returns:
      the probability of the given text tokens in the vocabulary

    • predictNextTokens

      String[] predictNextTokens(String... tokens)
      Predict the most probable output sequence of tokens, given an input sequence of tokens.
      Parameters:
      tokens - a sequence of tokens
      Returns:
      the most probable subsequent token sequence