Class TokenPatternFeatureGenerator

  • All Implemented Interfaces:

    public class TokenPatternFeatureGenerator
    extends Object
    implements AdaptiveFeatureGenerator
    Partitions tokens into sub-tokens based on character classes and generates class features for each of the sub-tokens and combinations of those sub-tokens.
    • Constructor Detail

      • TokenPatternFeatureGenerator

        public TokenPatternFeatureGenerator()
        Initializes a new instance. For tokinization the SimpleTokenizer is used.
      • TokenPatternFeatureGenerator

        public TokenPatternFeatureGenerator​(Tokenizer supportTokenizer)
        Initializes a new instance.
        supportTokenizer -
    • Method Detail

      • createFeatures

        public void createFeatures​(List<String> feats,
                                   String[] toks,
                                   int index,
                                   String[] preds)
        Description copied from interface: AdaptiveFeatureGenerator
        Adds the appropriate features for the token at the specified index with the specified array of previous outcomes to the specified list of features.
        Specified by:
        createFeatures in interface AdaptiveFeatureGenerator
        feats - The list of features to be added to.
        toks - The tokens of the sentence or other text unit being processed.
        index - The index of the token which is currently being processed.
        preds - The outcomes for the tokens prior to the specified index.