mlx.data.core.Tokenizer.tokenize_shortest

Tokenizer.tokenize_shortest(self: mlx.data._c.core.Tokenizer, input: str) → List[int]

Tokenize the input such that the sum of trie_key_scores is minimized.

Parameters:

input (str) – The input string to be tokenized.