mlx.data.core.Tokenizer.tokenize

Tokenizer.tokenize(self: mlx.data._c.core.Tokenizer, input: str) → mlx.data._c.core.GraphInt64

Return the full graph of possible tokenizations.

Parameters:

input (str) – The input string to be tokenized.