mlx.nn.SinusoidalPositionalEncoding

mlx.nn.SinusoidalPositionalEncoding#

class SinusoidalPositionalEncoding(dims: int, min_freq: float = 0.0001, max_freq: float = 1, scale: float | None = None, cos_first: bool = False, full_turns: bool = False)#

Implements sinusoidal positional encoding.

For more details see the paper Attention Is All You Need.

Parameters:

  • dims (int) – The dimensionality of the resulting positional embeddings.

  • min_freq (float, optional) – The minimum frequency expected. Default: 0.0001.

  • max_freq (float, optional) – The maximum frequency expected. Default: 1.

  • scale (float, optional) – A multiplicative scale for the embeddings. Default: sqrt(2/dims).

  • cos_first (bool, optional) – If True embed using [cos(x); sin(x)] instead of the reverse. Default: False.

  • full_turns (bool, optional) – If True multiply the frequencies with \(2\pi\). Default: False.

Methods