mlx.nn.gelu_fast_approx

Contents

mlx.nn.gelu_fast_approx#

class gelu_fast_approx(x)#

A fast approximation to Gaussian Error Linear Unit.

See gelu() for the exact computation.

This function approximates gelu with a maximum absolute error \(< 0.015\) in the range \([-6, 6]\) using the following

\[x = x \sigma\left(1.702 x\right)\]

where \(\sigma(\cdot)\) is the logistic sigmoid.

References: - hendrycks/GELUs - https://arxiv.org/abs/1606.08415

Contents