Posits, a New Kind of Number, Improves the Math of AI

Abstraction: Posit number format achieves higher numerical accuracy than floating-point for AI computations

Key points:

• Posits proposed by John Gustafson and Isaac Yonemoto (2017) as an improvement over IEEE floating-point; add a variable-length "regime" field (exponent of exponent) that packs more precision near 1 and -1
• Deep neural network weights are normalized (near 1/-1), making them ideal candidates for posit's precision sweet spot
• Complutense University of Madrid built the first FPGA hardware implementation of the posit standard, enabling direct hardware comparison
• 32-bit posit matrix multiplication showed a four-orders-of-magnitude accuracy improvement over 32-bit floats (benchmarked against 64-bit double precision), with no increase in computation time — only modest chip area and power overhead
• Quire register enables efficient multiply-accumulate accumulation without repeated truncation, reducing cascading rounding error in neural network layers
• Nvidia, Arm, and Intel are concurrently standardizing 8-bit floating-point (FP8) as an alternative efficiency approach, trading accuracy for smaller memory footprint

Connections: Nvidia · Intel · Numerical Computing · Hardware For AI · Neural Network Training

Source: https://spectrum.ieee.org/floating-point-numbers-posits-processor