AI model quantization reduces the precision used for numbers inside a model. The change can shrink the model, lower memory use, and speed up inference. It often helps a model run on phones, laptops, and smaller graphics processors.
A model stores learned values called weights. Training may use 32-bit floating-point numbers. Each value then takes 32 binary digits of storage. Quantization might represent many values with 16 bits, 8 bits, or 4 bits instead.
Smaller numbers create practical gains
Moving model weights between memory and a processor takes time and energy. A lower-bit model moves less data. Hardware can also perform some low-precision calculations faster. These gains may let a device generate more tokens per second or serve more users on one machine.
A rough example shows the storage change. One billion parameters at 32 bits need about four gigabytes just for the raw weights. The same count at 8 bits needs about one gigabyte. File formats add details, so real sizes do not match the simple estimate exactly.
Rounding can change an answer
Lower precision cannot represent every original value. The process maps a range of detailed numbers onto a smaller set of levels. Small differences disappear. Most may not matter, but some layers or rare inputs can react badly.
Quality tests should cover the model’s real tasks. A general text score may stay stable while code output declines. A vision model may keep common object accuracy and miss small features. Teams compare speed, memory, energy, and output quality before picking a format.
Two common routes
Post-training quantization changes a completed model. It needs no full retraining and offers a fast path to a smaller file. A calibration set may help choose good number ranges. That set should resemble real input.
Quantization-aware training simulates low precision while the model learns. The model can adjust around rounding effects. This route takes more work but may preserve quality at lower bit levels.
Weights are only part of memory use
Language model inference also stores temporary values and a key-value cache. The cache holds attention information from earlier tokens. Long context can make it large. Quantizing weights alone does not remove every memory limit.
Hardware support matters as well. A processor may handle 8-bit operations well and offer little gain for an unusual 4-bit format. Software might unpack compressed values during use. A small file can run slowly when the device lacks the right operations.
Quantization changes how a model represents numbers. It does not add knowledge or repair weak training data. It serves as an engineering tradeoff that makes an existing capability cheaper or more portable.