Posts by Colin Zeng
Online Quantization: Smooth Path from One Checkpoint to Any Desired Serving Scenario
- 05 November 2026
Quantization lowers the numeric precision of a model’s weights (and often its activations) — for example from bf16 to FP8 or MXFP4 — to shrink memory footprint and speed up inference. Traditionally this happens offline, before serving: you run a quantization script, apply accuracy-recovery algorithms (rotation, SmoothQuant, GPTQ, AWQ) with calibration data, write a new, smaller quantized checkpoint to disk, and later load that checkpoint into the serving engine. It is the right way to produce the final artifact for a model you will serve many times.