重新思考LLM FP4预训练中的压缩偏见：几何起源、系统性影响与UFP4方法
Rethinking Shrinkage Bias in LLM FP4 Pretraining: Geometric Origin, Systemic Impact, and UFP4 Recipe
摘要
FP4训练能够显著降低LLM预训练的记忆和计算成本。不过，目前使用的FP4硬件设备和技术方案，包括NVIDIA Blackwell/Rubin系列系统和AMD MI350系列GPU，仍然以E2M1数据元素为基础。在本研究中，我们发现了这种选择的一个根本缺陷：非均匀格式如E2M1存在“收缩偏差”问题，这种偏差是由其表示方式中的几何不对称性导致的系统性舍入误差。我们发现，这种偏差会在不同层之间逐层累积，而随机Hadamard变换则进一步加剧了这种偏差，从而解释了现有基于E2M1的FP4训练方法中出现的不稳定现象。相比之下，均匀网格格式（E1M2/INT4）可以避免这种几何错误，并能更好地将随机Hadamard变换带来的优化效果转化为更高的量化质量。基于这一发现，我们提出了UFP4——一种均匀的4位训练方法，它将随机Hadamard变换应用于所有三个训练矩阵中，同时仅对dY进行随机舍入处理。在Dense 1.5B、MoE 7.9B和MoE 124B规模的长期预训练中，UFP4所产生的BF16相对损失下降幅度明显低于基于E2M1的强基线方法，这一结果得到了规模分析以及消融实验的支持。我们的研究结果表明，未来的加速器应该支持类似E1M2/INT4的均匀4位网格格式作为与E2M1并列的主要训练方式。
English Abstract
FP4 training promises substantial reductions in memory and computation cost for LLM pretraining, yet current FP4 hardware paths and recipes, including NVIDIA Blackwell/Rubin-class systems and AMD MI350-series GPUs, remain centered on E2M1 data elements. In this study, we identify a fundamental limitation of that choice: non-uniform formats such as E2M1 inherently suffer from Shrinkage Bias, a systematic negative rounding error caused by the geometric asymmetry of their representable bins. We show that this bias accumulates multiplicatively across layers and is amplified by the Random Hadamard Transform (RHT), providing a unified explanation for the training instability observed in existing E2M1-based FP4 recipes. In contrast, uniform grids (E1M2/INT4) bypass this grid-geometry error and better convert the improved bucket utilization from RHT into higher quantization quality. Based on this finding, we propose UFP4, a uniform 4-bit training recipe that applies RHT to all three training GEMMs while restricting stochastic rounding to dY alone. On Dense 1.5B, MoE 7.9B, and MoE 124B long-run pretraining, UFP4 consistently achieves lower BF16-relative loss degradation than strong E2M1-based baselines, supported by scaling-law analysis and ablation studies. Our results suggest that future accelerators should support E1M2/INT4-style uniform 4-bit grids as first-class training primitives alongside E2M1.