There are several types of loss scaling techniques:
Even on older hardware (V100s) that don't support BF16, researchers have developed algorithmic approaches to avoid scaling. These involve accumulating partial gradients in higher precision (FP32) locally before converting to FP16 for communication or weight updates. While technically involving precision management, modern libraries abstract this away, making the user experience "scaling free." loss scaling free
The term "loss scaling free" typically refers to the Lossless Scaling software, a universal tool that allows users to apply advanced upscaling and frame generation to virtually any game or application on Windows, regardless of its original hardware or software limitations. While the core software is a paid application on Steam , there are open-source alternatives and plugins that offer similar "scaling-free" or "lossless" performance benefits. Core Technology: LSFG and Upscaling Lossless Scaling Frame Generation (LSFG) There are several types of loss scaling techniques:
Here’s a helpful write-up on training, aimed at practitioners who use mixed precision (FP16/BF16) and want to avoid the complexity of manual or dynamic loss scaling. While the core software is a paid application
Transitioning to a loss scaling free environment typically involves switching to the format. Developed by Google, BF16 uses the same 8-bit exponent as standard FP32, giving it a massive dynamic range of up to BF16 (Loss Scaling Free) Exponent Bits Max Value Loss Scaling Required (via GradScaler ) Not Needed Precision Higher (10-bit mantissa) Lower (7-bit mantissa)