BitDistill: Microsoft’s 1.58‑Bit Pipeline Cuts Memory by 10× and Speeds CPU by ~2.65×

What BitNet Distillation Does

Microsoft Research introduces BitNet Distillation (BitDistill), a practical three‑stage pipeline to convert pretrained FP16 large language models into 1.58‑bit BitNet students. The method targets downstream deployment: it preserves accuracy close to the FP16 teacher while producing CPU‑friendly ternary weights and INT8 activations, yielding substantial memory and inference improvements.

Why direct conversion fails and the pipeline goal

Prior work showed that BitNet trained from scratch can match full‑precision quality, but naively quantizing a pretrained FP16 model to 1.58 bits typically degrades accuracy — and that gap widens with model size. BitNet Distillation focuses on closing this gap so teams can convert existing FP16 models without full retraining and still get the efficiency benefits of extreme quantization.

Stage 1 — Architectural refinement with SubLN

Low‑bit models often suffer from large activation variance that destabilizes quantized projections. BitNet Distillation inserts SubLayer Normalization (SubLN) inside each Transformer block: specifically before the output projection of the multi‑head self‑attention (MHSA) and before the output projection of the feed‑forward network (FFN). This stabilizes the hidden state scales entering quantized projections, improving optimization and convergence when weights are restricted to ternary values.

Empirical loss curves reported by the authors support that SubLN reduces optimization instability during low‑bit training.

Stage 2 — Continued pretraining to shape weights

Fine‑tuning a student at 1.58 bits directly on a task provides relatively few task tokens and is insufficient to reshape FP16 weight distributions into ternary‑friendly shapes. To address this, BitNet Distillation performs a short continued pretraining pass on a general corpus (the team uses 10B tokens from the FALCON corpus) to push weights toward BitNet‑like distributions. Visualizations show mass concentrating near the ternary transition boundaries so that small gradients during subsequent task training can flip weights among [-1, 0, 1]. This step improves learning capacity without requiring a full pretraining run.

Stage 3 — Dual‑signal distillation during fine‑tuning

The student learns from the FP16 teacher through two complementary signals:

• Logits distillation: temperature‑softened KL divergence between teacher and student token distributions.
• Multi‑head attention relation distillation: following MiniLM and MiniLMv2 formulations, which transfer relational patterns among Q, K, V without needing identical head counts and allow selecting a single layer to distill.

Ablations indicate that combining both signals outperforms using either alone, and that choosing a well‑placed layer preserves flexibility while delivering strong transfer.

Evaluation and results

The team evaluated classification tasks, MNLI, QNLI, SST‑2, and summarization on CNN/DailyMail, comparing three settings: FP16 task fine‑tuning, direct 1.58‑bit task fine‑tuning, and BitNet Distillation. For Qwen3 backbones at 0.6B, 1.7B, and 4B parameters, BitNet Distillation matches FP16 task accuracy while the direct 1.58‑bit baseline falls further behind as model size increases.

On CPU, the BitNet students achieve about 2.65× higher tokens‑per‑second and roughly 10× lower memory usage. Activations are quantized to INT8, and gradients through the quantizer use the Straight Through Estimator. The framework is compatible with post‑training quantization tools like GPTQ and AWQ for additional gains. Distilling from a stronger teacher tends to help, suggesting pairing small 1.58‑bit students with larger FP16 teachers when available.

Practical implications and deployment

BitNet Distillation is a pragmatic recipe for moving pretrained FP16 models to extreme low‑bit deployment without a full retrain. The three stages — SubLN insertion, short continued pretraining, and dual‑signal distillation — map cleanly to known failure modes in extreme quantization. The reported 10× memory reduction and ~2.65× CPU speedup at near‑FP16 accuracy make BitDistill attractive for on‑premise, edge, and other constrained deployments. The project provides optimized CPU and GPU kernels in bitnet.cpp, lowering integration risk for production teams.

For more details check the technical paper: https://arxiv.org/pdf/2510.13998 and the project GitHub repository linked by the authors.