Author(s): Kyouma45

Originally published on Towards AI.

How BitNet Run a Transformer With (Almost) No Multiplication?

Paper-explained Series: 12

If you’ve read my earlier deep-dives on HRM, Mamba, and TRM, you know the recurring theme: the frontier isn’t only about making models bigger. Sometimes the most interesting research asks whether we’ve been paying for precision we never actually needed. BitNet is the purest expression of that idea. It asks a heretical question — what if every weight in a large language model were just −1, 0, or +1? — and then answers it by training models that match full-precision baselines while shrinking memory by an order of magnitude and turning the most expensive operation in deep learning, matrix multiplication, into plain addition.

Let me walk you through the whole story, from why a 70B model is so painful to run, through the math of quantization, to the ternary trick, and finally to the 2-billion-parameter model Microsoft trained on 4 trillion tokens and put on Hugging Face in 2025.

1. Just how heavy is a 70B model, really?

Let’s ground this in hardware you can actually buy. The NVIDIA RTX 4090 is the top consumer GPU, and it ships with 24 GB of VRAM. That’s your budget.

Now take a 70-billion-parameter model like Llama-3 70B. In FP16 (half precision), every parameter occupies 2 bytes. So the weights alone require:

70 × 10⁹ params × 2 bytes = 140 GB

That’s just the static weights. It doesn’t fit in 24 GB — it doesn’t fit in six RTX 4090s (144 GB), and once you account for overhead you need a seventh card. APXML’s Llama-3 70B VRAM guide is explicit: at FP16 with 1,024 tokens of context the model needs 148.85 GB → 7× RTX 4090 (24 GB each), rising to 151.32 GB → 8× RTX 4090 at 8,192 tokens (or, in the data center, “2× NVIDIA A100 · 80 GB”).

And weights are only part of the bill. During inference, transformers cache the key and value vectors for every past token — the KV cache — so they don’t recompute attention from scratch at each step. This cache grows linearly with sequence length and batch size, and for a 70B model at long context it can rival or exceed the model weights themselves. On top of that you have activations flowing through the network. A frontier 70B model is fundamentally a data-center citizen: two 80 GB A100s or H100s, minimum, wired together with fast interconnect.

For anyone who wants to run a capable model on a laptop, a phone, or a single consumer card, 140 GB of FP16 weights is simply a wall.

2. The first response: post-training quantization

The obvious fix is to stop storing weights in 16 bits. This is quantization — mapping high-precision floats into a smaller set of low-precision values.

The dominant flavor is post-training quantization (PTQ): you take an already-trained FP16 model and compress its weights after the fact, with no retraining (or just a short calibration pass over sample data). Two milestones defined the field:

Milestone 1: 8-bit Quantization (LLM.int8)

When researchers first tried compressing models larger than 6.7 billion parameters into 8-bit (INT8), the models suddenly broke down and output gibberish. The LLM.int8() paper discovered why: Outlier Features.

• The Outlier Problem: In large models, a tiny fraction of the network’s data (about 0.1%) suddenly develops massive, extreme values (“outliers”). However, these outliers are doing heavy lifting — if you delete them, the model’s accuracy plummets.
• The Scale Wrecking: Imagine trying to put the wealth of normal people and a billionaire on a scale of 1 to 10. If 10 is the billionaire’s wealth, everyone else gets rounded down to 0. You lose all the nuance of the normal people. This is what happens if you naively round a network with outliers into INT8.
• The LLM.int8() Solution (Mixed-Precision): The researchers created a sorting system. They isolate the few massive outliers and process them in their original, high-precision FP16 format. They take the remaining 99.9% of well-behaved, normal weights and compress them into INT8. The math is done separately and then combined. The result? A model that takes up half the memory but retains 100% of its intelligence.

Milestone 2: 4-bit Quantization (GPTQ, AWQ, QLoRA)

Compressing from 16-bit to 8-bit is hard, but going to 4-bit is brutal. In 4-bit, you only have 16 possible numbers to represent a weight. You have to be incredibly clever about how you round things. The text highlights three distinct strategies for surviving 4-bit compression:

A. GPTQ (The “Compensation” Trick):

When you round a weight down, you introduce a tiny mathematical error. GPTQ uses advanced calculus (Hessian matrices) to look at how that error ripples through a specific layer of the network.

• How it works: If GPTQ rounds one weight down, it instantly recalculates and slightly bumps the remaining unquantized weights up to compensate. By dynamically balancing the scales as it compresses, the final output of the layer stays almost identical to the original uncompressed version.

B. AWQ (The “VIP Protection” Trick):

Not all weights are equally important. AWQ (Activation-aware Weight Quantization) runs sample text (activations) through the model to see which weights are actually doing the most work.

• How it works: It identifies the “load-bearing pillars” of the model (salient channels). Instead of treating all weights equally, it protects those highly important weights, leaving them highly accurate, while aggressively compressing the less important weights.

C. QLoRA and NF4 (The “Bell Curve” Trick):

When you compress a model down to 4-bit, you only have exactly 16 available numbers to represent the billions of weights inside the AI.

If you look at the raw data of a neural network, the vast majority of its weights are tiny fractions clustered incredibly close to zero (e.g., 0.015, -0.04, 0.002). Very few weights are large numbers. They form a natural bell curve.

If you space your 16 available numbers evenly across a number line, you create a major problem:

1. You waste slots on large numbers that barely exist in the model.
2. You don’t have enough slots near zero to capture the delicate, subtle differences between all those tiny fractions. They all get aggressively rounded to the same number, and the model loses its intelligence.

NF4 (4-bit NormalFloat) solves this by changing the spacing. Instead of spacing the 16 numbers evenly, it clusters the majority of them tightly around zero and spaces them much further apart at the extreme edges. Because the spacing perfectly matches the bell-curve distribution of the data, the model retains maximum precision exactly where the bulk of its “brain” actually operates.

Adding LoRA to the Mix

Once you compress a massive model into 4-bit using NF4, it becomes “read-only.” The math is too restricted to teach it anything new (a process called fine-tuning). This is where LoRA (Low-Rank Adaptation) comes in.

To teach the AI a new skill without uncompressing it, researchers use a clever workaround:

1. Freeze the Base: You lock all the billions of 4-bit compressed weights so they can no longer be changed.
2. Attach the Adapter: You create a tiny, brand-new set of high-precision (16-bit) weights and attach them alongside the massive frozen model.
3. Train the Adapter: When you feed the model new training data, you only update this tiny new side-module.

Think of the 4-bit base model like a massive, printed encyclopedia. You can’t change the printed text, but LoRA allows you to slap a small sticky note of new, specialized instructions onto the page.

QLoRA is simply the combination of these two techniques: using NF4 to shrink the base model so it fits on your graphics card, and using LoRA to train it on new data without running out of memory.

3. Why a quantized big model beats a small full-precision one

Here’s the counterintuitive part that motivates everything after: given a fixed memory budget, you’re almost always better off taking a big model and quantizing it hard than training a small model at full precision.

The definitive study is Dettmers & Zettlemoyer’s “The case for 4-bit precision: k-bit inference scaling laws” (2023). In their words, they “run more than 35,000 experiments with 16-bit inputs and k-bit parameters to examine which zero-shot quantization methods improve scaling for 3 to 8-bit precision at scales of 19M to 176B parameters across the LLM families BLOOM, OPT, NeoX/Pythia, and GPT-2.” The headline: 4-bit precision is almost universally optimal for the trade-off between total model bits and zero-shot accuracy. In other words, for a fixed number of bits, a 4-bit model with more parameters beats an 8-bit or 16-bit model with fewer.

But there’s a floor. Their scaling curves show the trend reverses below 3 bits — at 3-bit and lower, quality can fall off a cliff rather than degrade gracefully, and several models (OPT, Pythia) became unstable. PTQ, being a lossy afterthought applied to a model that never “knew” it would be quantized, simply can’t push reliably past that barrier.

That’s the gap BitNet aims at. If 4-bit is the PTQ sweet spot and sub-3-bit is a minefield for PTQ, maybe the way to reach 1-ish bits isn’t to compress after training — it’s to train in low precision from the start, so the network learns to live within the constraint.

4. The math of quantizing weights

Before we get to ternary, let’s make the mechanics concrete, because BitNet reuses exactly these primitives.

Here is the step-by-step breakdown of how modern quantization works.

A. The Core Math: Symmetric Quantization

The most common method is absmax (symmetric) quantization. It assumes your data is roughly centered around zero, and it calculates a single multiplier to stretch or shrink your weights so they perfectly fit into your integer limits.

Let’s assume you want to compress your weights into b-bit integers.

• First, you find the single largest absolute number in your entire matrix of weights: max(|W|).
• Then, you divide the maximum possible integer your bit-depth allows (2^(b-1) — 1) by that largest weight.

Example: If you are using 8-bit quantization, the max integer is 127. If the largest weight in your network is 2.5, your scale is 127 / 2.5 = 50.8.

• You multiply every weight in the network by the scale $s$, and then round it to the nearest whole integer.

scale s = (2^(b−1) − 1) / max(|W|)
W_quant = round(s · W) # integers in [−(2^(b−1)−1), +(2^(b−1)−1)]

To use the weights, you dequantize by dividing back out:

W_dequant = W_quant / s

Because rounding throws information away, W_dequant ≠ W. The gap is the quantization error, and minimizing it is the whole game.

B. The Zero-Point: Asymmetric Quantization

Symmetric quantization forces the “zero” of the integer buckets to line up perfectly with the mathematical 0.0. This works beautifully if your weights form a bell curve centered on zero.

But what if your data is purely positive (like activations that have passed through a ReLU function, which deletes negative numbers)? If you use symmetric quantization, you will waste half of your integer buckets on negative numbers that don’t exist.

Asymmetric quantization introduces a “zero-point” — an offset that shifts the integer buckets left or right. Instead of centering on zero, it perfectly maps the minimum value of your data to the lowest bucket, and the maximum value to the highest bucket. No buckets are wasted.

C. Granularity: Managing the Outliers

The math above requires calculating a scale (s). The critical question is: how much data do you apply that single scale to?

If you use one scale for millions of weights, a single massive outlier will ruin the scale for everything else (like a billionaire ruining the income scale for a normal neighborhood). Granularity determines how you group the weights before calculating their scale.

• Per-tensor: You calculate one single scale for the entire weight matrix. It is extremely fast and computationally cheap, but highly vulnerable to outliers.
• Per-channel / Per-row: You calculate a separate scale for every single row in the matrix. If row 4 has a massive outlier, it only ruins the precision for row 4. The other rows get their own perfectly optimized scales. This is how LLM.int8() survives outliers.
• Group-wise / Block-wise: The most aggressive grouping. You chop every row into tiny blocks (e.g., 64 weights per block) and calculate a unique scale for each block. This provides incredibly high precision and minimizes quantization error, which is why QLoRA uses block-wise quantization.

Keep these three ideas — scale, rounding error, and granularity — in your head. BitNet’s ternary quantizer is just a very aggressive, very cleverly-scaled version of them.

5. Pushing to 1 bit: the original BitNet and BitLinear

In October 2023, Wang et al. published “BitNet: Scaling 1-bit Transformers for Large Language Models” (arXiv:2310.11453). Their claim to fame: the first architecture to do quantization-aware training (QAT) for 1-bit LLMs from scratch, using a drop-in module called BitLinear to replace nn.Linear.

A. Binarizing the weights

In this first version, weights were binary — just {−1, +1}. The recipe: center the weights to zero mean, then take the sign.

W̃ = Sign(W − α), where α = (1/nm) Σ W_ij (Eqs. 1–3)

α is simply the mean of the weight matrix. Subtracting it before applying Sign — rather than signing the raw weights directly — is the detail that makes this work well rather than just work.

Sign(x) = +1 if x > 0, −1 otherwise

Here’s why centering matters: a single bit carries the most information when its two outcomes are equally likely. If the raw weight distribution has any skew (and trained weight matrices almost always do), signing them directly produces an imbalanced mix of +1s and −1s — some of the 1-bit budget gets wasted encoding a bias that a scalar could have captured for free. Centering to zero mean before signing pushes the split toward 50/50, which maximizes the entropy of the resulting binary code. The mean α is discarded after it’s done its job of setting the decision boundary — it isn’t added back anywhere downstream.

B. Recovering magnitude with β

Binarization throws away magnitude entirely. A weight of 0.002 and a weight of 4.8 both collapse to the same +1 if they’re on the same side of α. To partially undo this, BitNet computes one scalar per weight matrix:

β = (1/nm) Σ|W_ij| = ‖W‖₁ / (nm)

This is just the mean absolute weight — but it isn’t an arbitrary choice of “some average.” It’s the exact solution to a least-squares problem:

min_β ‖W − βW̃‖²

Given that the sign pattern W̃ is already fixed, β is the one number that makes βW̃ the closest possible reconstruction of the real-valued W, in the least-squares sense. So the two-step recipe — sign first, then scale — isn't heuristic layering; each step is separately optimal given the one before it.

C. Quantizing activations to 8-bit

Weights collapse to 1 bit, but activations are left considerably more precision — 8 bits in BitNet’s experiments — because activations carry the actual input-dependent signal flowing through the network and are far more sensitive to aggressive rounding. The scheme is standard absmax quantization

γ = max(|x|)
Quant(x) = Clip(round(x · Q_b / γ), −Q_b, Q_b − 1), Q_b = 2^(b−1)

γ is the largest-magnitude value in the activation vector. Scaling by Q_b / γ stretches the activations so the biggest value uses the full available integer range, then rounding and clipping produces signed 8-bit integers in [−128, 127] for b = 8.

D. SubLN — normalizing before you quantize

Right before the quantization step, activations pass through a LayerNorm placed inside the sublayer (hence “Sub-LN”), rather than only at block boundaries the way a standard Pre-LN transformer does it.

This does two jobs at once. First, it keeps the input to Quant() well-scaled: without it, activation magnitudes can drift as they propagate through many binarized layers, and a single outlier value would inflate γ and crush the resolution available to every other entry in that vector. Second, it stabilizes training — a network built from 1-bit weights has meaningfully different variance dynamics than a full-precision one, and the extra normalization compensates for that so gradients don't blow up or vanish across depth.

E. Putting it together: the BitLinear forward pass

y = W̃ · Quant(LN(x)) × (βγ / Q_b)

Walking through it left to right:

1. LN(x) — normalize the incoming activations (this is SubLN).
2. Quant(LN(x)) — absmax-quantize the normalized activations into 8-bit integers.
3. W̃ · Quant(LN(x)) — the matrix multiply itself, but now between a matrix of pure ±1s and a matrix of small integers. Multiplying by ±1 isn't really multiplication — it's addition or subtraction. This is the entire point of binarizing the weights: the most expensive operation in a transformer, the matmul, becomes integer accumulation instead of floating-point multiply-accumulate, which is dramatically cheaper in both compute cycles and energy on real hardware.
4. × (βγ / Q_b) — a single scalar rescale applied once to the output, not per element. β restores the weight magnitude discarded by Sign(); γ / Q_b restores the activation scale compressed by Quant(). This is the only floating-point arithmetic left in the whole layer.

So the O(nm) part of the computation — the part that scales with model size — runs in cheap integer ops, and the only float math is a single scalar multiply on the output.

Importantly, BitNet keeps its gradients and optimizer states in full precision during training and uses a deliberately large learning rate — a small nudge to a latent weight often won’t flip a binary value, so aggressive steps are needed to make progress.

6. From 1 bit to 1.58 bits: the ternary leap

In February 2024, the same group published the paper that made the field sit up: “The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits” (Ma et al., arXiv:2402.17764). The change is deceptively small: add a third value, 0, so every weight is now ternary: {−1, 0, +1}.

The quantizer switches from sign to absmean: scale the weight matrix by its mean absolute value, then round each weight to the nearest of {−1, 0, +1}:

W̃ = RoundClip(W / (γ + ε), −1, 1)
RoundClip(x, a, b) = max(a, min(b, round(x)))
γ = (1/nm) Σ |W_ij|

Why does adding zero matter so much? Because 0 lets the network switch a connection completely off. A weight of 0 means “this input feature does not contribute to this output” — it’s built-in feature filtering and sparsity, learned during training. Binary {−1, +1} forces every connection to vote either for or against; ternary lets the model abstain. That extra expressive freedom is exactly why b1.58 closes the gap with full precision that binary BitNet couldn’t. You still keep the multiplication-free property — a ternary matmul is just conditional adds and skips.

7. The results: memory, speed, energy, accuracy

BitNet b1.58 was compared against a reproduced FP16 LLaMA, both trained from scratch on 100 billion tokens of RedPajama at matched sizes. The numbers:

The crossover is the headline: at 3B parameters, BitNet b1.58 matches full-precision LLaMA in perplexity while using 3.55× less GPU memory and running 2.71× faster. Zero-shot task accuracy tells the same story — the gap narrows with scale and b1.58 actually matches or exceeds FP16 from 3B onward (b1.58 3B scored 50.2 average vs LLaMA 3B’s 49.7 across seven tasks). And the 3.9B ternary model beats the 3B FP16 model on accuracy while being cheaper on every axis — a genuine Pareto improvement.

On energy, the paper estimates BitNet b1.58 saves 71.4× the arithmetic energy of matrix multiplication versus FP16 on a 7 nm chip (because it’s almost all INT8 addition, no FP16 multiplication). End-to-end energy savings grow with size, from ~18.6× at 1.3B to ~41.2× at 70B.

On throughput, comparing two 70B models on A100–80GB cards: BitNet’s smaller memory footprint let it run 11× the batch size and deliver 8.9× the throughput (2,977 vs 333 tokens/sec) of FP16 LLaMA.

The team also verified token-scalability: a b1.58 3B trained on 2 trillion tokens beat StableLM-3B (trained on the same 2T tokens) on every reported end task.

8. The Pareto claim: a 70B ternary model vs a 13B FP16 model

The most quotable consequence is the “equivalence table” the paper derives from its latency/memory/energy curves. Because the savings compound as models grow, they claim:

• 13B b1.58 is more efficient (latency, memory, energy) than 3B FP16
• 30B b1.58 is more efficient than 7B FP16
• 70B b1.58 is more efficient than 13B FP16

Read that last line slowly. A 70-billion-parameter ternary model is cheaper to run than a 13-billion-parameter FP16 model — while being vastly more capable, since it has more than five times the parameters. This is the inversion of everything the 140 GB wall implied. The bottleneck for LLM inference has historically been memory bandwidth (shuttling weights from DRAM to on-chip SRAM); when your weights are 1.58 bits instead of 16, that bandwidth cost mostly evaporates. (Note this is a projection from the paper’s scaling curves — see the caveats.)

9. Why “1.58” bits?

This trips people up, so let’s nail it. A single ternary weight can be in one of 3 states. The information content of choosing among 3 equally likely options is:

log₂(3) ≈ 1.585 bits

That’s it. Two states (binary) = log₂(2) = 1 bit. Three states (ternary) = log₂(3) ≈ 1.58 bits. The name “b1.58” is a precise, slightly nerdy way of saying “ternary.” You can’t store 1.58 bits directly, of course — in practice the inference code packs multiple ternary weights into one byte. Microsoft’s GPU kernel packs four ternary values into a single INT8 (four values × ~1.58 ≈ 6.3 bits, comfortably under 8).

10. The hardware problem: GPUs weren’t built for this

Here’s the catch that the papers are refreshingly honest about. Today’s GPUs — with their cuBLAS libraries and tensor cores — are exquisitely optimized for FP16, BF16, and INT8/INT4 GEMM (general matrix multiply). There is no native hardware instruction for a “1.58-bit × 8-bit” matrix multiply (W1.58A8). So even though ternary weights should be dramatically faster, off-the-shelf GPUs can’t fully cash in the theoretical win.

The workaround is custom kernels. For GPUs, Microsoft wrote a bespoke CUDA kernel that stores four packed ternary weights per INT8 in high-bandwidth memory, loads them into fast on-chip SRAM, and unpacks-then-computes (“pack-store-load-unpack-compute”). For CPUs, they built bitnet.cpp, an official inference framework (forked from llama.cpp, using lookup-table kernels pioneered in T-MAC). Per Microsoft’s own benchmarks, bitnet.cpp delivers 2.37×–6.17× speedups on x86 CPUs with 71.9%–82.2% energy reduction, and 1.37×–5.07× on ARM with 55.4%–70.0% energy reduction — all “lossless” relative to the training procedure. Strikingly, “bitnet.cpp can run a 100B BitNet b1.58 model on a single CPU, achieving speeds comparable to human reading (5–7 tokens per second).”

Both papers end with the same call to action: this is a new computation paradigm that “opens the door for designing specific hardware optimized for 1-bit LLMs.” They explicitly point to Groq’s LPU-style dedicated inference hardware as evidence that purpose-built silicon delivers, and argue for chips designed around ternary addition from the ground up.

11. Which layers actually go 1-bit (and which don’t)

A common misconception is that BitNet ternarizes everything. It doesn’t. Only the big matrix multiplications become BitLinear:

Replaced with 1.58-bit BitLinear:

• The attention projections (Q, K, V, and output projection)
• The feed-forward network (FFN) linear layers (up, gate, down projections)

Kept in higher precision:

• Input/output embeddings — the model needs high-precision probabilities to sample tokens.
• The output head / lm_head — same reason.
• Normalization layers (RMSNorm/SubLN) — negligible compute cost, but crucial for stability.
• Activations — quantized to 8-bit (INT8), not ternary.
• Optimizer states, gradients, and the master weights during training — kept in full precision (more on this in §15).

The paper’s justification is pragmatic: residual connections and layer norm contribute negligible compute at scale, and the QKV transformation cost shrinks relative to the parametric projections as the model grows. So you ternarize where the FLOPs and the memory actually live, and leave the cheap, sensitive parts alone.

12. BitNet a4.8: quantizing the activations too

Once weights are ternary, as in BitNet b1.58, the memory cost of loading weights mostly disappears. What’s left is the matmul itself — the activation-times-weight arithmetic — which was still running on 8-bit activations. In November 2024, Wang, Ma, and Wei released BitNet a4.8: 4-bit Activations for 1-bit LLMs (arXiv:2411.04965), aimed squarely at that remaining cost. Weights stay ternary; the goal is to push activations from 8-bit down to 4-bit so inference can run on fast INT4/FP4 kernels.

The obstacle is the one that haunts every activation-quantization scheme: outlier channels. A handful of unusually large activation values force the quantization scale wide enough that everything else gets crushed into just a few usable integer levels. BitNet a4.8’s contribution, based on studying activation distributions across a 7B BitNet b1.58 model, was noticing that different parts of the network behave very differently — so rather than one blanket policy, it applies a hybrid quantization-and-sparsification strategy:

• Inputs to attention and FFN sublayers follow clean, roughly Gaussian distributions → quantize directly to 4-bit.
• Intermediate states — specifically the inputs to the FFN’s down-projection and to attention’s output projection — are the problem children: full of large outliers alongside a mass of values near zero. Quantizing these to 4-bit blows up the error. So instead of quantizing them tightly, a4.8 sparsifies them first — keep only the top-K largest-magnitude entries (K = 50% for the attention output projection, in the paper’s setup) and zero out the rest — then quantizes the survivors at a more forgiving 8-bit. This “sparsify-then-quantize” approach is borrowed from the Q-Sparse method.

To push sparsity further, a4.8 also swaps the FFN’s SwiGLU nonlinearity for a gated squared-ReLU variant (ReLU²GLU). Because squared ReLU hard-clamps every negative input to exactly zero, it drives the down-projection’s inputs to over 80% exact-zero entries at 7B scale, with the gate branch alone reaching around 67.5% zeros.

That last detail is what actually produces the paper’s headline number. Every time an activation entry lands on exactly zero — whether from ReLU² or from top-K masking — the weight column it would have multiplied contributes nothing to the output for that token, and can be skipped outright. Averaged across all the sublayers, weighted by each layer’s share of the model’s parameters, this works out to roughly half the network per token: for a 7B model, about 3.4B of 6.5B parameters actually get touched on a given forward pass — close to the paper’s rounded headline that only around 55% of parameters are active at inference.

It’s worth keeping the two levers separate, since it’s easy to conflate them: sparsification (from ReLU² and top-K masking) is what creates the zeros and shrinks the active fraction below 100% in the first place; quantization (4-bit vs. 8-bit) is a separate axis that just determines how cheaply the surviving ~55% gets computed, via fast INT4/FP4 kernels. Both matter, but only one of them is why not all the parameters get used.

a4.8 also enables a 3-bit KV cache. KV cache stores past keys and values and grows linearly with sequence length — for long contexts it becomes the memory bottleneck, often exceeding the weights themselves (for a model like LLaMA3–70B, serving 32 requests at 128K context can need over 1.2 TB of KV cache). a4.8 quantizes the KV cache down to 3 bits: after applying RoPE, the K and V heads are quantized directly with absmax to unsigned low-bit integers, no calibration needed, with one small exception — the bos (beginning-of-sequence) token's heads are kept at 4-bit because they carry the most extreme outlier features. The result: negligible accuracy loss even at 3-bit KV, roughly halving KV-cache memory versus 8-bit and letting you serve far longer contexts on the same hardware. Training-wise, a4.8 is continue-trained from a b1.58 checkpoint in two stages (W1.58A8 → W1.58A4), needing only a few billion extra tokens, and matches b1.58 accuracy at every size from 700M to 7B.

13. The open problem: token scale

For all its promise, there was a nagging asterisk on the early BitNet results. The b1.58 and a4.8 experiments were run at a research scale of 100 billion tokens (with a single 2-trillion-token comparison against StableLM). Frontier models are trained on trillions — Llama 3 on 15T, Qwen2.5 on 18T. It was genuinely unknown whether the “matches full precision” story would survive when you trained a native 1-bit model at true frontier data scale, or whether some subtle instability would emerge over trillions of tokens. Training a ternary model at full scale remained the field’s big open question.

14. BitNet b1.58 2B4T: the first native 1-bit LLM trained at scale

In April 2025, Microsoft Research answered the open question with “BitNet b1.58 2B4T Technical Report” (arXiv:2504.12285): a 2-billion-parameter native ternary model trained on 4 trillion tokens — the “2B4T” name. It’s the first open-source, native 1-bit LLM at this scale, and the weights are on Hugging Face (microsoft/bitnet-b1.58-2B-4T).

Architecture. A LLaMA-3-style decoder: 30 layers, hidden size 2,560, 20 attention heads with grouped-query attention (5 KV heads), intermediate size 6,912, squared-ReLU FFN, RoPE positional embeddings, SubLN normalization, no bias terms, and the LLaMA-3 tokenizer (128,256 vocab). All the big linear layers are BitLinear (ternary weights, absmean); activations are 8-bit (absmax, per-token). It’s a W1.58A8 model.

Training recipe. Three phases:

1. Large-scale pre-training on 4T tokens of web data (DCLM, FineWeb-EDU) plus synthetic math, with a distinctive two-stage schedule — a high peak learning rate followed by an abrupt “cooldown” to a low rate halfway through, paired with weight decay that drops from 0.1 to 0.
2. Supervised fine-tuning (with summed rather than mean cross-entropy loss).
3. Direct preference optimization (DPO, 2 epochs). The high-learning-rate trick works precisely because 1-bit models are more stable than full-precision ones — a small nudge to a latent weight often doesn’t flip the ternary value, so you can afford aggressive steps.

Results. Benchmarked against LLaMA 3.2 1B, Gemma-3 1B, Qwen2.5 1.5B, SmolLM2 1.7B, and MiniCPM 2B (all instruction-tuned, full precision), BitNet b1.58 2B4T posts an average of 54.19 across 16 benchmarks — essentially tied with Qwen2.5 1.5B’s 55.23 and ahead of everything else — while leading the pack on several: ARC-Challenge (49.91), WinoGrande (71.90), GSM8K math (58.38), and PIQA (77.09).

The efficiency, though, is the whole point:

To actually get these numbers you must run it through bitnet.cpp (or the custom CUDA kernel); the vanilla transformers library has no ternary kernels and will show none of the speed or energy benefits.

15. How you train a model whose weights can only be −1, 0, +1

The deepest challenge in training a ternary network is that calculus and rounding do not mix. The derivative of a rounding function is zero on the flat parts and undefined at the jumps. If you try to run standard backpropagation through a rounding step, the gradient zeroes out. The learning signal dies instantly, and the model learns nothing.

The Solution: Quantization-Aware Training (QAT)

To survive this, the system maintains two separate versions of every weight simultaneously:

• Latent Weights (The Master): Full-precision (BF16) decimals. These run in the background to accumulate gradients and are the only weights actually updated by the optimizer.
• Quantized Weights (The Shadow): The strict ternary (-1, 0, +1) values. These are generated on the fly by rounding the latent weights during every single forward pass.

The Training Loop and The STE

Because of this dual-weight setup, the training process relies on a strict division of labor. The bridge that makes this possible is the Straight-Through Estimator (STE).

Here is exactly how the loop flows:

1. The Forward Pass (The Guess): The model makes its predictions using only the ternary shadow weights. This forces the model to learn under the exact low-precision constraints it will face when deployed.
2. The Backpropagation (The Grade): The loss is calculated. The calculus works backward and computes the gradient specifically for the ternary weights, because they were the ones that actually did the math and caused the error.
3. The STE (The Bridge): Here, the math hits the non-differentiable rounding function. The STE acts as a mathematical bypass. It takes the gradient calculated for the ternary weight, completely ignores the rounding function that should block its path, and passes the gradient straight through to the other side.
4. The Update (The Application): The optimizer receives this bypassed gradient and applies it directly to the high-precision latent weight.

The Result: The gradients are calculated based on the ternary weights, but applied to the latent weights. Over thousands of steps, these tiny updates accumulate in the full-precision latent weights. Once a latent weight drifts far enough across a threshold, its rounded ternary shadow flips to a new value (-1, 0, or +1), and the network successfully learns.

This is also why the “1-bit LLMs training tips” guidance emphasizes a large learning rate and the two-stage weight-decay schedule: with weight decay applied to the latent weights, its magnitude acts like a confidence score for each ternary weight, so decay is disabled in the second half of training to let the model settle.

BitNet keeps gradients and optimizer states in full precision for exactly this reason — accumulating updates in low precision would vanish or explode. And the beautiful punchline: once training is done, you throw the latent weights away. They exist only to make learning possible. At inference you keep only the ternary weights — which is why the shipped inference model is 0.4 GB while the BF16 “master” checkpoint (also on Hugging Face, for fine-tuning) is much larger.

References

1. Wang, H., Ma, S., Dong, L., et al. (2023). BitNet: Scaling 1-bit Transformers for Large Language Models. arXiv:2310.11453.
2. Ma, S., Wang, H., Ma, L., et al. (2024). The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits. arXiv:2402.17764.
3. Wang, H., Ma, S., & Wei, F. (2024). BitNet a4.8: 4-bit Activations for 1-bit LLMs. arXiv:2411.04965.
4. Ma, S., Wang, H., Huang, S., et al. (2025). BitNet b1.58 2B4T Technical Report. arXiv:2504.12285.
5. Wang, J., Zhou, H., Song, T., et al. (2025). bitnet.cpp: Efficient Edge Inference for Ternary LLMs. arXiv:2502.11880 / “1-bit AI Infra, Part 1.1” (arXiv:2410.16144).
6. Ma, S., Wang, H., et al. The Era of 1-bit LLMs: Training Tips, Code and FAQ. Microsoft unilm repository (reproduced in BitNet: 1-bit Pre-training for LLMs, JMLR 26, 2025).
7. Dettmers, T., & Zettlemoyer, L. (2023). The case for 4-bit precision: k-bit Inference Scaling Laws. ICML 2023 (arXiv:2212.09720).
8. Dettmers, T., Lewis, M., Belkada, Y., & Zettlemoyer, L. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. arXiv:2208.07339.
9. Frantar, E., Ashkboos, S., Hoefler, T., & Alistarh, D. (2023). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. ICLR 2023.
10. Lin, J., Tang, J., Tang, H., et al. (2023). AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. arXiv:2306.00978.
11. Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS 2023 (arXiv:2305.14314).
12. Bengio, Y., Léonard, N., & Courville, A. (2013). Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation. arXiv:1308.3432.

Until next time, folks…
El Psy Congroo

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI