LittleBit-2: Samsung’s Sub-1-Bit LLMs via Latent Geometry Alignment

Lead — On July 23, 2026, Samsung Research published LittleBit-2, the successor to a compression method that pushes large language models below one bit per weight. Presented at ICML 2026, LittleBit-2 keeps the original LittleBit architecture from NeurIPS 2025 but changes one thing: how the compressed weights are initialized. That single change takes the method from lagging the best 1-bit baselines to matching or beating them across the entire sub-1-bit regime — with, Samsung says, no additional cost at inference time.

Advanced

Key points

Samsung Research published LittleBit-2 on July 23, 2026, presenting it at ICML 2026 while keeping the original LittleBit architecture from NeurIPS 2025.
LittleBit factorizes each dense weight into low-rank latent factors, so fractional bit-rates below 1 bit per weight become possible simply by shrinking the rank.
At 0.1 bpp, LittleBit compresses a Llama2-13B model from 26.06 GB in FP16 to about 0.84 GB, a 31x reduction, and a Llama2-70B from 138 GB to 1.98 GB.
LittleBit-2 solves for an orthogonal rotation with Joint-ITQ that is folded into the factors before training, so it adds nothing to inference time.
On Llama-3-8B at 1.0 bpp, LittleBit-2 reaches 11.53 perplexity with 57.33% average zero-shot accuracy, versus 16.30 for the original LittleBit and 13.09 for OneBit, which scored 52.23% accuracy.

LittleBit-2 pipeline diagram showing a pre-trained weight matrix going through truncated SVD, a Joint-ITQ solver that applies a rotation to the latent factors, then quantization-aware training and deployment — The LittleBit-2 pipeline: truncated SVD yields latent factors, a Joint-ITQ solver optimizes a rotation that aligns them with the binary hypercube, then dual-SVID initialization and quantization-aware training produce the deployable layer. Image credit: Samsung Research

Why Sub-1-Bit Is Hard

The memory wall is the practical obstacle to running large models locally. A 70B-parameter model needs roughly 140 GB of VRAM in FP16, and even a fully 1-bit version still demands around 15 GB — too much for most phones and laptops. Conventional quantization has a structural floor here: with binary weights at ±1, every parameter still costs one bit, so 1 bit per weight is the smallest budget a standard scheme can offer.

LittleBit sidesteps that floor by decoupling the number of stored parameters from the dimensions of the weight matrix. It factorizes each dense weight into low-rank latent factors, so a matrix with millions of entries can be represented by far fewer numbers. Fractional bit-rates below 1 bit per weight (bpp) then become possible simply by shrinking the rank. At the extreme, LittleBit compresses a Llama2-13B model from 26.06 GB in FP16 to about 0.84 GB at 0.1 bpp — a 31× reduction — and a Llama2-70B from 138 GB to 1.98 GB.

How LittleBit Compresses a Model

Diagram comparing a standard transformer layer with the LittleBit architecture, which splits each weight matrix into a primary and residual binary path with scale vectors applied at row, latent, and column dimensions — LittleBit replaces each linear layer with two binary latent paths (primary and residual), surrounded by FP16 scale vectors. Image credit: Samsung Research

LittleBit factorizes the weight matrix W into low-rank factors U and V, then binarizes them to a matrix of ±1 values. To limit the resulting loss of information, it learns three separate FP16 scaling vectors: a per-row scale, a per-column scale, and a per-rank “latent” scale that assigns a distinct importance weight to each rank — the part Samsung highlights as the key addition over prior scale-only methods.

Two training details make this trainable. Dual-SVID initializes the binary factors from the signs of a truncated singular value decomposition, and derives the scales from the magnitudes via a rank-1 approximation, giving quantization-aware training a stable starting point. Residual compensation then splits the same bit budget into a primary path plus a second path trained on the primary path’s error. At inference, the design is MatMul-free: the binary factors let the model replace FP16 matrix-vector products with bitwise operations. Samsung reports an 11.6× kernel-level speedup over an FP16 baseline for a 70B MLP layer at 0.1 bpp, and 203.2 tokens/second end-to-end on Llama2-7B versus 82.6 for FP16. A bonus is a much smaller KV cache, since the model caches the reduced latent dimension rather than the full hidden dimension — roughly 21× smaller for Llama2-7B at 0.1 bpp.

LittleBit-2: Fixing the Geometry

Two-panel plot showing spiky zero-centered latent distributions under standard SVD on the left, and a bimodal distribution aligned to the corners of a square after Joint-ITQ rotation on the right — Standard SVD produces “spiky” latent factors that sit close to the binary decision boundary (left); LittleBit-2’s rotation pushes them into a bimodal distribution aligned with the hypercube vertices (right). Image credit: Samsung Research

The original LittleBit still trailed the strongest 1-bit model. At the 1-bit operating point it scored a perplexity of 9.08 on Llama-2-7B while OneBit scored 8.36. Samsung’s diagnosis, detailed in the ICML paper, is latent geometry misalignment: standard SVD singular vectors concentrate their energy in a few “spiky” outlier coordinates, which is the worst possible shape for a sign-based binarization. Values near zero flip signs under tiny perturbations, and a handful of large outliers force the shared scales to overestimate the useful range for everything else.

The fix exploits a simple property: the factorization is rotationally invariant, so the latent factors can be rotated without changing the reconstructed weight. Applying an orthogonal rotation leaves the model’s math intact while reshaping the distribution of the latents. LittleBit-2 solves for the best such rotation with Joint-ITQ, an alternating-minimization procedure that moves the factors toward the corners of the binary hypercube and produces a bimodal distribution with a wide margin from the unstable zero point. Critically, the rotation is folded into the factors before training, so it adds nothing to inference. That distinguishes it from incoherence-processing methods such as QuIP#, QuaRot, and SpinQuant, which rely on online Hadamard transforms at run time.

Results

Benchmark table comparing FP16, GPTQ, EfficientQAT, BiLLM, ARB-LLM, OneBit, LittleBit, and LittleBit-2 on Llama-3 8B and Llama-2 7B and 13B, reporting WikiText-2 perplexity, zero-shot accuracy, and memory footprint at 1-bit, 0.55-bit, and 0.1-bit settings — LittleBit-2’s main results across the 1-bit and sub-1-bit regimes. Image credit: Samsung Research

On the standard WikiText-2 perplexity benchmark (lower is better), LittleBit-2 closes the gap at 1.0 bpp and dominates below it. On Llama-3-8B at 1.0 bpp it reaches 11.53 perplexity with 57.33% average zero-shot accuracy, versus 16.30 for the original LittleBit and 13.09 for OneBit, which scored 52.23% accuracy. On Llama-2-7B at 1.0 bpp it reaches 8.27, effectively matching OneBit’s 8.36 under the same memory budget.

The gains widen as compression deepens. At 0.55 bpp on Llama-3-8B, STBLLM collapses to 241.96 perplexity and the original LittleBit struggles at 18.47; LittleBit-2 holds 14.01. At an extreme 0.1 bpp, where the model body shrinks to roughly 0.1 GB — under 1% of an 8B model’s original footprint — LittleBit-2 is still functional at 23.74 perplexity. It also generalizes beyond Llama: on Gemma-3-27B at 0.1 bpp, where a floating-point low-rank approximation collapses entirely, LittleBit-2 retains 16.38 perplexity.

What This Means

The headline number — 0.1 bits per weight — is best read as a floor rather than a free lunch. At that setting, 23.74 perplexity is far above the FP16 model’s 6.10, so the deep-compression regime trades real quality for extreme size. The more consequential result is at 1.0 bpp, where LittleBit-2 matches dedicated 1-bit baselines while retaining LittleBit’s MatMul-free inference and much smaller KV cache — a combination that matters for on-device deployment.

It is also worth placing this in context. LittleBit targets the sub-1-bit regime through learned quantization-aware training, whereas other recent compressed builds, such as PrismML’s Ternary Bonsai 2, push a single quantized model to a fixed ~1.76 bits per weight and report near-full-precision retention. LittleBit’s method is a training-time recipe rather than a downloadable model, and the code — publicly released on GitHub, with LittleBit-2 enabled by an opt-in --use_itq flag — still requires the compute to run quantization-aware training and knowledge distillation. The direction is clear, though: as Samsung frames it, the missing ingredient in extreme compression turned out to be geometry, not architecture, and geometric alignment is now a concrete lever for fitting foundation models onto memory-constrained devices.

Related Coverage

Bonsai 2 27B: Qwen3.8 in 5.9 GB at 98% of Full Precision — a different point on the compression spectrum: a 1.76-bit build of a 27B model.
JetBrains Ships Junie Local: A 27B Coding Agent That Runs Offline — a shipping example of a 4-bit on-device model.

This post was drafted with AI assistance and reviewed by RITS staff.