LLMs Think in Geometry, Not Language, New Study Argues

A new study from independent researcher David Noel Ng argues that large language models do most of their reasoning outside of language entirely. In “LLM Neuroanatomy III: Do LLMs Break the Sapir-Whorf Hypothesis?” — published in late March 2026 — Ng shows that across five architecturally distinct frontier models from five different labs, the middle transformer layers collapse language identity almost to zero and organize representations by meaning instead. The result has direct implications for multilingual evaluation, interpretability, and a long-running debate in linguistics.

Advanced

A Three-Phase Anatomy

Ng analyzed five models — Qwen3.5-27B, MiniMax M2.5, GLM-4.7, Gemma-4 31B, and GPT-OSS-120B — on a curated dataset of 64 sentences covering eight topics (science, poetry, history, cooking, law, medicine, sports, economics) across eight languages. For every layer of every model, he computed centered cosine similarity across all 2,016 pairwise combinations, categorizing each pair as same-language/different-topic, different-language/same-topic, or different on both axes.

The pattern that emerged is consistent across all five architectures. Early layers (roughly 0–5) act as a decoding stage where representations cluster tightly by language family. Middle layers (roughly 10–45) act as a reasoning stage where language identity essentially disappears: the centered cosine similarity between two sentences in the same language but on different topics drops to approximately zero, while two sentences in different languages describing the same topic become the most similar pairs in the network. Late layers (roughly 45–64) re-encode language-specific structure in preparation for token output.

An Anti-Whorfian Bottleneck

The Sapir-Whorf hypothesis, in its strong form, holds that the language you speak determines the way you think. Ng argues the data falsifies that version for transformer LLMs: regardless of input language, the models route representations through a shared geometric space where meaning is the dominant organizer. He calls the middle of the network an “anti-Whorfian bottleneck.” The weak form — linguistic relativity — survives, but only at the input and output ends, where language-specific structure is still legible.

A follow-up experiment extends the finding beyond natural language. Testing 12 concepts expressed in English, Python, and LaTeX (36 total samples) reproduces the same three-phase curve. That suggests the universal representation may be modality-agnostic, not just language-agnostic — English, code, and math all converge on the same middle-layer geometry.

Why This Matters for Model Surgery

The study is a direct theoretical extension of Ng’s earlier RYS method, which topped the HuggingFace Open LLM Leaderboard in 2024 by duplicating seven middle layers of a 72B-parameter model with no retraining, yielding gains of +17.72% on MuSR and +8.16% on MATH Level 5. The new paper explains why that worked: layers that operate in the format-agnostic reasoning space have similar input and output distributions, so looping back through them doesn’t produce catastrophic distribution mismatch. Duplicate a decoding or encoding layer instead, and the representational geometry breaks.

For practitioners, the implications cut in several directions. Multilingual benchmarks that probe the final layers may be under-measuring how much reasoning is actually language-independent. Interpretability work should be concentrated on the middle of the stack, where the shared semantic geometry lives. And architectural interventions — layer pruning, duplication, skip connections — should respect the three-phase anatomy rather than treating the network as uniform.

Related Coverage

• Anthropic Discovers Functional Emotions Inside Claude — related interpretability research on internal activation structure in frontier models.
• Understanding Neural Howlround: A Self-Reinforcing Bias in Large Language Models — earlier RITS coverage of novel LLM failure modes tied to internal representations.

This post was drafted with AI assistance and reviewed by RITS staff.

Sources

• LLM Neuroanatomy III: Do LLMs Break the Sapir-Whorf Hypothesis? — David Noel Ng
• LLM Neuroanatomy: How I Topped the LLM Leaderboard Without Changing a Single Weight — David Noel Ng
• LLM Neuroanatomy II: Modern LLM Hacking and hints of a Universal Language? — David Noel Ng
• dnhkng/RYS GitHub repository
• One Layer, +12%: What 667 Configs Reveal About Small LLM Anatomy