Tag: transformer
-
LLMs Think in Geometry, Not Language, New Study Argues
A new study from independent researcher David Noel Ng argues that large language models do most of their reasoning outside of language entirely. In “LLM Neuroanatomy III: Do LLMs Break the Sapir-Whorf Hypothesis?” — published in late March 2026 — Ng shows that across five architecturally distinct frontier models from five different labs, the middle […]
-
FlashAttention-4: Algorithm and Kernel Co-Design for Blackwell GPUs
Key points On March 5, 2026, Tri Dao and collaborators from Princeton University, Together AI, Meta, NVIDIA, and Colfax Research released FlashAttention-4, a ground-up redesign of the attention kernel optimized for NVIDIA’s Blackwell GPUs. The kernel reaches up to 1,605 TFLOPs/s on the B200 at 71 percent hardware utilization, running 1.3x faster than cuDNN 9.13 […]
-
Reverse Engineering Apple’s Neural Engine to Train Transformers on M4
Key points A developer reverse-engineered Apple’s Neural Engine (ANE) on M4 silicon and used it to train a transformer model, which Apple has never officially supported. The open-source project bypasses CoreML and talks directly to the hardware through private APIs, reaching 9.3ms per training step at 1.78 TFLOPS sustained on an M4 Mac Mini running […]
-
Understanding Apple’s Parallel-Track MoE Architecture
Apple recently introduced a novel Parallel-Track Mixture‑of‑Experts (PT‑MoE) architecture for their server-side language models, aiming to dramatically improve scalability and efficiency while preserving model quality (machinelearning.apple.com). Key points Apple recently introduced a Parallel-Track Mixture-of-Experts (PT-MoE) architecture for its server-side language models, aiming to dramatically improve scalability and efficiency while preserving model quality. Rather than a […]