Tag: inference
-
Qwen Releases FlashQLA: 2-3x Faster Linear Attention Kernels on Hopper
Qwen has open-sourced FlashQLA, a high-performance linear attention kernel library built on TileLang that targets the GDN (Gated Delta Network) Chunked Prefill workload. Released in late April 2026 under the MIT License, FlashQLA reports a 2–3× forward-pass speedup and roughly 2× backward-pass speedup over the FLA Triton kernels, and roughly 2× over FlashInfer 0.6.9 on […]
-
Luce DFlash Brings 2x Speculative Decoding to Qwen3.6-27B on a Single RTX 3090
Luce DFlash now runs Qwen3.6-27B on a single RTX 3090, delivering up to 2× throughput over autoregressive decoding. The Lucebox Hub project, which hand-tunes LLM inference for specific consumer GPUs, ported its DFlash speculative decoding stack to Qwen3.6-27B in late April 2026. Because Qwen3.6-27B reuses the Qwen3.5 architecture string with identical layer and head dimensions, […]
-
DFlash: Block Diffusion Delivers 6x Faster LLM Inference
DFlash is a new speculative decoding framework that uses block diffusion models to generate draft tokens in parallel rather than sequentially, achieving over 6× lossless acceleration on large language models — up to 2.5× faster than the previous state-of-the-art method EAGLE-3. The paper was published in February 2026 by Jian Chen, Yesheng Liang, and Zhijian […]
-
TinyGPU Brings NVIDIA and AMD eGPUs to Apple Silicon Macs
On March 31, 2026, George Hotz’s Tiny Corp announced that Apple has officially approved the TinyGPU driver extension — making it possible for Apple Silicon Mac users to run external NVIDIA and AMD GPUs over Thunderbolt/USB4 without disabling System Integrity Protection. After more than a year of engineering custom userspace GPU drivers, the approval marks […]
-
ZML: A Zig-Based Inference Engine Bringing LLMs to AMD GPUs
ZML, a Paris-based open-source project, is gaining traction as a production inference stack written almost entirely in Zig — bypassing the Python and PyTorch dependency chains that dominate AI infrastructure. With its v2 release on March 24, 2026, ZML now compiles large language models directly onto NVIDIA, AMD, Google TPU, and AWS Trainium hardware from […]
-
NVIDIA Releases gpt-oss-puzzle-88B: Up to 2.82× Faster Reasoning on a Single H100
On March 26, 2026, NVIDIA released gpt-oss-puzzle-88B — a deployment-optimized version of OpenAI’s gpt-oss-120B reasoning model that delivers up to 2.82× faster inference while matching or exceeding the original model’s accuracy. Created using NVIDIA’s Puzzle neural architecture search framework, the 88-billion-parameter model fits on a single H100 GPU and represents a significant step forward in […]
-
NVIDIA Nemotron 3 Super: 120B Hybrid Model Activates Only 12B Parameters for Agentic AI
On March 11, 2026, NVIDIA released Nemotron 3 Super — a 120-billion-parameter open-weight model that activates only 12 billion parameters per token, delivering frontier-class reasoning at a fraction of the compute cost. Designed for multi-agent AI systems, the model combines three distinct architectures into a single hybrid backbone and ships with a native 1-million-token context […]
-
FlashAttention-4: Algorithm and Kernel Co-Design for Blackwell GPUs
Key points On March 5, 2026, Tri Dao and collaborators from Princeton University, Together AI, Meta, NVIDIA, and Colfax Research released FlashAttention-4, a ground-up redesign of the attention kernel optimized for NVIDIA’s Blackwell GPUs. The kernel reaches up to 1,605 TFLOPs/s on the B200 at 71 percent hardware utilization, running 1.3x faster than cuDNN 9.13 […]
-
Inception Launches Mercury 2: Diffusion-Powered Reasoning at 1,000 Tokens per Second
Key points On February 24, 2026, Inception launched Mercury 2, which the company calls the first reasoning-capable diffusion large language model and the fastest reasoning LLM available today. Inception reports that Mercury 2 reaches 1,009 tokens per second on NVIDIA Blackwell GPUs with 1.7 seconds of end-to-end latency, over five times faster than leading speed-optimized […]
-
Taalas HC1: Hardwiring Llama 3.1 Into Silicon for 17,000 Tokens/Second
Key points On February 21, 2026, Toronto-based startup Taalas unveiled the HC1, a custom ASIC that hard-codes Meta’s Llama 3.1 8B language model directly into silicon. Taalas says the HC1 delivers up to 17,000 tokens per second per user, roughly ten times faster than today’s best GPU-based solutions, at a fraction of the cost and […]