High-Performance Serving Framework for LLMs and Multimodal Models

New models

Highlights

• PD instances can switch between prefill and decode on the fly, no restart needed.

  #28403

• The prefix cache now runs on a Rust core by default.

  #39627

• DeepSeek-V4.1 gets 22% faster first token on long prompts.

  #40352

• Kimi K3 gets 20.6% higher prefill throughput in PD serving.

  #40045

• GLM-5.3-Flash now runs on AMD MI355X, with FP8 / MXFP4 MoE and MTP speculative decoding.

  #38546#39341#39778

• More accurate results under PP, DP attention, and CP, with layer communication now handled by SGLang (guide).

  guide#41557

New models

Highlights

• Sampling masks for RL rollouts

  #36630#36631#38279

  With return_sampling_mask, each decode step returns the exact token support the sampler drew from and the log-probability of the sampled token under it, so a trainer can replay the rollout without reconstructing top-k or top-p. Masks now run under overlap…

• Unified radix tree

  #34565#37381

  Branching-point caching for the SWA component keeps the sliding-window state at the point where requests fork from a shared prefix, so branches reuse it instead of recomputing. On DeepSeek-V4-Flash with a shared system prompt, token hit rate rises from 43.8%…

• DSpark under PD with decode context parallelism

  #37709

  A DCP1 prefill can now transfer its DSpark draft KV to a DCP-N decode, so hybrid models such as Kimi-Linear run DSpark in disaggregated, context-parallel serving. Verified on 8x B300 over NIXL and Mooncake up to 256K input.

• Responses API storage is opt-in

  #39122

  /v1/responses no longer retains results in memory unless the server starts with --enable-response-store. Without it, retrieval, previous_response_id chaining, and background requests return 400; PD deployments cannot enable it.

• SGLang Simulator

  #33824

  A CPU-only simulator runs the real scheduler, radix cache, and hierarchical cache with a latency predictor in place of the model forward. Against measured serving traces it predicts TTFT within about 6% on most traces (up to 10% on the longest 32K to 128K…

• Prefill context parallelism v1 removed

  #36228

  The strategy-based implementation is now the only prefill CP path; the v1 runtime and its CLI options are gone. Prefill CP on HIP, NPU, and MUSA is rejected until those platforms are ported.

New models

Highlights

• Beam search

  #31626

  SGLang can now do beam search. Pass beam_width in your request and you get back the n best sequences instead of a single sample. It works out of the box next to regular requests, though it does not yet mix with speculative decoding, disaggregation, DP…

• DeepEP v2

  #35634#34923

  DeepEP's new ElasticBuffer engine is available as --moe-a2a-backend deepep_v2 for DeepSeek-V3/V4 and Qwen3-MoE in FP8. Its buffers have a fixed size, so decode can run under CUDA graphs even across nodes. Performance is on par with the classic backend.

• LayerNorm sequence parallelism

  #30915

  With --enable-layernorm-sp, each tensor-parallel rank normalizes only its own share of the prefill tokens instead of all of them. That takes 3.5% off Qwen3-8B prefill on H100 and 5.6% on B200, and the saving grows with the TP degree. Dense Qwen3 models only…

• W4A8 MoE on Hopper

  #34967

  If you serve MXFP4 experts on Hopper, you can now quantize the activations to FP8 as well with --flashinfer-mxfp4-moe-precision fp8. DeepSeek-V4-Flash gains about 12% output throughput with no change in GSM8K accuracy. Needs FlashInfer 0.6.18.

• DCP on the default Blackwell MLA backend

  #33926

  Decode context parallelism now runs on trtllm_mla, not just CuTe DSL and Tokenspeed. It pays off at long context: at 128K input, plain TP stops scaling around 680 tokens per second on eight B200s, while DCP keeps going as concurrency grows.

• Faster speculative kernels

  #35175#33722

  DSA prefill top-k moves to the v2 kernel, 1.3 to 1.8 times faster on B200. KDA models get an opt-in fused accept path, SGLANG_OPT_KDA_FUSED_ACCEPT_STATE=1, that cuts MTP verify-and-commit time by 45% to 63% on Kimi-Linear shapes with bit-identical output.

New models

Highlights

• Overlapped checkpoint staging at startup

  #32017

  Checkpoint pages now stage from storage while CUDA graphs capture. Qwen3-32B on H100 starts 8.6-11.7% faster than serial with prefetch, and 2.38x faster (35.6s vs 84.8s) than the plain default. Opt in with --startup-weight-load-mode overlap.

• TP LMHead with All-to-All

  #32313

  The TP LMHead's allgather + scatter becomes a single all-to-all for pure-DP dp-attention. On DeepSeek-V4-Pro B200 decode, LMHead time drops 320us to 169us and TPOT improves 36.97ms to 35.67ms.

• FlashInfer MNNVL for pure allreduce

  #30700

  Non-fused allreduce sites now reuse the FlashInfer MNNVL workspace instead of falling back to NCCL. DeepSeek-V4-Flash TP4 decode on Blackwell gains up to +6.9% at small batches. Auto-enabled for DeepSeek-V3/V3.2/V4; elsewhere…

• NVFP4 checkpoints run on AMD

  #29328

  --quantization quark_mxfp4 dequantizes ModelOpt and Quark NVFP4 weights and requantizes to MXFP4 at load, never holding a full-precision copy. 97.5-100.2% GSM8K recovery vs the NVFP4 reference across MiniMax-M2.7, GLM-5.1, Kimi-K2.6, Qwen3.5-397B, and…

• Kimi K3 tuned for MI355X

  #33981#34261#34837

  A grouped-head MLA verify kernel replaces the MHA-shaped split-KV path that re-read the shared latent once per head, for 1.37-1.77x throughput and 1.45-2.42x ITL at concurrency 2-32; the AITER MLA prefill kernel now accepts K3's 12-head shape (TTFT up to…

• One compiled-kernel cache directory

  #32434

  Triton, FlashInfer, Inductor, DeepGEMM, and CUDA driver caches all move under SGLANG_CACHE_DIR. The first launch after upgrading recompiles once; see Breaking Changes.

New models

Highlights

• Kimi K3 day-0 support

  BlogCookbook#32541

  A 2.8T-parameter multimodal LatentMoE (896 experts, top-16, routed in a 3584-dim latent space) with a 1M-token context, 69 KDA linear-attention layers interleaved with 24 MLA layers, and a MoonViT3d vision tower, shipping as a native MXFP4 checkpoint. SGLang…

• MiniMax-H3 day-0 support

  Cookbook#33275

  MiniMax's video generation model that produces a video and a synchronized stereo audio track in one request, served natively on SGLang-Diffusion across all three public task profiles: text-to-video-and-audio (t2va), first/last-frame conditioning (fl2va)…

• Initial support for the Rust frontend

  #29799

  Migrates the front half of the server, everything from network ingress up to the point a tokenized request is handed to the GPU scheduler, from Python to a multi-threaded Rust implementation.

• DCP communication backends and q-replicate (Helix)

  #21637

  The DeepSeek-MLA decode context-parallel path gains pluggable comm backends. a2a exchanges packed attention output plus fp32 LSE in a single NCCL collective per layer, with fp8 KV carried as uint8 byte transport; fi_a2a delegates the cross-rank exchange…

• DWDP for MoE prefill

  #29778

  A new prefill parallelism strategy that prefetches peer expert weights over NVLink P2P and computes all experts locally, removing EP all-to-all token dispatch. On 4x B200 with gpt-oss-120b, prefill-only, DWDP4 reaches 1.92x over DEP4 at MNT 32K / ISL 32K, and…

• Session-reference-aware Unified Radix Cache

  #29173

  For agentic and RL-rollout workloads, requests can carry a stable session_id so eviction knows which prefixes an active session still references, instead of evicting purely by cache policy. Release the references with /close_session. Opt in with…

New models

Highlights

• DSpark: confidence-driven speculative decoding

  Blog#30261#31434

  A new speculative algorithm. It drafts semi-autoregressively in blocks, then sizes each verify window from the draft's own confidence instead of a fixed draft length. Reaches 383.7 tok/s at accept length ~5 on DeepSeek-V4-Pro, TP8 on B300 (bs=1). Enable with…

• Inkling support

  BlogCookbook#31681

  A 975B-parameter multimodal MoE with a 1M-token context. It mixes sliding-window, full and Mamba2 linear attention, and adds an NVFP4 MoE, optional vision/audio towers and native MTP. On Blackwell it reaches up to 71.7k tok/s input and 171.0 tok/s per-user…

• UnifiedRadixTree is now the default

  #30468#30636#30626

  UnifiedRadixTree is now the default for SWA, Mamba and DSA models. Replay SSM and Mamba int8 checkpoints are synced onto it, and a cache hit now resets only the state it used.

• GLM-5.2 DSA cache layer split under prefill CP

  #29421

  KV and indexer cache layers are sharded across CP ranks. Each rank owns a disjoint layer range instead of all layers. That cuts per-rank KV memory by ~74% (0.77 to 0.20 GB/rank) at 8192 tokens on GLM-5.2-FP8, 78 layers, cp_size=4. Enable with…

• ReplaySSM Ring Spec-Verify (GDN)

  #28695

  Drops the per-draft SSM snapshot. Speculative scratch goes from 11.5 GB to 1.8 GB per GPU (6.4x smaller) on Qwen3.5-35B-A3B at TP1, at accuracy and throughput parity. Opt in with --enable-gdn-replayssm-spec (default off; GDN with a linear draft chain only…

• Linear attention on Blackwell (SM100)

  #30113#30169

  The first correct KDA MTP path. Its recurrent_kda decode kernel runs at 29.6 us vs 36.8 us for Triton (ncu, B=64). The full decode path reaches parity by B=128 and 1.35x at B=256, and is slower below that. Separately, GDN/KDA CuteDSL prefill fuses state I/O…

New models

Highlights

• GLM-5.2 NVFP4, tuned for production

  Cookbook#29413#29343

  We took time this cycle to tune GLM-5.2 NVFP4 on Blackwell for optimized production serving. It now runs at 500+ tok/s/user on 8x B300, 450 on 4x GB300 (bs=1). Run GLM-5.2 with our.

• Native web search (Exa)

  #29342

  Built-in web_search support backed by Exa.

• Breakable CUDA Graph on by default

  #29458#27988

  Breakable CUDA Graph is now the default capture path, reducing per-step kernel-launch overhead; full CUDA Graph support for the prefill phase lands as experimental.

• Linear-attention kernels (KDA / GDN)

  #29472#28451

  New FlashKDA prefill backend for safe-gate KDA linear attention, plus ReplaySSM buffered output-only decode for linear attention.

• FlashInfer A2A for routed MoE

  #22394

  Adds FlashInfer all-to-all with the flashinfer_trtllm_routed MoE runner.

• DeepSeek-V4 Optimization

  #28612#29775#29619

  Optimizes C128 state-pool allocation using the request state pool; FlashMLA sparse prefill is now enabled by default for DeepSeek-V4, reaching >10% throghput gain on long context; Non paged indexer support for long context prefill, with >5% e2e throughput gain

New models

Highlights

• DeepSeek-V4 on GB300 since Day 0

  Blog

  5x higher throughput at the same interactivity, serving DeepSeek-V4 on NVIDIA GB300 with SGLang.

• Waterfill & LPLB MoE load balancing

  Blog

  Two dispatch-time load-balancing methods for DeepEP expert parallelism: Waterfill for shared-expert dispatch and LPLB for redundant expert replicas, improving throughput for DeepSeek-V3/R1 and DeepSeek-V4.

• KDA CuteDSL prefill kernel on Blackwell (SM100)

  Kimi-Linear cookbook#27488

  New CuteDSL prefill kernel for Kimi-Linear (KDA), 1.08-1.52x faster than the Triton path via a reusable scratch workspace, plus a cuda-graph padding fix; see the Kimi-Linear cookbook.

• Linear-attention prefix-cache memory savings

  #28185#28302

  An int8 checkpoint pool stores recurrent states compactly in the Mamba radix cache, substantially increasing prefix-cache capacity for KDA / GDN models; the speculative conv-window intermediate cache is deduplicated with a sliding-window layout, halving its…

• LPLB: linear-programming load balancer for MoE expert parallelism

  #24515

  Balances token routing across redundant expert replicas by solving a per-layer LP; opt-in via --ep-dispatch-algorithm=lp, default behavior unchanged.

• MSCCL++ integration & MNNVL allreduce fusion

  #22734#23402

  MSCCL++ migrates to the upstream mscclpp Python package (Executor + DSL compiler) with auto-tuned collectives for TP=8 single-node and TP=16 two-node; FlashInfer fused allreduce + residual + RMSNorm re-enables an MNNVL backend behind…

New models

Highlights

• Spec V2 is now the default speculative-decoding path

  #26997#26972#27463

  Tree drafting with topk > 1 is production-ready across the triton / FA3 / MLA / aiter backends, including page_size > 1 and Mamba/hybrid-linear models. Spec V1 is deprecated, with EAGLE/MTP now running on the unified V2 worker, and topk = 1 drafting is…

• Lower per-step scheduler overhead

  #25945#25879#26380

  Unified async value passing through FutureMap plus moving prefill input transfer onto the forward stream reduced per-step launch overhead and improved stability under high concurrency.

• Piecewise & Breakable CUDA Graph coverage

  #23351#26382#25195

  Piecewise (PCG) and Breakable (BCG) CUDA Graph capture more of the model to cut per-step kernel-launch overhead, now extended to DSA models, Kimi-K2.5, and DeepSeek V4.

• Faster Qwen 3.5 on Blackwell

  #22921#23273#26200

  New FlashInfer Gated DeltaNet (GDN) kernels and a CuTeDSL GDN prefill kernel speed up Qwen 3.5 on Blackwell GPUs.

• HiCache for hybrid models by default

  #27759

  HybridModel (SWA/Mamba) launches HiCache through UnifiedTree by default, bringing hierarchical KV-cache offload to sliding-window and Mamba hybrids out of the box.

• Heterogeneous CPU + GPU EPD disaggregation (with Intel)

  Blog

  Offload VLM vision encoding onto Intel Xeon CPUs alongside GPUs, with up to ~1.3x P99 TTFT and request-throughput gains under load.

New models

Highlights

• DeepSeek V4 support

  LMSYS blogDeepSeek-V4 cookbook#23882

  Full inference path for DeepSeek-V4, including: Day-0 Features; Parallelism: Tensor Parallelism/Expert Parallelism/Context Parallelism/Data Parallel Attention; Hardware: Nvidia B300/B200/H200/H100/GB200/GB300, AMD MI35X; Prefill-Decode Disaggregation…

• TokenSpeed MLA attention backend (Blackwell, FP8 KV cache)

  #24925

  New MLA prefill/decode kernels integrated as an attention backend on SM100, with FP8 KV cache support for low-latency MLA serving

• DSv3.2 / GLM-5 FP4 low-latency perf

  #23965#23856#23590

  PDL enabled across DSv3.2 / GLM-5 kernels, torch.mm for the DeepSeek V3.2 indexer GEMM, and a reland of the Cute-DSL FP4 dense GEMM — materially trimming low-latency overheads on FP4 paths

• HiCache + UnifiedRadixTree

  #23316#23391#24691

  HiCache framework support for UnifiedRadixTree (with SWA), HiCache for DeepSeek V4, SSD offload through Mooncake store, and stability fixes across cascade eviction, tombstone replay, and partial-match paths

• Speculative Decoding V2 maturation

  #23336#24663#24664

  Adaptive Spec V2, EAGLE-3 SWA + newer drafters, Kimi K2.5 EAGLE-3 MLA, Gemma 3/4 + EAGLE-3, and an extensive naming / shape-handling refactor across draft-extend paths

• CUDA 13 DeepEP migration

  #25113

  Gateway DeepEP source swapped from a community fork to deepseek-ai/DeepEP@hybrid-ep so DeepEP builds and runs cleanly on the CUDA 13 default; FlashInfer pinned at 0.6.11.post1 alongside a gpt-oss triton-kernel fix

New models

Highlights

• CUDA 13 + Torch 2.11

  #21247#24162#24183

  Default CUDA version moves to 13.0 across SGLang, sgl-kernel, and Docker images, and PyTorch is upgraded from 2.9 to 2.11 — modernizing the build matrix and unlocking newer kernels (tracking issue)

• Speculative Decoding V2 by default

  #21062

  Spec V2 (with overlap scheduling to hide CPU overhead) is now the default, materially reducing per-step CPU cost for EAGLE/MTP/DFLASH paths

• Decode Radix Cache for PD Disaggregation

  #19746

  Decode-side prefix caching now works under prefill/decode disaggregation, recovering radix-cache hit rates and TTFT savings for long shared prefixes in disaggregated deployments

• DFLASH Speculative Decoding

  #22077#22358#22342

  New high-throughput spec-decode kernel from the kernel community, expanded across model backends and AMD ROCm

• FA3 Kernels from the Kernel Community

  #20796

  Drop-in FA3 kernels contributed by the community, integrated alongside FA4 to give users a high-performance option that's easy to maintain

• LoRA support for DeepSeek-V3 and Kimi-K2

  #22323#22381

  LoRA now works on the largest MLA-based MoE models, including DeepSeek-V3 MLA LoRA and Kimi K2 — enabling adapter-based fine-tuning of frontier-scale models