SGLang and Miles Add Day-0 Support for MiniMax M3.1 Flash

SGLang and Miles provide day-0 inference and reinforcement learning support for MiniMax M3.1 Flash. SGLang serves M3.1 Flash with MSA-V1 sparse attention kernels, ultra-low-latency MegaKernels, DSpark and DFlash2 speculative decoding, and HiCache. Miles trains M3.1 Flash with reinforcement learning, using SGLang for rollouts.

For agentic workloads, SGLang optimizes M3.1 Flash for both single-request latency and high-throughput serving of many concurrent sessions. With ultra-low latency MegaKernels, SGLang decodes a single request at about 1K tokens/s on 4× B300 at an acceptance length of 5. On the AgentX coding-agent trace, one TP4 replica sustains about 100K tokens/s per GPU at 96 concurrent sessions and an acceptance length of 3.07.

Model Overview

MiniMax M3.1 Flash is a natively multimodal MoE model. It reads text, images, and video over a million-token context. Weights are on Hugging Face, and full details are in the model card.

What's New in MiniMax M3.1 Flash

MiniMax M3.1 Flash keeps the MiniMax M3 backbone and vision encoder, with the following changes:

Get Started

Follow the MiniMax M3.1 Flash SGLang Cookbook for deployment instructions. Its command generator covers each supported GPU and serving strategy. On B300, we recommend this configuration for agentic workloads, with HiCache and DSpark / DFlash2 speculative decoding:

FMHA_SM100_ALLOW_JIT=1 \
sglang serve \
  --model-path MiniMaxAI/MiniMax-M3.1-Flash \
  --trust-remote-code \
  --reasoning-parser minimax-m3 \
  --tool-call-parser minimax-m3 \
  --tp 4 \
  --moe-runner-backend flashinfer_trtllm_routed \
  --speculative-algorithm DSPARK \
  --speculative-use-block-verification \
  --enable-hierarchical-cache \
  --hicache-ratio 2 \
  --cuda-graph-backend-prefill breakable \
  --chunked-prefill-size 32768 \
  --max-running-requests 128 \
  --cuda-graph-max-bs-decode 128 \
  --mem-fraction-static 0.8 \
  --tokenizer-worker-num 8

The command runs in the lmsysorg/sglang:dev-minimax-m31-flash image, which includes the CUDA 13.4 toolkit that the MSA kernels need. The first launch compiles these kernels and caches them for later launches.

SGLang Day-0 Support

Feature Support

The table below lists the serving capabilities integrated for M3.1 Flash. Availability of individual execution paths depends on the hardware and deployment configuration.

Hardware Support

Day-0 support covers NVIDIA Blackwell, NVIDIA Hopper, and AMD Instinct GPUs, each with its own kernel path.

Beyond the attention and MoE kernels, SGLang fuses all-reduce, normalization, activation, quantization, and cache writes on Blackwell and Hopper, and packs the token gathers for DP attention. On AMD, SGLang fuses QK-norm, RoPE and the KV-cache write, the FP8 quantization after RMSNorm and SwiGLU, and the router with its top-k and expert sort. The MXFP8 attention, dense and shared-expert projections run as block-FP8 GEMMs on AITER with tables tuned for M3.1. All-reduce uses AITER custom all-reduce for small messages and INT8/INT4 quick all-reduce for large ones.

On B300, TRT-LLM MoE at TP4 decodes faster than DeepGEMM MegaMoE at TP4/EP4 at every batch size we measured, by 59% at batch size 1 and 20% at batch size 32, so the recommended deployment uses it. DeepGEMM MegaMoE keeps higher prefill throughput. On H200 at 128K context, the Hopper optimizations raise TP8 decode throughput by 1.38× at batch size 1 and 1.49× at batch size 22. On MI355X without speculative decoding, individual optimizations improve decode by up to 61% at batch size 1 and prefill by up to 25%.

Launch recipes for each platform are in the SGLang Cookbook. For questions, join the SGLang Slack.

DFlash2 Speculative Decoding

In addition to the DSpark draft shipped with the checkpoint, SGLang supports DFlash2 for M3.1 Flash with a draft model trained by RadixArk. The draft conditions on hidden states from M3.1 Flash, projecting them into its own KV cache, and reuses the target model's token embeddings and LM head. It proposes several future tokens in a single forward pass. M3.1 Flash then verifies those proposals together, so an accepted prefix can advance generation by multiple tokens per target-model call.

DFlash2 combines parallel drafting with dynamic convolutions across positions in the draft block and a lightweight candidate selector that models dependencies between proposed tokens. SGLang captures drafting and candidate selection in the same CUDA graph and uses its overlap scheduler to reduce CPU overhead between GPU operations. The DFlash2 draft uses the DFLASH speculative-decoding path.

Block verification. For sampled DSpark and DFlash2 proposals, SGLang supports block verification through --speculative-use-block-verification. It uses the joint probabilities of draft prefixes under the draft and target models to decide how much of the block to accept. After accepting a prefix, the verifier draws a correction token, or a target-model bonus token if all proposals were accepted. This can increase the accepted prefix length while preserving the target sampling distribution. The benefit applies to sampled proposals. The flag does not change greedy verification.

AgentX Performance

AgentX replays 393 recorded coding-agent sessions (semianalysis_cc_traces_weka_062126). We served MiniMax M3.1 Flash with SGLang as one TP4 replica on 4× B300 with DSpark at 1 to 96 concurrent sessions, with the DSpark acceptance length fixed at 3.07 tokens per step and HiCache enabled from 48 sessions. Throughput counts input and output tokens per GPU, including cached prompt tokens. Per-user speed is the inverse of the P90 inter-token latency.

Throughput grows from 5.4K tokens/s per GPU at one session to 101.5K at 96 sessions, while per-user speed goes from 353 to 44 tokens/s. HiCache capacity matters at high concurrency. At 96 sessions, raising the host pool from 2× to 4× the GPU KV cache lifted throughput from 76.2K to 101.5K tokens/s per GPU and per-user speed from 26 to 44 tokens/s, because fewer evicted prefixes had to be prefilled again.

We ran the same AgentX setup on 4× MI350X at 1 to 64 concurrent sessions, with HiCache at a 2× host pool from 48 sessions. Throughput grows from about 5K tokens/s per GPU at one session to about 81K at 64 sessions, while per-user speed goes from about 318 to 61 tokens/s.

Ultra-Low Latency Inference

At small batch sizes, decode latency depends on both individual operators and the transitions between them. Separate kernel scheduling, intermediate writes and rereads, and waits for an entire preceding stage all add latency.

MegaKernel brings multiple stages into one GPU kernel, coordinating resources across operators. Different CTAs and clusters can execute different stages concurrently, with downstream tasks advancing as soon as their inputs are ready. Weight loading and initialization overlap upstream computation, while CTAs and clusters continue into subsequent tasks, reusing on-chip data and resources. Organizing execution around task dependencies shortens the critical path.

Following this approach, we implement each M3.1 Flash decode Transformer block at batch size 1 as two MegaKernels: MegaAttention and MegaMoE. Each covers its respective module, including communication and post-processing, to minimize per-layer latency.

SGLang runs these MegaKernels through the plugin framework of SGLang Low-Latency Mode. With --monokernel-backend msa, SGLang dispatches single-request verification steps to MegaAttention and MegaMoE and keeps the standard path for larger batches.

Reaching ~1000 TPS with MegaKernels

MegaAttention

MegaAttention integrates input normalization and quantization, QKV and indexer projections, RoPE and KV-cache updates, indexer scoring and TopK selection, sparse attention, output projection, and tensor-parallel reduction with residual addition and RMSNorm.

64 CTAs perform projection, while other CTAs begin indexer scoring once Qi is ready. CTAs that finish projection then join the indexing work. After indexing, 8 clusters execute TopK and FMHA, while another 6 prepare and execute OProj, with 8 CTAs per cluster. This partitioning reuses resources across stages and overlaps downstream preparation with upstream computation.

Once Qi is ready for indexer query tokens, the kernel publishes a shared readiness signal. Using the existing historical indexer K cache, the indexer can score historical pages concurrently with the remaining projection work, without waiting for the entire projection stage to finish. TopK results are then passed through shared memory within each FMHA cluster, reducing intermediate data movement.

OProj CTAs initialize their pipelines and preload weights during upstream computation, then wait for the corresponding activation-group readiness signals before loading FMHA output. Preparing ahead of input readiness shortens the subsequent execution path.

The communication tail reuses the per-token clusters that executed FMHA. OProj writes directly into the TP4 communication payload. Once all local OProj outputs and all 4 ranks are ready, 2 CTAs per token cluster perform NVLS reduction and residual addition, accumulating the sum of squares for RMSNorm. They pass residual slices and statistics through distributed shared memory to all 8 CTAs, which cooperate to complete RMSNorm. The residual result is written to global memory as a module output, while RMSNorm consumes the cluster-local handoff, avoiding a reload.

MegaMoE

MegaMoE integrates routing, input quantization, routed and shared expert computation, weighted combination, tensor-parallel reduction, and residual addition in 1 cooperative kernel launch per rank.

Routing overlaps input quantization. The shared expert starts from the packed input without waiting for route selection, allowing it to overlap routed expert computation.

The illustrated implementation assigns 128 CTAs to routed work and 16 to the shared expert. Routed CTAs perform input packing and router partial jobs at entry, then continue into expert computation. Tasks are assigned by token, selected-route slot, and slab. Actual expert IDs come from the router. Shared CTAs partition channels across all tokens.

4 CTAs cooperate on each selected route of a token. Each computes an Up/Gate projection (FC1) slice, applies activation and quantization, then produces an output slice through the Down projection (FC2). Every FC2 CTA consumes activation slices from all 4 CTAs. Producers publish epoch-tagged records, allowing consumers to identify ready data for the same token and route without waiting for unrelated routes.

FC2 weight loading overlaps the FC1 tail. Once the final FC1 MMA reads of a buffer region complete, the weight producer reuses that shared-memory region for FC2 weights, starting downstream transfers while remaining FC1 work continues. FC2 MMA still waits for the complete activation from all 4 CTAs and the corresponding weight values and scales.

For each token and output slab, the CTA assigned to selected-route slot 0 continues into combine and communication, giving 32 reused output owners. Each retains its own Down result in shared memory and gathers the other 3 routes’ matching outputs through epoch-tagged records. During Down computation, an idle consumer warp waits for the matching shared expert slice, overlapping readiness checking with computation. The owner still synchronizes before consuming that slice.

Each owner combines its rank’s shared and weighted routed contributions, then pushes BF16 records carrying epoch tags to peer ranks, delivering data and readiness together. Receivers alternate between 2 buffers and accept only current-epoch records. Once a slice’s peer records are ready, the owner sums them with its local contribution in fixed rank order and adds the residual to produce the final output.

Miles Day-0 Support

Miles supports MiniMax M3.1 Flash reinforcement learning on day 0, training on Megatron and generating rollouts with SGLang.

Sparse attention. The Megatron-Core plugin implements MSA on every layer: as in SGLang, the indexer selects 16 128-token blocks per KV head, and a Triton kernel attends over them. The indexer is frozen and its top-k is recomputed in training. It matches SGLang's selection except at near-ties.

Trainer–rollout consistency. Rollout Routing Replay fixes the MoE expert path, and q/k normalization and RoPE run in FP32 with a single BF16 cast, as in SGLang's fused kernel.

Colocated training and rollout. Training and rollout share 32 GPUs. Optimizer moments stream to node-local NVMe, and updated BF16 weights are written in place into SGLang's fused q/k/v + index GEMM, keeping captured CUDA graphs valid.

The Validated Run

The figure below shows steps 0–65 of a GRPO run on DAPO-Math-17K with a 4K-token response cap on 32 GB300 GPUs (TP4, PP4, EP8, 256 samples per step). The five-step mean reward rises from 0.64 to 0.77, and per-token KL between trainer and rollout stays between 0.00065 and 0.00101 without sustained growth.

Get Started with Miles

The Miles recipe downloads the release, converts it to BF16, and launches training:

# 4-layer model on one node of 4 GB300 GPUs, to check the setup
python scripts/run_minimax_m3_1_flash.py --model-name MiniMax-M3.1-Flash-4layer-bf16 --num-nodes 1

# Full model on 8 nodes of 4 GB300 GPUs, with the nodes joined to one ray cluster
MILES_SCRIPT_EXTERNAL_RAY=1 python scripts/run_minimax_m3_1_flash.py \
    --model-name MiniMax-M3.1-Flash-bf16 --num-nodes 8

Acknowledgments

Thanks to the MiniMax, NVIDIA, and AMD teams for collaborating with the SGLang & Miles team on day-0 support for MiniMax M3.1 Flash.

MiniMax: Zhi Zhang, Hai Cheng, Tong Chen, Liyuan Zhang, Yudong Pan, Jingxin Li