MoE vs dense LLM inference: vLLM benchmark on two RTX 5090s

What was compared, and on what

Two checkpoints, one engine, one machine. The dense side is the checkpoint from our Qwen3.8-27B production field report, unchanged. The MoE side is Nex-N2.5-mini in a plain NVFP4 conversion. Both ran on a rented 2x RTX 5090 host so the production machine kept serving; the dense model's remaining long-context cells ran on the production machine overnight at low scheduler priority, and the difference between the two hosts is measured and reported in section 8.

	Nex-N2.5-mini (MoE)	Qwen3.8-27B (dense)
Architecture	Qwen3.5-MoE family: 40 layers, 30 linear attention (Gated DeltaNet) + 10 full attention; 256 routed experts, 8 active per token plus a shared expert	64 layers, 48 linear attention + 16 full attention; one feed-forward block per layer
Parameters	35.1 B total (safetensors), about 3.3 B active per token (computed from config.json, S8); routed experts hold 32.2 B of the total	27 B, all active
Checkpoint	ProCreations/Nex-N2.5-mini-NVFP4: routed experts W4A4 NVFP4, attention, routers and shared experts BF16, 23.9 GB packed; no MTP head in the tensors (S2)	unsloth/Qwen3.8-27B-NVFP4: mixed NVFP4 and FP8, 21.8 GiB on disk (field report)
Context window	262,144 native	262,144 native
Weights per GPU as loaded	13.2 GiB (13,560 MiB, K1)	10.9 GiB (field report)
GPU KV cache at 95% memory	3,237,081 tokens, 12.35 contexts of 262,144 (K1)	1,038,029 tokens, 3.96 contexts (K2; production: 1,039,536)
Serving	vLLM 0.27.0, tensor parallel 2, FP8 KV, prefix caching, --mamba-cache-mode align, --max-num-seqs 12, --max-num-batched-tokens 6272, --watermark 0.08, priority scheduling, --scheduler-reserve-full-isl, qwen3 reasoning parser, qwen3_xml tool parser, 60 GiB CPU KV tier (S6)

The dense model's KV pool on the rental came out within 0.15% of the production machine's, and its single-stream decode within 15% of the field report's 83 tokens per second, so the rental is a faithful twin of production for everything except cold prefill, which section 8 covers. The benchmark client sent the same sampling to both (temperature 0.7, top_p 0.95); the MoE's card also recommends top_k 40 and qwen3_coder as the tool parser, and section 9 tests that recommendation against production's qwen3_xml. Model facts were read from the Hugging Face cards and config files on 2026-09-17 (S1 to S4).

How a cell was measured

A cell is one context length, one concurrency, one prefix mode, one engine. Every request in a cell carries a pseudo-random document of exactly the target length, asks for a three-sentence summary, and is forced to decode exactly 256 tokens with thinking off, so a cell measures prefill and decode and nothing else. Cold cells give every request its own document; cached cells share one document, warmed by an unmeasured request, and differ only in a 48-token tail. The client ran on the box itself, so no network is in the numbers.

Per cell the client records time to first token (queueing included, on purpose: when concurrency exceeds the 12-request cap the wait is real), inter-token latency, per-stream and aggregate tokens per second, and the engine's own counters before and after: KV usage, running and waiting requests, preemptions, prefix-cache hits. The grid was preregistered: a sweep of --max-num-seqs from 12 to 48 at 60K cold, then 60K, 120K and 200K, cold and cached, at 1, 4, 8, 12, 16, 24 and 32 concurrent requests. The sweep turned out to be flat, which is itself a result: at 60K cold with 32 streams, aggregate input ran at 7,910 tokens per second with the cap at 12 and 7,967 with the cap at 48 (rows M4 to M20); the cap-12 cell was run twice, M4 and M27, 7,910 and 7,861). The prefill budget, not the sequence cap, decides cold throughput, so the rest of the grid ran at production's 12.

One stream: the MoE is 2x faster, at any length

With one request in flight the MoE decoded at 172.7 tokens per second at 60K, 172.7 at 120K and 172.2 at 200K; the dense model did 80.9, 87.4 and 75.7. Decode speed is length-invariant on both, which is the hybrid linear-attention stack doing its job on both sides. Time to first token on a cold prompt is where the two separate most: 8.0 seconds against 26.4 at 60K, 17.6 against 62.5 at 120K.

One request, cold prompt	Nex-N2.5-mini	Qwen3.8-27B	Rows
60K: time to first token · decode	8.0 s · 172.7 tok/s	26.4 s · 80.9 tok/s	M21, M64
120K: time to first token · decode	17.6 s · 172.7 tok/s	62.5 s · 87.4 tok/s	M35, M78
200K: time to first token · decode	34.6 s · 172.2 tok/s	55.2 s · 75.7 tok/s (production machine)	M49, M98
Inter-token latency, p95, single stream	11.9 to 15.8 ms	13.9 to 27.8 ms	same rows

For an interactive agent turn, which is one stream reading a long session and writing a short tool call, that is the whole difference: on the rental, a first token in about a third of the time (8.0 against 26.4 seconds at 60K) and the answer at twice the speed; against the production machine the first-token gap shrinks to roughly half, as section 8 explains. If your workload were only that, the article could stop here. It isn't, so it doesn't.

Many streams: the advantage shrinks to 1.4x

With a warm shared prefix and 32 concurrent requests, the MoE produced 442.2 aggregate tokens per second at 60K against the dense model's 318.7 on the same host, 427.3 against 343.0 at 120K, and 365.7 against 269.6 at 200K, the last two with the dense model on the production machine at low scheduler priority (section 8). Per stream, the MoE fell from 208.3 tokens per second alone to 52.5 with 12 running, while the dense model went from 91.0 to 34.1. The 2x single-stream gap is a 1.4x aggregate gap under load.

Concurrent requests (60K, cached)	1	4	8	12	16	24	32
Nex per stream, tok/s	208.3	114.0	90.8	52.5	47.0	37.0	37.4
Qwen per stream, tok/s	91.0	65.0	48.3	34.1	35.9	30.3	29.6
Nex aggregate, tok/s	143.1	334.3	513.8	467.7	442.5	413.4	442.2
Qwen aggregate, tok/s	79.1	218.4	314.2	317.0	321.2	322.3	318.7
Nex time to first token, p95	0.5 s	1.1 s	1.8 s	2.6 s	7.4 s	9.6 s	16.1 s
Qwen time to first token, p95	0.4 s	1.0 s	1.8 s	3.0 s	9.0 s	12.1 s	20.6 s

Read the shape, and then the endpoints. The dense curve is flat from 8 streams on: once the batch fills, each additional request costs the others little, because a dense model reads the same weights per step no matter how many tokens are in the batch. The MoE curve peaks at 8 and drifts down, and its per-stream number keeps falling until the cap. Past 12 concurrent requests both time-to-first-token columns are queue time: the engine admits 12 and the rest wait, exactly as production does.

Cold prefill: the same knob caps both

On a cold prompt every token has to be computed, and both engines process cold prompts in 6,272-token chunks between decode steps. On the rental the MoE prefilled at about 7.9K tokens per second at 60K, 6.9K at 120K and 5.5K at 200K; the dense model at 2.3K and 2.1K on the same host, and 4.2K and 3.4K on the production machine. Both slow down as context grows, because the attention share of each chunk grows with the tokens already in the cache.

Cold prefill, aggregate input tok/s	60K	120K	200K
Nex-N2.5-mini, rental	7,861	6,919	5,542
Qwen3.8-27B, rental	2,273	2,092	not run
Qwen3.8-27B, production machine	not run	4,160	3,388
Rows	M27, M70	M41, M79 (4 streams), M90	M55, M100 (8 streams)

What the chunk budget does to decode under cold load is the same on both: with 32 cold 60K requests the MoE's aggregate decode sat at 33.6 tokens per second and its inter-token p95 at 592.8 ms; the dense model's at 9.7 and 2178.3 ms. Decode only runs between prefill chunks, so a machine full of cold prompts is a slow machine for everyone on it, whichever architecture is loaded. That is a scheduler property we already knew from production; the MoE just gets through the cold work three to four times sooner on the same host.

KV cache capacity: three times the room, and no preemptions

The MoE's KV pool holds 3,237,081 tokens on the two cards, 12.35 full 262,144-token contexts; the dense model's holds 1,038,029, 3.96 contexts. Only 10 of the MoE's 40 layers keep a per-token KV cache (the other 30 carry a fixed recurrent state), against 16 of 64 on the dense model, and its attention heads are smaller. Across all 107 cells the MoE was never preempted; the dense model was preempted twice per cell at 200K cold with four and eight requests, at 99% KV usage.

200K cold, 8 concurrent requests	Nex-N2.5-mini	Qwen3.8-27B (production machine)
Requests running at once	6	3
KV usage, peak	88%	99%
Preemptions	0	2
Wall time for the cell	273.1 s	472.1 s
Rows	M51	M100

Two details keep this honest. First, the KV percentage under --scheduler-reserve-full-isl reports what the scheduler has reserved for admitted requests, not tokens present, so the MoE reads 88% with six 200K requests admitted, although those six hold about 37.1% of its pool in actual tokens. Second, at 32 concurrent 200K cold requests the MoE's p95 time to first token was 18.7 minutes (M55), because everything past the sixth request queued. Admission control turned memory pressure into waiting on both engines and into thrashing on neither, which is what the watermark and reserve flags are for.

Why per-stream speed falls on a MoE

A decode step on a dense model reads every weight once, whether one token or twelve are in the batch, so adding streams is nearly free until compute saturates. A decode step on this MoE reads the router and the eight experts each token selected. One stream touches 8 of 256 experts per layer; twelve streams can touch up to 96, so the weight bytes read per step grow with the batch, and each stream's share of the step shrinks.

The numbers fit that picture. At one stream the MoE reads roughly 3.3 billion active parameters per token and the dense model 27 billion, and the MoE decodes about 2.1x faster rather than 8x, because attention, the recurrent layers, the embeddings and the kernel launch overhead don't shrink with sparsity. At twelve streams the MoE's per-stream rate had fallen to 52.5 tokens per second while the dense model held 34.1; the aggregate still favours the MoE, by 1.4x to 1.5x, because it started so far ahead. None of this is a defect. It is the trade: a MoE buys single-stream speed and cache room with total parameters, and pays some of it back under batching. Which side of that trade you want is a question about your traffic, not about the architecture.

The rental-versus-production correction

The dense model prefilled about twice as fast on the production machine as on the rental: 4.2K against 2.1K tokens per second at 120K, 3.4K at 200K where the rental wasn't run. Single-stream decode matched (75.7 to 80.4 tokens per second on the production machine, 80.9 to 87.4 on the rental) and so did the KV pool. Same image, same flags: the difference is the host.

The production machine has a persistent torch.compile cache and no other tenants; the rental host reported a load average around 160 from other jobs (H1, H2). We did not run the MoE on the production machine, so the honest statement of its cold-prefill advantage over production is the conservative one: 6.9K against 4.2K at 120K, about 1.7x, rather than the 3.3x the rental alone would suggest. Single-stream decode and KV capacity are host-independent, since the dense model's decode and pool matched across both hosts. The 120K and 200K aggregate rows compare the MoE on the rental with the dense model on the production machine and carry the same caveat in the MoE's favour; the 60K rows are same-host.

Quality as-is: where the dense model wins

Throughput is half the decision. The other half is whether the model does the job, so the same production-derived decisions were replayed against both endpoints with identical requests: 424 recorded tool decisions from Claude Agent SDK sessions in our runtime (264 clean decisions, 160 states just before a historically failed action), a frozen 35-case smoke set, and a tool-call suite at 100,000 tokens of context. On the 264 clean decisions the dense model chose the recorded tool 75.8% of the time and the MoE 68.9%. The MoE repeated the historically failed action far less often. And it stopped where the dense model kept going.

Offline replay, temperature 0, seed 42	Qwen3.8-27B	Nex-N2.5-mini
264 clean decisions: chose the recorded tool	75.8%	68.9%
264 clean decisions: argument keys exactly right	50.0%	38.6%
160 pre-failure states: repeated the historically failed tool	78.1%	51.9%
160 pre-failure states: repeated the exact failed action	20.0%	6.2%
Argument parse errors, endpoint errors	0, 0	0, 0
35-case smoke: recovery after a rejected tool (20)	12/20	1/20
35-case smoke: sanity subset (5)	3/5	0/5
35-case smoke: clean structural (10)	6/10	6/10

The recovery row is the one to read twice. The 20 cases are states where a tool was just rejected by a guard (credential access denied, a skill required first, a tool not found) and the recorded next move was a different tool. The dense model made that move in 12 of 20. The MoE made no tool call in 19 of 20 and ended its turn; a manual replay of the first case returned "Understood. I'll use the authorized tool path and won't attempt direct credential access." and nothing else (Q-probe). Enabling its adaptive thinking changed nothing (1 of 20 again), and a 5-case sanity subset of the same set went 3 of 5 against 0 of 5 the same way. Read the lower failed-action repeat rate in the table with this in mind: a model that yields the turn also repeats nothing. For a chat assistant that is polite. For an autonomous agent that is a stalled run. The parse and transport counters are zero on both, so this is the model's behaviour, not the wire.

The tool-call suite says the same thing from the other side. With thinking off, the MoE produced a 17,000 to 21,000-character Write call intact at 100,000 tokens of context, streamed and not, and followed a tool result with the correct Bash call, 8 of 8 passes across both parsers, so production's qwen3_xml parser is safe for it and the card's qwen3_coder gains nothing. With thinking at medium or high it ignored an explicit "call the Write tool now" and wrote the report inline, 8 of 8 times. Our production tier already runs tool loops with thinking off for a related reason, recorded in the field report: thinking starves tool calls on the dense model too. On the MoE it replaces them.

What we would change, and what didn't work

Two tuning experiments, one variable each, both against the production flags. Raising the prefill chunk from 6,272 to 16,384 tokens bought 5.0% cold prefill at 60K and multiplied inter-token p95 under cold load by 2.8 (1596.0 ms against 566.3 ms, rows M84 and M23), and it needed GPU memory utilization lowered to 0.90 to survive its first chunk (K3). Not worth it; the production value stays. Speculative decoding with a multi-token-prediction head did not run at all on this stack.

The MTP attempt is worth a paragraph because the failure is specific. The checkpoint we served has no MTP tensors despite its config advertising a one-layer head, so we pulled a second conversion that keeps the head. With the production flags the engine refused at init: full-and-piecewise CUDA graphs are unsupported with speculative decoding on the FlashInfer backend. With piecewise graphs the proposer failed an assertion at init (K4). On vLLM 0.27.0 with this architecture, MTP is not a lever, and on a hybrid stack it also excludes prefix caching, which carries the cached numbers above. The single-stream decode without it is already 172.7 tokens per second.

Reproducing it

The serve command is the production one from the field report with the model swapped; the client is about 200 lines of Python around the OpenAI-compatible endpoint and the engine's /metrics. The two lines below are the whole difference between the arms.

# the MoE arm (the dense arm is the field report's command, unchanged)
python3 -m vllm.entrypoints.openai.api_server \
  --model ProCreations/Nex-N2.5-mini-NVFP4 --served-model-name nex-n2.5-mini \
  --tensor-parallel-size 2 --max-model-len 262144 --max-num-seqs 12 --gpu-memory-utilization 0.95 \
  --kv-cache-dtype fp8 --enable-prefix-caching --mamba-cache-mode align --watermark 0.08 \
  --scheduling-policy priority --max-num-batched-tokens 6272 --scheduler-reserve-full-isl \
  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_xml \
  --kv-transfer-config '{"kv_connector":"OffloadingConnector","kv_role":"kv_both","kv_connector_extra_config":{"spec_name":"TieringOffloadingSpec","cpu_bytes_to_use":64424509440,"offload_prompt_only":false}}'

# one cell: 32 concurrent requests, 200K tokens each, shared warm prefix, 256 decode tokens, thinking off
python3 bench.py --model nex-n2.5-mini --model-path ProCreations/Nex-N2.5-mini-NVFP4 \
  --ctx 200000 --conc 32 --mode cached --seqs 12 --out results.jsonl

# the two startup lines that must match before any number is comparable
grep -E "GPU KV cache size|Maximum concurrency" vllm.log

Three things to copy from this run rather than rediscover. Restarting the engine in place on a host without a per-container /dev/shm leaves the 60 GiB CPU-tier memory map behind and the next start dies with shared memory full (K5); production avoids this with a private 72 GB /dev/shm per container. The chunk budget, not --max-num-seqs, sets cold throughput, so sweep the budget first if you sweep anything. And measure decode on the same host you measure prefill on, or say which host each number came from, because the two disagree by 2x on the same image.

What this changes for us: nothing in production, and something in the plan. The dense tier stays; the cost logic in what an AI agent actually costs is unchanged. The engine numbers are on file as the bar the next candidate has to clear, with the same grid and the same 459 replayed decisions, so that when the next hybrid MoE ships the comparison takes an afternoon.

Common questions

Are dense models better than MoE models?

Neither is better; they spend the same GPUs differently. In this test the MoE decoded a single stream at 172.7 tokens per second against the dense model's 80.9, held three times as many 262,144-token contexts in memory, and prefilled cold prompts faster. The dense model held its per-stream speed better as concurrency rose and scored higher on tool choice in the offline replay of 264 clean production decisions. Which one is better depends on whether your traffic is a few long streams or many short ones, and on how much you trust the model to keep acting after a tool is rejected.

Is this a benchmark of Claude or of a cloud API?

No. Both models ran locally on the same two GeForce RTX 5090s with the same vLLM 0.27.0 flags, and every number is a controlled cell with fixed prompts, seeds and a 256-token decode. Nothing here measures a hosted model, and the quality section compares the two local models with each other, not with a frontier model.

Why keep the production flags instead of tuning each model?

Because the question was what a MoE would do inside the deployment we already run, not what it could do in isolation. The flags that shape the results most, a 6,272-token prefill budget, a cap of 12 running requests, FP8 KV, prefix caching and priority admission, were left exactly as production has them. The only tuning experiments are in the section on what we would change, and they are labelled as deviations.

Why did the dense model prefill twice as fast on the production machine as on the rental?

Same image, same flags, same 1.04-million-token KV pool, different host. The rental prefilled 120,000-token prompts at about 2.1K tokens per second; the production machine, running the same cells overnight at low scheduler priority, reached 4.2K. Decode speed matched within 15%. The likely causes are the production machine's persistent compile cache and the rental host's CPU contention. The article quotes the smaller MoE advantage that this correction implies.

What is a mixture of experts model?

A model whose feed-forward blocks are split into many small experts, with a router choosing a few of them for each token. Nex-N2.5-mini has 256 experts per MoE layer and routes every token to 8 of them plus one shared expert, so about 3.3 billion of its 35.1 billion parameters do work on any given token. A dense model such as Qwen3.8-27B runs all 27 billion on every token. The smaller active set is why the MoE decodes a single stream twice as fast; the full weight set still has to sit in GPU memory, which is why it loads 13.2 GiB per GPU against the dense model's 10.9.

Is Qwen a MoE model?

Qwen ships both kinds. Qwen3.8-27B, the model in this test, is dense. Nex-N2.5-mini is built on Qwen's Qwen3.5 MoE architecture (its config names Qwen3_5MoeForConditionalGeneration), so this comparison is in effect a Qwen dense model against a Qwen-architecture MoE, served by the same reasoning parser and the same qwen3_xml tool-call parser.

How much KV cache memory does one token take?

Read it off the engine's own numbers: available KV memory divided by the pool it reports. On the MoE, 16.29 GiB per GPU holds 3,237,081 tokens, about 5,403 bytes per token per GPU, 10.6 KB across the pair; on the dense model, 16.5 GiB holds 1,038,029, about 17,068 bytes per token per GPU, 33.3 KB across the pair (K1, K2, K6). Both run FP8 KV. The 3.2x difference is architecture: only 10 of the MoE's 40 layers keep a per-token key-value cache, against 16 of the dense model's 64, and the MoE's attention heads are smaller.

Is the KV cache stored on the GPU, and should you offload it?

It lives in the GPU memory left after the weights, here about 16 GiB per card for either model, and that pool alone decides how many requests can run at once. Production also keeps a 60 GiB CPU tier through vLLM's OffloadingConnector, on both models in this test; evicted blocks go there so a returning session's prefix is restored instead of recomputed. The tier does not raise the running-request ceiling, and this test did not measure it separately: the cached cells hit the GPU prefix cache.

What is time to first token, and why did it differ 3x?

Time to first token is the wait between sending a request and receiving the first generated token; on a cold prompt it is almost entirely prefill. At 60,000 tokens the MoE returned its first token in 8.0 seconds and the dense model in 26.4 on the rental, 17.6 against 62.5 at 120,000. The dense model prefills about twice as fast on the production machine as on the rental, so against production the gap is closer to half than a third. Past 12 concurrent requests, time to first token on both is mostly queue time, because the engine admits 12 and the rest wait.

Will you switch the production tier to the MoE?

Not on this evidence. The throughput and context capacity are real, but in the offline replay of production decisions the MoE trailed on tool choice, stopped calling tools when thinking was enabled, and yielded the turn after a tool rejection where the dense model retried. Those are behaviours, not settings. The engine numbers are on file as a reference for the next candidate.