Qwen3.8-27B NVFP4 on two RTX 5090s: production field report

What each layer contributes

Five things have to agree for this endpoint to exist: a model whose architecture keeps the KV cache small, a checkpoint that puts the right tensors in the right precision, a serving engine that has kernels for that precision on this GPU, a GPU whose tensor cores run 4-bit floats natively, and a workload shaped so that the first four pay off. This section names what each one brings, with the source for every statement.

Qwen3.8-27B: the architecture

Qwen3.8-27B is a dense 27-billion-parameter model with a hybrid layer stack. The model card describes it as "16 × (3 × (Gated DeltaNet → FFN) → 1 × (Gated Attention → FFN))": 64 layers, of which 48 are linear attention (Gated DeltaNet) and 16 are full attention, one in every four. The full-attention layers use 24 query heads and 4 key/value heads of dimension 256; the linear layers use 48 value heads and 16 key heads of dimension 128. The checkpoint's config.json on our disk says the same thing (layer_types: 48 linear_attention, 16 full_attention, full_attention_interval 4), and it carries a vision encoder (27 blocks, hidden size 1152, patch size 16), a one-layer multi-token-prediction head (mtp_num_hidden_layers 1) and a native context of 262,144 tokens, extensible to 1,000,000 with YaRN. It is released under Apache 2.0; the repository was created on 2026-08-05.

The hybrid stack is what makes the rest of this report possible. Only the 16 attention layers hold a per-token KV cache; the 48 linear layers carry a constant-size recurrent state per sequence. That is why two 32 GB cards can hold a million cached tokens (section 2) and why a 262,144-token window is a real setting rather than a nominal one. Thinking is on by default and can be disabled per request, reasoning depth is set with reasoning_effort, and prior-turn reasoning can be kept with preserve_thinking; all three matter for agents and all three are used below.

Unsloth: the checkpoint

unsloth/Qwen3.8-27B-NVFP4 is not a uniform 4-bit model. Unsloth's own description of Dynamic NVFP4 is "selecting important layers to remain in FP8 (W8A8) or BF16 and the rest in W4A4 (not W4A16) instead of forcing every layer into FP4". Read from the checkpoint's quantization_config on our disk, that resolves to this precision map:

Tensors	Precision	Detail from the checkpoint
MLP gate/up/down projections, layers 0 to 55 (168 modules)	NVFP4, W4A4	4-bit float weights in groups of 16 with FP8 (E4M3) block scales; 4-bit activations with local dynamic scaling
Attention q/k/v/o (16 layers); linear-attention in/out projections (48 layers); MLP of layers 56 to 63; lm_head	FP8, W8A8	Per-channel static weight scales, per-token dynamic activation scales
Vision encoder (27 blocks), linear-attention state paths, norms, embeddings (303 modules in ignore)	BF16	Unquantized
KV cache	FP8, static per-tensor scales	k_scale and v_scale shipped for the 16 attention layers

Two consequences follow directly. The shipped KV scales are what make --kv-cache-dtype fp8 a calibrated choice rather than a guess (vLLM's own startup line warns that FP8 KV "may cause accuracy drop without a proper scaling factor"). And the FP8 lm_head is why the card says the model "works with SGLang, but only with the v0.5.19". The weights load through the compressed-tensors format (version 0.17.2.a20260716 in the config) and occupy 21.81 GiB on disk including the MTP head.

vLLM: the engine

vLLM 0.27.0 (released 2026-08-10) supplies everything between the checkpoint and the tensor cores: tensor parallelism across the two cards, the FlashInfer attention backend with an FP8 KV cache, the CUTLASS NVFP4 GEMM kernel for the 4-bit layers, a Triton kernel for the Gated DeltaNet prefill, prefix caching that is aligned to the hybrid architecture's page sizes, chunked prefill, priority scheduling, a CPU tier for evicted KV blocks, the Qwen3 reasoning parser and tool-call parser, an Anthropic-compatible /v1/messages endpoint next to the OpenAI-compatible one, and the Prometheus metrics that this whole report is built from. Section 8 goes through what each of those did in production.

NVIDIA Blackwell: the hardware

NVFP4 is NVIDIA's 4-bit floating-point format for Blackwell. The NVIDIA Technical Blog defines it as "4 bits (1 sign, 2 exponent, 1 mantissa) plus 1 shared FP8 scale per 16 value block" and states it "reduces the model memory footprint by approximately 3.5x relative to FP16, and approximately 1.8x compared to FP8", implemented by the fifth-generation Tensor Cores. The GeForce RTX 5090 is compute capability 12.0 (sm120), and the engine confirms at startup that it uses the real kernel path: "Using FlashInferCutlassNvFp4LinearKernel for NVFP4 GEMM". The official recipe makes the same point for this exact checkpoint on this exact GPU: "a cutlass path, not an emulation fallback".

What the consumer card does not have also shapes the configuration. The two GPUs have no NVLink and the engine reports "Custom allreduce is disabled because your platform lacks GPU P2P capability or P2P test failed", so tensor-parallel all-reduce runs through NCCL over PCIe. Each card has 32,607 MiB, of which the engine budgets 95%. The rest of the machine is an AMD Ryzen 9 7950X and 128 GB of RAM (124.9 GiB as the OS reports it), driver 595.58, CUDA 13.2. It is owned hardware on our own network.

Scalably: the workload and the guardrails

Scalably runs a multi-tenant agent runtime: one isolated container per workspace per run, tool access through gated integrations, and per-workspace routing that decides whether a workspace talks to a frontier model or to this endpoint. What the platform contributes to the stack is the shape of the traffic (long, repetitive, tool-bound prompts), the gateway that meters and protects the endpoint, and the verification layer that never lets a local model's output reach a person on the model's own word. That layer is described in section 3 because it changes what "good enough" means for a 27B model.

The reproducible configuration

Below is the production command with host, port and paths removed, plus the version matrix. It has run unchanged since 2026-08-29 (container restart count: 0). The comments state why each flag is there; the reasons are quoted from the operator log where they were recorded.

# container (paths, host and port removed)
docker run -d --gpus all --network host --shm-size 72g --restart unless-stopped \
  --env PYTHONHASHSEED=0 --env HF_HOME=/models/hf --env VLLM_USE_DEEP_GEMM=0 \
  --mount type=bind,src=<models-dir>,dst=/models \
  --mount type=bind,src=<compile-cache-dir>,dst=/root/.cache/vllm \
  vllm/vllm-openai:v0.27.0 \
  --model unsloth/Qwen3.8-27B-NVFP4 \        # revision 57926bac in production
  --served-model-name qwen3.8-27b \
  --tensor-parallel-size 2 \                  # one rank per RTX 5090
  --max-model-len 262144 \                    # the model's native window
  --max-num-seqs 12 \                         # queue knee measured at ~12 running (was 32)
  --gpu-memory-utilization 0.95 \             # fallback 0.92 if graph capture OOMs
  --kv-cache-dtype fp8 \                      # checkpoint ships k/v scales
  --enable-prefix-caching \
  --mamba-cache-mode align \                  # 'all' silently falls back on this architecture
  --watermark 0.08 \                          # keep ~8% of KV blocks free; stops admit-then-preempt storms
  --scheduling-policy priority \              # per-request priority; equal priorities are FCFS
  --scheduler-reserve-full-isl \              # default on; admit only if the full input fits
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \              # switched from qwen3_coder on 2026-08-17, see section 5
  --kv-transfer-config '{"kv_connector":"OffloadingConnector","kv_role":"kv_both",
    "kv_connector_extra_config":{"spec_name":"TieringOffloadingSpec",
    "cpu_bytes_to_use":64424509440,"offload_prompt_only":false}}'   # 60 GiB of pinned RAM as a second KV tier

Chunked prefill is on with max_num_batched_tokens 6272 (the engine reports it at startup). No --speculative-config is set: the engine initializes with speculative_config=None, so the checkpoint's MTP head is loaded on disk but never used. That is a deliberate omission in this report, not an oversight in the deployment: nothing has been measured with it, so nothing is claimed about it.

Component	Version or value	Where it is read from
Serving image	vllm/vllm-openai:v0.27.0 (vLLM 0.27.0, released 2026-08-10)	image tag, vllm.__version__
PyTorch / CUDA in the image	torch 2.13.0+cu130, CUDA runtime 13.0	pip list in the container
FlashInfer	0.6.16.post3 (python, cubin, jit-cache cu130)	pip list
compressed-tensors / CUTLASS DSL	0.17.0 / 4.6.0	pip list
Host driver / CUDA	595.58.03 / 13.2	nvidia-smi
GPUs	2 × GeForce RTX 5090, 32,607 MiB each, compute capability 12.0	nvidia-smi, startup log
Checkpoint revision	57926baca9a82b4d6906b43f2750d55315f5b10f, pulled 2026-08-29	HF cache refs/main, snapshot mtime
Model architecture class	Qwen3_5ForConditionalGeneration, model_type qwen3_5_text	config.json
Weights per GPU	10.87 GiB, loaded in 16.2 s	startup log
KV cache per GPU	16.51 GiB = 1,039,536 tokens; "Maximum concurrency for 262,144 tokens per request: 3.97x"	startup log
Attention block size	1,568 tokens (aligned to the Mamba page, padded 0.13%)	startup log
CUDA graphs	FULL_AND_PIECEWISE, capture sizes 1, 2, 4, 8, 16, 24; 0.54 GiB	startup log
torch.compile	69 s on first start, then served from the persistent cache	startup log
Speculative decoding	none (speculative_config=None)	startup log

The gateway in front of the engine is LiteLLM 1.96.2. It publishes four aliases of the same model: qwen3.8-27b with thinking off and a 16,000-token output ceiling; qwen3.8-27b-think with reasoning_effort medium and an 8,192-token thinking budget; a -think-preserve variant that replays each prior assistant turn's thinking back to the engine through the reasoning field; and an xhigh variant with a 65,536-token output ceiling for one maximum-quality lane. Agent containers speak the Anthropic Messages protocol to a pass-through path that forwards to vLLM's native /v1/messages, so there is no protocol conversion in the gateway on the generation path.

What the production workload looks like

Synthetic benchmarks send fixed prompts at fixed concurrency to an idle machine. This endpoint sees two very different clients, both bursty: agent runs with 30,000-token prompts that repeat their prefixes, and a document digitizer that sends page images with thinking off. Over fourteen days the mean request carried 30,100 prompt tokens and returned 983; the median request carried 6,466 and returned 422. The gap between mean and median is the workload.

Characteristic	Observed (2026-08-29 to 2026-09-12)	Why it differs from a benchmark
Prompt length	p50 6,466 · p90 90,530 · p99 183,800 tokens	Agent sessions accumulate tool results; the p99 is near the 262k window
Output length	p50 422 · p90 2,959 · p99 9,027 tokens; 4.2% of requests hit their token limit	Tool calls are short; drafts and transcriptions are long
Prefix reuse	82.6% of prefilled tokens served from cache (712.8M of 863.5M)	System prompt, tool schemas and memory repeat across turns and across subagents
Concurrency	≥1 request running in 15% of minutes; ≥4 in 4.6%; ≥8 in 0.9%; peak 12 running with 6 waiting	Bursts, then idle; a benchmark holds concurrency constant
Use pattern	Requests completed in 269 of 336 hours (80%); 182 to 14,022 requests per day	Scheduled agent work and a timer-driven batch job, not a load generator
Thinking	Per alias: off for tool loops and extraction, on with a budget for drafting and review	Benchmarks pick one mode
Modalities	Text through the gateway; page images plus JSON-schema output from the digitizer	Vision prefill shares the same batched-token budget

Sessions and tool calls. Agent runs are Claude Agent SDK sessions: a system prompt, tool schemas, memory files, then a loop of model turn, tool call, tool result. Tool calls come back through vLLM's qwen3_xml parser with --enable-auto-tool-choice. Every turn resends the whole session, which is why the prefix cache, not raw throughput, decides whether the machine keeps up. The 82.6% hit rate means a 30,000-token turn costs the GPUs a few thousand tokens of new prefill.

The digitizer. Every fifteen minutes a timer on another host wakes a low-priority job that takes up to forty new documents from an agricultural cooperative's paper archive (an internal partner), renders each page, asks the same model for a transcription with thinking disabled and temperature 0, then asks for typed fields (document number, date, net, VAT, total, partner id, currency) with vLLM's JSON-schema-constrained output, two pages in flight at a time. As of 2026-09-12 it had processed 2,526 documents and 7,183 pages.

Model reads, code decides. A local model in production is never the last word here. In the digitizer, a field value is written only if its printed form appears literally in the transcription; the model may not compute or guess, and dates are never written automatically because a misread handwritten digit in the pilot produced a 2018 date on a 2026 receipt. In the agent runtime, a tool-outcome guard classifies every tool result into one of eight anomaly kinds (empty result, failure reported in a success channel, invalid envelope, failed, blocked, partial, pending, no-op) before the model sees it as success. At the end of a run, an independent Stop-QA step runs deterministic checks first (unresolved placeholders, tool errors the final answer never acknowledged, counts that disagree with the tool results), then an evidence-bound reviewer that can run on a different endpoint than the worker, with an escalate verdict that forces a human disposition. It has five modes from off to enforce-cascade, is off by default, and is enabled per workspace. A deferred-replay capture path exists in the runtime and is also off by default; it "does not run Qwen, fetch tools, retain raw prompt or result text, call a teacher or authorize training".

Fourteen days of production observations

These are production observations, not benchmark results. They are cumulative counters from vLLM's /metrics endpoint and series from the Prometheus that scrapes it, read on 2026-09-12 at 10:15 UTC, covering everything since the container started on 2026-08-29 at 11:54 UTC. The mix of prompts, thinking modes and clients changed from hour to hour, so none of this transfers to another workload as a rate.

Counter (cumulative, 13.9 days)	Value	Note
Completed requests	28,097	26,920 stopped naturally, 1,177 (4.2%) reached their token limit, 0 aborted, 0 engine errors
Prompt tokens processed	860.6M	includes cached prefix tokens; mean 30,100 per request
Tokens generated	28.6M	mean 983 per request
Prefix-cache hit rate	82.6%	712,833,184 hits / 863,463,591 queries
Preemptions	2	29 more occurred between 2026-08-17 and 2026-08-29 while the flags were being iterated
Engine restarts	0	3 restarts in the two weeks before, none since the current flags

Latency, split the way the engine splits it. Quantiles are interpolated over the engine's cumulative histogram buckets, so treat them as approximate.

Stage	Mean	p50	p90	p99
Queue wait	0.13 s	0.15 s	0.28 s	5.2 s
Time to first token	1.44 s	0.61 s	2.4 s	18.9 s
Prefill	1.10 s	0.48 s	1.9 s	12.7 s
Decode	16.1 s	6.7 s	39.9 s	156 s
End to end	17.5 s	8.3 s	41.7 s	158 s
Inter-token latency	16.4 ms	17.6 ms	23.6 ms	25.0 ms

Two readings from that table. The p99 time to first token (18.9 s) is consistent with an uncached prompt near the p99 length (183,800 tokens) on a 6,272-token prefill budget; the operator log recorded 85.4 s cold at 255,000 tokens during the August rollout. And inter-token latency stays in a narrow band (17.6 ms median, 25.0 ms p99). That histogram is token-weighted, so it is dominated by the periods when several streams were decoding at once and each stream ran a little slower; the engine's own throughput line at one running request, in the next table, is the single-stream figure (about 83 tokens per second, or 12 ms per token).

Generation rate by concurrency. The engine logs a 10-second average of its prompt and generation throughput with the number of running requests. Grouping the 19,221 active samples since 2026-08-29 by that number gives the table below. It is the closest thing to a scaling curve that production can offer, and it is not a benchmark: prompt lengths, thinking modes and image inputs all vary inside each row.

Running requests	Samples	Generation tok/s, p50	p90
1	8,870	83	89
2	2,155	127	157
4	1,058	243	306
6	1,192	331	425
8	719	433	538
10	83	367	461
12	7	455	498

Aggregate generation scales close to linearly up to eight running requests and flattens after, where prefill for newcomers competes for the same batched-token budget. The peak 5-minute aggregate over the whole window was 517 generated tokens per second, and 34,230 prompt tokens per second (mostly cache hits). Both are ceilings the workload happened to touch, not capabilities of the hardware.

Day by day. UTC days, sums of hourly counter increases. The table ends with 2026-09-11; the first ten hours of the read day (a few dozen requests) are in the totals above but not tabulated.

Day	Requests	Prefix-cache hit rate	Day	Requests	Prefix-cache hit rate
08-29 (from 11:54)	344	78.5%	09-05	407	84.4%
08-30	632	92.1%	09-06	610	89.6%
08-31	3,916	90.2%	09-07	2,024	88.8%
09-01	756	85.2%	09-08	473	62.6%
09-02	246	71.4%	09-09	394	52.8%
09-03	3,412	88.5%	09-10	14,022	56.9%
09-04	182	66.0%	09-11	646	56.2%

The hit rate follows the mix. Days dominated by agent sessions sit near 90%; days with little traffic or with image-heavy digitizer work sit lower, because page images share no prefix. 2026-09-10 is the outlier: the digitizer re-processed the whole archive that day (2,518 of its output records carry that date), which produced 14,022 requests and 12.15 million generated tokens, 42% of all generation in the window, at a 56.9% hit rate. The GPUs absorbed it at the low-priority slot without a single preemption or engine error, which is the scheduling story of section 8 in one day.

Cache pressure and the CPU tier. KV cache usage averaged 1.8% over time and 7% over active 10-second samples, with a p99 of 53% and a single-sample maximum of 99.6%. The CPU tier wrote 12.76 TB of evicted KV blocks to pinned RAM in 571.7 s of total transfer time and read 194.4 GB back in 438 loads (7.8 s). At roughly 17 KB per token per GPU, that read volume is on the order of 6 to 11 million tokens restored from RAM (the counter's per-rank accounting was not verified, hence the range), against 712.8 million served from the GPU cache: the second tier contributed roughly one percent of cache hits, under two on the most generous reading. It is insurance for the long sessions, not the main mechanism, and its write cost was negligible.

Thermals and power. The GPU exporter recorded a 14-day maximum of 87 °C on one card and 71 °C on the other; the hotter card sat at or above 80 °C in 1.8% of samples. Peak board power was 411 W and 410 W, mean 76 W and 68 W, mean utilization 15.8%. The operator log records a smart-plug measurement from 2026-08-17 of 175 W for the whole machine idle and 876 to 891 W under full inference. Host memory: 124.9 GiB total, of which the serving container holds 78.6 GiB resident (the 60 GiB pinned tier plus weights and compile artefacts), leaving a 14-day minimum of 15.1 GB available.

Surprises, failure modes and operational lessons

Everything in this list happened. Each item is dated from the operator log, the gateway configuration or the engine's own counters; nothing is inferred from a benchmark or a forum.

• The queue has a knee, and it is the sequence cap (2026-08-18). With --max-num-seqs 32 the log records "queue p95 9.4s -> 233.6s at running~12". The cap went to 12 and the gateway admits at most 10 outstanding requests, spilling the rest. Priority scheduling does not displace an already-running request at the cap; it orders the waiting queue and preempts a running request only when a KV allocation fails, so admission control had to live in front of the engine.
• Admitting to a full cache causes preemption storms (2026-08-18). The --watermark 0.08 flag keeps about 8% of KV blocks free. 29 preemptions in the two weeks of iteration before it; 2 in the fourteen days since, with the cache touching 99.6% once.
• Timeout-based fallback fails mid-stream. A gateway-level stream timeout with a fallback route produced MidStreamFallbackError once the SSE stream had opened. The replacement decides before dispatch: a hook polls the engine's /metrics every second, counts running plus waiting plus requests admitted since the last poll, and spills to the cloud endpoint above the cap, above 85% KV usage, or whenever the telemetry is older than three seconds ("losing telemetry degrades to cloud cost, never to a rig outage").
• Cold prefill at the window edge is slow (2026-08-18). 85.4 s time to first token at 255,000 tokens with no cache, on the 6,272-token prefill budget. The production p99 of 18.9 s is lower only because most long prompts are mostly cached.
• The tool-call parser mattered (2026-08-17). qwen3_coder stalled on long inputs combined with very large write-tool arguments in 2 of 12 runs of one experiment (recovered by a watchdog). The switch to qwen3_xml passed the smoke set (single, parallel, 27 KB write, think plus write) and has run since.
• Thinking starves tool calls (August 2026). The gateway config records one task measured both ways: 11,170 tokens in 144 s with thinking versus 917 tokens in 11.7 s without, same answer; and a 200-token budget fully consumed by thinking with no tool-call block ever emitted. The default alias runs with thinking off; drafting and review aliases turn it on with an 8,192-token budget and reasoning_effort medium.
• Images inside tool results blew up as text (2026-08-17). Base64 images nested in tool results reached the engine as 185,000 characters of text. A gateway hook now lifts them into proper image content.
• The hybrid cache mode you ask for is not always the one you get. Prefix caching on a Mamba-style architecture defaults to all where supported and otherwise falls back to align; the deployment names align explicitly so the log states what actually runs. The engine labels the feature experimental at every start. In fourteen days it produced an 82.6% hit rate and no cache-related incident.
• Consumer cards, consumer interconnect. No P2P, no symmetric memory on capability 12.0, so all-reduce runs through NCCL over PCIe. The two cards also do not heat equally: 87 °C versus 71 °C peak, 1.8% of samples at or above 80 °C on the hotter one. Airflow is on the list.
• 4.2% of requests end on the token limit. Some are intended (bounded extraction), some are agents drafting past their ceiling. The counter exists so it can be watched per alias; today it is one number.
• Batch work changes the cache, not the SLA. The 2026-09-10 archive refresh halved the day's hit rate and produced 42% of all tokens generated in the window, and agent turns kept their priority through it.
• The only engine-side "errors" were two client requests for a model name that does not exist. request_success_total{finished_reason="error"} is 0.
• Persist the compile cache. torch.compile took 69 s on the first start; a bind-mounted cache directory means a restart does not pay it again. A private 72 GB /dev/shm per container keeps the KV memory maps from outliving the process.

Compared with the official vLLM recipe

The vLLM project publishes a recipe for this model (recipes.vllm.ai, "Updated 2026-09-10", read 2026-09-12) with a section for two RTX 5090s that uses the same Unsloth checkpoint. The recipe is right for what it measures: a clean, general-purpose configuration verified on a development build. The production command differs in eight places, and every difference is a consequence of the workload in section 3, not a correction.

# official recipe, 2x RTX 5090 (consumer Blackwell, sm120), quoted
vllm serve unsloth/Qwen3.8-27B-NVFP4 \
  --tensor-parallel-size 2 \
  --max-model-len 262144 \
  --kv-cache-dtype fp8 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder

Setting	Official recipe	This deployment	Why we differ
Tool-call parser	qwen3_coder	qwen3_xml	Stall on long input plus very large write arguments, 2026-08-17 (section 5)
Concurrent sequences	engine default	12	Measured queue knee; gateway admits 10
GPU memory utilization	engine default	0.95	More KV room; fallback 0.92 if graph capture fails
Scheduling	default (FCFS)	priority, watermark 0.08, reserve full input	Two clients with different urgency on one endpoint
Prefix caching	default	explicit, --mamba-cache-mode align	Log states the mode that actually runs
KV tier in RAM	none	60 GiB, OffloadingConnector / TieringOffloadingSpec	Long sessions that return after eviction
Speculative decoding	optional, {"method":"mtp","num_speculative_tokens":3}	none	Not yet evaluated on this mix (section 10)
Verified on	vLLM 0.26.1rc1.dev608	vLLM 0.27.0 release image	Pinned image with a persistent compile cache

The recipe's startup table reports 920,517 KV tokens and 10.64 GiB of weights per GPU for the Unsloth checkpoint at 262,144 context; this deployment reports 1,039,536 KV tokens and a 10.87 GiB model-loading footprint. The difference in KV room is the memory-utilization setting; the two weight figures measure slightly different things (the recipe's is weights, the log's is everything model loading allocated). The recipe also records an MTP acceptance of 0.788 for this checkpoint versus 0.897 for a uniform-W4A4 build, and says the two "are not interchangeable on 32 GB cards: unsloth is mixed-precision (FP8 channel-wise alongside the 4-bit groups) and leaves room for roughly twice the KV cache, while the uniform-W4A4 Inferact build drafts better. Both serve correctly." That trade is exactly the one this workload wants: cache room over drafting speed. How this dense tier compares with a mixture-of-experts checkpoint on the same GPUs and the same flags is measured in MoE vs dense inference on two RTX 5090s.

Two smaller points of agreement. The recipe notes that vLLM auto-disables DeepGEMM for this model type on Blackwell and that the flag is unnecessary; the production script still sets VLLM_USE_DEEP_GEMM=0 from an earlier iteration, harmlessly. And the recipe's single-card section states plainly that one 5090 serves this model only at 32,768 tokens with --enforce-eager, which is the clearest argument for the second card.

What the Unsloth checkpoint contributes

The specific thing this checkpoint buys is room: the FP8 attention and linear-attention projections keep the quality-sensitive paths at 8 bits while the 168 large MLP projections go to 4, and the recipe's own measurement says the result "leaves room for roughly twice the KV cache" of a uniform 4-bit build. For a workload whose median request is 6,466 tokens and whose p99 is 183,800, cache room is the metric.

• Calibrated FP8 KV scales in the checkpoint. The kv_cache_scheme is static per-tensor FP8 with k_scale and v_scale for all 16 attention layers. Unsloth's guide describes FP8 KV calibration as the path to "2x longer context lengths"; on this machine it is what turns 16.51 GiB per GPU into 1,039,536 tokens.
• The vision tower is untouched. All 27 encoder blocks are in the ignore list, so the digitizer's page images go through BF16 weights. 7,183 pages have been read that way.
• The MTP head ships with it. model_mtp.safetensors is in the snapshot and the recipe reports it drafting at 0.788 acceptance. It is available the day we decide to test it.
• An FP8 lm_head. Correct for accuracy, and the reason the card limits SGLang to one version; on vLLM it just loads.
• Revisions move. Production runs 57926bac (commit dated 2026-08-29). The repository's main branch is at f0b7c9e7 (2026-09-08), and the five commits since mid-August are all titled "Update README.md". Whether any tensor changed between them is not something this report checked, which is itself the point: pin the revision.

What is not claimed: any accuracy figure for this checkpoint against BF16. Unsloth publishes its own comparisons and speed claims ("~1.5× faster" than BF16 on its Qwen3.8 page); we have not reproduced them and do not repeat them as ours. What would help most from Unsloth, from a production seat: a changelog line per revision that says whether weights changed, and the calibration recipe (data mix, sequence lengths, whether tool calls and thinking traces were in it), because the next step here is post-training and re-quantizing on our own trajectories, and matching the calibration distribution to the workload is the part we cannot see.

What vLLM contributes

vLLM's contribution is that all of the following ran together for fourteen days on a consumer machine without an engine error: FP8 KV with FlashInfer on sm120, CUTLASS NVFP4 GEMMs, a Triton Gated DeltaNet kernel, hybrid-aware prefix caching, chunked prefill, priority scheduling, a CPU KV tier, two API surfaces, two parsers and a metrics endpoint. The observations below are the ones another operator can reproduce from the startup log and /metrics alone.

• Prefix caching on a hybrid model. With --mamba-cache-mode align the engine set the attention block size to 1,568 tokens "to ensure that attention page size is >= mamba page size" and padded the Mamba page by 0.13%. That is the granularity at which prefixes are reused, and it delivered 82.6% hits on agent traffic. Documentation calls the feature experimental; the counters say it worked.
• Chunked prefill with a 6,272-token budget. Prefill p50 is 0.48 s and p99 12.7 s. The budget bounds how much a long newcomer can delay running decodes, which shows up as a 25.0 ms p99 inter-token latency even at twelve running requests.
• Priority scheduling and the watermark. The digitizer's batch requests and agent turns share the engine; the scheduler orders by priority and then arrival, and the watermark stopped the admit-then-preempt cycle (2 preemptions in fourteen days). The documented limit stands: priority reorders the waiting queue and preempts only on allocation failure, so a cap on running sequences still has to be enforced in front.
• The CPU tier. OffloadingConnector with TieringOffloadingSpec and 60 GiB of pinned memory wrote 12.76 TB and read back 194.4 GB in 438 loads. In this workload it is a safety net for sessions that return after eviction, contributing roughly one percent of cache hits. The engine exposes the counters to know that.
• MTP: available, not enabled. vLLM supports the in-checkpoint draft head through --speculative-config '{"method":"mtp",...}' and exposes vllm:spec_decode_num_accepted_tokens_total and _draft_tokens_total; the recipe recommends reading those before tuning. This deployment has no observation to add yet.
• Two API surfaces on one engine. Agent containers speak the Anthropic Messages protocol to vLLM's /v1/messages through a gateway pass-through; the digitizer speaks OpenAI chat completions with response_format json_schema and chat_template_kwargs. The reasoning parser (qwen3) splits thinking from answer on both, and prior-turn thinking can be replayed through the reasoning field of an assistant message (the gateway note records that reasoning_content is dropped and reasoning is accepted, verified with the tokenizer endpoint on 2026-08-17).
• Metrics are the product. Every table in this report came from /metrics: request histograms split into queue, prefill, decode and inter-token; prefix-cache queries and hits; KV usage; preemptions; offload bytes and time. A serving engine that did not export these would have produced a marketing post instead of a field report.

What AI agents mean for everyday business work

AI agents are becoming useful for everyday tasks such as drafting customer emails, preparing follow-ups, and updating records. Sales professionals expect AI agents to reduce email-drafting time by 36% once fully implemented. For small teams, that could mean more time for customers and less time spent writing repetitive messages.

The challenge is making sure these agents work reliably. In this deployment, the inference system processed 28,097 requests over approximately 14 days without an engine error. Scalably also runs a verification layer, so a model's output never reaches a person on the model's own word.

For businesses adopting AI agents, the practical starting point is simple: choose a repetitive task, give the agent access to the tools it needs, and define when human review is required. Faster messages are a side effect; the goal is useful work automated without losing oversight.

Qwen3.8 as a production tool-using worker

The model's job here is not to be the smartest thing in the building; frontier models stay in the loop for hard reasoning and for review. Its job is to run tool loops, drafts and extraction reliably at a cost that makes repetition free. Three properties of Qwen3.8-27B carry that: native tool calling that survives a strict parser, per-request thinking control, and a vision head that reads paper.

• Tool calls. Through qwen3_xml and auto tool choice the model has run Claude Agent SDK sessions for fourteen days, including the write-heavy cases that stalled the other parser. The checkpoint card's release notes list "Tool calling improvements: Makes parsing nested objects to make tool calling succeed more", which matches the experience, with the parser choice as the caveat.
• Thinking as a dial, per request. enable_thinking, reasoning_effort and preserve_thinking are all exercised: off for the default agent alias and for extraction, medium with a budget for drafting and review, preserved across turns for one lane. The Qwen card warns that "in multi-turn agentic tasks, lower reasoning effort does not always reduce overall task completion time"; the operator log's August effort ladder (2026-08-22 and 08-23) concluded "LOW effort is the value pick on well-specified tasks; keep medium as fleet default", and recorded that unbudgeted xhigh reasoning ran to a "blowout" on a hard single-shot problem, ending in "NEVER run xhigh without a budget". Both can be true; the workload decides, and the per-request control is what lets it.
• Vision plus constrained output. The digitizer sends page images at temperature 0 with thinking off and receives schema-constrained JSON. Printed text needs almost no correction; handwriting is approximate and goes to a review queue. Every written value is grounded by code against the transcription.
• Output discipline. Median output 422 tokens, 4.2% of requests at the limit. For a worker that is mostly asked for the next tool call, short is correct.

What would help most from Qwen, from a production seat: a normative statement of the tool-call wire format that parsers are expected to implement (so that "which parser" stops being an operator experiment), and documented semantics for reasoning_effort in multi-turn tool use, since the effect we measured depends on it.

Planned work

Everything in this section is planned, not live. It is listed so that the next report can be held to it.

1. Trajectories to a verified dataset (planned). The runtime already leaves causal trajectories (session, tool call, tool result, outcome). A snapshot from 2026-08-24 counted 4,471 main sessions, 3,623 subagent runs and 143,145 tool-use blocks across five hosts. Deterministic evidence, not model judgement, labels which decisions were recoveries, which were repeats of a failed action and which were clean. Raw tenant data never enters a training set.
2. Specialized worker post-training (planned; first experiment done). Two rank-16 LoRA runs on rented GPUs, away from the production machine, cost $53.88 and were evaluated on the BF16 base model for about $10.47. Verified recovery improved from 31 to 38 of 73 decisions (paired p = 0.0156); exact repeats of a failed action fell from 32 to 24 of 160 (20% to 15%); clean-decision retention regressed from 49 to 45 of 73. The ledger's verdict: movement proven, "neither adapter is a serious or production candidate". The next curriculum is retention-heavy. The two-timescale architecture we built around that result is described in Fast Memory, Slow Weights.
3. NVFP4 with quantization-aware distillation (planned). NVIDIA Model Optimizer documents QAD as its "recommended strategy for accuracy recovery after quantization" and states that "full-precision fine-tuning followed by QAD delivers the best accuracy, especially at aggressive quantization levels (e.g., NVFP4)". The planned order is post-train in high precision, evaluate, quantize to NVFP4, evaluate the quantized checkpoint on the same frozen suite, apply QAD only if the second evaluation demands it.
4. Replay and evaluation before promotion (planned). The deferred-replay capture path exists and is off; the plan is to replay captured states against a candidate worker in a sandbox and gate promotion on the frozen suite plus replay regression. Promotion is per workspace, versioned and reversible through the same module gates that govern integrations today.
5. MTP on the production mix (planned). Enable the in-checkpoint draft head on a canary alias and read acceptance from the engine's speculative-decoding counters before believing any throughput change, as the recipe advises.
6. A small verifier model (planned). The Stop-QA reviewer role today can route to a separate endpoint; the target shape is a small specialized verifier between a local worker and delivery, with escalation to a larger model or a person. Sizes are proposed, not chosen.
7. Revision pinning and a thermal fix (planned, small). Pin the checkpoint by revision in the serve script, and improve airflow on the card that reaches 87 °C.

Reproducibility appendix

Everything needed to stand up the same endpoint and read the same counters, with hosts and secrets removed. The two diagrams in sections 1 and 3 are the architecture; the version matrix in section 2 is the software.

Prerequisites. Two Blackwell GPUs with 32 GB each (compute capability 12.0 or higher; NVFP4 does not run on earlier architectures), a recent NVIDIA driver with CUDA 13 support, Docker with the NVIDIA container toolkit, about 22 GiB of disk for the checkpoint and 60 GiB of RAM you are willing to pin. The --shm-size 72g in the command below exists because tensor parallelism and the CPU tier both use shared memory.

# 1. pull the pinned checkpoint revision into the mounted HF cache
HF_HOME=<models-dir>/hf hf download unsloth/Qwen3.8-27B-NVFP4 --revision 57926baca9a82b4d6906b43f2750d55315f5b10f

# 2. start the engine (full command in section 2)
docker run -d --gpus all --network host --shm-size 72g --restart unless-stopped \
  --env HF_HOME=/models/hf --env PYTHONHASHSEED=0 \
  --mount type=bind,src=<models-dir>,dst=/models \
  --mount type=bind,src=<compile-cache-dir>,dst=/root/.cache/vllm \
  vllm/vllm-openai:v0.27.0 --model unsloth/Qwen3.8-27B-NVFP4 --served-model-name qwen3.8-27b \
  --tensor-parallel-size 2 --max-model-len 262144 --max-num-seqs 12 --gpu-memory-utilization 0.95 \
  --kv-cache-dtype fp8 --enable-prefix-caching --mamba-cache-mode align --watermark 0.08 \
  --scheduling-policy priority --scheduler-reserve-full-isl --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml \
  --kv-transfer-config '{"kv_connector":"OffloadingConnector","kv_role":"kv_both","kv_connector_extra_config":{"spec_name":"TieringOffloadingSpec","cpu_bytes_to_use":64424509440,"offload_prompt_only":false}}' \
  --host <host> --port <port>

# 3. confirm what actually loaded (the lines this report quotes)
docker logs <container> 2>&1 | grep -E "NvFp4LinearKernel|attention backend|GDN prefill|KV cache size|Maximum concurrency|attention block size|Mamba cache|speculative_config|Custom allreduce"

# 4. health and identity
curl -s http://<host>:<port>/v1/models
curl -s http://<host>:<port>/metrics | grep -E "^vllm:(request_success_total|prefix_cache_(hits|queries)_total|num_preemptions_total|kv_cache_usage_perc)"

# 5. the per-request histograms behind the latency table
curl -s http://<host>:<port>/metrics | grep -E "^vllm:(request_queue_time|request_prefill_time|request_decode_time|time_to_first_token|e2e_request_latency|inter_token_latency)_seconds_(sum|count)"

# 6. a request with thinking off and a JSON schema (the digitizer's shape)
curl -s http://<host>:<port>/v1/chat/completions -H 'content-type: application/json' -d '{
  "model":"qwen3.8-27b","temperature":0,
  "chat_template_kwargs":{"enable_thinking":false},
  "response_format":{"type":"json_schema","json_schema":{"name":"fields","schema":{"type":"object","properties":{"total":{"type":"string"}}}}},
  "messages":[{"role":"user","content":"Return the total from: TOTAL 1,234.00"}]}'

What "reproduced" means here. The startup lines in step 3 should match this report exactly on the same image, revision and GPUs (kernel selection, 1,568-token block, 1,039,536 KV tokens at 0.95 utilization). The counters in steps 4 and 5 will not match, because they measure your traffic, not the machine.

Term	Meaning in this report
NVFP4	NVIDIA's 4-bit float format (E2M1) with an FP8 scale per block of 16 values and a per-tensor FP32 scale; executed natively by Blackwell tensor cores
W4A4 / W8A8	Weights and activations both quantized to 4 or 8 bits; W4A16 would keep activations at 16 bits and forgo the FP4 tensor cores
Tensor parallelism (TP)	Each layer's tensors split across the two GPUs; every layer needs an all-reduce between them, here over PCIe via NCCL
KV cache	Per-token key and value tensors of the 16 attention layers kept for reuse during decoding; FP8 here, with shipped scales
Prefix caching	Reusing KV blocks for a prompt prefix that an earlier request already computed; hit rate = cached tokens / tokens asked for
Chunked prefill	Processing a long prompt in slices bounded by max_num_batched_tokens so decoding requests keep making progress
Gated DeltaNet	The linear-attention layer type used in 48 of the 64 layers; constant-size state per sequence, no per-token KV
Mamba cache mode	How vLLM caches the linear layers' state for prefix reuse; align matches its page size to the attention block size
MTP	Multi-token prediction: a draft head trained with the model that proposes several tokens per step for speculative decoding; present, not enabled
TTFT / ITL	Time to first token (queue plus prefill) and inter-token latency during decode
Preemption	The scheduler evicting a running request's KV blocks to make room; it resumes later
Watermark	The fraction of KV blocks the scheduler keeps free before admitting more work
Priority scheduling	Requests carry an integer priority (lower first, default 0); ties are first-come first-served

Common questions

Is any number in this report a benchmark?

No. Every figure is a cumulative production counter or a Prometheus series read from the running vLLM engine on 2026-09-12, covering all traffic since the serving container started on 2026-08-29. The per-concurrency generation rates are 10-second engine averages over a mixed workload and are labelled as production observations. A controlled benchmark would need fixed prompts, fixed sampling and an idle machine, and none of that describes production.

Which checkpoint revision is actually running?

Revision 57926bac of unsloth/Qwen3.8-27B-NVFP4, pulled on 2026-08-29 when the container started. The repository's main branch moved on 2026-09-08; every commit since is titled Update README.md, and we have not diffed the tensors. Pin the revision if you want to reproduce what this report measured.

Why is speculative decoding (MTP) not enabled?

The checkpoint ships an MTP draft head and the official vLLM recipe reports acceptance rates for it, but this deployment has never run with speculative decoding, so we have no observation to publish. It is on the planned list, with the acceptance counters the recipe recommends as the first thing to read.

Why only twelve concurrent sequences?

Because that is where the queue stopped behaving. During the August rollout the operator log records queue p95 rising from 9.4 seconds to 233.6 seconds at around twelve running requests with the earlier cap of 32, so the cap was lowered to twelve and the gateway admits at most ten outstanding requests before spilling to a cloud endpoint. In fourteen days the cap was reached, six requests waited at the peak, and the engine preempted twice.

Can a single RTX 5090 serve this model?

The official vLLM recipe says yes with constraints: 32,768 tokens of context and --enforce-eager, because CUDA graph capture otherwise runs out of memory on one 32 GB card. Two cards with tensor parallelism are what make the 262,144-token window and twelve sequences possible in this report.

The endpoint this report describes carries production for one company's agents and one partner's archive on two consumer GPUs, and it does so because four other teams' work fits together: a hybrid architecture that keeps the cache small, a checkpoint that spends bits where they matter, an engine that runs the real kernels and counts everything, and a GPU generation that made 4-bit floats native. The cost side of why this matters is in what an AI agent actually costs; the runtime the agents live in is described in the Claude Agent SDK guide and on the platform page. The next report will be held to section 10.