Running Qwen3.8-27B NVFP4 on two RTX 5090s in production

Why serve a model locally at all

Agent traffic has a shape: very long prompts, short outputs, deep tool loops, and a lot of repetition. On our production rig the mean request carries about 30,100 prompt tokens and returns 983. That shape is expensive on a metered frontier API and cheap on owned hardware with a warm prefix cache, so we route the repeated work to a local model and keep frontier models for the reasoning that earns them.

The routing decision is made per workspace, not per company and not globally. A workspace that drafts and verifies content on a schedule can run entirely on the local model; a workspace doing fresh multi-step research can stay on a frontier model; the post-completion reviewer can run on a different endpoint than the worker it reviews. That flexibility is the whole point of owning the serving layer. I wrote up the cost side separately in what an AI agent actually costs: model mix and prompt caching are the two dials that move the inference bill, and a local endpoint gives you both.

There's a second reason that matters more to the companies we run inside: their data stays on hardware we own. The document archive described below never left the building to be read.

The hardware

Two NVIDIA GeForce RTX 5090s (32,607 MiB each, 64 GB aggregate), an AMD Ryzen 9 7950X, 128 GB of system RAM (124 GiB as the OS reports it), driver 595.58, CUDA 13.2. Owned, on our own network, not rented. That's the entire inference tier today.

Consumer cards have real limits (no NVLink, 32 GB each, PCIe between them), and the configuration below is shaped by them. It's also the reason the numbers are interesting: this is what a small company can put on a desk, and it carries production.

The serving configuration

vLLM 0.27.0 serving unsloth/Qwen3.8-27B-NVFP4 with tensor parallelism of 2, a 262,144-token context window, an FP8 KV cache, prefix caching, priority scheduling, and 64 GiB of CPU memory as a second KV-cache tier. The model is exposed with the Qwen3 reasoning parser and the qwen3_xml tool-call parser, so agents get native tool calling.

# the production command, host and port removed
vllm serve unsloth/Qwen3.8-27B-NVFP4 \
  --served-model-name qwen3.8-27b \
  --tensor-parallel-size 2 \
  --max-model-len 262144 \
  --max-num-seqs 12 \
  --gpu-memory-utilization 0.95 \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --mamba-cache-mode align \
  --watermark 0.08 \
  --scheduling-policy priority \
  --max-num-batched-tokens 6272 \
  --scheduler-reserve-full-isl \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --kv-transfer-config '{"kv_connector":"OffloadingConnector","kv_role":"kv_both",
    "kv_connector_extra_config":{"spec_name":"TieringOffloadingSpec",
    "cpu_bytes_to_use":64424509440,"offload_prompt_only":false}}'

Three choices carry most of the weight. FP8 KV cache halves the memory each cached token costs, which is what makes a 262k window sane on 64 GB. KV offloading to CPU through vLLM's OffloadingConnector with the TieringOffloadingSpec gives the cache a 64 GiB second tier in system RAM, so prefixes evicted from the GPU come back from RAM instead of being recomputed (verified on 2026-09-12 against docs.vllm.ai: cpu_bytes_to_use is the required primary-tier size, kv_role is kv_both). Priority scheduling lets production requests jump ahead of background work on the same endpoint, which the next section relies on.

--scheduler-reserve-full-isl makes the scheduler check that a request's full input length fits in the KV cache before admitting it, which stops long agent prompts from thrashing the cache under chunked prefill (it defaults to on in current vLLM; it's spelled out here because it matters). Twelve concurrent sequences is the honest ceiling for this memory budget with 30k-token prompts.

Two weeks of production, measured

From the serving container's start on 2026-08-29 to the reading on 2026-09-12: 28,095 completed requests, 860.5 million prompt tokens processed, 28.6 million tokens generated, and an 82.6% prefix-cache hit rate. Zero requests finished with an engine error.

Counter (vLLM /metrics, cumulative)	Value	Note
Completed requests	28,095	26,918 stopped naturally, 1,177 (4.2%) hit their token limit
Prompt tokens	860.5M	mean 30,100 per request
Generated tokens	28.6M	mean 983 per request
Prefix-cache hit rate	82.6%	712.8M hits / 863.4M queries
Mean time to first token	1.44 s	40,760 s over 28,372 requests
Mean end-to-end latency	17.5 s	agent turns, including generation
Engine-side errors or aborts	0	client-side timeouts aren't in this counter

Read the ratio, not the totals. Thirty prompt tokens in for every one out is what agents look like, and it's why the prefix cache is the single most important number on the table. With 82.6% of prefill served from cache, a 30,000-token request costs the GPUs a few thousand tokens of real compute. Without it, this machine couldn't carry the load at all.

What the table doesn't contain is tokens per second. I haven't run a controlled generation benchmark on this box, and dividing 28.6M generated tokens by wall-clock time would produce a number that measures the traffic pattern, not the hardware. When there is a proper benchmark it will get its own post.

What the GPUs actually do

Two kinds of work share the endpoint: agent workspaces from our platform, routed through a gateway, and a document digitizer for an agricultural cooperative's paper archive that calls the vision head of the same model directly. As of 2026-09-12 the digitizer has transcribed and classified 2,526 documents, 7,183 pages, on these two cards.

The digitizer is the more instructive workload because it's boring. Every fifteen minutes a timer wakes it at low priority (Nice=10), it takes up to forty new documents, renders each page to an image and asks the model for a transcription with thinking disabled and temperature 0, then asks for typed fields (document number, date, net, VAT, total, partner id, currency) using vLLM's JSON-schema-constrained output. A value is written into the archive only if its printed form appears literally in the transcription. The model may not compute or guess; a misread handwritten digit in the pilot produced a 2018 date on a 2026 receipt, so dates are never written automatically. Printed text needs almost no correction; handwriting is approximate and goes to a review queue.

That pattern, a local model doing the reading and deterministic code deciding what counts, is the same one our agents run under: the model proposes, validators check, humans handle what neither can prove. It also shows the scheduling story in miniature. The digitizer fills idle capacity; when an agent workspace sends a request it takes priority and the batch job simply runs a little slower.

What I'd tell someone doing this

Serve for the prefix cache first, memory second, throughput third. Agents repeat themselves; a configuration that keeps prefixes warm beats one that maximizes raw tokens per second, every time.

NVFP4 on consumer Blackwell works today with vLLM; a 27B model in 64 GB with a 262k window and room for twelve sequences wouldn't be possible at BF16 on these cards. FP8 KV plus CPU tiering buys context, not speed, so treat them as the price of the long window and skip both if your prompts are short.

• Turn thinking off for extraction. The same model serves agents with reasoning on and the digitizer with reasoning off per request; vision transcription gets neither better nor cheaper with a thinking trace.
• Priority scheduling is how one box serves two masters. Without it, a forty-document batch would sit in front of a human waiting in Slack.
• Measure with the engine's own counters. A provider dashboard's p50 throughput, or a spreadsheet estimate, is not a measurement. /metrics is.
• Know where it hurts. Twelve concurrent sequences, a first token that arrives slowest when a prefix isn't cached (the 1.44 s mean hides that), and no interconnect between the cards. Those are the limits of the machine, and they're the reason the next section exists.

What's next, and where NVIDIA comes in

The local model is being specialized on our own production trajectories. The first adapters, trained on rented GPUs off the production rig, improved verified recovery on a frozen evaluation set from 31 to 38 of 73 decisions and cut exact repeats of failed actions from 20% to 15%, while clean-decision retention regressed. Real movement, not yet a production candidate.

The next stage is the one where the quantization and the training have to meet. NVIDIA's Model Optimizer documents quantization-aware distillation as its "recommended strategy for accuracy recovery after quantization" and reports that "full-precision fine-tuning followed by QAD delivers the best accuracy, especially at aggressive quantization levels (e.g., NVFP4)" (Model Optimizer llm_qat README, read 2026-09-12). That is exactly the sequence we need: post-train in high precision, quantize to NVFP4, and evaluate the deployed checkpoint on the same frozen suite as the trained one, so a regression is blamed on the right stage.

Beyond that: multi-GPU serving for long-context agent loops that keeps tenants isolated, a small verifier model that sits between a local worker and delivery, and a path from two GPUs to an owned inference tier that still keeps training separate from production. Those are engineering problems, and they're the ones we'd most like to work on with the people who build the hardware. The platform this all serves is described on the platform page; the agents themselves run through the Claude Agent SDK inside isolated containers, and the local endpoint is just one of the places they can be pointed.

Common questions

Can two RTX 5090s really serve a 27B model?

Yes. Qwen3.8-27B in NVFP4 fits across two 32 GB RTX 5090s with tensor parallelism of 2, an FP8 KV cache, and a 262,144-token context window. It has served production agent traffic continuously since late August 2026.

Why not send everything to a frontier API?

Because most agent traffic is repeated, long-context, tool-bound work where a specialized local model is good enough and far cheaper per token, and because company data staying on owned hardware is a feature. Frontier models stay in the loop for the hard reasoning and for verification; routing is decided per workspace, not globally.

What does the 82.6% prefix-cache hit rate mean?

Of every token vLLM was asked to prefill in the last two weeks, 82.6% was already in the KV cache from an earlier request. Agent prompts share long prefixes (system prompt, tool schemas, memory), so the cache turns a 30,000-token request into a few thousand tokens of new compute.

Is this a benchmark?

No. Every number here is a cumulative production counter read from vLLM's /metrics endpoint on 2026-09-12, covering all traffic since the serving container started on 2026-08-29. There is no tokens-per-second figure because we have not run a controlled benchmark, and a mean over a mixed workload would not be one.

If there's one thing to take away: a 27B model on two consumer GPUs is a production inference tier when the workload is agents, because agents repeat themselves and the cache does the rest. The hard part isn't serving the model. It's deciding, per workspace, which work it's allowed to do.