Qwen 3.6 35B on DGX Spark: Tokens Per Second Benchmarks
Real DGX Spark benchmarks for Qwen 3.6 35B-A3B: about 30 to 120 tokens per second, why concurrency is the ceiling, and when to move to datacenter GPUs.
Qwen 3.6 35B on DGX Spark: Tokens Per Second Benchmarks
On an NVIDIA DGX Spark, Qwen 3.6 35B-A3B decodes at roughly 28 to 30 tokens per second with a standard vLLM FP8 setup, and 97 to 120 tokens per second with NVFP4 quantization and an optimized engine. That single-stream speed is genuinely usable for one developer, but it is not a production serving number.
The reason it runs well at all is architecture. Qwen 3.6 35B-A3B is a mixture-of-experts model that activates only about 3 billion of its 35 billion parameters per token, so a dense 70B model on the same box manages just 2.7 tokens per second by comparison. The real limit on the Spark is concurrency: its memory bandwidth caps aggregate throughput long before you reach multi-user production load. Here are the real numbers and where the ceiling actually is.
Tokens per second by setup
The headline figure depends almost entirely on quantization and the inference engine. Here is what independent benchmarks report for Qwen 3.6 35B-A3B on a single DGX Spark, single stream:
| Setup (quantization / engine) | Single-stream decode | Notes |
|---|---|---|
| Standard vLLM, FP8 | ~28-30 tok/s | About 32% of the ~91 tok/s theoretical bandwidth ceiling |
| FP8 + speculative decoding (MTP-3) | ~64 tok/s | Multi-token prediction |
| DFlash build | ~50 tok/s | Tuned community build |
| NVFP4 quantization | ~97 tok/s | Roughly doubles plain FP8 |
| Atlas engine, NVFP4 | ~120 tok/s | Held across a 1M-token context |
The spread is wide because decode on the Spark is bound by memory bandwidth, not compute. Anything that reduces the bytes read per token (lower-precision quantization, speculative decoding, leaner kernels) moves the number. NVFP4 roughly doubles the plain FP8 result, and the Atlas engine, a pure-Rust inference stack with custom kernels, holds about 120 tokens per second even across a one-million-token context (Flowtivity).
Why a 35B model runs at all on a 273 GB/s box
The DGX Spark pairs a GB10 Grace Blackwell chip with 128 GB of unified LPDDR5X memory, and that memory tops out at about 273 GB/s of bandwidth shared between CPU and GPU (LMSYS). For text generation, that bandwidth is the whole story.
Inference runs in two phases with very different profiles. Prefill, reading your prompt, is compute-bound and fast: the Spark pushes 800 to over 2,000 tokens per second there. Decode, writing the answer one token at a time, is memory-bound, because every token requires reading the active model weights out of memory. That is where the 273 GB/s ceiling bites.
This is why the mixture-of-experts design matters so much. A dense 70B model has to read all 70 billion weights for every token, which drops it to about 2.7 tokens per second on the Spark (LMSYS). Qwen 3.6 35B-A3B reads only its roughly 3 billion active parameters per token, moves far less data, and lands an order of magnitude higher. The MoE architecture is what makes a 35B-class model pleasant to use on a desktop at all.
Which setup should you actually run?
For day-to-day local use, NVFP4 is the setup worth chasing. It roughly doubles the throughput of a plain FP8 build and pushes Qwen 3.6 35B-A3B into the 97 to 120 tokens per second range, fast enough that the model feels responsive in a chat loop. The trade-off is setup effort: NVFP4 and engines like Atlas are newer and less turnkey than a stock vLLM install, so expect to spend time on quantization and kernel compatibility.
If you just want the model running with the least friction, standard vLLM in FP8 gets you to about 30 tokens per second and works out of the box. That is slower than the headline numbers, but for single-user development, code completion, or testing prompts it is perfectly workable. Speculative decoding, or multi-token prediction, is a middle path that lifts FP8 to around 64 tokens per second without changing quantization.
Whichever you pick, the single-stream figure is what you are optimizing. None of these setups change the concurrency story below, because that ceiling is set by the hardware, not the software.
What happens under concurrency
Single-stream speed is the number that gets quoted. It is also the most flattering one. The moment you serve more than one request at a time, every stream competes for the same 273 GB/s pipe, and the math changes.
With a standard vLLM FP8 deployment, aggregate throughput rises to about 156 tokens per second at 32 concurrent streams, but it scales sub-linearly and per-user latency degrades hard. At 32 streams, time-per-output-token reaches about 195 milliseconds, roughly 5 tokens per second per user (rikkarth benchmark). That is fine for an overnight batch job and far too slow for interactive chat. NVFP4 does better, reaching about 322 tokens per second aggregate at 8 concurrent streams, but that is still only about 42 tokens per second per user (NVFP4 concurrency benchmark), and the curve keeps flattening as you add load.
So the honest summary is not “it breaks above four users.” It is that the Spark’s aggregate ceiling lives in the low hundreds of tokens per second, and you trade per-user responsiveness to get there. For a developer, a small research team, or a privacy-sensitive single-tenant workload, that is plenty. For production traffic with dozens or hundreds of concurrent users and a latency target, it is nowhere near enough.
DGX Spark is a developer box, not a production server
None of this is a knock on the Spark. It is one of the best local AI development machines you can put on a desk. The 128 GB of unified memory lets you load models that will not fit on a consumer GPU, it sips power, and it keeps your data entirely local (NVIDIA). For prototyping against Qwen 3.6, fine-tuning experiments, offline research, or a single power user, it is close to ideal.
The mistake is treating that single-stream benchmark as a serving number. Production inference is a concurrency problem, and concurrency on the Spark is gated by a memory bus that a datacenter GPU beats by more than ten times. An H100-class accelerator with HBM moves north of 3 TB/s, so it sustains high single-stream speed and high concurrency at the same time, which is exactly what multi-tenant serving needs. The Spark is where you build and test. It is not where you put the model when real traffic arrives.
Serving Qwen 3.6 in production
When you outgrow the desk, you do not need to buy a rack. Tessera serves Qwen 3.6 35B-A3B on datacenter GPUs with high-bandwidth memory, so the per-user latency and concurrency limits you hit on a Spark do not apply. You get sustained throughput under concurrent load, an OpenAI-compatible API so existing client code works with a base-URL change, and flat monthly pricing instead of per-token metering that turns a traffic spike into a surprise bill.
Tessera runs open-source models only, on dedicated EU and LATAM GPU clusters, with EU data residency for teams that need it. If you are benchmarking Qwen 3.6 locally to decide whether it is good enough, it almost certainly is: see the full Qwen 3.6 35B benchmarks and the coding-specific results for the quality side. The Spark answers “is this model fast enough for me.” Tessera answers “is it fast enough for everyone, at once.”
FAQ
How many tokens per second does Qwen 3.6 35B run on a DGX Spark?
About 28 to 30 tokens per second single-stream with a standard vLLM FP8 setup, rising to roughly 97 to 120 tokens per second with NVFP4 quantization and an optimized engine like Atlas. Decode is limited by the Spark’s 273 GB/s memory bandwidth.
Why is Qwen 3.6 35B so much faster than a 70B model on the same hardware?
Qwen 3.6 35B-A3B is a mixture-of-experts model that activates only about 3 billion of its 35 billion parameters per token, so it reads far less data from memory per token. A dense 70B model reads all of its weights every token and drops to about 2.7 tokens per second.
Can a DGX Spark serve Qwen 3.6 to multiple users at once?
Only at small scale. Aggregate throughput tops out in the low hundreds of tokens per second (about 156 tok/s at 32 streams on FP8, about 322 tok/s at 8 streams on NVFP4), and per-user latency degrades as you add streams. It works for a developer or a small team, but production multi-user serving needs datacenter GPUs with high-bandwidth memory.
Is the DGX Spark good for running Qwen 3.6 locally?
Yes, for a single user or a small team it is excellent. The 128 GB of unified memory comfortably holds Qwen 3.6 35B-A3B, single-stream speed is responsive with NVFP4, and your data never leaves the machine. It stops being the right tool when you need to serve many concurrent users with a latency target, which is a job for datacenter GPUs.