δΈ­ζ–‡

How to Run Qwen3.5-9B Locally: From iGPU to RTX 5090, Every GPU Tier Covered

Β· By AIX Cove Β· Reviewed by AIX Cove Β· ai-tutorials-how-tos
How to Run Qwen3.5-9B Locally: From iGPU to RTX 5090, Every GPU Tier Covered

If the Qwen3.8-27B is the muscle car of local AI β€” brilliant but thirsty β€” the Qwen3.5-9B is the hot hatch: nearly as clever per dollar, and it starts on almost anything with a plug. Released March 2, 2026 under Apache 2.0, this 9B dense model pairs a hybrid Gated DeltaNet + Gated Attention architecture (only 8 of 32 layers carry a growing KV cache) with a native vision encoder, 262K context (YaRN-extended to ~1M), multi-token prediction, and 201 languages. The headline numbers: MMLU-Pro 82.5, GPQA Diamond 81.7, LiveCodeBench v6 65.6, MMMU-Pro 70.1 β€” beating the GPT-OSS-20B column in Qwen's own table on most rows while fitting in under 6 GB at 4-bit.

This guide gives every hardware tier a concrete answer: which quant to download, which context to set, what speed to expect, and the two or three switches that actually matter. All commands verified against the current llama.cpp/Ollama/MLX ecosystems.

Know the model before you download it

  • Hybrid attention = cheap context. The 8Γ—(3Γ—DeltaNetβ†’1Γ—Attention) layout means KV cache costs only ~32 KiB/token β€” a 32K context fits in ~1 GB, not the 8 GB a full-attention 9B would demand. This is why the model feels "impossible" on 8 GB cards.
  • Natively multimodal. Text, images, and video from the same weights β€” no separate VL adapter. In llama.cpp you add the vision encoder with --mmproj; in Ollama and MLX builds it's baked in.
  • Thinking is OFF by default in GGUF builds. Unlike the 27B (which thinks by default and eats your context), the small-model GGUF template ships with enable_thinking undefined, which renders an empty think block. Turn it on with --chat-template-kwargs '{"enable_thinking":true}'. Sampling per Qwen: thinking = temp 0.6/top_p 0.95/top_k 20; non-thinking = temp 0.7/top_p 0.8/top_k 20. Qwen3's /no_think soft switch does NOT work on 3.5.
  • Dense means all 9B run per token. Unlike the 35B-A3B MoE (3B active), you pay full bytes per token β€” speed scales purely with memory bandwidth. That's the physics behind every number below.

Quant selection: the whole ladder, honestly

From the Unsloth Dynamic 2.0 GGUF repo (the reference conversions):

  • UD-Q4_K_XL / Q4_K_M (~5.3–5.7 GB) β€” the default for a reason. Perplexity +3.5% vs FP16, MMLU βˆ’1.4, GSM8K βˆ’2.7. Fits 8 GB cards with room for context.
  • Q5_K_M (~6.6 GB) / Q6_K (~7.4–7.6 GB) β€” the 12 GB tier's picks. Q6 is effectively lossless (+0.4% PPL).
  • Q8_0 (~9.5 GB) β€” reference quality (+0.05% PPL). The 16 GB sweet spot; also correct for 12 GB cards that want max fidelity over context.
  • UD-Q3_K_XL (~4.5 GB) β€” the 6 GB escape hatch. +8% PPL is a real hit; acceptable for chat, painful for code.
  • UD-Q2_K_XL / IQ2 (~3.2–3.7 GB) β€” last resort. +18% PPL, GSM8K βˆ’24. On a 9B this is noticeably damaged; treat as an experiment.

One rule to remember: at equal memory, a higher-quant smaller model beats a lower-quant bigger one. A Q8 9B outperforms a Q3 27B at the same ~9 GB footprint β€” the 27B has no room left for context.

Hardware tiers: what to run and what to expect

Integrated graphics / CPU-only (no discrete GPU)

Verdict: works, with calibrated patience. The model runs from system RAM via llama.cpp Vulkan/CPU.

  • 16 GB RAM machine: UD-Q3_K_XL (4.5 GB), 4–8K context, expect 2–5 tok/s on DDR4, 5–8 on DDR5.
  • 32 GB RAM: Q4_K_M, 8K context, ~5–10 tok/s depending on RAM speed. Bandwidth is everything β€” DDR5-6000 vs DDR4-2666 is roughly a 2Γ— gap.
  • Modern iGPU (Radeon 780M/8060S, Intel Arc): Vulkan backend, Q4_K_M. Ryzen AI MAX 395-class hardware measures ~32 tok/s at Q4_K_M β€” genuinely usable. Ordinary laptop iGPUs land in the 5–12 tok/s range.

8 GB VRAM (RTX 4060/4070 Laptop, 5060, old 1070/1070 Ti-class)

Verdict: the surprise sweet spot β€” this model was practically tuned for these cards.

  • Quant: UD-Q4_K_XL (~5.3 GB). All 32 layers on GPU (-ngl 99), ~2 GB left for KV and vision.
  • Context: 8–16K comfortable; with llama.cpp's TurboQuant KV (q8_0 KV cache + flash attention) you can push 32K+ and even spill context to system RAM β€” the hybrid architecture makes this cheap (~32 KiB/token).
  • Speed: ~19–29 tok/s measured class on 4060 8GB; older 1070-class cards land lower (~12–15) via CUDA on Pascal β€” still usable.
  • Community consensus: "Qwen3.5:9b running on 8GB VRAM is insane" β€” it's the best-quality model that fully fits these cards.

12 GB VRAM (RTX 3060 12GB, 4070, 5070, 4080 Laptop)

Verdict: the comfort tier. Q6_K (7.4 GB) for coding/reasoning precision or Q8_0 (9.5 GB) for near-lossless β€” both fit with 16–32K context. Expect 30–48 tok/s. Measured reference: RTX 4070 Ti 12GB runs the 9B comfortably; laptop 4080-class ~48 tok/s at Q6.

16 GB VRAM (4060 Ti 16GB, 5060 Ti 16GB, 5080, 4080)

Verdict: reference-quality everything. Q8_0 + 32K context + vision projector loaded, still ~4 GB spare. Expect ~19 tok/s measured on 4060 Ti 16GB at Q8 (288 GB/s bandwidth card), scaling to 45–60 tok/s on 4080/5080-class bandwidth. This is also the tier where the model makes an excellent always-on coding agent via llama-server + OpenAI-compatible API.

24 GB VRAM (3090, 4090, 5090 Laptop)

Verdict: overkill, gloriously so. Q8_0, 128K+ context, vision loaded. Measured: ~60 tok/s on 4090 at Q8_0, ~92 at Q4_K_M. At this tier consider running the 9B as a fast agent/summarizer alongside a 27B for heavy reasoning β€” the two complement each other.

32 GB (5090) and beyond

RTX 5090: ~80 tok/s at Q8_0, ~122 at Q4_K_M. Server-class Blackwell datacenter cards do 200+ but nobody buying a 5090 needs to hear that. The 9B at this tier is a latency toy β€” sub-15ms first-token for agent loops.

Mac (Apple Silicon)

Verdict: MLX first, GGUF second.

  • 16 GB Mac (M4 base class): MLX 4-bit (~5.6 GB disk, ~6.2 GB peak). Measured ~20 tok/s on M4 10-core β€” about right for 120 GB/s bandwidth. Usable for chat; batch 4Γ— hits ~65–79 tok/s aggregate.
  • 24–36 GB (M4 Pro/Max, M3 Pro): 8-bit MLX (~9.6 GB) is near-lossless; expect 25–35 tok/s single-stream. This is the best local assistant hardware-per-dollar in the ecosystem right now.
  • 48 GB+ (M Max/Ultra): Q8 or BF16, and you can run the 9B as a background service alongside bigger models.

mlx-community/Qwen3.5-9B-MLX-4bit is the standard conversion; mlx-works/oQ4e-mtp adds native MTP acceleration (43.5 tok/s on M5 Air-class hardware with TurboQuant KV).

Serving: the three commands that matter

llama.cpp (recommended, most control):

llama-server -hf unsloth/Qwen3.5-9B-GGUF:UD-Q4_K_XL \
  --mmproj unsloth/Qwen3.5-9B-GGUF:mmproj-F16.gguf \
  -ngl 99 -c 32768 --port 8080 \
  --chat-template-kwargs '{"enable_thinking":true}'

Ollama (easiest): ollama run qwen3.5:9b β€” pulls a 6.6 GB Q4_K_M with vision baked in. Note: Ollama's Qwen3.5 support has had documented performance regressions vs raw llama.cpp (CPU-fallback bugs, 30–60% slower decode on some builds); if speed matters, prefer llama.cpp directly.

vLLM (serving, 24 GB+): vllm serve Qwen/Qwen3.5-9B --max-model-len 262144 --reasoning-parser qwen3 β€” add --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}' for MTP, or --language-model-only to skip the vision encoder and free KV memory.

Uncensored variants: what exists for the 9B

The 9B has a lighter uncensoring ecosystem than the 27B β€” fewer variants, less forensic scrutiny β€” but the options are real:

  • Huihui-ai abliterated β€” the classic method, in safetensors/GGUF/MLX-4bit. Community testing found the Ollama huihui_ai/qwen3.5-abliterated build "NOT actually uncensored" β€” prefer the HuggingFace files directly.
  • HauhauCS Aggressive (0/465 claimed) β€” same Aggressive profile as the 27B line, GGUF Q4_K_M (5.3 GB) through BF16 (17 GB), vision encoder included. Same caveat as always: the claims are louder than the documentation.
  • lukey03 abliterated + QLoRA β€” the most documented small-model approach: 3-pass abliteration killed 13/18 test refusals; a tiny QLoRA pass (20 examples, 45 seconds on an H100) cleared the remaining 5 stubborn categories. 18/18 post-merge, beating Dolphin-Mistral 7B on the same benchmark.

The selection framework from our 27B guide applies unchanged β€” prefer publishers who document KL and capability deltas, run Q5+ when you can, verify behavior at your actual quant, and diff any chat template against stock. The 9B is more sensitive to quant-level behavior drift than the 27B (less redundancy per layer), so "verify at your quant" matters double here.

Common failure modes

  • "Slower than advertised": you're on Ollama with a known Qwen3.5 regression, or your context is set to 262K (KV cache spilling to RAM). Set 8–32K explicitly.
  • "Thinking won't turn off": known llama.cpp issue #20182 β€” enable_thinking:false in request payloads gets silently ignored on some builds. Workaround: --reasoning-budget 0 at server start, or the chat-template-kwargs form.
  • "Images don't work": you forgot --mmproj, or mixed an F16 projector with a BF16 model (n_embd mismatch error). Match the projector dtype to the text model.
  • "OOM at 262K context": marketing context vs physical context. The hybrid cache is cheap but not free (~32 KiB/token β†’ 262K β‰ˆ 8 GB of cache alone). Budget context like VRAM, not like a wish.

Which tier should you buy / run?

The 9B's honest positioning in September 2026: it is the best model that fully fits 8 GB, the best quality-per-watt always-on assistant for 12–16 GB, and a latency play for 24 GB+ rigs that also own a bigger model. If you have 16 GB+ and want maximum capability per query, the 27B guide's recommendations stand β€” but for agents, routers, classification, vision OCR, and anything where tokens/second and RAM footprint dominate, the 9B is the smart money.

Sources & further reading

Sources: official docs & pricing pages, hands-on testing where noted, and community feedback. Prices verified August 2026 and may change.