How to Run Qwen3.8-27B Locally: A GPU-by-GPU Setup Guide

2026-09-10 · By AIX Cove · Reviewed by AIX Cove · ai-tutorials-how-tos

How to Run Qwen3.8-27B Locally: A GPU-by-GPU Setup Guide

Quick verdict: Qwen3.8-27B is a dense 27B vision-language model released August 14, 2026 under Apache 2.0, and the whole guide boils down to one decision: pick the quantized file that fits your memory, then cap the context window to what is left over. With 16 GB of VRAM or a 32 GB unified-memory Mac you get the full experience. Below that, you trade quality (smaller quants) or speed (CPU offload), and this article tells you exactly which trade each GPU makes.

It matters because this model rewrote the local-AI baseline: 73.0 on Terminal Bench 2.1, 61.7 on SWE-bench Pro, native image and video input, and a 262,144-token context window — on hardware many readers already own. The deployment math is unusual, so let's do it once before the per-GPU configs.

Why this model is cheaper on memory than it looks

Qwen3.8-27B looks like a memory hog — 27B parameters at 16-bit would be roughly 54-56 GB. But only 16 of its 64 layers use conventional attention; the other 48 use Gated DeltaNet, a linear-attention variant that keeps a small fixed-size state instead of a growing KV cache. Community measurements put the KV cost at about 64 KB per token — roughly a quarter of a conventional dense 27B — which is why a 262K context window is even discussable on consumer cards.

The practical consequence: your real constraint is weight size plus a modest KV budget. Get the weights to fit and context comes surprisingly cheap. Fail to fit them, and no flag saves you.

Full precision (BF16): ~55 GB — multi-GPU or 64 GB+ unified memory only.
Q8/FP8: ~29-31 GB — one 32 GB card, or 48 GB Mac.
4-bit (Q4/IQ4): ~14-18 GB depending on the exact quant — the sweet spot.
3-bit and below: ~10-13 GB — quality drops, but 16 GB cards and 24-32 GB Macs come alive.

The quantization cheat sheet

Community quantizers (Unsloth's Dynamic series is the current quality benchmark) publish measured file sizes. Two calibration points from side-by-side perplexity tests: Unsloth's UD-Q4_K_XL retains about 96% top-1 agreement with the original weights, and the quality curve is steep below ~14 GB and nearly flat above ~22 GB.

UD-Q4_K_XL / Q4_K_M: ~16.5-17.9 GB. The 24 GB default. Best size/quality balance.
UD-IQ4_XS: ~14.3-15.1 GB. The 16 GB pick — 4-bit quality with actual context room.
UD-Q3_K_XL: ~12.2-13.1 GB. Fits 16 GB with lots of context headroom; surprisingly close to Q4 in blind tests.
UD-Q2_K_XL: ~9.2-9.8 GB. Emergency tier for 12 GB cards and 16 GB Macs. Read again before you rely on it.
UD-Q6_K / Q8: 22-31 GB. For 32 GB cards and 48 GB+ Macs; measurable but small gains.

The one setting that ruins more setups than any GPU shortage

Qwen3.8 thinks by default, at the most expensive effort level (xhigh). Left alone it will burn tens of thousands of reasoning tokens on a one-sentence question, then crash into the context ceiling and return an empty answer that looks like a runtime bug. Before optimizing anything else, set reasoning_effort to medium (near-free quality-wise) or low, and disable thinking entirely for simple requests via enable_thinking: false. Community tests show the xhigh default can cost 5-10x the tokens of the answer itself on trivial prompts.

Sampling profiles worth copying: thinking mode uses temperature=1.0, top_p=0.95, top_k=20; instruct mode uses temperature=0.7, top_p=0.80, top_k=20 with a presence penalty of 1.5.

Pick your path: the hardware tiers

Speed expectations below come from published community benchmarks (llamabench.ai aggregates, Tom's Hardware, per-GPU deep dives), usually with llama.cpp and the model's built-in MTP speculative decoding. Your numbers will vary with context length and settings, but the orderings hold.

Integrated graphics and 8 GB laptops: possible, not pleasant

If your machine has only integrated graphics (AMD or Intel iGPU) or a discrete 8 GB laptop card, you will not fit a good quant in fast memory — the model will run mostly from system RAM no matter what you do. Two honest paths:

With 64 GB of system RAM: run the llama.cpp Vulkan or SYCL backend with a 2-bit to 3-bit quant (10-13 GB) and accept roughly 3-8 tokens/second. Usable for patient single questions, painful for agent work.
With 16-32 GB of RAM: even the small quants thrash. The realistic answer is a smaller model (a 7B-14B class, or a small MoE) and treating Qwen3.8-27B as a cloud/API experience. Fighting for 1-2 tokens/second is not worth your evening.

On laptops specifically: an RTX 4060 Laptop (8 GB) or 5060 Laptop (8 GB) behaves like the integrated case above — expect single-digit tokens/second with Q2/Q3 quants and a short context. A 5070 Laptop (8 GB) lands in the same 3-5 t/s range reported by owners. If local AI matters to you, 16 GB of VRAM should be the minimum spec on your next machine.

Mac (Apple Silicon): the quiet winner

Unified memory changes the math — there is no separate VRAM pool, so the question is only how much RAM your machine carries and what else it needs. Use the MLX builds (via LM Studio or mlx-lm) rather than GGUF; they are consistently faster on Metal.

24 GB Mac (base M-series): 2-bit to 3-bit quants run, slowly (~5-8 t/s). Works for experiments, not for daily driving.
32-36 GB: IQ4_XS/Q4_K_M (~15-16.5 GB) fits with a real context window; expect roughly 10-14 t/s depending on bandwidth.
48 GB (M4 Pro class and up): Q4 fully resident, ~17 t/s; Q6/Q8 possible with tight context. This is the entry point we recommend.
64-128 GB (M4/M5 Max, Studio): the comfortable home for this model — 27-30 t/s at 4-bit, Q8 available, and long-context sessions that behave.

Set the MLX memory limit explicitly (the wired_limit in LM Studio or the env variable for mlx-lm) so macOS does not evict weights mid-session, and keep the context at 16-32K unless you specifically need more.

16 GB desktop and laptop cards: 4060 Ti 16GB, 5060 Ti 16GB, 4070 Ti Super, 5080

This is the tier where the model's linear-attention design earns its keep. The correct quant is UD-IQ4_XS (~14.3 GB) or UD-Q3_K_XL (~12.2 GB) — never Q4_K_M, which fills the card and starves the context.

4060 Ti 16GB / 5060 Ti 16GB: IQ4_XS at 16-32K context runs around 17-20 t/s with MTP; Q3_K_XL buys you 64K+ context at similar speed. The best value tier for this model.
5080 (16 GB): one documented deep-dive runs UD-Q3_K_XL at a 90K context at 130 t/s prefill / 33-57 t/s decode fully on-GPU by moving the display to the iGPU and quantizing the KV cache to q8_0/q4_0. With Q4-class quants expect 16-28 t/s decode depending on context.
4080 / 4070 Ti Super (16 GB): similar recipe to the 5080, slightly lower numbers; the 4070 Ti Super's extra bandwidth helps at long context.

Two 16 GB tricks worth stealing: move your display output to the integrated GPU (frees 300-500 MB of VRAM that Windows/Linux desktops hold), and quantize the KV cache (--cache-type-k q8_0 --cache-type-v q8_0 in llama.cpp) — it roughly halves KV memory with no quality loss anyone has measured.

12 GB cards (4070, 5070 desktop): the awkward middle

Q4-class weights do not fit; Q3 barely does with almost no context. You are choosing between UD-Q3_K_XL at ~4-8K context (functional, cramped) or a 2-bit quant with real context but noticeably degraded quality (measured around 3-6 t/s). If this is your card, honestly evaluate whether Qwen3.6-27B at Q4 or a strong 14B model serves you better. The 27B is worth its memory demands.

24 GB cards: 4070 Ti, 5070 Ti, 3090, 4090, 5090 Laptop

This is the model's intended home and the config we recommend as default: UD-Q4_K_XL (17.9 GB) at 32K context, MTP speculative decoding on, KV cache at q4_0.

RTX 4090: ~46 t/s baseline decode, 125+ t/s with MTP speculative decoding — the fastest 24 GB card.
RTX 3090: ~40 t/s baseline; with MTP and KV quantization the community has pushed sustained agentic workloads to ~60-114 t/s. The second-hand value king.
5090 Laptop (24 GB): thermally limited versus the desktop 4090 but still solidly in the 30-40 t/s class — the right laptop choice if you want this model on the go.
5070 Ti / 4070 Ti (16 GB variants excluded): note that the 5070 Ti desktop ships with 16 GB, not 24 — it belongs to the tier above. Only the 3090/4090/5090 Laptop here get the full Q4 experience.

llama.cpp launcher for this tier:

llama-server -m Qwen3.8-27B-UD-Q4_K_XL.gguf \
  -ngl 99 -fa on --jinja -c 32768 \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  --spec-type draft-mtp --spec-draft-n-max 2 \
  --temp 1.0 --top-p 0.95 --top-k 20

Measured on a 5090 (32 GB) this exact config delivers ~137 t/s decode at full quality; on a 4090 expect 80-125 t/s; on a 3090, 45-70 t/s.

32 GB cards: 5080 Super, 5090 desktop

The RTX 5090's 32 GB is the single fastest consumer way to run this model: UD-Q6_K (22 GB) or even Q8 fits with real context, and published benchmarks show 74-155 t/s decode depending on quant and speculation settings. The long-context picture is less rosy: at 256K, even a 5090 needs ~12 minutes for the first token — fine for batch document analysis, useless for chat. Keep interactive sessions at 64-128K.

Setup walkthrough: the three runtimes worth using

Option 1: Ollama (simplest)

# macOS/Linux/Windows
ollama run qwen3.8:27b          # 18 GB, Q4_K_M default, 256K context, vision enabled

Ollama picks sensible defaults and handles the vision adapter automatically. For smaller cards pull a specific quant from Hugging Face instead: ollama run hf.co/unsloth/Qwen3.8-27B-GGUF:UD-IQ4_XS. The tradeoff: less control over KV cache quantization and MTP, which matter exactly on the 12-16 GB tiers.

Option 2: LM Studio (best GUI, best for Mac MLX)

Download the app, search for "Qwen3.8-27B", pick the quant matching your tier from the table above, and set the context slider to 16-32K. On Apple Silicon choose the MLX build. On Windows with NVIDIA, enable Flash Attention in the model settings and set GPU offload to maximum. Its per-model settings UI makes the reasoning_effort and context decisions visible, which is half the battle with this model.

Option 3: llama.cpp (maximum control, fastest)

# Linux build with CUDA
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON && cmake --build build -j --target llama-server
# download + serve in one line
./build/bin/llama-server -hf unsloth/Qwen3.8-27B-GGUF:UD-Q4_K_XL \
  -ngl 99 -fa on --jinja -c 32768 --port 8080

The server exposes an OpenAI-compatible API at localhost:8080 that anything — chat UIs, coding agents, your own scripts — can call. This is the runtime every benchmark above used; if you want the published numbers, use this path.

Uncensored variants: what exists and what to expect

The base Qwen3.8-27B ships with standard safety alignment, and abliterated ("uncensored") derivatives appeared within days of the open-weights release. Four routes are worth knowing, in descending order of quality transparency:

Huihui-ai abliterated (safetensors): the classic abliteration method (orthogonalizing the refusal direction out of the residual stream), published as huihui-ai/Huihui-Qwen3.8-27B-abliterated. Also on Ollama directly: ollama run huihui_ai/Qwen3.8-abliterated.
JonathanColetti Uncensored (measured, GGUF + bf16): an abliteration-style derivative with unusually honest documentation — bf16 weights here and imatrix GGUFs here (IQ2_M through Q8_0, MTP head preserved and verified, vision projector included). The published refusal rate drops from 98/100 to 12/100 on their held-out harmful-prompt set, with a mean benchmark delta of just -0.5 points (MMLU 83.4 → 83.3, ARC-Challenge -1.2). The maintainer is explicit that refusals are "substantially reduced, not eliminated."
OrcaRouter Uncensored-FP8 (vLLM serving): an FP8 build matching the official quantization scheme so it serves through the identical vLLM kernel path at 262K context with tools and MTP intact — see orcarouter/Qwen3.8-27B-Uncensored-FP8 (GGUF conversions at chimingw's mirror). This is the route if you are serving with vLLM/SGLang rather than running llama.cpp.
HauhauCS Aggressive MTP GGUF (0/465 refusals, fastest): the most aggressive option — HauhauCS/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-MTP-GGUF reports zero refusals across 465 test prompts with "direct answers, minimal preamble." It also ships the best speculative-decoding story of any variant: the native MTP head is preserved in every GGUF, and the bundled FastMTP 32K sidecar claims up to 3.02x document generation versus no MTP (35.2% over standard embedded MTP). Full quant ladder from IQ4_XS (15.7 GB) to Q8_K_P (31.5 GB). Caveats: the published speed numbers come from a 96 GB RTX PRO 6000 Blackwell rig, so treat them as upper bounds on consumer cards; and "aggressive" means exactly what it says — quality/regression data beyond the refusal tests is thinner than the JonathanColetti release. If a Balanced variant of the Qwen3.8 release exists when you read this, HauhauCS's own guidance is that it is the safer default for reliability-critical agentic work. One more entry for the ledger: the independent forensics project behind the benchmark below found HauhauCS's Reaper tool is a Heretic fork with attribution stripped and relicensed, measured refusals and capability degradation contradicting the "lossless" claims on models it could reverse (Qwen3.5-27B: TruthfulQA −8.2pp, MMLU −1.9pp), and discontinued it from future comparisons; the GGUF-only Qwen3.8 release was not part of that table.

How to choose: the four questions that actually matter

Comparing uncensored models by refusal rate alone is like comparing cars by top speed — true and mostly useless. Cross-architecture research on abliteration tools (a December 2025 arXiv study covering Heretic, DECCP, ErisForge and FailSpy) and independent forensic benchmarks of the Qwen 3.6/3.8 variant ecosystem point to four questions worth asking before you pick:

1. How much did the edit shift the model? (KL divergence) — This is the single best predictor of quality damage. Abliteration works by subtracting a "refusal direction" from the model's internal representations; the less the overall token distribution moves, the more of the original model survives. JonathanColetti publishes this number (first-token KL 0.1191 at their chosen operating point, with a full trade-off table from 12 to 98 refusals) — a maintainer who publishes a KL/refusal curve is telling you they optimized instead of guessing. Independent forensics on the wider ecosystem found the best methods (Heretic-class) keep capabilities within ~1% of base, while the worst implementations have crashed MMLU by 6 points on some architectures. Method quality is variant-dependent: a tool that is gentle on one model family can damage another.
2. Was it measured at all, and was the measurement honest? — Refusal rate is cheap to claim (0/465 looks great until you ask what the 465 prompts were and whether "answered with a disclaimer" counted as non-refusal; research on automated refusal-scoring found marker-based counting can understate refusal by 20+ points because disclaimer-style answers still comply while hedging). The gold standard is a maintainer who publishes refusals AND a capability benchmark AND a KL number against the same base revision. JonathanColetti and independent forensics do; "0 refusals, no changes to capabilities" with no methodology is a marketing claim.
3. Does the behaviour profile match your workload? — HauhauCS's own guidance is the clearest framing: Aggressive (raw answer, no preamble) for users who specifically want that; Balanced (still 0/465 refusals, but reasons out loud with occasional brief disclaimers) for agentic coding, tool use and long-context work where stability matters more than tone. Refusal removal and answer style are independent axes — a model can be fully uncensored and still think before it answers, and for most real work that is what you want.
4. What happens at YOUR quant level? — Refusal-rate and benchmark numbers are usually measured at high precision (Q6+). The refusal direction edit interacts with quantization: the same abliterated model can behave differently at IQ4_XS and materially worse at 2-bit, and derivative maintainers rarely test below Q4. JonathanColetti explicitly tells users to evaluate behaviour at Q6_K/Q8_0 rather than assume it holds at IQ4_XS. If you are running 16 GB or smaller, your honest expectation should be "uncensored but somewhat less coherent than the published numbers suggest."

Practical defaults that fall out of the research: for a first uncensored model, pick the variant that publishes a KL/refusal trade-off table and capability deltas (currently JonathanColetti), at Q6 or above if your VRAM allows. If you specifically want zero-friction direct answers and understand the trade, HauhauCS Aggressive with its MTP speedups is the strongest pick on paper — just keep a Balanced or JonathanColetti fallback around for long agentic sessions. Treat any abliterated 2-bit quant as an experiment, not a tool. And whatever you pick, spot-check TruthfulQA-style behaviour (adversarial false-premise questions): research on abliteration's inherent costs shows factuality-under-pressure is the capability most consistently degraded by refusal-direction removal, even when general benchmarks look fine.

An 11-day, 167 GPU-hour shootout: 8 variants, one base, measured

In September 2026 the independent forensics project Abliterlitics published the first systematic side-by-side of eight Qwen3.8-27B uncensored variants against the stock base — full report here, discussion here — roughly 167 GPU-hours on a single RTX 5090 across eleven days: weight forensics, KL divergence, a 13-task benchmark suite, and HarmBench 400 (temperature 0, all arms pinned to the stock chat template). It is the closest thing this ecosystem has to ground truth, and it confirms — and complicates — everything above.

The scoreboard (HarmBench ASR, higher = more requests actually completed, judged):

orcarouter — 82.2%, the winner. Arditi-style edits at 131 matrices, layer 38. The only variant whose model card survived contact with the weights: all 4 claims verified exactly, every capability delta inside the card's ±1.3pt promise. Best copyright unlock in the set (39%). Weights here.
apostate — 78.7%, best value. A new KCRN method, just 41 verified edits, lowest measured KL (0.0439), capabilities nearly identical to base. Packaging quirks to know about: text-only re-save with no vision tower and no MTP head, stored FP16. Weights here.
huihui — 75.6%, reliable. The classic method holds up at 27B: clean unlock everywhere except copyright, where it stays at 3%. As linked above.
ultra_heretic — 70.5%. Heretic v2 with MPOA; works, but the heaviest truthfulness drop outside obliteratus and 118 soft refusals (96 of them copyright deflections). Weights here.
coder3101 — 70.0%. Vanilla Heretic. Its card undersold itself: the card says 33/100 refusals, the judge measured 5 explicit refusals in 400. Weights here.
blackfrost — 68.5%, sneaky packaging. The claimed rank-k "direction bank" is not in the weights (single direction, 100% rank-1), and the chat template ships a 1,457-character jailbreak system prompt injected into every conversation. The forensics verdict: the weights alone do reach the claimed refusal floor, but run it with the stock template only. Weights here.
obliteratus — 63.9%, avoid. The most aggressive edit in the panel (841 of 850 tensors touched) and the only variant that got meaningfully dumber: HumanEval −33.6pp, LAMBADA perplexity +20%, TruthfulQA −6.3pp, and 44.8% of responses never finish their think block inside a 15,360-token budget — a model that delivers the goods only inside an unterminated monologue is not a usable model. The "0% refusals" claim only holds if you score an unfinished thought as an answer. Weights here.
trohrbaugh — 57.5%, the safe pick. Last place is deliberate: 122 explicit refusals, the most surviving alignment. In exchange, the cleanest capability profile of all (GSM8K −1.2pp, HumanEval +3.1pp, the only calibrated KL — card 0.0535 vs measured 0.0586, a 9% gap) and the lowest loop rate (15.3%). This is the variant the report's author runs at home. Weights here.
base — 4.5%. A wall: 381 explicit refusals in 400, zero compliance on chem/bio, harassment, harmful content and copyright.

Five findings from the study that should change how you shop for uncensored models:

Surgical beats heavy, and it is not close. The top two spots went to the two smallest verified edits (131 and 41 matrices); the heaviest edit of all landed second-to-last. At 27B, editing everything mostly buys you a model that thinks in circles.
The thinking-loop finding is new and matters. Qwen3.8 reasons before answering, and on the aggressive arms up to 45% of adversarial responses never close their think block. The same arms finish school math fine — every variant stays within 1.2pp of base on answered-only GSM8K. Math converges; adversarial deliberation does not. If a variant page reports refusal rates but not unterminated-think rates, you are seeing half the story.
Copyright is the new universal wall. Chem and bio — historically the hardest unlock — are now trivially unlocked (every arm clears 80%, the top two at 100%). Copyright is the opposite: no model exceeds 39%, five of nine sit at or below 3.2%. If your use case involves song lyrics, book passages, or periodical text, none of these variants reliably deliver.
Chat template forensics is now mandatory. Three of the eight variants ship modified templates (a hardcoded jailbreak, a thinking-off rewrite, a deleted reasoning-effort prompt), and on this model family the template is a stronger behavioral lever than most weight edits. The same weights behind two different templates behave like two different models. Whatever variant you download, diff its chat template against the base model's before trusting any benchmark claim.
Model-card honesty is checkable, and it varies wildly. orcarouter: 4/4 claims verified. trohrbaugh: KL calibrated within 9%. coder3101: card undersold by an order of magnitude (in the honest direction). obliteratus: "0% refusals" is a scoring artifact. blackfrost: the multi-direction story is contradicted by its own weights. Treat published refusal numbers as a claim to verify, not a fact to inherit.

A note on scope: JonathanColetti and HauhauCS were not in this particular eight-model table (the former is measured in its own card data cited above; the latter's GGUF-only release could not be reverse-engineered into the comparison, and the same forensics project has a separate page documenting why it discontinued HauhauCS models from its comparisons). And one commenter raised a fair methodological critique: some copyright "soft refusals" may just be a 27B model not knowing the material rather than refusing — the report itself flags this as a possible cause. The capability and KL forensics are unaffected either way.

The bottom line: which one should you actually run

Everything above — methods, forensics, the 167-hour shootout — collapses into one decision table:

Just want the best one: orcarouter. With the shootout data in, this is the evidence-backed default: highest real completion rate (82.2%), the only card whose every claim survived forensic verification, clean small edits, no thinking-loop problem, best copyright unlock — and it ships in every format: safetensors, FP8 for vLLM, GGUF conversions. One model covers the desktop and the serving case.
Want zero capability damage above all: apostate (78.7% ASR at the lowest measured KL, 0.0439 — accept the no-vision/no-MTP packaging) or trohrbaugh if you'd rather keep some refusals on purpose (57.5%, the cleanest capability profile and lowest loop rate — the author's own daily driver).
llama.cpp / LM Studio with published KL curves: JonathanColetti. Still the best-documented first pick — the benchmark table rewards claims you can verify, and this is the repo that publishes its own trade-off data.
Raw direct answers at maximum speed: HauhauCS Aggressive — with eyes open: MTP acceleration is real, the forensics record is mixed, and the maintainer's own advice is Balanced for anything reliability-critical. Keep a fallback model installed.
Avoid: obliteratus (44.8% thinking loops, real capability damage) and blackfrost (jailbreak hidden in the chat template, unverified direction-bank claim).

Three settings matter more than the variant you pick: run Q6 or better if VRAM allows (behaviour diverges at low quants); keep the stock chat template (diff any you download); and spot-check TruthfulQA-style false-premise questions after install — it is the capability every measured method damages most. And keep expectations calibrated by category: math and reasoning survive uncensoring essentially intact, copyright does not (measured ceiling 39%), and adversarial-style prompts are where aggressive variants fall into thinking loops.

Everything in the GPU tiers above applies unchanged — file sizes, quant ladders, and settings are identical because abliteration only edits a small set of weight directions, not the architecture. Two honest caveats: refusal removal can slightly degrade benchmark scores (about half a point in the best-documented case) and abliterated weights can change how the model behaves at very low bit-rates (2-bit quants of ablitered models are less tested). Community discussion of the alternatives lives at r/LLM's comparison thread.

A practical note on legality and responsibility: these are Apache 2.0 weights modified by third parties, not by Qwen. The original model card, terms, and applicable local laws still apply to what you generate. Running a model that refuses less does not change what you are responsible for.

Common failure modes, and their one-line fixes

"The model loads but answers are empty or truncated": the default xhigh reasoning ate your context. Set reasoning_effort: medium or disable thinking.
"Slower than the benchmarks": you are partially offloaded to CPU. Check the runtime log for the GPU-layer count; if it is not 99, your quant is too big — drop one tier.
"Out of memory at 32K context": enable Flash Attention and quantize the KV cache to q8_0. If still OOM, drop to IQ4_XS.
"Repetitive or incoherent output on old builds": llama.cpp builds before roughly b10450 produce fluent garbage from this architecture. Update.
"Why is it thinking forever?": it is xhigh by default; see the first bullet.

Bottom line

Qwen3.8-27B is the first model in its weight class where the deployment decision is genuinely interesting: the linear-attention architecture makes context cheap, so your only real decision is weight size against memory. A 16 GB card runs it well (IQ4_XS/Q3_K_XL, ~17-30 t/s). A 24 GB card runs it great (UD-Q4_K_XL + MTP, 40-125 t/s). A 32 GB card or 64 GB Mac runs it at full quality. Integrated graphics and 8 GB machines should wait — or run it in the cloud where it costs fractions of a cent.

Sources: Qwen official model card (Aug 2026), Unsloth GGUF documentation, llamabench.ai community aggregates, Tom's Hardware RTX 5090 benchmarking, mberatsanli/qwen38-27b-local (RTX 5080 deep dive), KGP Talkie 45-configuration llama.cpp sweep, AMD Day-0 support notes. Fact-checked 2026-09-11.

Sources: official docs & pricing pages, hands-on testing where noted, and community feedback. Prices verified August 2026 and may change.