Introduction
I was curious how capable a local LLM could be on the hardware I already have, so I started this project. I wanted to see what would fit on one GPU, how fast it would run, and how the models compared once I put them through the same basic tests. So I tried a bunch of them, ran the benchmark samples, and kept track of the failures too. Below are my results.
Hardware And Test Setup
- GPU: NVIDIA RTX 6000 Ada Generation with 49,140 MiB (48 GiB) VRAM.
- Story runs used Dynamo in
dynamo:latest-vllm-local-dev with vLLM 0.25.1, one GPU, --gpu-memory-utilization 0.90, --max-model-len 32768, --max-num-batched-tokens 2048, --max-num-seqs 4, and --enforce-eager. Benchmark clients used different concurrency values, and the 70B runs used a 4K context cap and one sequence.
- Successful models were reached through Dynamo's OpenAI-compatible
POST /v1/chat/completions endpoint at the local frontend. Each model was loaded alone and stopped before the next one.
- Story measurements are one-request eager-mode smoke tests, not tuned throughput benchmarks. Time to first token (TTFT) and tokens per second are measured at the client through Dynamo.
Working Models
The models below are ordered by measured LiveCodeBench score first, then by the broader benchmark mean. This ordering is provisional because the LiveCodeBench runs used different runtime settings and only Qwen3 0.6B has a thinking/no-thinking pair.
| Model | Checkpoint and Specs | Measured VRAM and Context Capacity | Basic Test | Story Performance | Quality Signal | Caveats |
|---|
| Qwen3-VL-30B-A3B Instruct | AWQ 4-bit, QuantTrio/Qwen3-VL-30B-A3B-Instruct-AWQ; 262K native context; tested text-only | 15.93 GiB weights; 26.39 GiB KV cache; 288,256 KV tokens | API request passed and returned the exact requested response | Not measured | Local LiveCodeBench 38%; Text Arena not recorded | Best measured LiveCodeBench result in the historical sample. Image input was disabled. |
| Qwen2.5-Coder 32B Instruct | Official AWQ 4-bit, Qwen/Qwen2.5-Coder-32B-Instruct-AWQ; 32K native context | 18.14 GiB weights; 23.99 GiB KV cache; 98,240 KV tokens | API request passed and returned the exact requested response | Not measured | Local LiveCodeBench 35% | Conventional instruct model; BF16 version did not fit. |
| OpenAI GPT-OSS 20B | Official MXFP4; 131K native context | 13.8 GiB weights; 28.51 GiB KV cache; 1,167,515 KV tokens | Partial: guesses 5 and 7 returned higher and correct | TTFT 0.120 s; 768 output tokens; 24.465 s end to end; 31.55 tokens/s | Local LiveCodeBench 27%; Text Arena 1317 +/- 6 | Fastest measured story result; historical LiveCodeBench run had 44 transport errors. |
| DeepSeek-R1-Distill-Qwen-32B | AWQ 4-bit, casperhansen/deepseek-r1-distill-qwen-32b-awq; 131K native context | 18.17 GiB weights; 23.96 GiB KV cache; 98,144 KV tokens | Partial: guesses 5 and 7 reached correct final answers | TTFT 0.088 s; 768 output tokens; 32.553 s end to end; 23.66 tokens/s | Local LiveCodeBench 25%; AIME24 72.6; MATH-500 94.3 | Lowest measured TTFT; long reasoning and 3 transport errors in the historical LiveCodeBench run. |
| Qwen3 32B | Official AWQ 4-bit, Qwen/Qwen3-32B-AWQ; 32K native context, 131K with YaRN | 18.24 GiB weights; 23.92 GiB KV cache; 97,968 KV tokens | API request passed with a 1,024-token allowance | Not measured | Local LiveCodeBench 20%; Text Arena 1347 +/- 9 | Emits <think> content; historical result was not mode-controlled. |
| Qwen3-Coder 30B-A3B Instruct | AWQ 4-bit, stelterlab/Qwen3-Coder-30B-A3B-Instruct-AWQ; 262K native context | 15.69 GiB weights; 26.63 GiB KV cache; 290,832 KV tokens | Passed exactly: higher, higher, correct | TTFT 0.192 s; 548 output tokens; 35.821 s end to end; 15.38 tokens/s | Local LiveCodeBench 20% | Best short deterministic smoke-test behavior; historical LiveCodeBench run had 41 transport errors. |
| Gemma 3 27B IT | AWQ 4-bit, gaunernst/gemma-3-27b-it-int4-awq; 128K input and 8K output; tested text-only | 17.0 GiB weights; 25.18 GiB KV cache; 210,825 KV tokens | Passed: higher, higher, correct | TTFT 0.180 s; 588 output tokens; 62.274 s end to end; 9.47 tokens/s | Local LiveCodeBench 20%; Text Arena 1365 +/- 4 | Slowest tested model; one historical response was unparsed. |
| NVIDIA Nemotron 3 Nano 30B-A3B | NVFP4; 262K model/configuration context; this run used 32K | 18.22 GiB weights; 23.98 GiB FP8 KV cache; 1,624,473 KV tokens | Passed logically: higher, higher, correct; each included visible reasoning markup | TTFT 0.207 s; 768 output tokens; 56.183 s end to end; 13.72 tokens/s | Local LiveCodeBench not collected | The Ada GPU has no native FP4 compute; the LiveCodeBench run did not complete. |
| Qwen3.5-35B-A3B | AWQ 4-bit, QuantTrio/Qwen3.5-35B-A3B-AWQ; 262K native context; tested text-only | 21.38 GiB weights; 20.8 GiB KV cache; 889,909 KV tokens | API request reached the response only with a larger output allowance | Not measured | Local LiveCodeBench 17%; Text Arena not recorded | Historical result; explicit no-thinking run still required. |
| Qwen3 0.6B | Official BF16; 32K native context | 1.12 GiB weights; 40.59 GiB KV cache; 380,032 KV tokens | Not ranked by model quality | Not measured | LiveCodeBench: 15% thinking; 8% no-thinking | Same prompts and seed, but different historical runtimes, concurrency, token limits, and no explicit no-thinking field in saved payloads. |
| DeepSeek-R1-Distill-Llama-70B | AWQ 4-bit; 128K native context; test limited to 4K | 37.09 GiB weights; 5.34 GiB KV cache; 17,488 KV tokens | API request returned an OpenAI JSON response | Not measured | LiveCodeBench not collected | Fits only with a 4K request cap and one sequence. |
| Llama 3.3 70B Instruct | AWQ 4-bit; 128K native context; test limited to 4K | 37.09 GiB weights; 5.34 GiB KV cache; 17,488 KV tokens | API request passed and returned the exact requested response | Not measured | Local LiveCodeBench not collected | Fits only with a 4K request cap and one sequence. |
Benchmark Comparison Progress
These are 100-question samples run through Dynamo's OpenAI-compatible API with AIPerf 0.12.0 and temperature 0. Each model uses the largest practical per-model settings tested or supported by the local GPU and model mode, so completion limits and concurrency may differ. They are not full benchmark scores. The ranking below uses the arithmetic mean of MMLU-Pro, GSM8K, MATH-500, and a usable local LiveCodeBench percentage; it is a local comparison score, not an official composite benchmark. GPQA-Diamond is blocked by gated dataset access. The fixed runner used a persistent environment with AIPerf 0.12.0 and datasets 3.6.0 installed in separate steps. In these exports, AIPerf reports unparsed=100 for every GSM8K row even when correct answers were recorded, so that field is not used to interpret GSM8K results.
What The Benchmark Measures Mean
MMLU-Pro samples broad academic and professional knowledge questions, with harder questions and less reliance on simple recall than the original MMLU. GSM8K samples grade-school math word problems and checks whether the model reaches the correct numerical answer. MATH-500 samples challenging competition-style mathematics problems and checks the final answer. LiveCodeBench samples recent programming problems and runs extracted code against tests. GPQA-Diamond samples difficult graduate-level questions in subjects such as biology, physics, and chemistry, where answering usually requires expert reasoning. The simple mean is the arithmetic average of the available MMLU-Pro, GSM8K, and MATH-500 percentages; the rank orders models by that average and is not an official overall score.
For LiveCodeBench, unparsed means the grader could not extract a runnable code block from the response. A response may contain a correct idea or solution but still be unparsed when it fails the expected code format, is truncated, or includes extra text that prevents extraction. These are 100-question samples, not official full benchmark scores. Comparisons are limited because sampling, prompts, model settings, response lengths, parsing, software versions, and grader behavior can change results; the simple mean also weights the three included benchmarks equally and does not include LiveCodeBench or GPQA-Diamond.
| Rank so far | Model | MMLU-Pro | GSM8K | MATH-500 | LiveCodeBench | Four-benchmark mean | LiveCodeBench - thinking | Status |
|---|
| 1 | Qwen3-VL-30B-A3B AWQ | 77% | 94% | 84% | 38% | 73.3% | Not mode-controlled: 38% official; 22 unparsed | Historical result; mode not recorded; tested text-only |
| 2 | Qwen3-Coder 30B-A3B AWQ | 73% | 94% | 81% | 36% | 71.0% | 36% official; 8 unparsed; 0 transport errors; 8,192-token cap | Corrected 100-question run; settings selected for best practical local result |
| 3 | GPT-OSS 20B MXFP4 | 79% | 89% | 85% | 27% | 70.0% | Not mode-controlled: 27% official; 21 unparsed; 44 transport errors | Historical result; mode not recorded; GSM8K unparsed export is not meaningful |
| 4 | DeepSeek-R1-Distill-Qwen-32B AWQ | 81% | 89% | 83% | 25% | 69.5% | Not mode-controlled: 25% official; 72 unparsed; 3 transport errors | Historical result; mode not recorded; GSM8K unparsed export is not meaningful |
| 5 | Qwen3 32B AWQ | 81% | 92% | 84% | 20% | 69.3% | Not mode-controlled: 20% official; 80 unparsed | Historical result; mode not recorded |
| 6 | Qwen2.5-Coder 32B AWQ | 72% | 84% | 80% | 35% | 67.8% | Not mode-controlled: 35% official; 2 unparsed | Historical result; mode not recorded; GSM8K unparsed export is not meaningful |
| 7 | Gemma 3 27B AWQ | 61% | 83% | 81% | 20% | 61.3% | Not mode-controlled: 20% official; 1 unparsed | Historical result; mode not recorded; GSM8K unparsed export is not meaningful |
| 8 | NVIDIA Nemotron 3 Nano 30B NVFP4 | 80% | 90% | 36% | Not collected | 68.7% | Not supported: no successful LiveCodeBench run | No successful LiveCodeBench run |
| - | Qwen3 0.6B | 35% | 76% | 71% | 15% thinking | 60.7% | 15% official; 77 unparsed; 0 transport; 28,000-token cap | Historical pair; same prompts and seed, but different runtime, concurrency, token limits, and no explicit no-thinking field in the saved payloads |
| - | Qwen3.5-35B-A3B AWQ | 82% | 91% | Invalid | 17% | Not ranked | Not mode-controlled: 17% official; 68 unparsed; 0 transport; 4,096-token cap | Excluded: the saved MATH-500 run processed only 6 valid responses and 94 errors |
Conclusions
- Qwen3-Coder 30B-A3B AWQ is the most dependable tested option for the short deterministic smoke test because it follows that response format cleanly and leaves substantial KV-cache capacity. In the corrected 100-question local sample it reached 36%, with 8 unparsed responses and 0 transport errors. The result is not a published-score reproduction and is not directly comparable to the earlier 20% diagnostic run because the runtime settings and transport reliability differed.
- GPT-OSS 20B is the fastest measured local model at 31.55 output tokens per second, but the current raw Dynamo response needs reasoning extraction.
- DeepSeek-R1-Distill-Qwen-32B AWQ has the strongest published reasoning signal among the tested models, but its long reasoning content reduces its suitability for short interactive responses without response parsing.
- Gemma 3 27B AWQ is usable in text-only mode, but was the slowest model tested and does not reliably return bare JSON without structured-output enforcement.
- The documented BF16 attempts from 27B through 70B exhausted the 48 GiB GPU. Quantized checkpoints made every tested model family loadable, including both 70B AWQ models, but the 70B models only leave enough capacity for a 4K request cap and one concurrent sequence.
Sources
- GPU measurement:
nvidia-smi on 2026-08-27.
- Context, architecture, and checkpoint details: GPT-OSS 20B, Nemotron 3 Nano NVFP4, Qwen3 32B, Qwen3-Coder 30B-A3B AWQ, DeepSeek-R1-Distill-Qwen-32B AWQ, Gemma 3 27B AWQ, Qwen3.5-35B-A3B, Qwen3-VL-30B-A3B AWQ, DeepSeek-R1-Distill-Llama-70B AWQ, and Llama 3.3 70B AWQ.
- Preference rankings: Arena AI Text Arena leaderboard retrieved 2026-08-27. Arena scores are configuration-dependent and are not coding benchmarks.
- Final cleanup measurement: 2 MiB GPU memory used and 48,510 MiB free; no Dynamo vLLM validation worker remained running.
