What a Coding-Agent Benchmark Taught Me About Local and Cloud Models
I have been experimenting with replacing part of my local coding-agent setup with cloud models, so I built a reproducible benchmark and ran the same six autonomous coding tasks across several models.
This is not a benchmark of raw model intelligence in the abstract. It is a benchmark of complete coding-agent runs: the model received a repository, a task, tools, and a starting state, then had to investigate, make changes, and finish without human steering.
The local baseline
My local baseline was Qwen3.8-27B running on an RTX 4080. The model was aggressively optimized to fit and run quickly on a single 16 GB GPU.
- Intel Core Ultra 7 265K
- NVIDIA RTX 4080 16 GB
- 96 GB system RAM
- Gigabyte Z890 AORUS ELITE WIFI7
- 850 W PSU
The actual llama.cpp configuration was:
[qwen3.8-27b]
chat-template-kwargs = {"reasoning_effort":"medium", "preserve_thinking":true, "reasoning_budget":8192}
reasoning-budget = 8192
load-on-startup = true
model = C:/llm/models/Qwen/Qwen3.8-27B/Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp.gguf
cache-type-k = q8_0
cache-type-v = q8_0
main-gpu = 0
split-mode = none
fit = on
fit-ctx = 65536
spec-type = draft-mtp
spec-draft-n-max = 4
spec-draft-p-min = 0.8So this was not Qwen3.8-27B at BF16, FP8, or even Q6. It was an IQ3_S / GSQ-RCO quantized model running entirely on one RTX 4080, with a 65,536-token context, Q8 KV cache, an 8K reasoning budget, MTP speculative decoding, no multi-GPU split, and llama.cpp automatic fitting enabled.
Depending on the prompt and speculative acceptance rate, generation speeds could approach 80 tokens per second. That made this a fairly aggressive test of how much useful coding capability survives when a modern 27B model is squeezed into 16 GB of VRAM.
The Cloud Models
I compared the local model with:
- GPT-5.6 Luna Max
- GPT-5.6 Terra Medium
- Claude Sonnet 5
- GLM-5.3
- DeepSeek V4.1 Flash
- Local Qwen3.8-27B IQ3_S
The cloud models were accessed through OpenRouter. Each model received the same starting repository, task prompt, and tools. There was no human steering after a run started. Hidden tests were executed only after the agent stopped.
A disclaimer about the evaluation: I was too lazy to review every generated change myself. I asked GPT Sol Light to review the results instead. Its review noted that the routing recommendation is not established by six cases with a correctness ceiling effect. The analysis below should therefore be read as a report of task completion and measured behavior—not as a claim that every generated change was manually audited for quality.
Across the six sessions per model, the harness recorded 509 model requests in total:
| Model | Sessions | Model requests |
|---|---|---|
| Local Qwen | 6 | 100 |
| DeepSeek | 6 | 109 |
| GLM | 6 | 104 |
| Terra | 6 | 61 |
| Luna Max | 6 | 59 |
| Sonnet 5 | 6 | 76 |
| Total | 36 | 509 |
How far would OpenCode Go go?
OpenCode Go is a useful way to think about subscription economics because it does not promise one fixed number of requests. Its current limits are dollar-denominated—$12 over five hours, $30 per week, and $60 per month—and the equivalent request count depends heavily on the selected model. The service currently advertises $10/month pricing after the introductory month.
Applying the request counts from this benchmark to OpenCode’s published monthly estimates gives this rough number of equivalent six-session runs:
| Go model | Requests in this benchmark | Published monthly estimate | Equivalent runs / month |
|---|---|---|---|
| GPT 5.6 Luna | 59 | 10,250 | ~174 |
| GLM-5.3 | 104 | 4,300 | ~41 |
| DeepSeek V4 Flash | 109 | 158,150 | ~1,450 |
| Qwen3.8 Max | 100 | 810 | ~8 |
These are not guarantees. They assume that every request has roughly the same token and cache profile as this experiment, that the model remains available in Go, and that the monthly limit is the binding limit. The five-hour and weekly caps can matter first for concentrated workloads. Terra and Sonnet are not included in this table because they are not listed among the current Go models. OpenCode also notes that the model list and estimates can change.
For my actual mixed benchmark, the total was 509 model requests. That is only about 5% of the published monthly Luna allowance, but it is not meaningful to divide the whole number by one model’s estimate because the six models have different request profiles. The practical takeaway is that a $10 Go subscription could cover many benchmark-sized runs when using its cheaper included models, while a premium included model gives fewer—but still substantial—runs. The subscription’s value comes from the aggregate dollar allowance and routing convenience, not from a universal request quota.
The benchmark
To make the comparison less ad hoc, I first used a prompt to ask GPT to generate another prompt for creating a frozen benchmark suite. The resulting benchmark repository is available at github.com/aresowj/pi-agent-benchmark. I then used that fixed suite as the source of the six tasks below.
The six cases were designed to exercise different coding-agent abilities:
- Configuration propagation across multiple layers
- An overnight time-range edge-case bug
- A retry/idempotency bug causing duplicate delivery
- A cross-cutting retry abstraction and refactor
- Algorithmic performance optimization
- A SQLite concurrent job-claiming race
The biggest result was also the biggest limitation:
Every serious model passed all six cases.
That means this benchmark has a ceiling effect. It tells me a lot about efficiency and agent behavior, but it does not yet establish a meaningful capability ranking. I need harder tasks before claiming that one model is more capable than another.
| Model | Pass | Total time | Output tokens | Tool calls | API cost |
|---|---|---|---|---|---|
| Claude Sonnet 5 | 6/6 | ~352s | 28,468 | 78 | ~$0.66 |
| GPT-5.6 Luna Max | 6/6 | ~380s | 14.0K | 86 | ~$0.04 |
| GPT-5.6 Terra Medium | 6/6 | ~384s | 16.4K | 84 | ~$0.45 |
| GLM-5.3 | 6/6 | ~374s | 39.8K | — | ~$0.42 |
| Local Qwen IQ3_S | 6/6 | ~1,146s | 45.4K | 107 | $0 API |
| DeepSeek V4.1 Flash | 6/6 | ~1,184s | 56.9K | 150 | ~$0.07 |
Claude’s aggregate usage report was 146 fresh input tokens, 575,601 cache-read input tokens, and 28,468 output tokens—575,747 processed input tokens in total. The per-task report also makes it possible to compare individual cases rather than only the aggregate run.
The per-case Sonnet 5 report provides the more useful breakdown:
| Case | Fresh input | Cache read | Output | Cost | Time |
|---|---|---|---|---|---|
| case01 | 27 | 125,135 | 4,809 | $0.1285 | 56.9s |
| case02 | 11 | 18,348 | 938 | $0.0260 | 16.3s |
| case03 | 29 | 148,348 | 5,566 | $0.1439 | 72.0s |
| case04 | 27 | 120,553 | 6,492 | $0.1425 | 70.9s |
| case05 | 29 | 89,255 | 6,103 | $0.1159 | 73.2s |
| case06 | 23 | 73,962 | 4,560 | $0.1000 | 62.9s |
1. The local Q3 model did much better than I expected
I expected the combination of a 27B model, roughly 3-bit weights, only 16 GB of VRAM, a not-so-large 65K context, and speculative decoding to show obvious reasoning degradation on architecture or concurrency tasks.
It did not.
The local model successfully solved the SQLite race condition, the retry/idempotency bug, the abstraction refactor, and the performance optimization. That does not prove IQ3_S is equivalent to a full-precision model. It means only that these six tasks were not difficult enough to expose the difference.
Still, the result makes me less interested in buying additional GPUs simply because I assumed Q3 quantization was too compromised for serious coding work.
2. 80 tokens per second does not mean the agent is faster
This was the most useful result.
The local model could reach around 80 generated tokens per second under favorable conditions. Yet complete agent runs took roughly three times as long as Luna, Terra, Claude, and GLM.
The reason is that the stronger models often needed dramatically fewer tokens and fewer interactions to reach the same result. In the concurrency case:
- Local Qwen: ~356 seconds and ~15.8K output tokens
- Claude: ~63 seconds and ~4.6K output tokens
- Luna: ~71 seconds and ~3.2K output tokens
- Terra: ~82 seconds and ~3.8K output tokens
All of them solved it. The local model was generating tokens quickly; it just needed far more of them.
For coding agents, I think the more meaningful metric is successful tasks per hour, not tokens per second.
3. Luna was the standout value result
Luna was nearly as fast as Claude, in roughly the same overall range as Terra and GLM, and produced dramatically less output than almost everything else.
- Luna: ~14K output tokens
- Terra: ~16K
- Claude: ~28K
- Local Qwen: ~45K
- DeepSeek: ~57K
The measured API-equivalent cost for Luna was around $0.04 for all six tasks. More importantly for me, I already pay for ChatGPT Plus/Codex. While I am within the included Codex allowance, Luna’s marginal cost is effectively zero.
For now, that makes it look like an unusually strong default coding-agent model.
4. Claude Sonnet was excellent
Claude Sonnet 5 was the fastest overall model in this run and used the fewest tool calls. It behaved decisively and had no obvious weak case.
But on this benchmark it was only about 7% faster than Luna while costing substantially more. That does not mean Claude is not stronger. The problem is that everything went 6/6. I need harder cases where Luna starts failing before I can determine whether Claude’s higher cost buys meaningfully higher autonomous success.
5. DeepSeek was cheap, but slow in my harness
DeepSeek V4.1 Flash was correct, but surprisingly verbose and interaction-heavy:
- ~57K output tokens
- 150 tool calls
- ~1,184 seconds total
That made it roughly as slow as the local model at completing the actual tasks. Its API cost is tiny, so I can still imagine using it for cheap background or batch agents. I am less interested in using it as my interactive default.
6. GLM looked fast and capable
GLM completed the benchmark in roughly the same wall-clock range as Luna, Terra, and Claude. It therefore looks more attractive to me than DeepSeek for interactive work.
Its efficiency profile was less impressive than its completion time. GLM used about 39.8K output tokens and 104 model requests across the six sessions—considerably more interaction than Luna’s 59 requests and 14.0K output tokens. In other words, GLM reached the finish line at a competitive speed, but it took a more verbose route to get there.
That makes GLM interesting as a capable interactive option, but not an obvious default for my setup. The completed GLM evaluations finished reliably and within the same broad time window as the other cloud models. It does not show that the extra requests or output buy better results, since the completed evaluations reached the same correctness ceiling. I would want to test longer tasks and more difficult debugging cases before deciding whether GLM’s behavior is a useful style difference or simply unnecessary agent overhead.
My current routing plan
After this run, I am leaning toward:
This is an unvalidated policy suggestion rather than a benchmark-derived ranking. The six cases were useful for comparing efficiency, but the correctness ceiling means I have not yet shown that the more expensive models solve harder tasks better.
- Luna as the default
- Terra Medium for difficult tasks; Claude Sonnet 5 also performed excellently, but I am not planning to use it because I do not have a Claude subscription
- Local Qwen for parallel agents, offline/privacy use, or quota fallback
- DeepSeek as a cheap background worker
I am also keeping the RTX 4080 setup. A 27B IQ3 model that can run at up to 80 tokens per second and still go 6/6 on this benchmark is too useful to throw away simply because cloud models complete complex agent loops faster.
What does local actually cost?
The API bill is only one side of the comparison. To estimate the local cost, I used a deliberately simple assumption: the complete computer draws an average of 600 W while the model is actively running, it runs for eight hours every day, and electricity costs $0.15 per kWh. That works out to:
0.6 kW × 8 hours × $0.15 = $0.72 per day
$0.72 × 30 days = $21.60 per month
$0.72 × 365 days = $262.80 per yearThis is an estimate, not a wall-meter measurement. The GPU may be busy while the rest of the machine is not, and idle time, cooling, monitor power, and other household electricity are not included. It is also worth separating the marginal cost of using hardware I already own from the cost of buying that hardware.
| Scenario | Up-front hardware | Electricity / year | Three-year monthly cost | Three-year total |
|---|---|---|---|---|
| Already own the machine | $0 incremental | $262.80 | $21.60 | $788.40 |
| Buy used system | Assume $1,200 | $262.80 | $55.00 | $1,988.40 |
| Buy new system | Assume $2,000 | $262.80 | $77.16 | $2,774.40 |
The used and new rows are planning assumptions for a complete RTX 4080-class system, not quotes. They amortize the purchase over 36 months and add the electricity cost. If the machine is also used for gaming or ordinary work, only part of that purchase price should really be assigned to the model.
Comparison with inexpensive APIs
For a rough throughput comparison, 80 output tokens per second for eight hours is about 2.30 million output tokens per day. Using current standard output prices, that much output would cost approximately:
| Service | Published output price | 8-hour output equivalent | 30-day equivalent |
|---|---|---|---|
| Local Qwen on the 4080 | Electricity at $0.15/kWh | $0.72 | $21.60 |
| Gemini 2.5 Flash-Lite | $0.40 / 1M output tokens | ~$0.92 | ~$27.65 |
| DeepSeek V4.1 Flash, off-peak | $0.60 / 1M output tokens | ~$1.38 | ~$41.47 |
| GPT-5 Mini | $2.00 / 1M output tokens | ~$4.61 | ~$138.24 |
The API figures exclude input tokens, cached-input charges, and any provider-specific tool fees. DeepSeek’s peak output price is currently $1.20 per million tokens, which would make the same output about $2.76 per day. The price references are Google’s Gemini pricing, DeepSeek’s pricing page, and OpenAI’s GPT-5 Mini model page; they can change, so this section should be treated as a dated snapshot.
Equal tokens are not equal work
In my benchmark, the local model needed about 45.4K output tokens for six completed tasks, while Luna needed about 14.0K. A cheaper API can still win economically if it reaches a correct result with far fewer tokens and less waiting. Conversely, the already-owned local machine has a very low marginal cost and offers privacy, offline operation, and unlimited parallel use within its hardware limits. My current conclusion is therefore conditional:
- If I already own the machine, local inference costs roughly $22 per month at this usage level and is easy to justify.
- If I must buy a $1,200 used system, a low-cost API can be cheaper until the machine is used heavily or its non-AI value is included.
- A new $2,000 system is difficult to justify on electricity savings alone; I would buy it for local control, privacy, availability, or other workloads.
How cached input changes cloud costs
Cloud model pricing usually separates three things: fresh input tokens, cache-read input tokens, and output tokens. Fresh input is the part of the prompt the provider has not seen before. Cache-read input is repeated context that the provider can reuse—often the repository instructions, system prompt, or unchanged conversation history. Output is the model’s generated reasoning and answer.
Cache reads are usually much cheaper than fresh input, but they are not free. A rough cost calculation is:
cost = fresh input × fresh-input price
+ cache read × cache-read price
+ output × output priceClaude’s aggregate report makes the effect visible in this benchmark: 146 fresh input tokens, 575,601 cache-read input tokens, and 28,468 output tokens. Although the processed input was enormous, almost all of it was cache-read context rather than newly billed at the full input rate. That is one reason a long-running coding-agent session can be much cheaper than multiplying the total context size by the standard input price.
This also makes simple “cost per request” comparisons misleading. Two requests can both count as one model request while having very different costs depending on how much context is fresh, how much is cached, and how much the model outputs. In practice, good prompt caching rewards agents that preserve a stable system prompt and repository context across turns. It also means that the cost estimates in this post should be read as measurements of this particular harness and provider routing, not universal prices for the model itself.
What I want to test next
The current benchmark is clearly too easy. The next version should include:
- A much larger unfamiliar repository with 30–50 relevant files
- Misleading nearby abstractions
- Undocumented invariants that must be inferred from callers and tests
- Architectural changes with several superficially valid approaches
- Long-horizon tasks requiring 20–40 meaningful tool calls
- Multiple test/fix cycles
- Bugs where the obvious fix is intentionally wrong
That is where I expect to see a meaningful difference between Qwen3.8-27B IQ3 on a 4080 and models like Luna, Terra, and Claude Sonnet.
The biggest takeaway so far is simple:
Raw inference throughput and coding-agent throughput are very different things.
My local model can generate around 80 tokens per second, yet a stronger cloud model can finish an engineering task several times faster simply by generating fewer unnecessary tokens and taking a better path through the problem.
Comments