Doing the sums before buying a machine for a local model

In one week my game-making agents used what would have cost about $1,460 at API list prices. I pay a flat $200 a month for the subscription, so that figure is not a bill, but the weekly Sonnet allowance did run out partway through a release. This note is the arithmetic I did before deciding whether a machine at home should take over part of the work.

The model I would start with is Qwen3.6-35B-A3B with 4-bit quantisation, a compressed form that needs less memory. It has open weights under Apache 2.0 and takes text and images. Of its 35 billion parameters only 3 billion are active per token, which is why it is fast: measurements published by someone else on an RTX 3090 graphics card give about 155 tokens per second when writing and over 3,000 per second when reading a prompt. The model file is about 21 GB.

I could run it on the laptop I already own, a MacBook Pro with an M5 Pro chip and 64 GB of unified memory. The whole model fits, the software is installed and it costs nothing, but it is my work machine, the agents use it all day, and night runs would mean leaving it on at home every night.

Or I could buy a small dedicated machine. The cheaper build I would accept:

PartChoicePrice
Mini PCMinisforum AI X1 Pro-470, sold without memory or disk; has OCuLink, a connector for an external graphics cardabout $760
Memory32 GB DDR5$430 to $490
Disk512 GB NVMeabout $90, my estimate
Graphics dockAOOSTAR AG02, OCuLink, with its own 800 W power supplyabout $230
Graphics cardused RTX 4060 Ti, 16 GB$380 to $470
Totalabout $1,950 (middle of the ranges)

A 16 GB card cannot hold a 21 GB model. This model can keep some of its layers in the computer's main memory, off the card, which is why the build needs 32 GB and not 16. That split also makes it slower, and the estimates I have seen for 16 GB cards are 30 to 50 tokens per second. For night runs that is enough. A 512 GB disk holds the system and three or four models.

The faster build swaps the card for a used 24 GB RTX 3090, which costs about $1,500. Then the whole model is on the card, which is the setup behind the 155 tokens per second above, and the total comes to about $3,000.

The limit in both builds is context, the amount of text the model can hold at once. In the published test the 3090 had about 2.5 GB of memory free with 8,000 tokens of context loaded, so I expect room for roughly 16,000 to 32,000 tokens, less with image input. That is too small for the roles I want to move, as they run today. A blind-test episode peaked at a median of 34,000 tokens of context, a research run at 54,000 and a QA run at about 100,000. Each step adds only 2,000 to 2,300 tokens, but the steps pile up. These roles can move only after I cut their work into small calls, one part of a transcript or one screenshot with a short history.

The roles that only read, file, summarise or test were 1.5% of the week's cost and 5.5% of its fresh tokens. Adding the research and art roles and the general Sonnet helpers brings that to about 8% of cost and 25% of fresh tokens, and because of the context limit less than that will actually move.

Eight per cent of $1,460 is about $120 a week at list prices. If all of it moved, which the context limit rules out for now, that would repay the $1,950 build in about four months and the $3,000 build in about six. But I pay a flat $200 a month, and moving 8% of the work does not put me on a cheaper plan, so in cash the machine saves nothing.

More projects do not change the cash sum either. The week I measured already held two games and the knowledge base, and more projects on the same subscription do not raise my bill. What they do is use up the Sonnet allowance sooner, and at that point a second $200 subscription becomes the real alternative. Compared with that, and allowing $25 to $30 a month for electricity (my estimate), the cheaper build pays for itself in about eleven months and the faster one in about a year and a half. The machine's price is also fixed, so every extra project that sends it work lowers the cost per task.

So I would not buy it for the money. What it would buy is quota, since the roles that would move are Sonnet roles and Sonnet is the allowance that ran out (by list price they were about three quarters of the week's Sonnet use, though I do not know how the subscription weights them), and, more than that, repetition. My QA calibration is two runs and my blind test is eight episodes, and several numbers in the main article are weak for the plain reason that a run costs quota and so I ran it once or twice. A local model should be able to play the same episode fifty times overnight for the price of electricity. That says nothing about the recall of the hosted tester, the one that runs on Claude, until I run both on the same episodes. It would show me how much one episode differs from the next, and so whether a result like four out of twelve is stable.

So the laptop goes first, since it costs nothing to find out whether the model is good enough at these tasks (it will not tell me about the 16 GB card's speed or context limit), and the dedicated machine waits until the night runs are worth more to me than a free laptop.

Claude Code can be pointed at another endpoint for a whole session, but I want only some roles to go local. The local model will therefore run behind my own scripts, served by llama.cpp, and the agents will call those scripts as tools. For the testers that means my script has to run the whole loop itself: screenshot, model, action.

Code written from a ticket stays hosted. I have not run a local model on my own tickets, the published coding scores for this model come from its vendor's own harness, and a wrong change in game code costs me more than a wrong summary. The blind tester stays hosted as a gate until a local one has been compared with it on the same episodes. Nothing inside the lead's own loop moves, because that loop needs far more context than a 24 GB card allows.

About meAll articlesLinkedInpavel@klinkov.com