Coding with LLMs (Claude Code, OpenAI Codex) is often presented as the ‘killer app’ for Generative AI. But looking at data, it seems the one piece of the puzzle missing is actual cost. …
Easy to set up, but still needs a 15k $ graphics card and electricity bill. The price you pay openai/anthropic is much cheaper than that for that quality of model.
Sure, you can setup a small model on a consumer graphics card, but the output will be considerably worse and the processing speed considerably lower.
For 240€/year you got a subscription to anthropic which will happily ingest a whole repository and process it in about one minute.
No matter what latest model GPU you installed on your computer, you won’t be able to do that.
There is a middle ground. Crypto farmers have transitioned into running AI workloads for money. There are things sort of like folding@home but you can let people use your GPU and you earn tokens which are used to buy compute or sold to people who want to buy compute on the network. So you can setup a bigass open source model for private on demand use it’s still not cheap but a lot closer to reality for a lot of people than a 15k initial purchase.
If you were paying the real price it would be 2 grand a year though. And in 5 years that 15k graphics card will be $200 and sip on electricity by comparison.
A100 is 6 years old and is now sold at over 10k $.
If you were paying a higher price it could be cheaper to buy the card, since the prices are low that is not the case.
Easy to set up, but still needs a 15k $ graphics card and electricity bill. The price you pay openai/anthropic is much cheaper than that for that quality of model.
Sure, you can setup a small model on a consumer graphics card, but the output will be considerably worse and the processing speed considerably lower.
For 240€/year you got a subscription to anthropic which will happily ingest a whole repository and process it in about one minute. No matter what latest model GPU you installed on your computer, you won’t be able to do that.
Sure, this guy was able to run a 26B model on an old CPU: https://point.free/blog/gemma-4-on-a-2016-xeon/
But that was not easy at all and the speed you get is definitely not the same as the one provided for a very cheap price.
There is a middle ground. Crypto farmers have transitioned into running AI workloads for money. There are things sort of like folding@home but you can let people use your GPU and you earn tokens which are used to buy compute or sold to people who want to buy compute on the network. So you can setup a bigass open source model for private on demand use it’s still not cheap but a lot closer to reality for a lot of people than a 15k initial purchase.
If you were paying the real price it would be 2 grand a year though. And in 5 years that 15k graphics card will be $200 and sip on electricity by comparison.
A100 is 6 years old and is now sold at over 10k $. If you were paying a higher price it could be cheaper to buy the card, since the prices are low that is not the case.
Currently nearly 5 year old used graphics cards are being sold for their initial price. Not sure how much they’ll get cheaper…