Coding with LLMs (Claude Code, OpenAI Codex) is often presented as the ‘killer app’ for Generative AI. But looking at data, it seems the one piece of the puzzle missing is actual cost. …
You’re right if we’re talking about the entire population of Earth. With these local models though, other people have already done all the hard stuff. Anyone with an RTX card and just a minimum level of patience can get going.
Minimum for local models is 12gb imo. There are several “rtx” cards that have 8gb. Also, why nvidia? AMD works well too. My previous point stands, still. If you don’t already have the hw, buying a pc today is very expensive. I don’t know if you go out much but it ain’t pretty out there. People arrn’t precisely swimming in cash.
Also, patience isn’t the only requirement. Keep in mind that some people struggle to even install a program.
Easy to set up, but still needs a 15k $ graphics card and electricity bill. The price you pay openai/anthropic is much cheaper than that for that quality of model.
Sure, you can setup a small model on a consumer graphics card, but the output will be considerably worse and the processing speed considerably lower.
For 240€/year you got a subscription to anthropic which will happily ingest a whole repository and process it in about one minute.
No matter what latest model GPU you installed on your computer, you won’t be able to do that.
There is a middle ground. Crypto farmers have transitioned into running AI workloads for money. There are things sort of like folding@home but you can let people use your GPU and you earn tokens which are used to buy compute or sold to people who want to buy compute on the network. So you can setup a bigass open source model for private on demand use it’s still not cheap but a lot closer to reality for a lot of people than a 15k initial purchase.
If you were paying the real price it would be 2 grand a year though. And in 5 years that 15k graphics card will be $200 and sip on electricity by comparison.
A100 is 6 years old and is now sold at over 10k $.
If you were paying a higher price it could be cheaper to buy the card, since the prices are low that is not the case.
I can’t imagine paying for AI when the open source tools have made it so easy to set up a model locally.
Don’t be daft. The vast majority of people don’t have the knowledge or resources to set that up locally.
You’re right if we’re talking about the entire population of Earth. With these local models though, other people have already done all the hard stuff. Anyone with an RTX card and just a minimum level of patience can get going.
Minimum for local models is 12gb imo. There are several “rtx” cards that have 8gb. Also, why nvidia? AMD works well too. My previous point stands, still. If you don’t already have the hw, buying a pc today is very expensive. I don’t know if you go out much but it ain’t pretty out there. People arrn’t precisely swimming in cash.
Also, patience isn’t the only requirement. Keep in mind that some people struggle to even install a program.
Easy to set up, but still needs a 15k $ graphics card and electricity bill. The price you pay openai/anthropic is much cheaper than that for that quality of model.
Sure, you can setup a small model on a consumer graphics card, but the output will be considerably worse and the processing speed considerably lower.
For 240€/year you got a subscription to anthropic which will happily ingest a whole repository and process it in about one minute. No matter what latest model GPU you installed on your computer, you won’t be able to do that.
Sure, this guy was able to run a 26B model on an old CPU: https://point.free/blog/gemma-4-on-a-2016-xeon/
But that was not easy at all and the speed you get is definitely not the same as the one provided for a very cheap price.
There is a middle ground. Crypto farmers have transitioned into running AI workloads for money. There are things sort of like folding@home but you can let people use your GPU and you earn tokens which are used to buy compute or sold to people who want to buy compute on the network. So you can setup a bigass open source model for private on demand use it’s still not cheap but a lot closer to reality for a lot of people than a 15k initial purchase.
If you were paying the real price it would be 2 grand a year though. And in 5 years that 15k graphics card will be $200 and sip on electricity by comparison.
A100 is 6 years old and is now sold at over 10k $. If you were paying a higher price it could be cheaper to buy the card, since the prices are low that is not the case.
Currently nearly 5 year old used graphics cards are being sold for their initial price. Not sure how much they’ll get cheaper…