So are we assuming here that LLMs won’t become more efficient over time?

Mostly. Moore’s law ran up against the physical limits of the materials we make chips out of - so desktops of today just do what the desktops of yesterday do, mostly.

We should keep seeing improvements in highly specialized models. There’s interesting outcomes to have here, with the right setup and ollama.

• but -

The really promising impressive models today are just running with long contexts on shithloads of hardware - which is neither coming to home PCs any time soon nor going to actually be profitable to run any time soon.

There’s an argument to be made that running the really interesting model on a ton of hardware might make money for really specific uses - but then when we talk about specific uses that are worth lots of money, those use cases tend to tolerate difficult interfaces and reward accuracy. LLMs invariably reduce accuracy in exchange for ease of use. There might be a sweet spot for a huge expensive hallucination prone LLM in some of these uses, but I doubt it (the entire approach) competes, long term.

There’s a few specific use cases where inaccuracy is desirable - largely forms of shifting accountability and some kinds of gambling. Things that either are or should be crimes have a higher tolerance for AI hallucination.

But - a small cheap local model has all the desirable attributes for doing these things (crimes) poorly as a big expensive model. So there’s probably not even much money to be made there.

I expect that this tech is not going away, but it’s also not earning back the current investment.