The edge is where it’s at
Interview with Nick about the post:
https://www.youtube.com/watch?v=a5rLzNxRjEQ&list=UU9rJrMVgcXTfa8xuMnbhAEA - video
https://pivottoai.libsyn.com/20251107-nicholas-weaver-the-futile-future-of-the-gigawatt-datacenter - podcast
time: 26 min 53 sec
Let me see if I got this right: Because use cases for LLMs have to be resilient to hallucinations, large data centers will fall out of favor for smaller, cheaper deployments at the cost of accuracy. And once you have a business that is categorizing relevant data, you will gradually move away from black box LLMs and towards ML on the edge to cut costs and also at the cost of accuracy.
I read it this way: because LLMs inevitably hallucinate, no matter how resource intensive the LLM is, it makes economic sense to deploy smaller, cheaper LLMs that hallucinate a little more. The tradeoff isn’t “hallucinations vs no hallucinations”, it’s “more hallucinations vs fewer hallucinations”, and the slight gain in accuracy from using the big data center isn’t worth the huge expense of using those big data centers.