The cost of hybrid inference is very low; You can squeeze Qwen 35B on a 16GB RAM machine as long as it has some GPU. Check out ik_llama.cpp and ubergarm’s quants in particular:
But if you aren’t willing to even try, I think that’s another bad omen for local models. Like the Fediverse, it won’t be served to you on a silver platter, you gotta go out and find it.
It’s really unfortunate how a lot of people have a knee jerk reaction towards anything LLM related right now. While you can make good arguments for avoiding proprietary models offered as a service, there’s really no rational reason to shun open models. If anything, it’s important to develop them into a viable alternative to corporate offerings.
I think it’s an extension of people only conceiving these things within capitalism (although they might call it techno feudalism or some shit), I remember the phrase “if you aren’t paying for something you’re the product” and thinking that so many people don’t realize we already have things that fall outside of that like so much of the FOSS ecosystem including Linux. It doesn’t help that this kind of messaging is so amplified by liberals on social media who refuse to see the real cause behind our current issues with AI and instead focusing on idealism.
Also, if y’all are interested, run local models!
It’s not theoretical.
The cost of hybrid inference is very low; You can squeeze Qwen 35B on a 16GB RAM machine as long as it has some GPU. Check out ik_llama.cpp and ubergarm’s quants in particular:
https://huggingface.co/ubergarm/models#repos
But if you aren’t willing to even try, I think that’s another bad omen for local models. Like the Fediverse, it won’t be served to you on a silver platter, you gotta go out and find it.
It’s really unfortunate how a lot of people have a knee jerk reaction towards anything LLM related right now. While you can make good arguments for avoiding proprietary models offered as a service, there’s really no rational reason to shun open models. If anything, it’s important to develop them into a viable alternative to corporate offerings.
I think it’s an extension of people only conceiving these things within capitalism (although they might call it techno feudalism or some shit), I remember the phrase “if you aren’t paying for something you’re the product” and thinking that so many people don’t realize we already have things that fall outside of that like so much of the FOSS ecosystem including Linux. It doesn’t help that this kind of messaging is so amplified by liberals on social media who refuse to see the real cause behind our current issues with AI and instead focusing on idealism.