I finally sat down and set this up last night. The Qwen 3.6 35b is very impressive and following this vid will get you running at workable speeds. I haven’t bumped it up to the 256k context window yet, but even with the 128 I’m able to have it work on projects I’d rather not be putting up on the cloud.
I have a eight gigabyte GPU and I wanted to run a thirty billion model or thirty-two or thirty-four or whatever. And being able to run that big models with actually decent speed is super cool.
As long as you’ve got the ram somewhere it seems to be just a matter of tweaking where your layers are. I still gotta get a proper handle on that but even the defaults this guy gives is a good start with my 1080.
Yeah. I have been considering getting a better card for gaming and AI, but I might consider delaying it now