I finally sat down and set this up last night. The Qwen 3.6 35b is very impressive and following this vid will get you running at workable speeds. I haven’t bumped it up to the 256k context window yet, but even with the 128 I’m able to have it work on projects I’d rather not be putting up on the cloud.
I’ve been using
llama benchinstead of actually using it for something so I haven’t messed with context too much. For actual work I’m still using the macbook which does really well.I have to give some credit to Apple here. It seems like they’re betting on the AI market crashing and positioning themselves as the easiest local AI product. AMD has something similar called Strix Halo but its all soldered on memory.
Ah, gotcha.
I’m not too familiar with Apple’s place in all this beyond some folks (organically?) figuring out that loading up some mac minis with a bunch of ram would let them run the larger models. Has Apple themselves been making moves marketing themselves for local AI?
All Apple computers come with Unified Memory meaning the GPU and CPU shares the same memory pool, have very high memory bandwidth, and they provide their own CUDA/ROCm/Vulkan equivalent SDK called Metal.
So any Mac can decently run any AI model that fits into its system RAM. For the same price as a high end AI card you can get a whole computer that can run larger models but at slower speeds.
Oh, I see! I thought that was just an efficiency thing in the mac mini. Got it.