- YouTube
youtu.be
Auf YouTube findest du die angesagtesten Videos und Tracks. Außerdem kannst du eigene Inhalte hochladen und mit Freunden oder gleich der ganzen Welt teilen.

I finally sat down and set this up last night. The Qwen 3.6 35b is very impressive and following this vid will get you running at workable speeds. I haven’t bumped it up to the 256k context window yet, but even with the 128 I’m able to have it work on projects I’d rather not be putting up on the cloud.

  • ZWQbpkzl [none/use name]@hexbear.netEnglish
    2·
    1 month ago

    Q4_K_M only got me to like 21.5tk/s My graphics card is a 5700XT which benches slightly below your GTX 1080. My Ryzen 9 5XXX is significantly stronger than your i7-6XXX so I’m can’t blame that bottle neck. Seems like CUDA does have an edge here.

    • JoeByeThen [he/him, they/them]@hexbear.netOPEnglish
      2·
      1 month ago

      Neat. Were you able to try the 256k context window? I’m still using the 128 but I have noticed that once it gets to be about 88% (smh. suss!) full, the reasoning has a tendency to get a bit loopy and the model can’t get itself out of reasoning. Like the same 4 lines over and over, type shit. Still, very capable until then! I’ll be curious to see if the 256 has the same issues and how much of the bigger context window will actually be usable.

      • ZWQbpkzl [none/use name]@hexbear.netEnglish
        1·
        1 month ago

        I’ve been using llama bench instead of actually using it for something so I haven’t messed with context too much. For actual work I’m still using the macbook which does really well.

        I have to give some credit to Apple here. It seems like they’re betting on the AI market crashing and positioning themselves as the easiest local AI product. AMD has something similar called Strix Halo but its all soldered on memory.

        • JoeByeThen [he/him, they/them]@hexbear.netOPEnglish
          1·
          1 month ago

          Ah, gotcha.

          I’m not too familiar with Apple’s place in all this beyond some folks (organically?) figuring out that loading up some mac minis with a bunch of ram would let them run the larger models. Has Apple themselves been making moves marketing themselves for local AI?

          • ZWQbpkzl [none/use name]@hexbear.netEnglish
            1·
            1 month ago

            All Apple computers come with Unified Memory meaning the GPU and CPU shares the same memory pool, have very high memory bandwidth, and they provide their own CUDA/ROCm/Vulkan equivalent SDK called Metal.

            So any Mac can decently run any AI model that fits into its system RAM. For the same price as a high end AI card you can get a whole computer that can run larger models but at slower speeds.

            • JoeByeThen [he/him, they/them]@hexbear.netOPEnglish
              1·
              1 month ago

              All Apple computers come with Unified Memory meaning the GPU and CPU shares the same memory pool,

              Oh, I see! I thought that was just an efficiency thing in the mac mini. Got it.