Apple Machine Learning Research has a new preprint: “The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity.” [Apple, PDF] “Lar…
Another thing that’s been annoying me about responses to this paper… lots of promptfondlers are suddenly upset that we are judging LLMs by abitrary puzzle solving capabilities… as opposed to the arbitrary and artificial benchmarks they love to tout.
Yeah any time its regurgitating an IMO problem it’s a proof it’salmost superhuman, but any time it actually faces a puzzle with unknown answer, this is not what it is for.
Another thing that’s been annoying me about responses to this paper… lots of promptfondlers are suddenly upset that we are judging LLMs by abitrary puzzle solving capabilities… as opposed to the arbitrary and artificial benchmarks they love to tout.
Yeah any time its regurgitating an IMO problem it’s a proof it’salmost superhuman, but any time it actually faces a puzzle with unknown answer, this is not what it is for.