Insights ¦ The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity

Published by: Apple
Search for original: Link

Key Take Aways

Current frontier Large Reasoning Models (LRMs) exhibit fundamental limitations in developing generalisable reasoning capabilities, especially beyond certain complexity thresholds.
Empirical analysis reveals three distinct reasoning regimes: standard LLMs outperform LRMs at low complexity; LRMs gain advantages at medium complexity; both collapse at high complexity.
Scaling propensity shows a counterintuitive pattern: reasoning effort (token usage) increases up to a threshold but then declines despite facing more complex problems.
Performance collapse occurs even when models operate well below their token limits, indicating intrinsic scaling limitations.
Models frequently overthink simple problems by exploring incorrect solutions early, highlighting inefficiencies in their reasoning processes.
At moderate difficulty levels, models often identify correct solutions only after extensive exploration of incorrect paths, demons...