1. In a controlled test of 600 answers to one lateral thinking riddle, Claude Haiku 4.5 refused to answer 87% of the time in Ukrainian and 0% of the time in English.
2. Across the same 600 answers, Fable 5 found the full solution 83 times out of 120, Opus 5 found it 37 times, Opus 4.8 found it 25 times, and Sonnet 5 and Haiku 4.5 found it zero times.
3. Sonnet 5 produced a physically correct partial answer in all 120 attempts and never once used the tool the riddle handed it, which is the difference between a right answer and the right answer.
4. Raising reasoning effort from low to high took full solutions from 14 to 35 out of 120 per level, and the two effort levels above high added three more between them.
5. Adding one paragraph defining the correct answer to the judges' instructions moved the overall full solution rate from 83% to 24%, with no change to the underlying answers.
6. That same paragraph moved Sonnet 5 from an apparent 1.98 out of 2 to a flat 1.00, and turned reasoning effort from a waste of budget into a lever that nearly tripled results.
7. An LLM judge given no reference answer grades confidence and fluency rather than correctness, which is why reference free evaluation runs systematically generous.
8. The entire experiment cost $31.32 in compute and roughly two hours of work, which is less than most teams spend deciding which public leaderboard to believe.
9. Opus 5 at maximum reasoning effort took a median of 95 seconds per answer and cost about $0.14, roughly 27 times the price of a Haiku 4.5 answer.
10. One riddle is a probe rather than a benchmark, so the conclusions about evaluation method travel much further than the conclusions about any individual model.

Read the full piece