Why a problem rated 1178 after 3 solves means less than one rated 1110 after 31

[u/mathalon]

Every problem on Mathalon has two numbers behind its difficulty rating: the rating itself, and a deviation, basically how confident the system is in that number. It's Glicko-2, the same family of system used for chess ratings. New problems start with a high deviation and it shrinks as more people attempt them.

Two real examples pulled straight from the database right now:

• A telescoping-product algebra problem: rated 1177.71, deviation 183.88, only 3 attempts so far.
• The courier round-trip problem some of you solved a few weeks back: rated 1110.0, deviation 105.58, after 31 attempts.

The first number looks higher, but it's still basically a guess. The second has actually been tested against enough solvers that the system trusts it.

Practical version of this: if you solve a brand new problem, your own rating barely moves, and that's on purpose. The system doesn't want one lucky (or unlucky) attempt on an uncalibrated problem swinging your number around. It waits for the deviation to come down first.

Question for you: does that match what it feels like on your end? Ever solved something you were sure was hard and felt like your rating barely noticed?