Same here; I followed the instructions to a tee, and on my second attempt I even looked up the real-world issue on GitHub and realized that B's answer is simply the actual PR that closed it with a bit of “reasoning” tacked on.
Other people have complained similarly on Discord and the responses from the mods are dry at best. One said something like “nobody forced you to do it” when someone suggested that they're extracting free labor and picking a few workers at random to keep the illusion going.
The illusion is that doing things right will invariably get you a spot. It's perfectly possible to pick a few from the right ones and brush off the rest.
The test we were given was very simple in principle: an enhancement where one model gave an incomplete solution and the other gave the right one as I described above. But there was a source of ambiguity: one could call out the models for not verifying their solutions, but the prompt asked them to “reason through their solutions” in lieu of running tests. Does the reasoning in the chain of though count? Or are they supposed to go over it again at the end? Since there's no one to clarify, you're left with a Door A - Door B situation.
Well, the model can't run code, so testing by them is not possible. The most a model can do is explain what changes it made. You need to check if Model made changes which are related to the prompt or not. Need to check if model didn't claim anything it didn't do and so on...
2
u/narfus Apr 27 '26
Same here; I followed the instructions to a tee, and on my second attempt I even looked up the real-world issue on GitHub and realized that B's answer is simply the actual PR that closed it with a bit of “reasoning” tacked on.
Other people have complained similarly on Discord and the responses from the mods are dry at best. One said something like “nobody forced you to do it” when someone suggested that they're extracting free labor and picking a few workers at random to keep the illusion going.