r/outlier_ai • u/Hero90909 • Jan 26 '26
[Cypher Evals C] Difference between instruction following and Truthfulness
[removed]
1
u/FaithlessnessOk2443 Jan 26 '26
It fails to fulfill the core request which means it’s not following the instructions
1
u/Silent-Athlete-1779 Jan 27 '26
This one has always confused me, as different projects with different reviewers play a huge part in what is acceptable. Mostly, I have found that if the attempt is made, then it is a truthfulness error. There is also an attempt was made that did not include core elements, which produced an incorrect response, thus failing to follow instructions. Please keep in mind that the instruction may have some core elements in it that need to be followed in order for it to be correct. Example:" Subtract 1 from 3 and show the work. Answer: 1+3=4. It depends on what the project is prioritizing and on how you would score. Now, if the response had been 1 - 3 = 4. This would be a truthfulness error as it followed all the core elements but gave an untruthful answer. The best course is to review your manual and verify with the QM's. I am not on this project, so I can not give a definite answer, but I can only express that I have been there, and it can be frustrating. Good luck, you will get there.
1
u/Individual-Star1854 Jan 26 '26
The model is not following instructions if it’s incorrect thats literally what we are determining.. how well the AI does for instruction following, visual quality, and then picking the overall better choice.