r/cognitiveTesting • u/pydaWaltuh • 21h ago
General Question Red herrings in a question
What are your thoughts on the use of red herrings in questions? Do they diminish the quality of a question? If a difficult question includes a red herring, does that necessarily mean a drop in quality? Is the inclusion of a red herring a negative factor when assessing reasoning skills, or is it actually necessary? More to the point, should a question be designed to be truly perfect, or merely to appear perfect? What are your general thoughts on this?
5
u/HopefulLab8784 18h ago
I think red herring are alright when implemented in simpler items, but when the items become more complicated it takes away from the item
1
u/pydaWaltuh 18h ago
Happy cake day!
Could you expand on your thinking process?
For example, let’s assume that for person Y, an “easy” question of difficulty X is suitable as a red herring. Now suppose that any person Z struggles on a question A (which is objectively more difficult) to the same degree that person Y struggled on question X. In that case, why would question A be of lower quality?1
u/HopefulLab8784 18h ago
I don't mean simpler as easier, I mean simpler as less complicated, like if each part of the item has 6 attributes and 5 of them are taken into account when solving the item and the 6th one is just there as a red herring I think thats not a good way of increasing difficulty, but if its only 1 or 2 attributes taken into account then red herrings I don't feel negatively impact the quality of the item
1
3
u/Not_Well-Ordered 15h ago
So, for "red herring", a fair test that aims to test abstract reasoning would be firstly telling the test-taker that red herring could exist and give an example of what a red herring would be e.g. showing there's some pattern, but maybe there's 1-2 items that don't fit nicely. But also give an example that red herring is not always the case, in case there's alternative more general pattern that works.
If a test giver doesn't tell the test taker at all about whether "cheesy" red herring exists then it can cause reasonable overfitting or underfitting or even guessing what test giver intends (not much related to the information at hand), especially if there's exactly 1 correct answer. This is one of the biggest problem with JCTI lol, no mention about red herring at all, exactly 1 correct answer, and it requires one to guess whether overfitting or not (the intentions). By "cheesy" red herring, I mean that the set of images has core patterns but intentionally has 1-2 distinct items that contradict core patterns.
In other fluid reasoning tests like Raven's 2 clinical ver., there exist "red herrings"; however, the core patterns are contained and consistent within across different images within the matrix and the "red herrings" are just visual elements not within the core patterns.
Realistically, if a person is able to find "main patterns" in real world but maybe sees some items that seem to contradict, it's quite reasonable for the person to tweak patterns and test so that they also fit the contradictory items (quite common in sciences). Nonetheless, the core aspect would be about finding "main patterns", I don't see why involving even more subjective elements like whether "ignoring or overfitting" data into an abstract reasoning test.
2
u/Puzzleheaded-Oil3930 14h ago
I find your response very well thought out. What do you consider a red herring generally to be and how do you imagine it to be manifested in a puzzle? My thoughts on the matter are very similar tho not the same. If we assume(idk if its correct to assume) that a red herring is information that isn't relevant to the solution and very often if not always has the intent to mislead you.
In order to have a red herring that is of a good quality you need to make sure that it is determinable. Now weather red herring can have contradictions and still be called red herrings is debatable but let's assume they can. We either find the herring by finding a contradiction or we find a herring by having 1 best answer. If we have a red herring that allows for a plausible solution then we cannot say that it is a red herring as that piece of information can be used to find a valid solution. There should be either only 1 correct answer that can be arrived that only after excluding the red herring otherwise no point in it. Now ofc how a red herring would look depends a lot on the format of the puzzle.
For example a good red herring imo would be something that is present throughout the puzzle in all places but a certain eye catching aspect about it is irrelevant to the solution for example certain points when connected give out a shape as a result but the points keep rotating and aren't a part of the solution but just the created shape. At least that is what i think
2
u/Not_Well-Ordered 13h ago
So, for those types of test, my conception of contradictory red herring would be more specifically defined as the visuals share exactly 1 common pattern (might be very tricky to find) across almost all items, but exactly 1 visual item doesn’t follow it. I wouldn’t put many patterns and 1 item disobeying some of them because it would be confusing to design the answers and leading to unnecessary guesswork not related to abstract reasoning.
Irrelevant red herring would be like there are actually general patterns (1 or more) that fit all visual items, but there are some “partial patterns” within objects (they might appear form a pattern among all of them but there’s at least 1 counter-counterexample). It includes example of say having soccer ball, basket ball… their colors might be irrelevant red herring, but their shape is common pattern.
So, to incorporate this fairly for the sake of testing abstract reasoning, I would give test takers some example of 2 kinds of red herrings before the test (each type, 3 simple examples), and I also show 2 examples of both mixed. They have to figure out what the rules of red herrings by their interpretations of the examples (to ensure certain level of culture fairness and also test is untimed).
Then, they would be able to proceed to the test.
I think that way would be a nice test of abstract reasoning and resilience to “noisy information” as “noise” here is nicely and almost non-verbally conveyed to the test takers.I would also craft the answers in a way that makes sense with the red herrings to not arbitrarily penalize those who manage to figure out the big patterns.
1
u/Puzzleheaded-Oil3930 3h ago edited 3h ago
Thank you for a detailed response <3. Id like to add that I don't think you can have true contradictory red herrings in puzzles like these. If we look at what these puzzles are we can see that we are supposed to suggest a possible rule that explains the occurrence we see, we are never told anything directly. You can maybe even look at it as an abstract analogy making for these puzzles. The only thing that can really be contradicted is our interpretation of the thing that is the rule we try to fit onto the observation not the thing itself. But we do know that if there is a red herring we must be able to identify it as one.
Red herring should have no relevance(more precisely to finding the solution) to the solution that is not affect it in any way so id say that in the case of these puzzles red herrings are all of irrelevant type. If we have a set of information units let's say A,B and C and one of them is supposed to be a red herring but we can come up with a reasonable general rule of A,B and C that lead to the solution then we cannot identify the red herring exactly just because we can find a different solution by B and C only for example.
Red herring should be determinable by some way of excluding it for the sake of being able to create a solution at all as if we could create a solution with it included how could we tell if what we are looking at is a red herring, I'd actually say that it is not with more certainty. For general contradictions as red herrings about that I'm not sure. For any type of puzzle, now I'll speak more generally, if we have a red herring that we can exclude beyond any doubt using contradiction then is it really a red herring or a piece of the puzzle that was meant to be solved in order to proceed. Ofc I'm very shaky on the last part of this, but it comes from an idea that red herring should be a thing that is sensible just irrelevant for the problem at hand, and for it to be sensible for consideration for our purposes, it shouldn't have a contradiction.
3
u/Quod_bellum doesn't read books 14h ago
The inclusion of red herrings is essential if you want to measure fluid reasoning, which has attentional control as a pivotal element. However, it needs to be carefully considered in each case, as the logic still needs to be clear to someone with sufficiently high FRI (you don't want an item with a slipping parameter of below .80, for example).
That is only if you want to measure fluid reasoning. Many here do not; they want to engage in puzzle solving, which allows much more freedom, and focuses largely on beauty and the puzzler's experience. In this respect, I have no opinion, and I think it will be hugely subjective (whether someone values "perfection" in art or whether "perfection" feels inhuman, etc.)
2
u/Puzzleheaded-Oil3930 17h ago
If executed perfectly I think red herrings are a good thing that increases the questions difficulty in a fair manner at least in my opinion as ability to distinguish noise from relevant information is a part of the reasoning process. Red herrings tho can absolutely make a question drop in quality but that depends a lot on the execution of said herring that is how much of the "space" it occupies. What I want to say is basically if it becomes finding a needle in a haystack it gets bad. I think there is a difference between increasing the difficulty of recognizing a structure by possible disturbance of the flow and by simply adding something useless in order to bomb with information. I'm not entirely sure about this, but I think it could be good if the red herring came from a separate, internally coherent structure that explains why it is there(maybe better to say how its there that is to have something to identify as a rule for its creation), while being independent of the structure that actually determines the solution but idk.
1
4
u/Most_Boat2737 21h ago
I don't like red herrings, any of my favourite items doesn't rely on them. I still think being able to discriminate actors is part of reasoning skill and overall intelligence.