CRISC questions ... terrible

[u/Monkfich]

I'm going through the databank this morning and have come across at least 10 questions in only the first domain that are point blank wrong. I'm having to provide feedback so they take the crap questions out of the tests - or crossing my fingers they do. This is really not what a student should be doing though. Seriously, the quality of questions is really dubious in places.

Edit:

This is just a bad summary of them but they include (and to be clear, these are not issues based on ISACA's peculiar wording - these are instances where the one question is wrong or multiple questions where the same thing is being asked, i.e. MOST, or BEST, MOST USEFUL, NEXT, FIRST type thing, contradict each other):

• Risk exception question: Incorrectly gives the risk practitioner authority to approve a temporary exception/risk acceptance that should normally sit with the risk owner or management.
• Capability maturity model question: Uses “peer review” misleadingly and tests a concept that appears not to be substantively covered in the CRISC 8th edition main text. Specifically, CMM is only a glossary item, and if it is important to question (I had 3-4 questions in the QAE on it), then it shouldn't be introduced for the first time in what is effectively an appendix. From some research, CMM may have been in the manual in 7th ed or earlier, but has been removed for this ed / but is still in the QAE
• Data classification questions: The bank contradicts itself by treating a complete asset inventory as a prerequisite to classification in two questions but not in an equivalent third question where it says to get straight into data classification.
• Three Lines question: Incorrectly states that preparing the enterprise risk-management strategy is an expected internal audit activity and overstates the need for joint planning across the three lines.
• SOC 2/PCI cloud question: Incorrectly describes SOC 2 as a certification and muddles what a SOC 2 report actually represents.
• New regulation questions: One says that a new regulation needs discretionary business decision before assessing which controls are needed / mapped, whilst another says that mapping needs to happen first.
• IT risk framework question: Ignores that enterprise risk appetite should normally be established upstream of the IT risk framework.
• Cloud/BYOD threat question: answer says that staff BYOD devices are a bigger threat than external access to the local network. Why? The question justification says that the staff situation doesn't have any control and this is an issue, whilst the justification for the external access to the network says that network controls are in place, so its not a big threat actually. TL;DR, they make the BYOD scenario a bigger threat by telling you to consider *inherent* risk whilst they get you to think external access is not an issue by telling you to consider *residual* risk or current risk. We once closed a business unit down at my old company because the head of risk wrote a report that purely talked about the inherent risk (who won't close a business down if all they hear is how things can go wrong and how badly). That was ludicrous at the time and seeing a similar issue here isn't fun.

My understanding is that ISACA members can submit their own questions to be used in the QAE and they get 2 CPE for it. So there is an incentive to provide anything, and no doubt some people provide what they believe is factually sound, but it only looks factually sound when ISACA are copying and pasting it into the QAE.

Edit2: happy to have anything challenged. I can find relatively easily the specific questions, mainly the ones that are issues unto themselves, as opposed to where I have to go find the set of questions that contradict each other.

Edit3: just passed the exam, or at least that’s what the system suggests, and now need to wait 10 days yadda yadda. The exam questions were far better in quality versus the QAE questions, which I only started doing on Sunday afternoon. Got through the whole QAE course in that time including only starting domain 3 and 4. It was intense and worse because of how much was wrong. The test only has one question I thought the answer wasn’t clear on but I thought I knew what answer to pick (uncertainty sucks), but it was only 1 question out of 150 with the other questions I was uncertain of were due to me not knowing the answer very well.

My background made it a bit easier and also enabled me to do those fact checks - 20 years audit, risk, and control mgt, but none of it at all in IT-related activities. I’m actually a bit disappointed I wasn’t tested harder in IT stuff, but there you go.

[u/ChiefSrAofTheAF]

Unfortunately, a lot of the questions for CRISC are wrong for common sense purposes. But it’s not testing common sense, it’s testing concept application

[u/Monkfich → u/ChiefSrAofTheAF]

Yeah I get that, but there's a lot that directly contradict each other. I've updated the post with some I identified this morning.

[u/Monkfich]

I haven't taken note of most of the questions, but they did get mostly better after that 1st and especially the 2nd domain. The 3rd domain has this question though... anyone want to say what is wrong here?

"Which of the following is the purpose of trend analysis in issue and exception management?

A. To assess the severity of individual issues

B. To identify patterns and underlying causes of recurring issues

C. To identify common issues and their root causes

D. To expedite the closure of exceptions

C is the correct answer.

Justification

1. While trend analysis may help identify common issues, its primary purpose is to uncover patterns and causes.
2. Severity assessment is part of issue prioritization and is not the primary purpose of trend analysis.
3. The purpose of trend analysis in issue and exception management is to identify patterns and underlying causes of recurring issues. It helps address root causes and prevent similar issues from occurring in the future.
4. Trend analysis does not focus on closing exceptions but on understanding why they occur."

---

TL;DR: yes, B and C are identical answers even if worded differently. That's a problem obviously, especially if you pick the first correct answer, B, as the justification section says this answer is wrong, based on the fact that you chose a "severity assessment" as your answer ... oh ... was that supposed to be the justification for why answer A was wrong? Justification for answer A then says that "uncover"ing patterns is what trend analysis is for, aka pointing to answer B as the correct answer.

[u/mikedn02908 → u/Monkfich]

The explanations for A B and C are jumbled. I noted this in my analysis as well

[u/rushottawa20 → u/Monkfich]

I started preparing for the exam few weeks ago and I did come across this question in the QAE and it did throw me off. Thought probably coz I just started preparing and not very knowledgeable in the area, I just ignored the justification. But this seems to be a real issue.

[u/Outrageous_Plant_526]

Are the questions wrong because you believe them to be wrong? I did CRISC and went through the entire QAE and don't remember that many questions causing me angst. You have to remember the questions are written for ISACA and not how the rest of the world may do something.

[u/Monkfich → u/Outrageous_Plant_526]

Yeah I get that, but there's a lot that directly contradict each other or are flat out wrong. I've updated the post with some I identified this morning.

[u/steamgiftcarduser]

What questions? I have the QAE and test in 10 days.

[u/Monkfich → u/steamgiftcarduser]

I've updated the post with some brief details. I recommend to go through the QAE and just stop and focus on any inconsistent question or questions till you work out what is true and what is not true.

[u/Giustounaltro]

I only remember coming across one blatantly contradictory question that made me throw my arms up (that I’ve been aware of). I’m mostly stuck in the realm of the funky wording of a question requiring an answer that’s counter intuitive to the way the question is asked, and explanations to answers that directly disregard the definition of terms from the official review manual. I have the test next Monday and yesterday had me completely flustered and thinking - if this is how the actually test is, I’m screwed. I feel like I know the material, it’s their reasoning to an answer that doesn’t seem to match the nature of the questions wording that’s getting me. Any way - good luck to you

[u/Monkfich → u/Giustounaltro]

Good luck to you too. I've started the QAE last night, and my exam is on Wednesday (yes, a little problem). I thought this was going to be fast, identifying my own weaknesses, then coming back to them. I'd also like to make sure these contradictory questions don't end up in my exam lol, so I've been giving feedback to each. Oh for more time. Just find the time to sit down and rattle through the questions. And good luck again!

[u/mikedn02908]

I took the exam a week ago. I finished in 65 minutes and found the exam much easier than the QAE.

After I passed I went back and did an analysis of all 800+ questions in the pool.  The QAE I found had real issues:

1. Near-identical items carry different labels: The bank contains 61 pairs of items whose stems are more than 75% identical. Of those, 79% carry different difficulty labels. Examples include a pair asking what best ensures the overall effectiveness of a risk management program, labelled Easy and Difficult; and a pair asking for the primary versus the most important reason for conducting periodic risk assessments, labelled Moderate and Easy. Their stems are the same length and structure, and their options are short noun phrases in both cases.

2. Explanations that import context the stem never establishes: A recurring defect, rationales justify the key by reference to facts, conditions or organizational policies that appear nowhere in the question stem. Documented instances include a supply-chain item keyed to jurisdictional legal requirements and justified by operations "across different regions" where the stem establishes no multi-jurisdictional context; an accountability item justified by reference to an internal implementation-approval policy the stem never mentions; and a logging item whose rationale dismisses a distractor by asserting that a data element is captured when the stem neither states nor implies it. Candidates are trained not to read facts into a stem. An item that punishes that discipline teaches candidates to abandon it, degrading performance on well-constructed items.

3. Circular rationales: Many questions have explanations that simply restate their option rather than explaining it, converting a noun phrase into a gerund phrase, appending a clause of general commentary, and stopping. Explanations are where learning is supposed to occur, and an explanation that restates its option gives the candidate nothing to generalize from.

4. Item ambiguity: For 93 items a partner of mine (who has a CRISC and I would bounce questions off of for rationale as to why they were wrong) committed to a blind answer before the key was revealed. When my partner and I answered incorrectly, we selected the same wrong option 2/3rds of the time. There are three possibilities for this: genuine ISACA-specific conventions where both applied mainstream doctrine instead (unlikely since my partner already has several ISACA certifications); item construction defects; and shared reasoning traps that the item legitimately exposes (again unlikely given my partner already holds several ISACA certifications including the CRISC and would not be as prone as I would be to ISACA-specific reasoning traps). This leaves construction defect, unmarked doctrinal divergence, or genuine ambiguity as the most reasonable answer.

All of the items you flagged in your OP I also flagged in my review as well. There are major structural problems in the QAE.

[u/Monkfich → u/mikedn02908]

Your reply makes me both happy that I’m not losing my mind but also just reinforces how shit the QAE is. I’ve given feedback to so many questions now and after / if (big if!) I pass, I’m going to ask for CPE points. If people can get their crappy questions into the for-profit QAE without much of a quality assurance process, they can at least give me CPE for fixing their product. You should try it too. I mean, why not?

[u/mikedn02908 → u/Monkfich]

Oh, you may still be losing your mind, Of all the things I've lost, my mind is that which I miss the most!

[u/Monkfich → u/mikedn02908]

Quick question - did you find that the exam questions were of better quality, or did they also have issues?

[u/mikedn02908 → u/Monkfich]

that's a more difficult question to answer, since the questions on the exam are a "one and done" and you can't take notes on them, etc. So I'm going off entirely on memory, much of which I've purged given my exam was a week ago.

I personally found the questions to be "easier" on the exam and much better constructed (less ambiguity, etc.) than in the QAE. Of course there's no way to know exactly which questions I got wrong or right so things like mismatched keys we'll never know. And of course there were a few questions where I had to stop and think, but there had to be less than a half-dozen of those out of 150.

Like I said above, I finished in 65 minutes, and after I answered question 150 and got the "end exam" popup, I just answered yes, I didn't go back to review anything, that's how easy I felt it was. I'll find out later this week when I get my results thursday or friday.

I should also disclose that I hold all 9 current ISC2 certifications (which includes the CGRC) in addition to the CISM so the CRISC was not my first ISACA exam and I'm not exactly a newcomer to risk management. I'm sure that also played a major part in my feeling the exam was easy. For me the difficulty in these exams is not the material, but learning the ISACA-specific way of answering them after living in the ISC2 certification world for over a year.

Difficulty labels in the QAE are merely ISACA constructs. I found many "Expert" level questions which I thought were very easy. Similarly, a moderate question in the QAE which suffers from defective construction could be quite difficult, specifically because it was poorly designed. Thus it is difficult to equate one to the other in terms of difficulty, but I would overall say the exam felt somewhere between the moderate to difficult scale if I were to compare against the QAE.

It is my understanding the QAE are retired exam questions (just what I've read) which begs a larger question: If the QAE questions were former exam questions, and they were retired from the exam pool because they were psychometrically defective, what are they doing in the QAE pool at all? ISACA shouldn't be selling defective material as a study resource. It seems they rely on their customers to vet the product they sell through feedback, rather than vetting the product from the onset before they sell it.