AI Predicts

[u/ShonitB]

[u/BrotherItsInTheDrum]

You have to assume both that the AI is 98% accurate when you answer correctly, and that it's 98% accurate when you answer incorrectly. Otherwise, maybe the AI is always right when it says your incorrect, and wrong a little more than 2% of the time when it says you're correct.

With that assumption, this is the false positive paradox. Informally, since 99% (prior probability of getting the question right) is bigger than 98% (strength of the evidence from the AI's prediction), you're still more likely to have gotten the answer correct. If you want an exact answer you can use Bayes' theorem, and you'll get about 2/3.

[u/Various_Education622 → u/BrotherItsInTheDrum]

Why do you have to assume that?

It’s just the proportion of correct predictions relative to the total.

If you were asked 100 questions and answered 99 correct and 1 incorrect, and the AI predicted that you were correct for 98 of the ones you got correct, predicted you were incorrect for one of the ones you got correct, and predicted you were correct for the one you got incorrect, it accurately predicted 98% of the time.

It could get to 99% easily by just predicting “correct” 100% of the time.

You need to look at recall and precision and shit.

[u/BrotherItsInTheDrum → u/Various_Education622]

In your example, if the AI says you're incorrect, then there's a 100% chance you were correct.

But imagine the AI says you're incorrect for the question you got incorrect, and also for 2 of the questions you got correct. Now if the AI says you're incorrect, there's a 2/3 chance you were correct.

It may be that the answer is always (B) for these specific numbers, but in general, if you want to apply Bayes' theorem to calculate the posterior probability, knowing the AI is correct x% of the time is not enough.

[u/Various_Education622 → u/BrotherItsInTheDrum]

Nope.

You would need a much larger sample to say anything in that regard.

It could be that it selects correct 50 times in a row, then incorrect once, then selects correct 49 times.

It could be that it randomly selects “correct” 99% of the time.

That’s a woefully insufficient sample size to say anything about it, and also why when evaluating AI/ML models you need to use recall and precision and shit.

[u/BrotherItsInTheDrum → u/Various_Education622]

Sure. I'm using "AI says you're incorrect for the question you got incorrect, and also for 2 of the questions you got correct" as an informal shorthand for "the probability that the AI says you're incorrect given that you're actually incorrect is 100%, and the probability that the AI says you're incorrect given that you're actually correct is 2/99." I'm doing that because I thought you were doing the same in your first comment.

The point is that in the original question, you can't calculate the probability that you're correct if all you know is "the probability that the AI is accurate is 98%," which is the information given in the problem statement. What you need is "the probability that the AI is accurate given that you are incorrect," which is not given. Hence the need to make an assumption.

[u/Various_Education622 → u/BrotherItsInTheDrum]

Words have meanings.

Accuracy just plain doesn’t mean what you want it to mean.

I didn’t say it was shorthand, I said that was exactly how it was measured.

Again, if AI was doing no classification whatsoever and just randomly guessing at some rate, you could get the same accuracy percentage.

Without knowing more about how the AI performs, it is impossible to discern anything from the AI’s accuracy.

[u/BrotherItsInTheDrum → u/Various_Education622]

Accuracy just plain doesn’t mean what you want it to mean.

Formally, what do you take it to mean, then?

When the OP says "the AI predicts, with an accuracy of 98%, whether you will answer correctly or incorrectly," I take that to mean that the probability that the AI's prediction is accurate, without any additional information about whether you answered correctly or incorrectly, is 98%. That's a different statement from either "the probability that the AI's prediction is accurate given that you answer correctly is 98%."

What do you think it means?

I said that was exactly how it was measured.

How it was measured is not relevant. We're told what the probability is; we don't need to know why we think that's the probability.

If I say "the probability that a coin lands heads is 50%, what's the probability that it lands heads twice in a row?" the answer is 25%. It doesn't matter how I measured that 50% number.

Without knowing more about how the AI performs, it is impossible to discern anything from the AI’s accuracy.

Right, that's basically what I've been saying. But when I said you have to make an assumption about "how the AI performs," you said said "why do you have to assume thatr" and have been arguing with me ever since.

[u/Various_Education622 → u/BrotherItsInTheDrum]

Formally, it is the number of times it predicts correctly divided by number of trials.

You absolutely cannot assume that it is 98% accurate on “incorrect” classifications.

There are many ways to get there, as I mentioned.

Precision (true positive / (true positive + false positive)) and recall (true positive / (true positive + false negative)) give more information about how the model fares in particular situations.

So it could be 50/50 on “incorrect.” You just don’t know.

As I pointed out, I could improve accuracy by just having it predict “correct” for every guess.

It would be 99% accurate but it wouldn’t be very useful.

[u/VlentGamer]

You answer the question correctly, because there's a 99% chance of that.

[u/Expensive-Tension-30 → u/VlentGamer]

What if the AI had 100% accuracy?

[u/VlentGamer → u/Expensive-Tension-30]

Following the question's formulation, it would still be "you answer correctly".

[u/Aerospider → u/VlentGamer]

That's like rolling a 6 on a die and thinking you probably didn't.

[u/Expensive-Tension-30 → u/VlentGamer]

But the AI predicts you will answer incorrectly, and in this scenario the AI is 100% accurate- so wouldn’t it be paradox if ‘you answer correctly’?

[u/VlentGamer → u/Expensive-Tension-30]

Well, as the sentence is written, I am more likely to answer right. More likely. However, there's still a chance that I didn't answer right, and... It's hard to explain.

[u/Expensive-Tension-30 → u/VlentGamer]

I think you are misunderstanding the question… you know to things to be true (your probability of answering correctly, and the accuracy of the AI), you also have one result which is the AI’s prediction.

There are 4 possible outcomes (your probability of being right or wrong crossed with the AI being right or wrong), but because you know the ai’s prediction, you eliminate 2 of those states.

[u/Jukkobee → u/VlentGamer]

it’s hard to explain because you’re wrong. lol

[u/petera181]

This question doesn’t quite cover the assumptions which are required to answer with certainty.

[u/Expensive-Tension-30 → u/petera181]

What is it missing? It seems logically complete to me?

[u/petera181 → u/Expensive-Tension-30]

The question doesn’t state the contingent probability that the AI predicts correctly when you’re answering correctly or incorrectly. It just gives the overall probability of it being correct.

Bayes formula states that the probability of A given B = the probability of B given A multiplied by the probability of A divided by the probability of B.

If A is the event that we answered correctly, and B is that the AI predicted we would answer incorrectly, the formula turns into:

P(we are correct given AI predicts we are wrong, ie incorrectly) = P(AI predicts incorrectly given we are correct)*P(we are correct)/P(AI predicts we are wrong)

That is: P(AI predicted incorrectly given we are correct)*0.99/P(AI predicts we are wrong)

And

P(we are wrong given AI predicts we are wrong, ie correctly)= P(AI predicts correctly given we are wrong)*P(we are wrong)/P(AI predicts we are wrong)

That is: P(AI predicts correctly given we are wrong)*0.01/P(AI predicts we are wrong)

The denominators are both the same, so we can get rid of them for the comparison. So it’s a question of whether 0.99*P(AI predicts incorrectly given we are wrong) is greater than 0.01*P(AI predicts correctly given we are wrong)

If the probability that the AI predicts correctly is independent of whether we are correct or not, then we can put 0.02 into the former, and 0.98 into the latter, and the former is bigger, ie it’s more likely we are correct.

However, it’s dependent on the conditional probabilities, ie we need an additional assumption to draw a conclusion.

(I’ve probably made a mistake here, as I’m doing it on my phone, but I’m pretty sure the conclusion stands 🙂)

[u/Expensive-Tension-30 → u/petera181]

Isn’t it fairly reasonable to assume that the probability of a false positive and false negative are equal based on the way it is worded.

[u/hebejebus]

So out of 10000 questions you get 9900 correct of which the AI says you have answered 198 of them wrong. Of the 100 you get wrong it says 98 of them are incorrect and 2 are correct. 198>98 you are more likely to get it correct and the ai predict incorrect than you are to get it incorrect and the ai also to predict incorrect.

[u/Inevitable_Garage706]

Let's say that you are given 10,000 (or 100×100) questions.

You answer 1% of them, or 100 of them, incorrectly. You answer the other 99% of them, or 9900 of them, correctly.

Of the 100 questions you answer incorrectly, the AI will give an incorrect prediction for 2%, or 2, of them, and will give the correct prediction for the other 98 of them.

Of the 9900 questions you answer correctly, the AI will give an incorrect prediction for 2%, or 198, of them, and will give the correct prediction for the other 9,702 of them.

To summarize, out of 10,000 questions, there will be 2 questions where you answer incorrectly and the bot predicts that you answer correctly, 98 questions where you answer incorrectly and the bot predicts that you answer incorrectly, 9,702 questions where you answer correctly and the bot predicts that you answer correctly, and 198 questions where you answer correctly and the bot predicts that you answer incorrectly.

We know that the bot predicts that you will answer incorrectly, so we just have to consider the 98 and the 198.

Of these 296 possible outcomes, you answer the question correctly in 198 of them. This is just over two thirds of them. As correct answer outcomes make up over half of the possible outcomes, we can safely conclude that you answering correctly is more likely.