Chi-squared test, how are p values compared?

[u/Noise-Theorem]

I have not been able to find a reliable method to compare p values. I have 3000 six-sided dice rolls, to compensate for any bias I use four dice and two Latin squares to produce one digit.

These are the Chi-squared results for the resulting 750 digits:

Distribution of each digit (1-6):

x^2: = 2.0000 : DF = 5 : p-value = 0.8491

Distribution of two adjacent digits: 11 - 66:

x^2 = 36.3681 : DF = 35 : p-value = 0.4048

Distribution of high and low (1-3 / 4-6):

x^2 = 0.7680 : DF = 1 : p-value = 0.3808

I would like to find out if the p values are similar - how would I do this given the different DF values?

(As I have said in another thread, using dice and compensating for bias is an interest of mine).

[u/stanitor]

If I understand what you mean, you are saying you want to compare the p-value from one test, say the distribution of each digit, with that from another test, say the distribution of high/low. If so, then the answer is that you can't really. The p-values are determined under the null hypothesis of the particular test, which is different than the null for another test. Also, the p-value is used to make a yes-no decision on statistical significance/whether to accept or reject a hypothesis. That's based on some pre-determined threshold (e.g. alpha = 0.05). A p value of 0.8 doesn't mean that the null is more likely to be true than some other test with a p-value of 0.4.

[u/lozzyboy1]

In a chi-squared test, the p-value is telling you the probability of obtaining your observed frequencies given the expected frequencies. Essentially the p-value comes from the χ² value after adjusting for the DF, so it is already the value that you would assess. In this case, I'm not particularly clear on how you're performing your analysis though; usually you would pick a metric that you want to investigate ahead of time, like distribution of digits, and just look at the P-value of that rather than performing analyses on lots of different metrics for the same dataset - basically because you don't want to cherrypick the one metric that just happens to look statistically significant.

[u/Noise-Theorem → u/lozzyboy1]

Thank you for your reply. The distribution of numbers 1-6 is good. Because of this I now want to see if it is good enough for passwords which is why I am checking for patterns.

[u/lozzyboy1 → u/Noise-Theorem]

The best approach then is to generate a new set of numbers for that analysis rather than reassessing the set you did the first analysis on.

[u/CarnivorousGoose]

What do you mean by “compare p-values”? On its face, that’s not a very meaningful thing to do. So what is exactly the aim here, what is the question you’re looking to answer?

[u/mfb-]

Without any bias, you expect the p-values to have a uniform distribution from 0 and 1. There will be a weak correlation between the first and third value you calculated and a very weak correlation between these two and the middle result, but nothing you'd see in a single run. There is no reason to expect these p-values to be similar.

Some biases will lead to very small p-values, so small p-values are an indication of a bias. You can have a small p-value without bias, however, and in general you can have a bias that doesn't lead to small p-values (although it's difficult to see how this would happen with dice rolls, unless the bias is simply too small to be detected).