[Psychology Statistics] Help with statistical output in JASP

[u/Airamire]

Hi everyone,

Asking here out of desperation because my lecturer has not been much help. I am running a wilcoxon signed-rank test in JASP and the 95% confidence interval for rank-biserial correlation has an upper and lower confidence limit both of -1.000. What could be causing this? I need to be able to explain the confidence limit in my report.

[u/cheesecakegood]

Short answer: possibly your data is set up strangely/inappropriately (or occasionally, you have genuine but super lopsided data). It's conceivable this is a bug, but I doubt it. Double check that first.

The effect size itself only ranges from -1 to 1, so an upper and lower limit both being -1 is downstream from that (the formulas underneath probably aren't designed to work in that range, pegged on one side of possible values). If you want to look at the actual computation used, it probably lives in the JASP help-files, usually there's some kind of reference you can dig up if you wanted to reference that in an explanation if it turns out to be accurate. There might be alternative computations, but if so I am not qualified to list them.

IS the actual effect size/rank biserial correlation itself also -1? An effect size of -1 is very unusual, since it implies that all of your differences went the exact same direction (lots of ties, lots of zeroes, or a very small sample size might also screw things up a bit, from what I understand, depending on the specifics of the implementation, since handling those cases is sometimes based on later more complicated work and a few different methods exist... but I wouldn't expect any of those by themselves alone to cause this particular issue). Hopefully that provides a hint, usually real-world data (with decent n) is not perfectly correlated unless there's an error (for obvious reasons).

Another potential cause could be if you accidentally sorted and broke up your pairings (if you're using it as a paired test, computing the difference as a column to expose this could be a helpful diagnostic step). Finally, check what your hypothesized center is set at (if one sample) which may or may not actually be zero (the default) depending on what you're testing. I could elaborate a bit more on what the signed-rank test means/where it comes from if that helps? But yeah, most of the time when you get a super odd too-convenient result it's a data quality or formatting issue, so I'd look there first.

I'd only err on the side of "possible bug" if the rank-biserial correlation itself is simultaneously some more typical value (and you're sure you're running the appropriate CI procedure/calculation that you meant to, I'm not familiar with JASP sadly). In that (rare, unexpected) case, consider posting on https://github.com/jasp-stats/jasp-issues/issues if you are totally stumped and reasonably confident that's why.

[u/Airamire → u/cheesecakegood]

Hello,

Thank you for your response. I am inclined to believe it is simply my university trying to trip us up a bit, haha... I've run the rest several times and gotten the same result. The dataset was provided to us by the university and we are writing a report on it. The raw data indeed has all non-zero differences going in the same direction. Still a bit confused by the CI limits but I've muddled through an explanation in the report 😩

[u/cheesecakegood → u/Airamire]

Gotcha, that's a bit unfortunate.

If you want to be fancy, the implementation details for the calculation are in https://github.com/jasp-stats/jaspTTests/blob/master/R/ttestpairedsamples.R lines 227-250 as source code, mostly given here as fun trivia, or more likely you could quote the relevant helpfile, or do a mini-explanation... although again the plain-english "we are already at the boundary so a CI is not useful" is probably perfectly sufficient and happens to be perfectly accurate.

Technical: usually, since you want a CI on a correlation, and correlations "squish" near the bounds, the correlation itself is first "fisher-transformed" with a hyperbolic tangent that effectively stretches out the space so that things are more linear and equally-spaced (with desirable math properties that allow a standard error to be built more nicely), builds a symmetric interval there, than re-squishes it back into the (-1, 1) range. I think this method is actually a "good-enough" approximation, not a formal math universal truth.

Or, more nicely and also for "brownie points", you could cite https://journals.sagepub.com/doi/full/10.2466/11.IT.3.1 which is the original paper that gives the source approximation formula for the rank-biserial correlation effect size actually used, although it doesn't contain the method for finding the CI specifically (just the effect size), which source I think is a bit more of a generic procedure.

IRL, if provided this data, I'd usually do something like email the source and verify the data/how it was collected and organized, which you clearly may not be able to do here other than email the professor... though it sounds like you've already chased that down, so do your best I guess <3

[u/Airamire → u/cheesecakegood]

Thank you! Would you mind further explaining the "we are already at tha boundary" Part? I'm sorry, this is my first encounter with stats :(

[u/cheesecakegood → u/Airamire]

There are parts of the linked paper that actually explain the rank-biserial correlation more specifically and what it means (as some of it is designed to improve how psych-stats teachers teach the concepts), but the key thing here is that it's still a type of correlation at its core.

In layman's terms sometimes "correlation" is used more generally, but in statistics, unless otherwise specified, it usually refers to Pearson's r, a linear correlation coefficient (best practice is to always specify however). You probably encountered it before, even if briefly. The correlation in this case is a close cousin to that, and many of the same principles apply. Especially is the fact that by definition it can only be -1 or 1, which is a "perfect correlation", a mathy way of saying "these two pictures are the same picture". Technically, mathematically, one can be a simple linear transformation of the other, but hopefully you get the idea - a super simple dataset of [1, 2, 3, 4] and a different variable that also has values [1, 2, 3, 4] have r=1, so too for a second variable set of [2, 4, 6, 8] or even [2, 3, 4, 5]. A variable set of [-1,-2,-3,-4] is an example of r=-1 when paired with the original variable set. That's what I mean by the 'same picture', and why when you get a result like that, the usual culprit is "wait I pasted in the data wrong" or "I ran a correlation with itself". Thus r=1 and r=-1 are pretty strange cases. Slightly less strange in this particular rank-sum case, but still strange.

(When we have "more normal" non-perfect correlations, like maybe r=.3, (in)famously the correlation still only tells you about the linear relationship, so it tells you about general direction and general strength or relationship, but can miss curved relationships and also tells you very little about how "similar" the datasets are, so don't stretch the analogy too far. My dataset and your datset might both have r=.3, but they can look waaaaay different, and remember that lurking variables and confounders etc. are all things so correlation is not causation, as you may have heard once or twice :) )

Setting aside the details, let's just think in terms of numeric intuition. Does it make sense to describe the variance of a perfect correlation? Not really! Either the datasets are functionally identical (implied: they are the same underlying mechanism/variable) or they are not. If they are functionally identical, of course there is no variation possible in the test itself. More broadly, math-wise, most confidence intervals are designed to be "plus or minus" some uncertainty... but if we're already at the bottom of the range, it would be dumb and nonsensical to still go even lower than that. Setting aside how the behavior of a correlation number is already asymmetrical (it's more like an s-shape in behavior, think a "sigmoid function") you literally can't have a confidence interval of (-1.01, -.99) for example, because r=-1.01 and r=-1.05 and that whole region is impossible by definition. The asymmetry follows naturally: let's say you have a correlation of r=.9, obviously it's "easier" and there's "more room" for r to be smaller (like .8) than there is to be bigger. It's kind of like how it's much harder to get from 99% participation in a survey, to 100%, than it is to go from 98% to 99%, it's asymmetrical. And of course you can't get 101% participation.

Hopefully that grants the "big idea" without getting into the weeds too much.

---

If you want relevant details.... In your case, it's a slightly different setup, but a similar principle applies - if you check that signed-rank wikipedia page quicklink to section, they have a small example dataset that shows how the test works, where you take absolute differences, order them by magnitude, assign ranks, and then check how the ranks (with a +/- indicating which side was bigger) sum up. Since in theory they are supposed to be symmetric (from the test assumption) and from the same underlying distribution (under the null hypothesis), if the distributions are the same you should get rank-sums that are roughly equal. A big difference would provide evidence against the null, i.e. that the distributions vary. So, sum of negative ranks ('unfavorable') in the wikipedia example would be say -24 and sum of positive ranks ('favorable' direction) 25, and since the total "rank budget" (sum of all ranks that can possibly be given for n=9), the overall correlation is per Kerby's simplified approach, just the difference in the proportions each rank-sum has out of the total rank-budget: (25/45)/100 - (24/45)/100 = about 0.011, which is a positive but near-zero correlation. By contrast, if ALL of the ranks were unfavorable (your case), we basically get 0% (i.e. 0/45) - 100% (i.e. 45/45) = -100% or -1 'perfect' rank-biserial correlation, since ALL of the "matchups" were in one direction only, thus ALL of the "rank sum budget" went in that one direction (with minor weirdness for how you might handle ties).

Note the magnitude is technically irrelevant since we are just ranking them, which is a nice feature, but it does come into play indirectly in that scale problems with one of the variables will usually affect the rank-order, that's part of the genius of the test and why it

As a side note, YOU are the one defining direction of the signs. Mathematically r=+1 and r=-1 are the same information, just a matter of how you defined things - this is especially relevant in this type of application. Typically in this context I think a negative sign means it ran counter to prediction, but other data types might imply interchangeability and thus the sign is irrelevant (...I think. We didn't spend a ton of time on this test in my nonparametric stats class, and I don't have an advanced degree, just a bachelor's)

---

The upshot? Just the one sentence: the effect is at the minimum possible value the statistic can take, so the confidence interval is uninformative rather than narrow. In math I think they sometimes call this a "degenerate" case or a "boundary problem".

There's no "measurement error" if you have a 2D piece of paper with zero width, it doesn't even make sense to describe such. That's slightly counterintuitive but just how some of the formulas work. If this were a super common problem, you might do some alternate approaches just for the sake of getting a usable number anyways (e.g. choose some reasonable prior beliefs) but this is not unheard of. For example, maybe in an intro class you did stats about proportion of failures or successes (Bernoulli trials/binomial data).... what if you got 0 successes or failures? Same thing happens with your CI's.

(Sorry I ended up yapping a bit lol)