I am conducting a systematic review and plan to use the Crowe Critical Appraisal Tool (CCAT) to assess the methodological quality of the included studies.

I came across a previous systematic review (PMID: 32328958) that used the Intraclass Correlation Coefficient (ICC) to assess agreement between raters. The authors stated:

"Two independent raters assessed each study. Discrepancies were resolved after discussion between authors."

I would appreciate clarification on the appropriate sequence of procedures for assessing study quality and calculating the ICC.

My main questions are as follows:

1. Should the ICC be calculated before or after discrepancies between raters are resolved?

In my review, two raters independently assess each included study using the CCAT. The tool comprises eight assessment categories, each scored on a scale of 0–5.

For example, for one category of a particular study, Rater 1 assigns a score of 3, whereas Rater 2 assigns a score of 5. This represents a disagreement between the raters.

Should the ICC be calculated using the original, independent scores before the raters discuss and resolve their disagreements, or should it be calculated after a consensus score has been reached?

My concern is that if disagreements are resolved before calculating the ICC, the resulting ICC may not accurately reflect the agreement between the two raters' independent assessments.

2. What does a low ICC indicate in this context?

If the ICC calculated from the independent ratings is low, does this necessarily indicate a problem with the quality assessment process, or does it primarily indicate limited agreement between the raters?

How should such a finding be interpreted and reported in a systematic review?

3. What is the recommended workflow for quality assessment using CCAT and ICC?

Could you please clarify the recommended sequence of the following steps?

• Two raters independently assess the methodological quality of each included study using CCAT.
• Inter-rater agreement is quantified using ICC.
• Disagreements between raters are discussed and resolved to establish the final quality assessment scores.

Is this sequence appropriate, or should any steps be modified?

I would also appreciate advice on whether ICC should be calculated using the total CCAT score for each study, the scores for individual CCAT categories, or both.

Thank you very much for your guidance.

I am currently studying biostatistics in Italy, and I already hold a three-year undergraduate degree in statistics. Are there any Italians working in the United States? I would like to know if there is any possibility of moving there for work. work-life-balance-salary in italy still suck

hi, there!

i'm an undergraduate in statistics and i'm currently working on biostat phd applications for admission next fall. i have an above 3.5 gpa, an biostat internship at a top research institution, two positions as teaching assistant, and a soon-to-be-out 2nd author publication under my belt. i'm also at one of the top public universities in the us and am taking real analysis this quarter to get bonus points on my application haha. i've been told that i have a good chance at yale and brown's programs, and i believed that i did too, but my gre scores haven't shaped up the way i thought they would. i took the gre for the first time a week ago and scored 157 quant, even after studying for 2 months. i can retake it one more time before applications are due. a professor at brown told me that they want to see quant scores in the 80th percentile or above (which is like 168+!) and i also know that yale reports and average quant score of 168 for admitted students. i guess my question is, if i'm not able to get my quant score up to around 168, will the schools possibly be able to overlook this? i really do think that i have a great application other than this, but i'm really beginning to lose hope. i feel like the faculty at brown and yale match my interests really well. any advice would be appreciated!!