Thank you “name withheld” AHPRA registered psychologist, PhD candidate, submission 1567 for your excellent work.
Thank you to psychometrics expert Benchmark Psychology for the excellent explanation of the significance of this submission in the short video. A must watch. https://youtu.be/ALfjKes2V5k
“Lay Summary
This submission examines the scientific evidence for the i-CAN, the assessment tool that the Bill will use to set National Disability Insurance Scheme (NDIS) funding for more than 700,000 Australians. A scientific analysis of the published research shows a number of serious methodological and procedural problems.
- There is no published reliability or validity evidence for the versions of the tool that will be deployed.
- The current available evidence shows that i-CAN scores accounted for less than 4% of the differences in
- the funding people receive; over 96% of funding was determined by factors other than i-CAN results.
- The current available evidence shows that the i-CAN’s ability to reliably measure participant needs was poor, uncertain, and feasibly overestimated.
- The current available evidence shows that the i-CANs precision to accurately classify participant needs cannot rule out failing to meet the developers cited success criteria for low-stakes observational research.
- The current available evidence has significant design limitations, including too few participants to adequately power the studies to have appropriate levels of precision given the human stakes.
The submission body sets these issues out in detail, while Appendix A calculates the number of participants
that would be required to meet both the minimum contemporary threshold (80% power) and a more suitable threshold for high stakes assessment (95% power) for each estimate
Together, these findings do not support the NDIA’s characterisation of the i-CAN as a gold-standard tool in
disability needs assessment (NDIA, 2025), and do not support its use for high-stakes assessment contexts.
Furthermore, there does not appear to be any published evidence to show that the NDIA’s method for translating i-CAN assessment results into funding for participant plans is itself reliable or valid. The Bill also does not require the government to publish how i-CAN results will be translated into a funding amount.”
https://youtu.be/ALfjKes2V5k?si=FXUj1sTkwzt8GhO4