SambaSafety, powered by Vigillo, just announced the development of a new CSA scorecard utilizing item response theory methodology for the CSA Fast Act score model before its planned release.
The new SambaSafety CSA Scorecard powered by Vigillo has been in development for the past year and will be available to the company’s customers as an educational tool as early as November, according to Executive Vice President of SambaSafety Transportation Steve Bryan.
FMCSA released a report in June stating that it hopes to run a small scale IRT model in September, run a full scale IRT model in April 2019 and evaluate results and effectiveness of the full scale model in June 2019.
Bryan said the goal of SambaSafety CSA Scorecard powered by Vigillo is to help carriers get ahead of the curve and start adapting to the new methodology.
“When CSA was first being tested in 2008, Vigillo was the industry leader in analyzing what fueled CSA scores and made the first CSA scorecards available to the entire industry, nearly two years before the FMCSA program went live in December 2010,” Bryan said. “Today, SambaSafety is leveraging Vigillo’s expertise to help customers understand the new CSA scoring model long before it will be released.”
IRT methodology is widely used across various industries, including education and psychology. It is often used to access standardized tests and evaluate professional programs.
“An IRT model can evaluate motor carriers with varying safety data and characteristics by using violations to assess the safety performance of each carrier. The IRT model develops this assessment by determining the prevalence of each violation and the violation’s effectiveness in distinguishing between carriers with and without a strong safety culture,” the FMCSA report reads. “In addition, an IRT model will provide a measure of confidence in a carrier’s safety assessment based on the types of violations they commit.”
While the methodology is highly respected, its implementation will completely overhaul the way CSA scores are calculated. It will eradicate violation weights, CSA points, BASIC measures and safety event groups in favor of creating a single CSA BASIC score to represent a carrier’s overall safety culture.
“We believe that FMCSA’s approach utilizing the IRT methodology is moving in the right direction,” Bryan said. “Our analysis shows that the IRT model is an effective means of identifying a poor safety culture at motor carriers and will represent a significant improvement in the overall effectiveness of the CSA program.”
During a webinar announcing SambaSafety CSA Scorecard powered by Vigillo Wednesday, Bryan acknowledged that IRT is complex and difficult to understand, meaning users will have to out a certain amount of trust in the method without understanding exactly how it works. He hopes, however, that the outcomes will be understandable and inspire trust.