Following the publication of a National Academy of Sciences (NAS) report last June that recommended changes to the way the Federal Motor Carrier Safety Administration (FMCSA) records and displays data in its SMS program, the agency has decided to scrap proposed changes. Instead, FMCSA said it will further study the NAS recommendations and work toward implementing them into its SMS website.
“As a result of the ongoing implementation of the NAS recommendations, FMCSA removed the preview from the SMS website and will not be proceeding with the proposed changes at this time,” FMCSA said in a Federal Register Notice.
Mandated by Congress in the Fixing America’s Surface Transportation Act (FAST Act), the NAS report suggested FMCSA look at whether public display of Safety Measurement System (SMS) scores – the scores that support the CSA program – is warranted and that the agency reconfigure its modeling when generating those SMS scores, perhaps replacing the current system with a model based on “Item Response Theory.”
Today, FMCSA formally rescinded two previous notices of enhancements to the CSA program in favor of further exploring the NAS recommendations.
The NAS study advised FMCSA to not make changes to the program using “ad hoc analysis” and instead adopt the Item Response Theory (IRT) model. IRT is a data-driven approach that has been used in other settings, such as ranking hospitals, says Science Daily. It is relatively unproven in trucking, but FMCSA and NAS believe that if utilized correctly, it can provide improved modeling data to identify carriers at the most risk.
In the NAS report last June, the group recommended six approaches FMCSA could take to improve the CSA program. Those are:
- FMCSA should develop an Item Response Theory (IRT) model over the next 2 years. If it is then demonstrated to perform well in identifying motor carriers for alerts, FMCSA should use it to replace SMS.
- FMCSA should continue to collaborate with States and other agencies to improve the quality of Motor Carrier Management Information System (MCMIS) data in support of SMS.
- FMCSA should investigate ways of collecting data that will likely benefit the recommended methodology for safety assessment.
- FMCSA should structure a user-friendly version of the MCMIS data file used as input to SMS without any personally identifiable information to facilitate its use by external parties, such as researchers, and by carriers.
- FMCSA should undertake a study to better understand the statistical operating characteristics of the percentile ranks to support decisions regarding the usability of public scores.
- Given that there are good reasons for both an absolute and a relative metric on safety performance, FMCSA should decide on the carriers that receive SMS alerts using both the SMS percentile ranks and the SMS measures, and the percentile ranks should be computed both conditionally within safety event groups and over all motor carriers.
The Federal Register Notice today did not specify exactly the approach FMCSA will take, but in a report issued at the Transportation Research Board’s annual meeting in January, FMCSA detailed its approach to the NAS recommendations. The report anticipated small scale IRT modeling to begin this summer followed by full-scale modeling this winter and continuing into summer 2019 with a final implementation plan in place by the summer/fall of 2019.
The notices FMCSA rescinded today included potential changes to the underlying BASICs to better identify carriers are risk. The October 2016 notice included a preview of six changes to SMS methodology, including those proposed in the 2015 notice.
- SMS Intervention Thresholds were adjusted to better reflect correlation to crash risk.
- Changes to the HM Compliance BASIC to segment by CT and non-CT carriers and to post motor carrier percentile rankings under the HM Compliance BASIC to the public.
- Reclassifying violations for operating while OOS under the Unsafe Driving BASIC rather than the BASIC of the underlying OOS violation.
- Increasing the maximum vehicle miles traveled used in the Utilization Factor to more accurately reflect the operations of high-utilization carriers.
- Increasing the minimum number of crashes in the Crash Indicator BASIC from two to three.
- Assigning BASIC percentiles only to carriers that have had an inspection with a violation in the past year.
