NAS issues recommendations on reforming CSA program

The National Academies of Science, Engineering and Medicine released a report that gave recommendations on reforming the Federal Motor Carrier Safety Administration’s Compliance, Safety and Accountability program.

   The National Academies of Science, Engineering and Medicine (NAS) earlier this week published preliminary findings that gave recommendations on reforming the Federal Motor Carrier Safety Administration’s (FMCSA) Compliance, Safety and Accountability (CSA) program.
   The CSA program is designed to improve safety and prevent commercial motor vehicle crashes, injuries and fatalities, according to the FMCSA. The program is comprised of three main components: the safety measurement system (SMS), interventions, and a safety fitness determination rating system.
   CSA scores were removed from public view with the enactment of the Fixing America’s Surface Transportation (FAST) Act, which President Barak Obama signed into law in December 2015.
   Looking at the current SMS, NAS’ report said it “is structured in a reasonable way, and its method of identifying motor carriers for alert status is defendable. However, much of what is now done is ad hoc and based on subject-matter expertise that has not been sufficiently empirically validated. This argues for FMCSA adopting a more statistically principled approach that can include the expert opinion that is implicit in SMS in a natural way.”
   The report also said the FMCSA should develop an item response theory (IRT) model over the next two years, and if it performs well in identifying motor carriers for alerts, the FMCSA should use it to replace SMS in a manner similar to the way SMS replaced SafeStat.
   Steve Bryan, president and founder of Vigillo, a SambaSafety company that analyzes fleet safety data in the trucking industry to provide insights that impact safety performance, believes IRT can improve CSA, but while IRT is well understood by statisticians and has been widely used for years in many different applications, it is far more complex than the current CSA methodology.
   The report also said the FMCSA should continue to collaborate with states and other agencies to improve the quality of motor carrier management information system (MCMIS) data in support of SMS. It also noted that two specific data elements – carrier exposure and crash data – require immediate attention, because “current exposure data are missing with high frequency, and data that are collected are likely of unsatisfactory quality.”
   The “FMCSA should structure a user-friendly version of the MCMIS data file used as input to SMS without any personally identifiable information to facilitate its use by external parties, such as researchers, and by carriers,” the NAS report added. “In addition, FMCSA should make user-friendly computer code used to compute SMS elements available to individuals in accordance with reproducibility and transparency guidelines.”
   Bryan emphasized how MCMIS data “is a mess” and how Vigillo spends 20-30 percent of its time cleaning data before it begins to analyze it.
   “Good job FMCSA on what you have done with CSA so far, we (NAS) give you a solid 5 out of 10 for your efforts so far,” he said. “But there needs to be a sound statistical model applied to replace the ad hoc nature of how CSA is formulated today. More, and more reliable data needs to be gathered, and the data utilized needs to be more transparent and software developed to make such data more useful by all stakeholders.  Only then should CSA Scores be released to the public.”