Use of Item Response Theory approach will begin in September, with possible recommendations by next summer
The Federal Motor Carrier Safety Administration (FMCSA) has agreed to implement a series of actions as it works toward the possible implementation of an Item Response Theory (IRT) approach to identifying at-risk carriers for intervention.
IRT is a data-driven approach that has been used in other settings, such as ranking hospitals, says Science Daily. It is relatively unproven in trucking, but FMCSA and NAS believe that if utilized correctly, it can provide improved modeling data to identify carriers at the most risk.
Since its launch, FMCSA’s CSA program and the underlying SMS methodology used to determine carrier risk scores has been under fire from the industry, saying that is improperly identifies carriers for interventions. In February 2014, the GAO issued a report, “Modifying the Compliance, Safety, Accountability Program Would Improve the Ability to Identify High Risk Carriers,” that suggested FMCSA revise Safety Measurement System (SMS) methodology to better account for data limitations and that the agency consider those data limitations when determining a carrier’s fitness to operate.
The National Academy of Sciences (NAS) followed that report with one of its own, saying that FMCSA’s approach was defensible, but a more statistically driven approach to data collection, something known as IRT, would improve the process.
NAS made 6 recommendations that FMCSA acknowledged could possibly improve the process. FMCSA issued its formal response to those recommendations on Tuesday and has scheduled a public meeting for Aug. 29, 2018, in Arlington, VA, to discuss its new approach.
“The FMCSA is prioritizing the work to be completed in response to the NAS study and to meet the requirements of the FAST Act,” the agency said in a 10-page document. “FMCSA, with the input of the NAS Standing Committee, will develop and run a small scale IRT model by September of 2018 and after evaluating the results, run a full scale IRT model by April of 2019. Additionally, FMCSA will work with the NAS Standing Committee on planning and scheduling public meetings throughout IRT development to further discuss the recommendations.”
Mandated by Congress in the Fixing America’s Surface Transportation Act (FAST Act), the NAS report suggested FMCSA look at whether public display of SMS scores is warranted and that the agency reconfigure its modeling when generating those SMS scores, perhaps replacing the current system with a model based on IRT.
In response to the NAS report, in July FMCSA formally rescinded two rulemakings that addressed enhancements to the CSA program until further study could be conducted. The agency has now outlined its path forward in addressing the NAS recommendations and potential changes to CSA.
FMCSA said its action plan will cost the agency $2.5 million, not including staff time, over 2 years, including $1 million for new contract labor costs for “additional expertise in statistical modeling.”
A Standing Committee established by NAS will assist FMCSA in its research over the next year, which will include both small-scale and large-scale IRT studies.
“While this type of model has not been widely used within the transportation industry, it may provide [FMCSA] with tools to make decisions about the importance of different variables within SMS, such as which violations to include in a BASIC, severity weights and others,” NAS wrote in its initial report.
In response, FMCSA has agreed to move forward with an IRT study of carrier data, starting with a small-scale study in September and a large-scale study by April 2019. After evaluation, FMCSA anticipates making recommendations on the future use of IRT data collection methods by summer 2019.
Addressing some of the additional points made by NAS, FMCSA agreed that better collection of data from states would be beneficial and it will explore agreements with the International Registration Plan (IRP) and the International Fuel Tax Agreement (IFTA) to help in those efforts. NAS said that collecting information by state would aid in smoothing out enforcement variations. Collecting better data, FMCSA said, would reduce the need to use “substitute values and would improve the quality of the data in SMS.”
To improve the data process, FMCSA will also consider improved collection of data during registration and the biennial update processes; voluntary reporting of data by fleets through incentives; conduct more outreach; and seek out better inspection software options.
FMCSA will also explore the potential benefits and costs (including privacy concerns) of collecting more detailed data from fleets such as driver turnover rates, type of cargo, method and level of compensation. NAS believes data such as this can be beneficial in identifying at-risk carriers, although FMCSA has some concerns.
An updated webpage for MCMIS data will be among the first steps FMCSA takes, agreeing with NAS that a more user-friendly system for researchers and carriers can assist in the safety mission of fleets.
Additionally, FMCSA will collect comments and data from carriers, insurers and shippers for a study on how SMS data is used and whether percentile ranks are beneficial to the public.
Finally, FMCSA said that once its IRT modeling is complete, it will evaluate whether absolute measures should be used for carrier intervention thresholds. NAS recommended the agency utilize a hybrid model incorporating both SMS percentile ranks and SMS measures.
In the meantime, FMCSA plans to update its MCMIS database to make all possible corrections to data prior to starting its IRT modeling study. It noted that existing violation codes could be removed or consolidated for modeling purposes.