From 2009 to 2012, large commercial trucks and buses have averaged about 125,000 crashes per year, with about 78,000 injuries and over 4,100 fatalities. In 2010, FMCSA replaced its tool for identifying the riskiest carriers--SafeStat--with the CSA program. CSA is intended to reduce the number of motor carrier crashes by better targeting the highest risk carriers using information from roadside inspections and crash investigations. CSA includes SMS, a data-driven approach for identifying motor carriers at risk of causing a crash. GAO was directed by the Consolidated Appropriations Act of 2012 to monitor the implementation of CSA. This report examines the effectiveness of the CSA program in assessing safety risk for motor carriers. GAO spoke with FMCSA officials and stakeholders to understand SMS. Using FMCSA's data, GAO replicated FMCSA's method for calculating SMS scores and assessed the effect of changes--such as stronger data-sufficiency standards--on the scores. GAO also evaluated SMS's ability to predict crashes. The Federal Motor Carrier Safety Administration's (FMCSA) Compliance, Safety, Accountability (CSA) program has helped the agency contact or investigate more motor carrier companies that own commercial trucks and buses and has provided a range of safety benefits to safety officials, law enforcement, and the industry than the previous approach, SafeStat. Specifically, from fiscal year 2007 to fiscal year 2012, FMCSA more than doubled its number of annual interventions, largely by sending warning letters to riskier carriers. A key component of CSA--the Safety Measurement System (SMS)--uses carrier performance data collected from roadside inspections or crash investigations to identify high risk carriers for intervention by analyzing relative safety scores in various categories, including Unsafe Driving and Vehicle Maintenance. FMCSA faces at least two challenges in reliably assessing safety risk for the majority of carriers. First, for SMS to be effective in identifying carriers more likely to crash, the violations that FMCSA uses to calculate SMS scores should have a strong predictive relationship with crashes. However, based on GAO's analysis of available information, most regulations used to calculate SMS scores are not violated often enough to strongly associate them with crash risk for individual carriers. Second, most carriers lack sufficient safety performance data to ensure that FMCSA can reliably compare them with other carriers. To produce an SMS score, FMCSA calculates violation rates for each carrier and then compares these rates to other carriers. Most carriers operate few vehicles and are inspected infrequently, providing insufficient information to produce reliable SMS scores. FMCSA acknowledges that violation rates are less precise for carriers with little information, but its methods do not fully address this limitation. For example, FMCSA requires a minimum level of information for a carrier to receive an SMS score; however, this requirement is not strong enough to produce sufficiently reliable scores. As a result, GAO found that FMCSA identified many carriers as high risk that were not later involved in a crash, potentially causing FMCSA to miss opportunities to intervene with carriers that were involved in crashes. FMCSA's methodology is limited because of insufficient information, which reduces the precision of SMS scores. GAO found that by scoring only carriers with more information, FMCSA could better identify high risk carriers likely to be involved in crashes. This illustrative approach involves trade-offs; it would assign SMS scores to fewer carriers, but these scores would generally be more reliable and thus more useful in targeting FMCSA's scarce resources. In addition to using SMS scores to prioritize carriers for intervention, FMCSA reports these scores publicly and is considering using a carrier's performance information to determine its fitness to operate. Given the limitations with safety performance information, determining the appropriate amount of information needed to assess a carrier requires consideration of how reliable and precise the scores need to be for the purposes for which they are used. Ultimately, the mission of FMCSA is to reduce crashes, injuries, and fatalities. GAO continues to believe a data-driven, risk-based approach holds promise; however, revising the SMS methodology would help FMCSA better focus intervention resources where they can have the greatest impact on achieving this goal. GAO recommends that FMCSA revise the SMS methodology to better account for limitations in drawing comparisons of safety performance information across carriers. In addition, determination of a carrier's fitness to operate should account for limitations in available performance information. In response to comments from the Department of Transportation (USDOT), GAO clarified one of the recommendations. USDOT agreed to consider the recommendations. (Author/publisher)
Abstract