FMCSA Safety Program Effectiveness Measurement: Compliance Review Program CR Impact Assessment Model Results for 2001 and 2002 with Additional Analysis May 2003 FMCSA-RI-03-016 Prepared for: Federal Motor Carrier Safety AdministrationOffice of Data Analysis and Information SystemsData Analysis Division, MC-RIA400 Seventh Street, S.W.Washington, DC 20590 Prepared by: John A. Volpe National Transportation Systems CenterMotor Carrier Safety Assessment DivisionDTS-47Kendall SquareCambridge, MA 02142 PREFACE This report documents the methodology and results from an improved model to measure the effectiveness of one of the key safety programs of the Federal Motor Carrier Safety Administration (FMCSA). The research was conducted by the Research and Special Programs Administration's (RSPA) John A. Volpe National Transportation Systems Center (the Volpe Center) in Cambridge, MA under a project plan agreement with the FMCSA. The work on FMCSA Program Effectiveness Measures addresses the requirements of the Government Performance and Results Act (GPRA) of 1993, which obligates federal agencies to measure the effectiveness of their programs as part of the budget cycle process. Work on FMCSA Program Effectiveness Measures was initiated during FY 93. In December 1994, a report titled "Office of Motor Carriers Safety Program - Performance Measurement" was prepared. That report provided a comprehensive breakdown of Office of Motor Carriers (OMC) safety programs and activities and described about a dozen potential evaluation models. (Note: The OMC later became the FMCSA.) Based on the OMC's review, the Volpe Center revised the report and recommended four evaluation models to assess the key OMC programs: roadside inspections conducted by participating states under the Motor Carrier Safety Assistance Program (MCSAP), on-site compliance reviews conducted by the OMC field offices and the states, commercial vehicle traffic enforcement also performed by the states under the MCSAP, and a comprehensive assessment of combined effects. Two initial evaluation models covering the roadside inspection program and the compliance review program were described in detail in a December 1998 report titled "OMC Safety Program Performance Measures." A review panel was convened to evaluate these models and made recommendations for improvement. The Volpe Center incorporated these recommendations together with other Volpe Center defined improvements into two "second-generation" models that measure the effectiveness of these two programs. A previous report1 described the implementations of the Compliance Review Impact Assessment Model covering 1999 and 2000. This report describes the implementations of the model for 2001 and 2002. At the FMCSA, the project is managed by Dale Sienicki, Chief of the Office of Data Analysis and Information Systems, Data Analysis Division. The Volpe Center project manager is Donald Wright, Chief of the Motor Carrier Safety Assessment Division in the Office of System and Economic Assessment. The analysis was performed at the Volpe Center by Jon Ohman, with assistance from Nancy Kennedy of the Volpe Center and Basav Sen of EG&G Services, under contract to the Volpe Center. TABLE OF CONTENTS Section Page EXECUTIVE SUMMARY.................................................................................................................... v 1. INTRODUCTION.......................................................................................................................... 1-1 1.1. Project Objective................................................................................................................. 1-1 1.2. Project Scope....................................................................................................................... 1-2 2. COMPLIANCE REVIEW IMPACT ASSESSMENT MODEL................................................. 2-1 2.1. Compliance Reviews.......................................................................................................... 2-1 2.2. Methodology of Model....................................................................................................... 2-1 2.3. Results of Implementations of Model for 2001 and 2002................................................. 2-2 2.4. Historical Comparison of Results....................................................................................... 2-9 3. ADDITIONAL ANALYSIS.......................................................................................................... 3-1 3.1. Overview............................................................................................................................. 3-1 3.2. Carrier Size.......................................................................................................................... 3-1 3.3. Other Analyses.................................................................................................................... 3-3 APPENDIX A. 2002 COMPLIANCE REVIEW FOLLOW-UP................................................... A-1 A.1. Purpose.............................................................................................................................. A-1 A.2. Plan.................................................................................................................................... A-1 A.3. Response........................................................................................................................... A-2 A.4. Results - Crash Reduction................................................................................................ A-3 A.5. Results - Carrier Attrition................................................................................................ A-4 APPENDIX B. CARRIER ATTRITION FROM 2001 TO 2002................................................... B-1 B.1. Purpose of Calculations........................................................................................... B-1 B.2. Calculation of Carrier Attrition Mileage from 2001 to 2002....................................... B-1 TABLE OF CONTENTS (continued) Section Page APPENDIX C. ADDITIONAL ANALYSES FROM 2000 COMPLIANCE REVIEW FOLLOW-UP......................................................................................................... C-1 C.1. Overview........................................................................................................................... C-1 C.2. Carrier Size........................................................................................................................ C-2 C.3. SafeStat Category.............................................................................................................. C-3 C.4. Number of Compliance Reviews...................................................................................... C-4 C.5. Planned Course of Action................................................................................................. C-5 LIST OF ILLUSTRATIONS FIGURES Figure Page 2-1. Compliance Review Impact Assessment Model........................................................................ 2-3 TABLES Table Page ES-1. Compliance Review Impact Assessment Model Implementations - Results from 1999 to 2002........................................................................................................ vi 2-1. Compliance Review Impact Assessment Model Implementations - Results from 1999 to 2002......................................................................................................... 2-9 3-1. Pre-CR and Post-CR Average Crash Rates by Carrier Size................................................ 3-2 3-2. Estimated Number of Crashes Avoided by Carrier Size..................................................... 3-2 A-1. 2002 CR Follow-up Stratification............................................................................................ A-1 A-2. 2002 CR Follow-up Response.................................................................................................. A-2 A-3. 2002 CR Follow-up Results - Crash Rates and VMT............................................................. A-3 A-4. 2002 CR Follow-up Results - Active and Known Inactive Carriers in Strata A4 and N4..... A-5 A-5. 2002 CR Follow-up - Stratum A4 - Active Carriers.............................................................. A-5 A-6. 2002 CR Follow-up - Stratum A4 - Sample Carriers............................................................. A-5 A-7. 2002 CR Follow-up Results - Carrier and VMT Attrition Rates............................................ A-6 C-1. Pre-CR and Post-CR Average Crash Rates by Number of Power Units................................. C-2 C-2. Pre-CR and Post-CR Average Crash Rates by SafeStat Category........................................... C-3 C-3. Pre-CR and Post-CR Average Crash Rates by Number of Compliance Reviews Received in 1996-1998.............................................................................................................. C-5 C-4. Pre-CR and Post-CR Average Crash Rates by Type of Planned Course of Action................ C-5 EXECUTIVE SUMMARY Background This report documents the methodology and results from an improved model to measure the effectiveness of one of the key safety programs of the Federal Motor Carrier Safety Administration (FMCSA), the compliance review (CR) program. The research was conducted by the Research and Special Programs Administration's (RSPA) John A. Volpe National Transportation Systems Center (the Volpe Center) in Cambridge, MA under a project plan agreement with the FMCSA. The work on FMCSA Safety Program Effectiveness Measurement addresses the requirements of the Government Performance and Results Act (GPRA) of 1993, which obligates federal agencies to measure the effectiveness of their programs as part of the budget cycle process. This report describes the methodology of the Compliance Review Impact Assessment Model and presents the results from the implementations of the model based on data collected from CRs performed in 2000 and 2001. Finally, the results of an additional analysis designed to examine the relationship between crash rate change following a CR and carrier size are presented.. Methodology of Model The on-site compliance review (CR) is perhaps the single greatest resource-consuming activity of the FMCSA. Thousands of CRs are conducted each year. In the year 2001, federal and state enforcement personnel conducted nearly 12,000 CRs on individual motor carriers. It is intended that through education, heightened safety regulation awareness, and enforcement effects of the CR, carriers will improve the safety of their commercial vehicle operations, and, ultimately, reduce their crash rates. The CR Impact Assessment Model was developed to determine the effectiveness of the CR program. The model shows the direct impact of compliance reviews on carrier safety, but not the "deterrent" effects (i.e., the "threat" of having a CR). The model is based on the individual and cumulative "before and after" changes in the safety performance of carriers that received CRs. The model compares a motor carrier's crash rate in a time period after an on-site compliance review to its crash rate prior to that review. To make this comparison, the model uses crash and mileage data collected during compliance reviews. As part of the CR procedure, investigators are required to obtain the number of recordable crashes (crashes involving fatalities, injuries, or "towaways," in which an involved vehicle cannot leave the crash scene due to damage) in which the carrier was involved over the past 12 months as well as the number of vehicle miles traveled (VMT) by the carrier's fleet over the same 12 months. Therefore, crash rates (in the form of the number of recordable crashes per million VMT) for all carriers having received CRs can be calculated. Since the model determines the change in crash rates from before to after CRs, it requires not only pre-CR crash rates but also crash rates after the CRs. To obtain post-CR crash rates for all reviewed carriers, the Compliance Review Follow-up is conducted. The Follow-Up involves collecting additional data from carriers that had CRs. The CR Impact Assessment Model uses the results of the Follow-up. Implementations of Model for 2001 and 2002 The CR Impact Assessment Model was implemented for (1) CRs conducted in 2000 to estimate the number of crashes (and associated fatalities and injuries) avoided in 2001, and for (2) CRs conducted in 2001 to estimate the number of crashes (and associated fatalities and injuries) avoided in 2002. The 2002 CR Follow-up data, which were obtained from a sample of carriers that received CRs in 2000, were used to estimate three parameters in the model: (1) change in average crash rate, (2) change in VMT, and (3) decrease in VMT due to carrier attrition. The results are shown in Table ES-1 together with results previously obtained for 1999 and 2000. Table ES-1. Compliance Review Impact Assessment Model Implementations - Results from 1999 to 2002 Model Implementation for: 1999 2000 2001 2002 Based on CRs Conducted in: 1998 1999 2000 2001 Number of Carriers Receiving Compliance Reviews 6,055 8,877 11,340 8,924 Total Vehicle Miles Traveled (VMT) (million miles) 13,844 17,409 22,610 18,455 Pre-CR Average Crash Rate (crashes per million VMT) .823 .804 .757 .715 Model Results Estimated for: 1999 2000 2001 2002 Number of Crashes Avoided 1,200 1,500 2,200 1,600 Number of Fatal Crashes Avoided 43 54 79 58 Number of Injury Crashes Avoided 480 600 880 640 Number of Towaway Crashes Avoided 677 846 1,241 902 Number of Lives Saved 51 64 91 67 Number of Injuries Avoided 822 1,028 1,395 1,015 Note: Model implementations for 2000 and 2002 are based on estimates from the previous year's follow-up study. Precision of Estimates It should be noted that the estimates of crashes avoided for 2001 and 2002 are based on data from the 2002 CR Follow-up, which was conducted on a stratified sample of carriers that received CRs in 2000. Therefore, sampling variability may affect the significance of the differences between the post-CR average crash rates and the average crash rates measured at the time of the CRs. Sampling variability may also affect the precision of the estimates derived from the post-CR average crash rates, i.e., percent reduction in crash rate, number of crashes avoided, and associated program benefits (i.e., lives saved and injuries avoided). Additional Analysis To further measure the effectiveness of the compliance review program, an additional analysis was performed to examine the relationship between crash rate change following a CR and carrier size. The results of the implementation of the model were broken out by carrier size, i.e., the number of power units. It was found that carriers with 20 or fewer power units had the largest reductions in their crash rates in the year following a CR. 1. INTRODUCTION 1.1. PROJECT OBJECTIVE Since the early 1980s, Congress has passed several acts that strengthened federal motor carrier safety regulations and led to Federal Motor Carrier Safety Administration (FMCSA) programs to enforce them. The Surface Transportation Assistance Act of 1982 established the Motor Carrier Safety Assistance Program, a grants-in-aid program to states to conduct roadside inspection and enforcement programs aimed at commercial motor vehicles. The 1984 Motor Carrier Safety Act directed the Department of Transportation (DOT) to establish safety fitness standards for carriers. In response to this legislation, the DOT, in conjunction with the states, implemented the Motor Carrier Safety Assistance Program (MCSAP) to establish and fund the roadside inspection and enforcement program and the Safety Fitness Determination Process (SFDP) and rating system based on on-site safety audits (called compliance reviews). It is expected that a major benefit of these programs has been and will continue to be an improved level of safety in the operation of commercial motor vehicles. Previously, however, there was no means to measure the benefits and effectiveness of these programs. This project was established to identify major functions and operations (programs) associated with the FMCSA mission and to develop results-oriented performance measures for those functions and operations, as called for in the Government Performance and Results Act (GPRA) of 1993. Program evaluation should be viewed as a continuous management process that encourages the organization to reflect periodically upon how it is implementing its programs. Program effectiveness should be reassessed in light of the mission, available resources, changing requirements, political climate, technological change, public demands, and costs. Periodic review of the results of the evaluations will ensure that the activities are working, i.e., that they are delivering what was promised. This report is intended to satisfy the desire of the FMCSA to verify the effectiveness of one of its motor carrier safety programs, the compliance review program. The immediate objective of this effort is to measure how much of an impact the safety program activities have on avoiding crashes involving interstate motor carriers and reducing resulting injuries and fatalities. One of the long-term objectives is to provide a baseline of the effectiveness of the selected programs through the use of standard safety performance measures that can be compared to future safety performance. This baseline will allow the FMCSA to judge the relative performance of its programs on a periodic basis by reflecting the benefits resulting from changes in each program. This capability will provide the FMCSA with a powerful analytical tool that can estimate the effects of changes within an activity and the effects of changes in resources between program activities. The results of these analyses will provide a basis for FMCSA resource allocation and budgeting decisions that will more closely optimize the effectiveness and efficiency of its motor carrier safety programs. 1.2. PROJECT SCOPE The scope of this overall effort is limited to the major identifiable programs and their effectiveness in reducing crashes and avoiding injuries and fatalities. It is hypothesized that the FMCSA safety program elements exert a positive influence causing changes in driver behavior and carrier operations ultimately leading to improvements in the level of motor carrier safety. It is recognized, however, that motor carriers are also affected by the highway environment and factors other than the influences of the FMCSA safety program elements that may intervene, impact, or influence motor carrier safety. No attempt is made here to account for these other exogenous influences on motor carrier safety performance, crash rates, and their associated consequences, i.e., fatalities and injuries. The Safety Program Effectiveness Measurement project includes the roadside inspection, compliance review, and traffic enforcement activities and programs performed and supported by the FMCSA. This report is concerned with the compliance review program and describes the CR Impact Assessment Model. An improved Roadside Inspection and Traffic Enforcement program effectiveness measurement model, called the Intervention Model, has also been developed and is described in a companion report.1 An objective of the project is to continue to improve these safety program effectiveness measures and models and run them on a recurring basis. The models will serve the program specific requirement to measure program effectiveness as well as the broader function of supporting annual budget requirements and helping to determine the best resource allocation among program elements. This report describes the methodology of the Compliance Review Impact Assessment Model. The results from the implementations of the model for carriers receiving CRs in 2000 and 2001 are derived and presented. Also, the results of an additional analysis designed to examine the relationship between crash rate change following a CR and carrier size are presented. 2. COMPLIANCE REVIEW IMPACT ASSESSMENT MODEL 2.1. COMPLIANCE REVIEWS The on-site compliance review (CR) is perhaps the single greatest resource-consuming activity of the FMCSA. Thousands of CRs are conducted each year. In the year 2001, federal and state enforcement personnel conducted nearly 12,000 CRs on individual motor carriers. In addition to actually conducting CRs, the FMCSA invests in: extensive analysis of the requirements of the Federal Motor Carrier Safety Regulations (FMCSRs), enhancements to the design of the CR to better assess safety performance and compliance with the FMCSRs, continued safety investigator training, enhancements to prioritization methodologies such as SafeStat1 to determine who should receive CRs, and enhancements to information systems to report and store the results of the CRs that are conducted. When performing CRs, FMCSA and state safety investigators spend many hours examining the safety records of individual motor carriers to assess their compliance and safety performance. The investigators also discuss their findings with the carriers' safety managers to improve understanding of their safety programs. After a review is completed, the carrier is assigned a safety rating (i.e., satisfactory, conditional, or unsatisfactory). If serious violations are discovered, an enforcement case is initiated and a fine may be imposed. The CR results are also incorporated, with other safety data (i.e., crashes, roadside inspection results, moving violations, and closed enforcement cases), into SafeStat to reassess the carrier's safety status. It is intended that through education, heightened safety regulation awareness, and the enforcement effects of the CR, carriers will improve the safety of their commercial vehicle operations, and, ultimately, reduce their crash rates. 2.2. METHODOLOGY OF MODEL The CR Impact Assessment Model was developed to determine the effectiveness of the CR program. The model shows the direct impact of compliance reviews on carrier safety, but not the "deterrent" effects (i.e., the "threat" of having a CR). The model is based on the individual and cumulative "before and after" changes in the safety performance of carriers that received CRs. The model compares a motor carrier's crash rate in a time period after an on-site compliance review to its crash rate prior to that review. To make this comparison, the model uses crash and mileage data collected during compliance reviews and stored in the FMCSA's Motor Carrier Management Information System (MCMIS). As part of the CR procedure, investigators are required to obtain the number of recordable crashes (crashes involving fatalities, injuries, or "towaways," in which an involved vehicle cannot leave the crash scene due to damage) in which the carrier was involved over the previous 12 months as well as the number of vehicle miles traveled (VMT) by the carrier's fleet over the same 12 months. Therefore, crash rates (in the form of the number of recordable crashes per million VMT) for all carriers having received CRs can be calculated. Since the model determines the change in crash rates from before to after CRs, it requires not only pre-CR crash rates but also crash rates after the CRs. To obtain post-CR crash rates for all reviewed carriers, the Compliance Review Follow-up is conducted. The CR Follow-up was conducted in 2002 on a sample of carriers that received CRs during 2000. The 2002 CR Follow-up data, which were obtained from a sample of carriers that received CRs in 2000, were used to estimate three parameters in the model: (1) change in average crash rate, (2) change in VMT, and (3) decrease in VMT due to carrier attrition. The 2002 CR Follow-up is described in Appendix A. 2.3. RESULTS OF IMPLEMENTATIONS OF MODEL FOR 2001 AND 2002 A diagram of the CR Impact Assessment Model, as implemented for 2001 and 2002, is shown in Figure 2-1. The CR Impact Assessment Model was implemented for (1) CRs conducted in 2000 to estimate the number of crashes (and associated fatalities and injuries) avoided in 2001, and for (2) CRs conducted in 2001 to estimate the number of crashes (and associated fatalities and injuries) avoided in 2002. The 2002 CR Follow-up data, which were obtained from a sample of carriers that received CRs in 2000, were used to estimate three parameters in the model: (1) change in average crash rate, (2) change in VMT, and (3) decrease in VMT due to carrier attrition. Since, however, there was no 2003 follow-up on carriers that received CRs in 2001, the parameters from the 2002 CR Follow-up were extrapolated one year. Since the 2002 CR Follow-up was conducted on a stratified sample of carriers, the model was implemented for 2001 and 2002 separately for each stratum. To illustrate the implementation procedure, the implementations for stratum A4 are described in this section. Stratum A4 consists of carriers with more than 100 power units that chose 'Authorized For-Hire' as one of their operation classifications when completing Form MCS-150 - Motor Carrier Identification Report. (For a description of all the strata, see Appendix A.) Figure 2-1. Compliance Review Impact Assessment Model (1) Identify carriers with one or more compliance reviews (CRs) in 2000/2001. 2001 Results: There were 322 carriers in stratum A4 that received CRs in 2000. 2002 Results: There were 230 carriers in stratum A4 that received CRs in 2001. (2) Calculate pre-CR average crash rate for the carriers with one or more CRs. 2001 Results: The 322 carriers in stratum A4 with CRs in 2000 had a pre-CR average crash rate of .626 crashes per million vehicle miles traveled (VMT). This average was obtained from the carriers' 2000 CR data by multiplying the total number of the carriers' crashes by 1 million and then dividing by the carriers' total VMT. This aggregate rate is equivalent to averaging each carrier's crash rate weighted by its VMT. 2002 Results: The 230 carriers in stratum A4 with CRs in 2001 had a pre-CR average crash rate of .633 crashes per million vehicle miles traveled (VMT). This average was obtained from the carriers' 2001 CR data by multiplying the total number of the carriers' crashes by 1 million and then dividing by the carriers' total VMT. This aggregate rate is equivalent to averaging each carrier's crash rate weighted by its VMT. (3) Calculate pre-CR and post-CR average crash rates for the carriers with follow-up data. 2001 Results: The carriers in stratum A4 with follow-up data had the following pre-CR and post-CR average crash rates: * Pre-CR: .639 crashes per million VMT * Post-CR: .618 crashes per million VMT 2002 Results: No follow-up of carriers that received CRs in 2001 was planned to obtain the post-CR (i.e., 2002) average crash rate. (4) Calculate the reduction in the average crash rate. The percentage change in the average crash rate is calculated as follows: Percentage Change in Average Crash Rate Post-CR Average Crash Rate - Pre-CR Average Crash Rate = --------------------------------------------------------------------------- X 100 Pre-CR Crash Rate 2001 Results: Percentage Change in Average Crash Rate .618 - .639 = ----------------- X 100 .639 = -3.3% (i.e., a reduction of 3.3 percent) 2002 Results: The 3.3 percent crash rate reduction (from 2000 to 2001) obtained in the 2002 CR Follow-up was used again for this implementation of the model. (5) Calculate total vehicle miles traveled (VMT) in 2001/2002 for the carriers with CRs in 2000/2001. 2001 Results: The estimate of the total 2001 VMT by the 322 carriers in stratum A4 with CRs in 2000 was calculated as follows: 2001 VMT = (2000 VMT - AVMT) X (1 + C) where AVMT = Decrease in carrier VMT from 2000 to 2001 due to carrier attrition, and C = Percentage change in VMT from 2000 to 2001. The 322 carriers in stratum A4 with CRs in 2000 had a total of 10,737 million vehicle miles traveled in 2000. The 2002 CR Follow-up found the decrease in carrier VMT from 2000 to 2001 due to carrier attrition, i.e., the 2000 CR VMT of reviewed carriers that ceased operations before or during 2001, to be 824 million miles. (Details of this calculation can be found in Appendix A.) The carriers in the 2002 CR Follow-up reported a 0.9 percent increase in VMT from 2000 to 2001. Therefore, the estimated total VMT for 2001 was: 2001 VMT = (10,737 - 824) million miles X (1 + .009) = 9,913 million miles X 1.009 = 10,002 million miles 2002 Results: The estimate of the total 2002 VMT by the 230 carriers in stratum A4 with CRs in 2001 was calculated as follows: 2002 VMT = (2001 VMT - AVMT) X (1 + C) where AVMT = Decrease in carrier VMT from 2001 to 2002 due to carrier attrition, and C = Percentage change in VMT from 2001 to 2002. The 230 carriers in stratum A4 with CRs in 2001 had a total of 9,006 million vehicle miles traveled in 2001. The decrease in carrier VMT from 2001 to 2002 due to carrier attrition, i.e., the 2001 CR VMT of reviewed carriers that ceased operations before or during 2002, was estimated to be 441 million miles (Details of this calculation can be found in Appendix B.) Since no follow-up was planned for carriers receiving CRs in 2001, the 0.9 percent increase in carrier VMT from 2000 to 2001 found in the 2002 CR Follow-up was used as an estimate of the percentage change in carrier VMT from 2001 to 2002. Therefore, the estimated total VMT for 2002 was: 2002 VMT = (9,006 - 441) million miles X (1 + .009) = 8,565 million miles X 1.009 = 8,642 million miles (6) Estimate the number of crashes avoided in 2001/2002. 2001 Results: The estimated number of crashes avoided in 2001 by the 322 carriers in stratum A4 with CRs in 2000 was calculated as follows: Crashes avoided in 2001 = Pre-CR Average Crash Rate X Crash Rate Reduction X 2001 VMT = .626 crashes per million miles X 3.3% X 10,002 million miles = 207 crashes The total estimated number of crashes avoided in 2001 in all eight strata was 2,230, which was rounded to 2,200 crashes. The estimate was rounded to the nearest 100 crashes, due to the limited precision of the estimates produced by the model. 2002 Results: The estimated number of crashes avoided in 2002 by the 230 carriers in stratum A4 with CRs in 2001 was calculated as follows: Crashes avoided in 2002 = Pre-CR Average Crash Rate X Crash Rate Reduction X 2002 VMT = .633 crashes per million miles X 3.3% X 8,642 million miles = 181 crashes, The total estimated number of crashes avoided in 2002 in all eight strata was 1,625, which was rounded to 1,600 crashes. The estimate was rounded to the nearest 100 crashes, due to the limited precision of the estimates produced by the model. Next, estimates were made of the number of crashes avoided in 2001 and 2002 by severity, i.e., fatal, injury, and towaway. The most recent estimates of the proportions of crashes by severity are found in the "Truck and Bus Crash Factbook 1995."2 This report was published in 1997 by the University of Michigan Transportation Research Institute under contract to the Federal Highway Administration's Office of Motor Carriers, which later became the FMCSA. According to the report, of the trucks involved in crashes on U.S. roads in 1995, 3.6 percent were involved in fatal crashes, 40.0 percent were involved in injury crashes, and 56.4 percent were involved in towaway crashes.3 Applying these proportions to the estimates of 2,200 and 1,600 crashes avoided produced the following results: 2001 Results: Fatal crashes = 2,200 X 3.6% = 79 Injury crashes = 2,200 X 40.0% = 880 Towaway crashes = 2,200 X 56.4% = 1,241 2002 Results: Fatal crashes = 1,600 X 3.6% = 58 Injury crashes = 1,600 X 40.0% = 640 Towaway crashes = 1,600 X 56.4% = 902 (7) Find the average numbers of fatalities and injuries per crash. 2001 and 2002 Results: The average number of fatalities per fatal crash was calculated from data from the Fatality Analysis Reporting System (FARS), which is maintained by the National Highway Traffic Safety Administration (NHTSA). For 2001 crashes involving large trucks or motorcoaches (i.e., cross-country or intercity buses), the ratio was 1.15 fatalities per fatal crash. The number of injuries per crash involves fatal as well as injury crashes, since fatal crashes can also result in injuries. State-reported crash data in the MCMIS were used to compute the average numbers of injuries in fatal and injury crashes. For 2001 large truck and bus crashes, the averages were as follows: * Fatal crashes: 1.06 injuries per crash * Injury crashes: 1.49 injuries per crash (8) Calculate benefits. 2001 Results: The estimated number of lives saved in the crashes avoided in 2001 was calculated as follows: Number of lives saved in fatal crashes in 2001 = Number of fatal crashes avoided X Average number of fatalities per fatal crash = 79 X 1.15 = 91 lives saved The estimated number of injuries avoided in 2001 was calculated as follows: Number of injuries avoided in 2001 = Number of fatal crashes avoided X Average number of injuries per fatal crash + Number of injury crashes avoided X Average number of injuries per injury crash = 79 X 1.06 + 880 X 1.49 = 1,395 injuries avoided 2002 Results: The estimated number of lives saved in the crashes avoided in 2002 was calculated as follows: Number of lives saved in fatal crashes in 2002 = Number of fatal crashes avoided X Average number of fatalities per fatal crash = 58 X 1.15 = 67 lives saved The estimated number of injuries avoided in 2002 was calculated as follows: Number of injuries avoided in 2002 = Number of fatal crashes avoided X Average number of injuries per fatal crash + Number of injury crashes avoided X Average number of injuries per injury crash = 58 X 1.06 + 640 X 1.49 = 1,015 injuries avoided It should be noted that the estimates of crashes avoided for 2001 and 2002 are based on data from the 2002 CR Follow-up, which was conducted on a stratified sample of carriers that received CRs in 2000. Therefore, sampling variability may affect the significance of the differences between the pre-CR and post-CR average crash rates. Sampling variability may also affect the precision of the estimates derived from the post-CR average crash rates, i.e., percent reduction in crash rate, number of crashes avoided, and associated program benefits (i.e., lives saved and injuries avoided). 2.4. HISTORICAL COMPARISON OF RESULTS The CR Impact Assessment Model has been implemented on an annual basis since its initial implementation to estimate 1999 results. The CR Impact Assessment Model uses data from the CR Follow-up to calculate post-CR crash rates. The CR Follow-up provides estimates of the change in the average crash rate, the change in vehicle miles traveled (VMT), and the decrease in VMT due to carrier attrition. The 1999 and 2001 results were based on the 2000 and 2002 CR Follow-up estimates, respectively. The 2000 and 2002 results were based on extrapolating estimates from the previous year's follow-up study. It should be noted that the 2000 CR Follow-up was conducted on all carriers that had CRs in 1998, whereas the 2002 CR Follow-up was conducted on a sample of carriers that had CRs in 2000. Table 2-1 shows the results of the implementations of the model from 1999 to 2002. Table 2-1. Compliance Review Impact Assessment Model Implementations - Results from 1999 to 2002 Model Implementation for: 1999 2000 2001 2002 Based on CRs Conducted in: 1998 1999 2000 2001 Number of Carriers Receiving Compliance Reviews 6,055 8,877 11,340 8,924 Total Vehicle Miles Traveled (VMT) (million miles) 13,844 17,409 22,610 18,455 Pre-CR Average Crash Rate (crashes per million VMT) .823 .804 .757 .715 Model Results Estimated for: 1999 2000 2001 2002 Number of Crashes Avoided 1,200 1,500 2,200 1,600 Number of Fatal Crashes Avoided 43 54 79 58 Number of Injury Crashes Avoided 480 600 880 640 Number of Towaway Crashes Avoided 677 846 1,241 902 Number of Lives Saved 51 64 91 67 Number of Injuries Avoided 822 1,028 1,395 1,015 Note: Model implementations for 2000 and 2002 are based on estimates from the previous year's follow-up study. 3. ADDITIONAL ANALYSIS 3.1. OVERVIEW The results of the implementation of the model were analyzed to determine the degree that carriers' crash rates change after the carriers receive CRs. The results were broken out by carrier size (i.e., number of power units). The results of this analysis revealed the types of carriers that will most likely respond positively to CRs. By focusing on carriers that are likely to respond positively to CRs, the effectiveness of the compliance review program may be improved. Alternative treatment approaches may be suggested for carriers that are at risk, but will most likely not respond positively to CRs. The 2002 CR Follow-up was conducted on a sample of carriers that received CRs in 2000. To perform the analyses in this section, the follow-up data were weighted. In each analysis, for each of the 547 carriers with usable data obtained in the 2002 CR Follow-up, the numbers of pre-CR and post-CR crashes and vehicle miles traveled (VMT) were multiplied by the inverse of the carrier's probability of being selected for the sample. (This probability depended on the carrier's stratum.) The weighted crash and VMT numbers were summed for each attribute category. Two weighted average crash rates were then calculated: * the pre-CR average crash rate - calculated using weighted data from the initial (2000) CR, and * the post-CR average crash rate - calculated using weighted data from the 2002 CR Follow-up. Each average crash rate was obtained by multiplying the weighted total number of crashes for a set of carriers by 1 million and then dividing by the weighted total number of VMT for that set of carriers. 3.2. CARRIER SIZE This analysis examined the relationship between crash rate change following a CR and carrier size. The results of the implementation of the model were broken out by carrier size, as measured by the number of power units. Table 3-1 shows the pre-CR and post-CR average crash rates broken out by size of carrier, i.e., the number of power units. The pre-CR average crash rate was inversely related to carrier size. In other words, as carrier size increased, the pre-CR average crash rate decreased. Table 3-2 shows, for each size group, the numbers of carriers that received CRs in 2000 and 2001 and the resulting numbers of crashes avoided in 2001 and 2002. These estimates are the results of the implementations of the model described in Section 2.3. The estimate for each size group is the sum of the estimates for the two applicable strata that make up that size group. For example, the number of crashes avoided in 2001 by carriers with 5 or fewer power units (658) is the sum of the crashes avoided in strata A1 (429) and N1 (229). (For a description of all the strata, see Appendix A.) Table 3-1. Pre-CR and Post-CR Average Crash Rates by Carrier Size Number of Power Units Number of Carriers in the Sample Pre-CR Average Crash Rate* Post-CR Average Crash Rate* Percent Change in Crash Rate <5 133 1.405 0.678 -51.7 6 - 20 149 1.026 0.633 -38.3 21-100 132 0.803 0.748 -6.8 >101 133 0.660 0.635 -3.8 All Carriers 547 0.775 0.668 -13.8 * - Crashes per million miles Table 3-2. Estimated Number of Crashes Avoided by Carrier Size Number of Power Units Number of Carriers with CRs in 2000 Estimated Number of Crashes Avoided in 2001 Number of Carriers with CRs in 2001 Estimated Number of Crashes Avoided in 2002 <5 4,805 658 3,784 386 6 - 20 4,099 1,023 3,225 762 21-100 1,997 276 1,592 202 >101 439 273 323 275 All Carriers 11,340 2,230 8,924 1,625 The smaller carriers, those with 20 or fewer power units, had the greatest reductions in average crash rates as well as the largest number of estimated crashes avoided as a result of the program. For carriers with 5 or fewer power units, the post-CR average crash rate showed a decrease of 51.7 percent from the pre-CR average crash rate. For carriers with 6-20 power units, the decrease was 38.3 percent. For carriers with more than 20 power units, the reductions in the average crash rates were smaller than the reductions for the smaller carriers. For carriers with 21-100 power units, the post-CR average crash rate showed a decrease of 6.8 percent. For carriers with 101 or more power units, the decrease was 3.8 percent. This additional analysis was first performed as part of the 2000 CR Follow-up. The current results follow the pattern of that previous analysis. The smaller carriers, however, showed much larger crash rate reductions in the current analysis than in the previous analysis. The results of the previous analysis are presented in Appendix C. 3.3. OTHER ANALYSES The previous report,1 which described the implementations of the model for 1999 and 2000, contained additional analyses of crash reduction by three other attributes: * carrier safety status (i.e., the carrier's SafeStat2 category before receiving the initial CR), * the number of compliance reviews that the carrier received, and * the planned course of action for the carrier following the initial CR (i.e., enforcement or no enforcement). This analysis was performed using the data from the 2000 CR Follow-up. Since this follow-up covered the entire population of carriers receiving CRs in 1998, the data could support all such analyses. The 2002 CR Follow-up, however, was performed on a sample of carriers that received CRs in 2000. This sample could not produce accurate analyses for these three attributes, because it did not have representative distributions of these attributes. The analysis by size of carrier could be performed because the sample was selected based on that attribute. All four additional analyses from the previous report, i.e., the analyses performed using the data from the 2000 CR Follow-up, can be found in Appendix C. APPENDIX A. 2002 COMPLIANCE REVIEW FOLLOW-UP A.1. PURPOSE The purpose of the 2000 Compliance Review (CR) Follow-up was to measure the post-CR crash rate change (from 2000 to 2001) for all carriers receiving CRs in 2000. The results of the 2002 CR Follow-up were used by the CR Impact Assessment Model, which is described in Section 2. A.2. PLAN The 2002 CR Follow-up was conducted on a sample of the carriers receiving CRs in 2000, unlike the 2000 CR Follow-up, which was conducted on all carriers receiving CRs in 1998. The smaller sample size reduced overall costs and allowed more time and effort to be devoted to data quality control. The questionnaire was redesigned and new procedures were instituted to address the problem of the overreporting of crash data in the 2000 CR Follow-up, There were 11,340 interstate or intrastate (hazardous materials) motor carriers that received CRs in 2000 that resulted in overall safety ratings. As of February 2002, 10,571 of those carriers were still active. The population of 10,571 active carriers was stratified by size and classification of carrier. Size was determined by the number of power units the carrier had at the time of its 2000 CR. Classification was determined by whether or not the carrier chose 'Authorized For-Hire' as one of its operation classifications when completing its most recent (as of September 2001) Form MCS-150 - Motor Carrier Identification Report. If the carrier indicated 'Authorized For-Hire' as one of its classifications, then it was defined as "authorized." If not, then it was defined as "non-authorized." The population was divided into the eight strata listed in Table A-1. Table A-1. 2002 CR Follow-up Stratification Stratum Classification Size (Number of Power Units) A1 Authorized <5 A2 Authorized 6 - 20 A3 Authorized 21-100 A4 Authorized >101 N1 Non-authorized <5 N2 Non-authorized 6 - 20 N3 Non-authorized 21-100 N4 Non-authorized >101 A stratified sample of 608 carriers was selected from the 10,571 active carrier that received CRs in 2000. In late February 2002, follow-up packages were mailed to the 608 carriers in the sample. Each follow-up package consisted of a cover letter, a questionnaire, and a postage-paid return envelope. The questionnaire asked for: * the number of recordable crashes by severity (fatal, injury, and towaway) in 2001, and * the total vehicle miles traveled (VMT) in 2001. In late March, a reminder package was mailed to the carriers that had not responded to the initial mailing. This package consisted of the same items as the original follow-up package, except for a reminder letter that was substituted for the original cover letter. Finally, the carriers that had still not responded to the follow-up were contacted by telephone. A.3. RESPONSE Table A-2 shows the response to the 2002 CR Follow-up. As shown in Table A-2, 520 questionnaires with usable data were received. Another 18 carriers did not return usable questionnaires, but received CRs between July 2001 and June 2002. These data were considered to be comparable to the calendar year 2001 data that were obtained from the questionnaires, and were used in the analysis. Table A-2. 2002 CR Follow-up Response Item Number Percent of Total Carriers in follow-up 608 100.0 Questionnaires with usable data received 520 85.5 Carriers with recent CR data (recent = July 2001 - June 2002) 18 3.0 Carriers with estimated data 9 1.5 Carriers with usable data 547 90.0 Carriers excluded from analysis (out of business, etc.) 61 10.0 Carriers accounted for 608 100.0 Data were estimated for another 9 carriers that either did not respond to the follow-up, or were active throughout 2001 but went out of business in 2002. The data that were used to estimate for these carriers included data from previous CRs (e.g., CRs in the first half of 2001) and state-reported crash data in the MCMIS. Therefore, usable data were obtained from or estimated for 547 (90.0 percent) of the 608 carriers in the follow-up. Another 61 carriers were excluded from the analysis. Of these carriers, 57 were either confirmed to be out of business or were inaccessible (i.e., unable to be located) and presumed to be out of business. Three other companies were no longer operating as interstate motor carriers, while one company was no longer operating as a motor carrier at all. A.4. RESULTS - CRASH REDUCTION Table A-3 shows the results of the 2002 CR Follow-up by stratum. Table A-3. 2002 CR Follow-up Results - Crash Rates and VMT Stratum Number of Carriers with Usable Data Pre-CR Average Crash Rate* Post-CR Average Crash Rate* Percent Change in Crash Rate Percent Change in VMT A1 73 1.043 0.596 -42.9 +16.6 A2 87 0.996 0.655 -34.2 +2.1 A3 89 0.793 0.727 -8.3 -2.3 A4 98 0.639 0.618 -3.3 +0.9 N1 60 2.214 0.918 -58.5 -11.2 N2 62 1.148 0.547 -52.4 +4.9 N3 43 0.881 0.901 +2.3 +7.0 N4 35 0.914 0.824 -8.2 +9.4 All Carriers 547 0.775 0.668 -13.8 +1.2 * - Crashes per million miles The overall weighted average crash rate reduction was 13.8 percent. To calculate this reduction, for each of the 547 carriers with usable data, the numbers of pre-CR and post-CR crashes and vehicle miles traveled (VMT) were multiplied by the inverse of the carrier's probability of being selected for the sample. (This probability depended on the carrier's stratum.) The weighted crash and VMT numbers were summed. Two weighted average crash rates were then calculated: * the pre-CR average crash rate - calculated using weighted data from the initial (2000) CR, and * the post-CR average crash rate - calculated using weighted data from the 2002 CR Follow-up. Each average crash rate was obtained by multiplying the weighted total number of crashes by 1 million and then dividing by the weighted total number of VMT. The overall average increase in VMT was 1.2 percent. This estimate was calculated by comparing the total weighted pre-CR VMT with the total weighted post-CR VMT. It should be noted that, because the 2002 CR Follow-up was conducted on a stratified sample of carriers that received CRs in 2000, sampling variability may affect the significance of the differences between the pre-CR and post-CR average crash rates. Sampling variability may also affect the precision of the estimates derived from the post-CR average crash rates, i.e., percent reduction in crash rate, number of crashes avoided, and associated program benefits (i.e., lives saved and injuries avoided). There was no sampling variability in the estimates produced by the implementations of the model for 1999 and 2000, which were described in the previous report.1 Those estimates were based on data from the 2000 CR Follow-up, which was conducted on all carriers that received CRs in 1998, not just a sample. A.5. RESULTS - CARRIER ATTRITION Implementation of the CR Impact Assessment Model requires an estimate of carrier attrition during the year under examination, i.e., the year whose crash count is affected by the compliance reviews performed the year before. In this case, to estimate the number of crashes avoided in 2001, one needs to estimate the decrease in VMT from 2000 to 2001 due to carrier attrition. That is, one needs to estimate the 2000 CR VMT of the carriers that became inactive before the end of 2001. There are two categories of attrition carriers: 1) Carriers that were found to be inactive before the 2002 CR Follow-up was conducted, i.e., pre-follow-up attrition carriers, and 2) Carriers that were discovered to be inactive during the 2002 CR Follow-up, i.e., follow-up attrition carriers. Since the 2002 CR Follow-up was conducted on a stratified sample of carriers, the model was implemented for 2001 separately for each stratum. This implementation procedure included the estimation of the decrease in VMT due to carrier attrition. To illustrate this procedure, the calculation of this quantity for stratum A4 is described in this section. The estimate resulting from these calculations was used in the implementation of the model for stratum A4, which is described in Section 3.2. Pre-Follow-up Attrition Prior to the follow-up, it was determined from the MCMIS Census File that 769 carriers that had received CRs in 2000 were no longer active. These pre-follow-up attrition carriers' total 2000 CR VMT was 1,065 million miles. Of these 769 carriers, 18 had 101 or more power units at the time of their 2000 CRs, i.e., were in stratum A4 or stratum N4. These carriers had a total of 459 million 2000 CR VMT. Since classification information on these carriers was no longer available, it was impossible to determine which of these carriers were in stratum A4 and which were in stratum N4. To estimate this breakdown, the data on these carriers were prorated based on the data for all active carriers in these two strata. Table A-4 shows the data on number of carriers for all active carriers in strata A4 and N4. Based on the percentages shown, the numbers of known inactive carriers and associated VMT in strata A4 and N4 were estimated. In stratum A4, it was estimated that there were 13 known inactive carriers with a pre-follow-up attrition mileage of 420 million VMT. Table A-4. 2002 CR Follow-up - Active and Known Inactive Carriers in Strata A4 and N4 Stratum Number of Active Carriers Active Carrier 2000 CR VMT (million miles) Percent of Active Carriers Percent of Active Carrier 2000 CR VMT Prorated Number of Known Inactive Carriers Prorated Known Inactive Carrier 2000 CR VMT (million miles) A4 309 10,317 73.4 91.5 13 420 N4 112 954 26.6 8.5 5 39 Total 421 11,271 100.0 100.0 18 459 Follow-up Attrition Table A-5 shows the number of active carriers and associated VMT broken out by whether or not the carriers were in the 2002 CR Follow-up sample in stratum A4. Of the 309 carriers in stratum A4 thought to be active at the time of the follow-up, 103 were selected for the follow-up. Table A-5. 2002 CR Follow-up - Stratum A4 - Active Carriers Status Number of Carriers Total 2000 CR VMT (million miles) Sample 103 3,255 Not in Sample 206 7,062 Total 309 10,317 Of the 103 carriers in stratum A4 selected for the 2002 CR follow-up, 5 carriers were found to be inactive during the follow-up. As shown in Table A-6, these carriers constituted 4.9 percent of the sample carriers, and their VMT constituted 1.8 percent of the total sample carrier VMT. These percentages will be referred to as the carrier attrition rate and the VMT attrition rate. Table A-6. 2002 CR Follow-up - Stratum A4 - Sample Carriers Status Number of Carriers Total 2000 CR VMT (million miles) Percent of Carriers Percent of VMT Active 98 3,197 95.1 98.2 Inactive 5 58 4.9 1.8 Total 103 3,255 100.0 100.0 The next step is to estimate the VMT of the non-sample carriers that would have been found to be inactive during the follow-up if they had been in the sample. In the 2000 CR Follow-up, which included all carriers that received CRs in 1998, the VMT attrition rate was used to estimate the VMT of the non-responding carriers that would have been found to be inactive. Since the 2000 CR Follow-up accounted for over 90 percent of the carriers that received CRs in 1998, the VMT attrition rate was based on data from nearly all nearly all the eligible carriers. The 2002 CR Follow-up, however, was conducted on a sample of carriers that received CRs in 2000. Consequently, the VMT attrition rates calculated in the strata were based on very few inactive carriers. Thus, the rates could vary greatly depending on the 2000 CR VMT of the carriers selected for the follow-up. A small number of carriers with very high or low VMT could have a disproportionate effect on the VMT attrition rate in a stratum. Table A-7 shows the carrier and VMT attrition rates by stratum found in the 2002 CR Follow-up. As noted above, in each stratum, the carrier attrition rate is the percentage of carriers found to be inactive, while the VMT attrition rate is the percentage of the associated 2000 CR VMT found to be inactive. Each overall rate is the percentage obtained after multiplying the total VMT in each stratum by the stratum attrition rate, summing the products, and dividing the sum by the total 2000 CR VMT. Table A-7. 2002 CR Follow-up Results - Carrier and VMT Attrition Rates Stratum Number of Carriers with Usable Data Carrier Attrition Rate (%) VMT Attrition Rate (%) A1 73 22.3 36.5 A2 87 9.4 11.2 A3 89 8.2 13.2 A4 98 4.9 1.8 N1 60 10.4 5.6 N2 62 8.8 7.6 N3 43 4.4 6.3 N4 35 7.9 3.2 All Carriers 547 7.3 7.7 Table A-7 shows while the two overall attrition rates were about the same (7.3% vs. 7.7%), the two rates varied greatly in some strata. For example, in stratum A1 the two rates varied by over 14 percentage points (22.3% vs. 36.5%) due to the effect of three carriers with extremely high 2000 CR VMT numbers. Since the carrier attrition rate appeared to be a more stable indicator than the VMT attrition rate, the carrier attrition rate was used to estimate follow-up attrition VMT in non-sample carriers. Therefore, the follow-up attrition mileage was calculated as follows: Follow-up Attrition Mileage = Attrition VMT of Sample Carriers + Attrition VMT of Non-sample Carriers = Attrition VMT of Sample Carriers + (Follow-up Attrition Rate X VMT of Non-sample Carriers) = 58 million miles + (4.9% X 7,062 million miles) = (58 + 346) million miles = 404 million miles Total Attrition Therefore, the estimated total attrition mileage for stratum A4 for 2001 was: Total Attrition Mileage = Pre-Follow-up Attrition Mileage + Follow-up Attrition Mileage = (420 + 404) million miles = 824 million miles APPENDIX B. CARRIER ATTRITION FROM 2001 TO 2002 B.1. PURPOSE OF CALCULATIONS This section describes the estimation of the VMT lost to carrier attrition from 2001 to 2002 in stratum A4. This quantity is needed in the implementation of the CR Impact Assessment Model for 2002 for stratum A4, i.e., the calculation of the number of crashes (and associated fatalities and injuries) avoided in 2002 as a result of CRs performed in 2001. This implementation is described in Section 2.3. B.2. CALCULATION OF CARRIER ATTRITION MILEAGE FROM 2001 TO 2002 There were 230 carriers in stratum A4 that received CRs in 2001. These carriers had a total of 9,006 million VMT in 2001. The decrease in carrier VMT from 2001 to 2002 due to carrier attrition, i.e., the 2001 CR VMT of carriers that ceased operations before or during 2002, consists of two components: 1) VMT of carriers that were found to be inactive before a CR Follow-up would have been conducted in 2003, i.e., pre-follow-up attrition mileage, and 2) VMT of carriers that would have been discovered to be inactive during the follow-up, i.e., follow-up attrition mileage. Pre-Follow-up Attrition According to the latest data available (September 2002) at the time of the analysis, none of the 230 carriers in stratum A4 had become inactive. Therefore, the pre-follow-up attrition mileage for stratum A4 was zero. Follow-up Attrition Since no follow-up was planned for 2003, the follow-up attrition mileage was estimated, using the results of the 2002 CR Follow-up. In that follow-up, the VMT of the carriers that were found to be inactive during the follow-up amounted to 4.9 percent of the VMT of all the active carriers responding to the follow-up. If a follow-up of carriers that received CRs in 2001 were to be conducted in 2003, it would include all 230 carriers in stratum A4, since there are no known inactive carriers in that stratum. Applying the 4.9 percent follow-up attrition rate obtained in the 2002 CR Follow-up produced the following estimate: Follow-up Attrition Mileage = Follow-up Attrition Rate X VMT of Carriers in Follow-up = 4.9% X 9,006 million miles = 441 million miles Total Attrition Thus, the estimated total attrition mileage for stratum A4 in 2002 was: Total Attrition Mileage = Pre-Follow-up Attrition Mileage + Follow-up Attrition Mileage = (0 + 441) million miles = 441 million miles APPENDIX C. ADDITIONAL ANALYSES FROM 2000 COMPLIANCE REVIEW FOLLOW-UP C.1. OVERVIEW The previous report,1 which described the implementations of the model for 1999 and 2000, contained additional analyses of crash rate reduction by four carrier attributes. These analyses were performed using the data from the 2000 CR Follow-up. Since this follow-up covered the entire population of carriers receiving CRs in 1998, the data could support all such analyses. The 2002 CR Follow-up, however, was performed on a sample of carriers that received CRs in 2000. This sample could not produce accurate analyses for three of these attributes, because it did not have distributions of these attributes that were representative of the entire population of carriers that received CRs in 2000. The analysis by size of carrier could be performed because the sample was selected based on that attribute. For reference, the descriptions and results of the additional analyses from the previous report are included in this appendix. The results of the implementation of the model for 1999 and 2000 were analyzed to determine the degree that carriers' crash rates change after the carriers receive CRs. The results were broken out by four attributes: * carrier size (i.e., number of power units), * carrier safety status (i.e., the carrier's SafeStat category before receiving the initial CR, * the number of CRs that the carrier received, and * the planned course of action for the carrier following the initial CR (i.e., enforcement or no enforcement). The results of these analyses revealed the types of carriers that will most likely respond positively to CRs. By focusing on carriers that are likely to respond positively to CRs, the effectiveness of the compliance review program may be improved. Alternative treatment approaches may be suggested for carriers that are at risk, but will most likely not respond positively to CRs. In each analysis, for each of the 4,937 carriers with usable data obtained in the 2000 CR Follow-up, two average crash rates were calculated: * the pre-CR average crash rate - calculated using data from the initial (1998) CR, and * the post-CR average crash rate - calculated using data from the 2000 CR Follow-up. Each average crash rate was obtained by multiplying the total number of crashes for a set of carriers by 1 million and then dividing by the total number of vehicle miles traveled (VMT) for that set of carriers. C.2. CARRIER SIZE The first analysis examined the relationship between crash rate change following a CR and carrier size. The results of the implementation of the model were broken out by carrier size, as measured by the number of power units. Table C-1 shows the pre-CR and post-CR average crash rates broken out by size of carrier, i.e., the number of power units. In each case, the average crash rate was inversely related to carrier size. In other words, as carrier size increased, the average crash rate decreased. Table C-1. Pre-CR and Post-CR Average Crash Rates by Number of Power Units Number of Power Units Number of Carriers Pre-CR Average Crash Rate* Post-CR Average Crash Rate* Percent Change <5 1,507 1.593 1.150 -27.8 6 - 20 2,031 1.185 0.848 -28.4 21-100 1,137 0.826 0.761 -7.9 >101 242 0.695 0.684 -1.6 Unknown 20 0.902 0.773 -14.3 All Carriers 4,937 0.833 0.747 -10.3 * - Crashes per million miles The smaller carriers, those with 20 or fewer power units, had the greatest reductions in average crash rates. For carriers 5 or fewer power units, the post-CR average crash rate showed a decrease of 27.8 percent from the pre-CR average crash rate. For carriers with 6-20 power units, the decrease was 28.4 percent. For carriers with more than 20 power units, the reductions in the average crash rates were smaller than the reductions for the smaller carriers. For carriers with 21-100 power units, the post-CR average crash rate showed a decrease of 7.9 percent. For carriers with 101 or more power units, the decrease was 1.6 percent. C.3. SAFESTAT2 CATEGORY The results of the implementation of the model were broken out by carrier safety status, i.e., the carrier's SafeStat category before receiving the initial (1998) CR. In this case, the results were studied to see if carriers in the higher risk categories, A and B, reduce their crash rates more than carriers in the lower risk categories, C-G, and carriers with no known safety deficiencies (i.e., no category). Carriers in the higher-risk categories currently receive priority for CRs. They are often deficient in the SafeStat Safety Evaluation Areas (SEAs) reflecting safety performance (e.g., crashes), while carriers in the lower risk categories often have more safety compliance deficiencies (which may lead to safety performance problems if not addressed). Table C-2 shows the pre-CR and post-CR average crash rates broken out by carrier safety status, i.e., the carrier's SafeStat category before receiving the initial (1998) CR. Table C-2. Pre-CR and Post-CR Average Crash Rates by SafeStat Category SafeStat Category Number of Carriers Pre-CR Average Crash Rate* Post-CR Average Crash Rate* Percent Change A-B 1,666 1.099 0.818 -25.6 C 461 0.693 0.610 -12.0 D-G 1,640 0.764 0.764 0.0 None 1,170 0.641 0.680 +6.1 All Carriers 4,937 0.833 0.747 -10.3 * - Crashes per million miles Carriers in Categories A and B, the higher-risk categories, had the highest pre-CR average crash rate as well as the greatest reduction in the average crash rate. The post-CR average crash rate showed a decrease of 25.6 percent. For carriers in Category C, i.e., carriers deficient in two SEAs, but not the Accident SEA, the post-CR average crash rate showed a decrease of 12.0 percent. The pre-CR average crash rate for this group was much lower than for the carriers in Categories A and B, probably because none of the carriers in Category C were deficient in the Accident SEA. For carriers in Categories D-G, i.e., carriers deficient in one SEA, the post-CR average crash rate showed no decrease. In other words, the post-CR average crash rate was equal to the pre-CR average crash rate. For carriers not in any SafeStat category, i.e., carriers not deficient in any SEAs, the post-CR average crash rate showed an increase of 6.1 percent. These carriers had the lowest pre-CR average crash rate of any group. In fact, over 60 percent of these carriers had pre-CR average crash rates of zero. Since the crash rates of more than 60 percent of the carriers in the group could only increase or stay the same, it is not surprising that the average crash rate for the entire group increased. In summary, the reduction in the average crash rate was directly related to SafeStat category. Carriers in Categories A and B, which are identified and prioritized first for compliance reviews by SafeStat, had the largest crash rate reduction. Carriers in Category C, which are identified and prioritized for CRs after the carriers in Categories A and B, also had a reduction in their average crash rate. Carriers in Categories D-G, which are next in priority for CRs, had no reduction in their average crash rate. Carriers not in any SafeStat category, which have the lowest priority for CRs, actually showed an increase in their average crash rate. The results indicate that the carriers that SafeStat is identifying and prioritizing for compliance reviews are the carriers that show the greatest reductions in crash rates following CRs. C.4. NUMBER OF COMPLIANCE REVIEWS The results of the implementation of the model were examined to determine if multiple CRs result in crash rate reductions. For purposes of this analysis, the model results were broken out by the number of CRs the carriers received during the 1996-1998 time-period. SafeStat is used by the FMCSA to identify and prioritize carriers for CRs based on deficiencies in safety performance and compliance. The majority of CRs are conducted on carriers that were identified through SafeStat. The carriers identified through SafeStat often have multiple CRs, because they continue to show deficiencies in their safety performance and compliance, and, therefore, continue to be identified and prioritized for CRs. A carrier can also receive additional CRs as follow-ups to enforcement actions. Table C-3 shows the pre-CR and post-CR average crash rates by the number of compliance reviews received in 1996-1998. The pre-CR crash rates are based on the most recent (latest) CRs conducted within the 1996-1998 time-period. Table C-3 shows that the reduction in the average crash rate increased as the number of CRs increased. Carriers that received only 1 CR had a crash rate reduction of 8.2, while carriers that received 2 CRs had a crash rate reduction of 11.0 percent. The crash rate reductions increased to 14.0 percent for carriers that received 3 CRs, and 20.6 percent for carriers that received 4 or more CRs. The results show that, on average, carriers benefit from multiple CRs in terms of reductions in crash rates. It should be noted, however, that the carriers that receive multiple CRs are those carriers with persistent safety deficiencies. Table C-3. Pre-CR and Post-CR Average Crash Rates by Number of Compliance Reviews Received in 1996-1998 Number of Compliance Reviews Number of Carriers Pre-CR Average Crash Rate* Post-CR Average Crash Rate* Percent Change 1 3,041 0.839 0.770 -8.2 2 1,363 0.816 0.726 -11.0 3 423 0.808 0.695 -14.0 4+ 110 1.033 0.820 -20.6 All Carriers 4,937 0.833 0.747 -10.3 * - Crashes per million miles C.5. PLANNED COURSE OF ACTION The results of the implementation of the model were also broken out by the course of action planned by the FMCSA for a carrier following its initial CR. More than one course of action may have been planned for a carrier. A carrier with prosecution and/or an out-of-service order indicated as the planned course of action was classified as an "enforcement" carrier. A carrier with only monitoring indicated as the planned course of action was classified as a "non-enforcement" carrier. It should be noted that these courses of action are the ones that were anticipated by the FMCSA at the conclusions of the CRs the carriers received in 1998, and may be different from the actual actions taken. The data in the MCMIS Compliance Review File do not indicate the actual actions taken after the CRs. Table C-4 shows the pre-CR and post-CR average crash rates by type of planned course of action. Table C-4. Pre-CR and Post-CR Average Crash Rates by Type of Planned Course of Action Type of Planned Course of Action Number of Carriers Pre-CR Average Crash Rate* Post-CR Average Crash Rate* Percent Change Enforcement 1,269 0.967 0.790 -18.3 Non-Enforcement 3,668 0.796 0.735 - 7.7 All Carriers 4,937 0.833 0.747 -10.3 * - Crashes per million miles Table C-4 shows that it was anticipated that 1,269 (25.7 percent) of the 4,937 carriers with follow-up data would undergo enforcement actions. The "enforcement" carriers showed a crash rate reduction of 18.3 percent, compared to a 7.7 percent reduction for the "non-enforcement" carriers. 1 John A. Volpe National Transportation Systems Center, Motor Carrier Safety Assessment Division, DTS-47, FMCSA Safety Program Effectiveness Measurement: Compliance Review Program, CR Impact Assessment Model, Results for 1999 and 2000 with Additional Analysis, January 2003. 1 John A. Volpe National Transportation Systems Center, Motor Carrier Safety Assessment Division, DTS-47, FMCSA Safety Program Performance Measures, Intervention Model: Roadside Inspection and Traffic Enforcement Effectiveness Assessment, September 2002. 1 SafeStat (Safety Status Measurement System) is an automated, data-driven analysis system that is designed to incorporate on-road safety performance information and enforcement history with on-site compliance review information in order to measure the relative safety fitness of interstate motor carriers. A thorough description of SafeStat methodology can be found in: John A. Volpe National Transportation Systems Center, Motor Carrier Safety Assessment Division, DTS-47, SafeStat, Motor Carrier Safety Status Measurement System, Methodology: Version 8.5, January 2003. This document is available at ai.volpe.dot.gov/SafeStat/safestat.asp?file=method.pdf. 2 Center for National Truck Statistics, University of Michigan Transportation Research Institute, Truck and Bus Crash Factbook 1995, 1997. 3 A fatal crash results in at least one fatality. An injury crash results in no fatalities, but bodily injury to at least one person who, as a result of the injury, immediately receives medical treatment away from the scene of the crash. A towaway crash results in no fatalities or injuries requiring transport for immediate medical attention, but in one or more motor vehicles incurring disabling damage as a result of the crash, requiring the vehicle(s) to be transported away from the scene by a tow truck or other motor vehicle. 1 John A. Volpe National Transportation Systems Center, Motor Carrier Safety Assessment Division, DTS-47, FMCSA Safety Program Effectiveness Measurement: Compliance Review Program, CR Impact Assessment Model, Results for 1999 and 2000 with Additional Analysis, January 2003. 2 SafeStat (Safety Status Measurement System) is an automated, data-driven analysis system that is designed to incorporate on-road safety performance information and enforcement history with on-site compliance review information in order to measure the relative safety fitness of interstate motor carriers. A thorough description of SafeStat methodology can be found in: John A. Volpe National Transportation Systems Center, Motor Carrier Safety Assessment Division, DTS-47, SafeStat, Motor Carrier Safety Status Measurement System, Methodology: Version 8.5, January 2003. This document is available at ai.volpe.dot.gov/SafeStat/safestat.asp?file=method.pdf. 1 John A. Volpe National Transportation Systems Center, Motor Carrier Safety Assessment Division, DTS-47, FMCSA Safety Program Effectiveness Measurement: Compliance Review Program, CR Impact Assessment Model, Results for 1999 and 2000 with Additional Analysis, January 2003. 1 John A. Volpe National Transportation Systems Center, Motor Carrier Safety Assessment Division, DTS-47, FMCSA Safety Program Effectiveness Measurement: Compliance Review Program, CR Impact Assessment Model, Results for 1999 and 2000 with Additional Analysis, January 2003. 2 SafeStat (Safety Status Measurement System) is an automated, data-driven analysis system that is designed to incorporate on-road safety performance information and enforcement history with on-site compliance review information in order to measure the relative safety fitness of interstate motor carriers. A thorough description of SafeStat methodology can be found in: John A. Volpe National Transportation Systems Center, Motor Carrier Safety Assessment Division, DTS-47, SafeStat, Motor Carrier Safety Status Measurement System, Methodology: Version 8.5, January 2003. This document is available at ai.volpe.dot.gov/SafeStat/safestat.asp?file=method.pdf. ?? 2 vii 1-2 2-9 3-3 A-7 B-2 C-5