SUNflowerNext : Towards a composite road safety performance index

Wegman, F.; Commandeur, J.; Doveh, E.; Eksler, V.; Gitelman, V.; Hakkert, S.; Lynam, D.; Oppe, S.

Background and aim

One of the aims of international cooperation in the field of road safety is to make oneself familiar with performances and progress in other countries and to understand if and how these can be of guidance to policymaking, in an adapted form if appropriate. Comparisons can be a starting point to learn from each other.

The learning includes subjects such as monitoring and explaining road safety developments, and gaining good insights in the impacts of interventions as a basis for speeding up road safety improvements in one's country or jurisdiction.

Benchmarking is a process in which countries or sub-national jurisdictions evaluate various aspects of their performance in relation to that of other counties or jurisdictions, including the so-called 'best-in-class'. The benchmark results provide countries or sub-national jurisdictions with information about others that can be used as a basis for developing measures and programmes to increase their own performance.

Two important tasks can be identified in this process:

  1. defining the key components of a road safety performance and investigating if and how these key components can be combined in a composite index;
  2. finding a meaningful reference (best-in-class) and defining procedures for identifying such a meaningful reference.

Comparing performances and, one step further, benchmarking performances seems to be an appropriate approach for road safety. This approach should help us to go beyond the rather traditional methods of comparing performances by only using mortality rates or fatality rates or risks. Ranking countries by using only these rates is a useful first step, but not very meaningful as a start to learn from each other.

The SafetyNet project aims to build the framework of a European Road Safety Observatory, which will be the primary focus for road safety data and knowledge, as was specified in the Road Safety Action Programme 2003. In the SafetyNet project it was decided to develop a method of benchmarking road safety by using road safety indicators. To this end, the SUNflower approach was used, more precisely the information captured in the SUNflower pyramid and earlier attempts to elaborate on this in developing the SUNflower footprint, as well as other SUNflower studies. We gave this project the name SUNflowerNext.

Hence, the aim of the SUNflowerNext project is to develop a knowledge-based framework for comprehensive benchmarking of road safety performances and developments of a country or of sub-national jurisdictions.

SUNflowerNext has made use of existing data that was relatively easily available. This ensured that the study could be carried out in a relatively short time. However, one important concession needed to be made. Because this study used an innovative approach with only existing data that was not always available, it was decided to set up the research in such a way that all the steps required for benchmarking a country's performance are taken, but to refrain from presenting the actual results of the benchmark as they are of insufficient quality. The experiences gained from this study are such that SUNflowerNext's ambition — benchmarking the safety performance of countries - is realistic once reliable data is available. Therefore, it is recommended to carry out this benchmarking in Europe in the near future, to widely disseminate the results, and to consequently use them for policy making in the European Member States.

Benchmarking of road safety performances

Benchmarking is a process in which actors evaluate various aspects of their performance in relation to others, and to the so-called 'best in class'. In the SUNflowerNext study we researched whether countries in the European Union could all be placed in one class, or whether we should consider working with two or more classes. Three procedures were used to find out whether meaningful groups could be made: safety experts were asked to group countries, secondly, countries were grouped based on road safety outcome indicators (grouping obtained with a Singular Value Decomposition (SVD) of the annual fatality risks in the years 1980-2003 of countries), and, thirdly, countries were grouped using general statistical data from a Multiple Correspondence Analysis (MCA) about a country in the most recent years.

In the SUNflowerNext project we concluded that it is better not to make comparisons between all European countries as one group, but to attempt grouping comparable countries and to then compare the countries within a specific group or class. The results of the three methods have many points of agreement. The grouping results have a preliminary character and it is recommended to elaborate on this topic before coming to a final decision on the grouping. The approach explored in SUNflowerNext could be used for this purpose.

Towards a composite road safety performance index

SUNflowerNext decided to develop an integral and comprehensive set of indicators to measure the road safety performance of a country while including all information in the SUNflower pyramid. SUNflowerNext distinguishes three types of indicator: the road safety performance indicator, the implementation performance indicator, and the policy performance indicator.

The first type of indicator captures a country's road safety quality. It has been named the Road safety performance indicator. Other names such as outcome indicator and product indicator are also used. In SUNflower the three top layers of the SUNflower-pyramid are included: final outcomes (numbers of killed and injured), intermediate outcomes (such as the safety performance indicator), and social costs.

The second type of indicator specifies the quality of the implementation of road safety policies: the Implementation performance indicator. For this implementation quality indicator the term process indicator can also be used. Basically, this indicator follows a vertical line in the pyramid linking 'safety measures and programmes', safety performance indicators and numbers of killed and injured people.

The third type of indicator deals with the quality of policy to improve road safety: the Policy performance indicator. Here SUNflowerNext distinguishes two components: the quality of conditions (strategies, programmes, resources, coordination, institutional settings, etc.) and the quality of action plans and individual (counter)measures) in the perspective of the ambitions expressed in road safety targets.

There are several reasons why it is attractive to combine all information in one indicator, a so-called composite index. A composite index includes all components of the SUNflower pyramid, more specifically the three types of indicator. The pros and cons of working with composite indices are rather well known and are presented in the report. Three words can summarize the main characteristics: 'simplification, quantification and communication'. Road safety will not be the first policy field to successfully attempt to capture performance in one single value. To mention a few: the Human Development Index, the Environmental Sustainability Index, and the Overall Health System Index. Based on these examples it was decided to also explore the opportunities for a composite index for road safety performance.

Weights based on statistical models were used to combine the basic indicators into a composite index. Both Principal Component Analysis (PCA) and Common Factor Analysis (FA) weighting were examined. Both methods group collinear indices to form a composite index that captures as much as possible of the information that is common among sub-indicators. The analysis was made on the data collected for 27 European countries. The composite index enables us to rank the countries in accordance with their safety performance.

The analysis revealed that the countries' ranking based on the combination of indicators is not necessarily similar to the traditional ranking of countries based only on mortality rates or fatality rates. We believe that adding information on policy performance and implementation performance to the ranking and grouping process improves the results beyond the established methods and makes them more comprehensible. Furthermore, it was observed that the indicators belonging to the final outcomes and intermediate outcomes, both part of the road safety performance indicator, are not uniform in their behaviour. The indicators that were found to be more consistent and termed 'core set of basic indicators' are recommended for future uses.

The general conclusion is that the design of a composite road safety performance index, for example the SUNflower index in which relevant information from the different components of the road safety pyramid has been captured and weighted, is realistic and meaningful. In addition, such an index gives a more enriched picture of road safety than a ranking only based on data on mortality or fatality rates, which is common practice at present. Grouping countries using this process is promising and seems to be preferable to simply ranking countries. Before defining the SUNflower index and actually applying the results to policy making, two improvements should be made: indicators must be developed for the Implementation performance indicator and procedures must be developed to make available high quality and comparable data for EU Member States.

Time series analysis

Safety developments are interesting because they may give us a better insight in underlying forces and, hopefully, also in the effectiveness of road safety interventions. Different approaches were used in this part of the study, among which state space modelling. The first attempt to compare developments in fatality rates (fatalities per 10,000 motorized vehicles) and mortality rates (fatalities per 100,000 inhabitants) was made at a macroscopic level. Although European countries do have a remarkably different history when it comes to the development of fatality rate vs. mortality rate, our data suggests that all countries seem to be moving to the same road safety position, although not at the same pace. Leading countries in the field of road safety generally keep ahead of the other countries, albeit with decreasing advantage.

Three types of disaggregate developments were compared (age, transport mode and road type). In this comparison countries were grouped. Looking at the results of the analyses, we may conclude that, although all European countries tend towards the same aggregated or macroscopic level of road safety, there are important differences between the individual countries as well as between groups of similar countries. These differences relate to how they reach this level of road safety when considering their focus on avoiding special types of accidents. In other words, the general policies of improving road safety in different countries ultimately seem to move towards the same safety level, but for different countries that level of road safety is achieved at a different pace and in different ways.

Sub-national comparisons

There are two basic reasons for comparing the safety performance of sub-national jurisdictions. In the first place, a ranking of relative performance of each area will be very useful for comparison within countries. In the second place, it will provide better understanding of the factors affecting safety improvement, so that safety practitioners can achieve more effective programmes. This requires greater focus on understanding how the effects of programmes are modified by the nature of the safety problems faced by each area. Lessons can not only be learned from comparison of areas within countries, but also from comparison of similar areas in different countries.

The study clearly identifies factors which have effects on risks at a regional and local level. Based on a literature review it was concluded that structural and cultural differences, the bottom layer of the pyramid, can considerably affect road safety at a regional and local level. The results of this part of the study are considered sufficiently interesting for recommending continuation of this work in an international/ European project. In addition, it is recommended to use different approaches for studies at both the regional and the urban level.

X + 126 + 48
SWOV, Leidschendam

SWOV publication

This is a publication by SWOV, or that SWOV has contributed to.