The first part of this research project aimed to provide an analysis of the suitability of existing benchmarking methodologies for use in the highway operations and maintenance sector in New Zealand. Ideally this should be a translatable best practice benchmarking methodology, rather than creating a bespoke model. In addition, the recommended methodology needed to translate performance, quality and cost into a level of service and value-for-money equation-based measure that could be compared across the New Zealand roading sector. The recommended methodology also needed to normalise for unique network characteristics, outside the control of the supplier/maintainer, that might impact on cost and quality. The recommended value-for-money equation-based measure is based on a generic framework for designing management control systems in not-for-profit organisations. The report details how it could be adapted for use in the highway maintenance and operations sector. The final framework includes measures of cost or expenditure on highway maintenance and operations, achievement in terms of quantity of maintenance and operations work undertaken and performance or quality of the service provided. It should also be noted that the framework aligns with the NZ Transport Agency’s value-for-money objectives and the accountability notions of effectiveness, efficiency and economy therein. A review of existing benchmarking methodologies used in highway operations and maintenance services, the wider transportation sector and similar industries both in New Zealand and overseas was undertaken to determine candidate benchmarking techniques. These included the partial efficiency measure, total factor efficiency measure, regression analysis and data envelopment analysis (DEA). An analysis of their suitability for use in the roading sector in New Zealand was then undertaken. Each candidate method had advantages and disadvantages to their use in the areas of their ability to handle multiple inputs or outputs, choice of weights, the format of the benchmark produced, their method for dealing with unique network characteristics, the complexity of the technique and usefulness of the outputs. DEA was recommended due to its ability to incorporate multiple inputs or outputs, the optimisation of weights as part of the analysis, the production of an efficient frontier of best performers, its ability to normalise for such unique network characteristics, the usefulness of the outputs and the fact that it has been shown to work in the highway maintenance and operations sector. In such cases, DEA has been supported by other analysis techniques, such as the analytic hierarchy process, regression analysis or peer groups, where appropriate. The only major disadvantage of DEA is its complexity; however, this is offset by the fact that the outputs of the analysis are much easier to explain to decision makers. If adopted, such a model will be able to compare operations and maintenance costs and performance between networks within New Zealand and will have the potential to do so against similar overseas organisations. It is not envisaged that the resulting model will be ubiquitous in the same way as the Road Assessment and Maintenance Management (RAMM) database. Instead, the model is seen as a high level management tool to help inform NZ Transport Agency (the Transport Agency) policy on issues such as procurement approaches and funding, as well as to help drive efficiency improvements within the regions, territorial local authorities and service providers. However, it is worth noting that the potential efficiency improvements identified by the benchmarking will only be achieved through management’s use of this information. The second part of this research project aimed to collect benchmarking data from two overseas road agencies, to both assess the availability and ease of collection of such data and to enable initial comparisons to be undertaken with the New Zealand roading sector, should the Transport Agency wish to do so. To retain anonymity of the participating road agencies they are referred to simply as international comparator A and international comparator B, in the report. Contextual data, data on inputs (expenditure), outputs (achievement) and outcome indicators (performance) are provided as per the Transport Agency’s value-for-money framework. All expenditure is reported in New Zealand dollars purchasing power parity equivalents. In collecting the benchmarking data from two overseas road controlling authorities, significant challenges were faced including a lack of timely cooperation, composed of delays due to obtaining approvals to release the data followed by delays in interrogating the road and financial databases, as well as differences in performance measurement, definition of maintenance tasks and accounting systems. These challenges are in line with international experiences in this area. Some basic ratio comparisons between the two overseas road agencies were undertaken, akin to the partial efficiency measure mentioned above. This resulted in a number of individual input-to-output ratios. This is a major drawback of such an approach in that each ratio only tells part of the efficiency story. As a decision maker it is very difficult to draw a definitive conclusion from such partial information, even if all possible partial efficiency ratios are produced. Although all too often the different ratios will provide conflicting information, in this case international comparator B consistently outperformed international comparator A, for the ratio measures reported, by virtue of the fact that they spent less per network length, per lane kilometre, per vehicle kilometre travelled, per head of population and per land area. However, highlighting another major drawback of this approach, such basic ratios do not take into account the service provided or performance. Although it is evident that although international comparator A commits more expenditure to pavement maintenance they also provide better performance than international comparator B — as measured by the international roughness index and wheel track rutting, the only comparable performance measures. In addition, the operating conditions are not taken into consideration. For example, international comparator A has higher rainfall and lower temperatures to contend with. In particular, a sizeable proportion of the routine maintenance budget is dedicated to winter maintenance. Finally, only one normalising factor is considered in each of the basic ratio comparisons. This further demonstrates the need to adopt a best practice benchmarking technique, such as DEA. If international benchmarking is to be implemented, then the recommendation is to first form a benchmarking club of similarly committed road agencies and then to agree on (or adopt existing) data processing standards and metadata standards. Benchmarking at the national level in New Zealand presents far fewer challenges than international benchmarking, particularly at the state highway level. The relative ease of availability, and potential for comparability, of performance and benchmarking data for New Zealand’s state highway networks would suggest that benefits from benchmarking can be realised much earlier than from an international benchmarking exercise. Although not without its own challenges, standardisation of data, if not actually metadata standards, already exists in the form of the RAMM database. Some data processing standards also exist (eg calculation of smooth travel exposure). A top down approach whereby the final performance, expenditure and contextual measures are the starting point and only the data required to calculate these measures is harmonised, is recommended. Finally, although DEA has been recommended as the technique with which to develop a benchmarking model for the operations and maintenance of New Zealand’s road network, considerable further research needs to be undertaken to ‘build’ the benchmarking model. The suggested next steps and methodology for further development are included in the report. (Author/publisher)
Samenvatting