L3Pilot is a large-scale, European, real-world pilot study of SAE Levels 3 and 4 automated driving functions with the focus on level 3. The goal of L3Pilot is to demonstrate and assess automated driving functions (ADF) in close to real or in real use contexts and environments. To this purpose, a series of on-road tests are performed, using passenger cars of different classes with ad-hoc equipment capable of implementing automated driving. The functions to be tested include motorway functions, traffic jam functions for motorways, urban functions, and parking functions, all designed to enable the driver to move safely and efficiently through selected road environments. This deliverable describes the methods to be used for evaluating the ADFs and answering the research questions developed and described in L3Pilot deliverable D3.1 (Hibberd et al., 2018).
In L3Pilot, the impact of ADFs on various theoretical concepts will be tested. The topics are partly interlinked and or methods depend on each other. Therefore, methods need to be defined thoroughly to ensure that all planned analysis will be feasible in the end. There will be four primary areas of analysis:
- Technical and traffic evaluation assessing the effect of the ADF on vehicle behaviour and the surrounding traffic based on data logged directly in the on-road tests.
- User and acceptance evaluation assessing the evaluation of the tested functions by the driver and their impact on drivers’ behaviour and drivers’ state (e.g. stress, fatigue).
- Impact assessment transfers the results to a more general level and assesses the potential impacts of so-called mature ADFs on personal mobility, traffic safety, traffic efficiency and the environment.
- Socio-economic impact assessment takes the results from all previous parts of the analysis and determines monetary values for the estimated effects, weighting expected costs and benefits.
The structure of this deliverable mostly follows these four areas of analysis. It starts with methodological considerations relevant for all topics, describing in particular the basic principles of assessment (e.g. the definition of baseline and treatment conditions) as well as the characteristics of the tested functions. These common parts are followed by individual chapters that describe in detail the primary methods for each of the four areas. In summary, the following methodological approaches are chosen:
- Technical and traffic evaluation: A driving scenario-based approach has been chosen for the analysis of functions’ driving behaviour, which will be compared to human driving. Furthermore, critical driving situations (incidents) will be analysed.
- User and acceptance evaluation: The primary tool for assessing user-related aspects is the pilot site questionnaire (one per ADF) that has been developed within L3Pilot. The questionnaire is supported by other measures such as take-over controllability rating, focus groups, interviews, and annual surveys.
- Impact assessment combines results from on-road tests, traffic simulations, statistics and databases and literature to estimate the potential impacts of ADF on road traffic in Europe.
- The socio-economic impact assessment will further elaborate the results from the impact assessment in order to estimate the potential impact of ADF on society. This will be done by using a snapshot approach that estimates the expected change in terms of benefit to cost ratio compared to today’s situation in case part of today’s fleet would be equipped with Level 3-ADFs.
The goal of the work presented here is to define methods which can be used for datasets collected at the different pilot sites in L3Pilot with the aim to finally combine results across pilot sites. The set methods will ensure that the research questions of L3Pilot are addressed with the analysis. Furthermore, the methods within L3Pilot need to ensure that the parts of the analysis dealing directly with the data from the on-road tests (technical and traffic evaluation, user and acceptance evaluation) provide the necessary information needed for estimating the impact of ADF on road traffic in Europe (impact assessment) and, in a next step, on society (socio-economic impact assessment).
The development of the methods is based on state-of-the-art literature on the various topics and the expert knowledge of the partners involved in the work. As a consequence, decisions on details of the methods as part of implementing and running the data analysis will be done later. The final methodology deliverable D3.4 Evaluation Plan will add details to the methods presented in this report. Within the structure of L3Pilot, decision on final details of the methodology will be amongst others the task of SP7 Evaluation.
