Within MEDIATOR the Testing & Evaluation activities aim to evaluate the functionality of the Mediator system under simulated and real road conditions with real drivers in different parts of Europe. This includes assessing the system’s performance, reliability, functionality, effects on driving safety as well as the acceptance, trustworthiness, perceived safety and user-friendliness for different groups of users. This deliverable covers the on-road evaluations of the vehicle-integrated Mediator system, which consists of three studies.
This public Deliverable 3.4 describes the on-road evaluations of the vehicle-integrated Mediator system. Three real-life on-road studies were conducted to test the overall performance of the Mediator system and its effects on safety-relevant behaviours, driver reactions and driver opinions.
First Study in Italy
The first study used the basic Automated Driving System (ADS)-level prototype and focused on the functionality of the HMI solutions in different driving conditions as well as on the acceptance/trust of the users. To reach this purpose, a Wizard of Oz (WoOz) vehicle prototype (starting from a righthand drive Jeep Renegade fully equipped, with ADS level 1) was created, integrating a nonfunctional double set of pedals and steering wheel on the left side and a prototype shifter to allow to simulate the driving mode change (from manual to automated and from automated to manual). This WoOz prototype was always driven by a professional driver, who was on the right seat and
drove using the standard Renegade primary controls, but the Wizard of Oz methodology guaranteed participants could experience an “automated” vehicle HMI behaviour, without being in a real automated vehicle.
16 naïve participants took part to an on-road test on a 46 km scenario, during which they tested the MEDIATOR HMI solutions (visual, vocal, acoustic, luminous, haptic and cushion inflation) integrated in the WoOz vehicle cabin (e.g., centre dashboard display, participant frontal display, shifter, steering wheel, seat belt) and designed to cope with the needs of MEDIATOR use cases (e.g., handover, takeover).
This on-road user testing allowed to evaluate the MEDIATOR HMI solutions designed in the MEDIATOR project and to understand their advantages and disadvantages. Both the acceptability (before the trial) and the acceptance of automated vehicles and MEDIATOR HMI were positive. The HMI usability and the users’ trust in these HMIs were positively evaluated too. Some MEDIATOR HMIs weaknesses emerged too, and they were used to understand what had to be fixed in the HMI solutions. This first study in Italy was useful to select the most appropriate HMIs to be tested during the on-road study in Sweden.
Second Study in Sweden
The second study was also conducted in the basic ADS-level prototype and focused on the functionality, validity, and reliability of the Technical Integration vehicle prototype under different degraded driver performance conditions, including conditions of degraded automation.
An on-road study with 50 naïve participants, using the same Wizard of Oz setup as in the first study in Italy, compared the full MEDIATOR HMI with a baseline HMI that was based on existing HMI designs (i.e., mainly using simple icons and sounds for interaction). Several novel features in the Mediator system showed great potential. The participants appreciated the time budget which provided information on when a change in automation level (i.e., responsibility) would take place. The combination of warnings and continuous mode and time budget information decreased the total duration in which participants were distracted and the maximum uninterrupted period of distraction. Especially the latter can have a significant effect on road safety, as long continuous periods of being distracted severely reduces situation awareness. Also, the elaborate Mediator takeover ritual, which included reasons for why a takeover was happening, was more appreciated and understood than the takeover ritual of the Baseline system. For fatigue, there were no differences in the development of fatigue between the MEDIATOR and Baseline HMIs. However, the results indicate that corrective alert-messages used in the transfer of control ritual somewhat reduced task-related fatigue, but not sleep-related fatigue. Finally, the Mediator system included an active proposal to increase the level of automation which was much appreciated by participants. They also strongly agreed that they would increase automation use if such a feature was available in their car. If automation is indeed safer than manual driving, this feature could therefore potentially improve road safety. Overall, the Mediator system was preferred over the Baseline system, but the results indicate a need for increased cohesion in the HMI and improved clarity of icons and warnings.
Third Study in Sweden
In the third study, the Mediator system’s performance was examined using the higher ADS-level prototype. Reliability and validity of the automation-status detection in relation to driving context, decision logic and timing of the Mediator system’s actions and messages were evaluated. Seven professional drivers experienced different configurations of the Mediator system. Over several weeks, they drove ten times on a specific route of one hour length covering use cases and scenarios the study focused on. Both quantitative data (from the test vehicle) and qualitative data (from structured interviews) were collected and analysed. Results indicate that drivers evaluated the Mediator system and it’s HMI quite positively. Some drivers argued that a system like Mediator is more useful for higher automation levels (level 3 and above). Overall reliability was good, although the distraction warning implemented in the vehicle was too sensitive. The simplified HMI implemented in the prototype did not affect drivers’ gaze behaviour; in fact, they looked at the Mediator display less overtime. However, due to limited data collected during the study this is an exploratory study and has limited external validity.
MEDIATOR has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 814735.
