Vehicle automation is developing rapidly, with substantial potential to improve mobility effectivity and efficiency of road transport. Vehicle automation can also have a large effect on safety, since it reduces the influence of human fallibility. However, at the same time, vehicle automation is likely to introduce new risks, particularly during the transition phase to full automation when vehicle control is shared by human drivers and automation. By sharing control, the task of the driver changes from an active to a more passive, supervisory role. Unfortunately, humans are not very good in monitoring and supervising tasks. It reduces the workload of the primary driving task and if reduced too much, this might lead to boredom, reduced attention and situational awareness, mode confusion, loss of skills and, eventually, to degraded performance with responses that are too late or even non-existent. In these cases, it will be dangerous to transfer the control from the automation system to the human driver. However, in other situations humans are more reliable than the automation system. In unusual, unexpected, or complex situations, humans are often better than systems. A human driver could quickly find a more tailored, unique solution while an automation system must rely on (still) inappropriate rules or algorithms. In those case, it would be safer to have the human driver in control rather than the system.
MEDIATOR aims to develop an intelligent ‘mediator’ support system, enabling safe, real-time switching between the human driver and automation, integrating the best of both. This system continuously and in real time monitors and weighs the information about the driving context, the driver state, as well as the automation status, while taking account of the general competences of the driver and the vehicle capabilities.
Schematic summary of the functioning of the Mediator system
By weighing these elements, the Mediator system will determine how and to what extent the human or the machine should be in control, enabling safe, comfortable, real-time switching between the two by a reliable, understandable and user-friendly human machine interface (HMI). To allow for timely announcement of a transfer of control the system will not only assess the current situation but also make a prediction of the status in the near future. For reasons of acceptability the system will take account of driver’s preferences, however without compromising safety.
This document, which is the final deliverable of task 1.4, describes the functions of the Mediator system in more detail and derives a set of functional requirements from them. The information presented in this document is based on the work done on state of the art and preliminary requirements in task 1.1 (Christoph et al., 2019), driver state in task 1.2 (Borowsky et al., 2020), automation state in task 1.3 (Mano et al., 2021) and Human Machine Interaction in task 1.5 (Van Grondelle et al., 2021).
Three levels of automation
The Mediator system will be developed for three automation levels, that reflect the required activity of the driver and the type of support needed:
- Continuous Mediation (CM) – Driver In the Loop describes ‘assisted driving’. Drivers are responsible but supported by the automation. The automation generally performs the active control tasks, while the driver has a monitoring task. Challenges in this level of automation are creating mode awareness and supporting the driver with their part of the driving task by creating an optimal task load.
- Driver Standby (SB) – Short Out of the Loop describes ‘conditional automation’. Drivers can be out of the loop for a short time but must remain ‘on standby’ to take back control when needed. Challenges here are related to regaining driver fitness and balancing the time until the automation or driver becomes unfit, making sure always one is fit enough for the driving task. This challenge extends to the HMI challenges of communicating these time budgets and mediating comfortable and safe takeovers over a relatively short time span.
- Time to Sleep (TtS) – Long Out of the Loop describes ‘highly level automation’. Drivers can be out of the loop for long periods of time and truly immerse themselves in non-driving related tasks and even fall asleep. Challenges in this level of automation are to bring the driver back in the loop after full disengagement and to predict when this will be required long enough in advance.
Ten use cases
During the project, for these three levels of automation, the Mediator system will be developed to cover ten specific scenarios or use cases. These use cases include both urban and motorway scenarios and focus on the detection and mitigation of driver degraded performance due to both distraction and fatigue.
The driving context, e.g., weather and traffic conditions and the presence of other road user types,will vary between and within use cases, depending on the exact aim and setup of related experiments.
The human-machine interface (HMI) is of crucial importance for a Mediator system, both from a safety and an acceptability point of view. The HMI must not only be usable throughout the various situations, but at the same time it must address issues of trust, transparency, and personal preferences. Four focus points of the Mediator HMI were identified:
Different types of takeover take place from driver to automation and vice versa and can be subject to different levels of necessity (e.g., comfort vs safety related takeovers) and different levels of urgency. The HMI should be able to handle all these different takeover situations.
In the conditional automation and the high-level automation conditions drivers must be informed, clearly and at all times, how long the current level of automation will last and when they are supposed to take over. The continuous, transparent communication about the time budget to the driver by the HMI is a key ‘preventive’ activity of the Mediator system.
In the continuous mediation automation level, the HMI must help to combat underload-induced driver distraction and fatigue which is likely to occur in situations where people mainly have a monitoring task. A focus of the current project is to research and develop ideas of balancing the driver workload for optimal task performance and driver comfort for this level of automation.
Personalisation will be an important aspect of the Mediator system and the related HMI. Personalisation can refer to the detection of driver states, the optimisation of decisions, and the communication. Personalisation helps to 'automatically' account for possible differences between drivers due to, for example, age, gender, or cultural differences.
Resulting high level functional requirements of the Mediator system
At a high level the functional requirement of the Mediator system can be structured chronologically as ‘event detection’, ‘action selection’ and ‘action execution’: first an event that requires a Mediator action is detected, then the appropriate action is selected based on safety and comfort requirements, and finally, the selected action is executed. In more detail the high-level system functions were summarised as follows.
- Detect: user requests, current automation level
- Estimate: time to automation/driver (un)fitness, time to driver (dis)comfort
- Estimate and predict driver fatigue and distraction
- Predict relevant context changes
- Decide: who is fittest to driver, considering driver safety and driver comfort
- Select: action for optimal driving conditions related to safety and comfort
- For takeovers and corrective actions, determine available time scale and necessity of action (urgency level)
- Execute: selected action
- Promote optimal cooperation between driver and automation through direct and indirect communication with the driver.
- Monitor: action execution
- Adjust: action or urgency level during execution if needed
The four modules of the Mediator system
The Mediator system consists of the integration of four different modules or components: the driver module, the automation module, the HMI module, and a central component.
Driver module: has multiple functions. It 1) estimates the time to driver (un)fitness and time to driver (dis)comfort used to decide who is fittest to drive, 2) provides information on possible actions for improving driver fitness or comfort 3) estimates the corresponding expected fitness improvement and available time scale, and, finally, 4) describes the reason for the reduced fitness (e.g., fatigue or distraction). Instruments needed include in-vehicle cameras an intelligent steering wheel and/or seatbelt equipment which is currently under development.
Automation module: ensures the interface between the Mediator system and the available or operational vehicle automation functions. It provides information on current automation state and can adjust automation state if needed. It also provides data about the driving context, e.g., weather related of traffic related.
HMI module: communicates between the vehicle related systems, including the Mediator system, and the human driver. Among other things it facilitates negotiations between driver and automation, guides control transfers between driver and automation (takeovers), informs which systems are switched on or switched off, executes actions to increase or maintain driver fitness, and takes account of driver preferences. The HMI should have high usability and transparency for its users and help to improve driver comfort and safety.
Central component: contains the decision logic and context modules and serves as a central gateway to transfer information between all modules in the Mediator system:
- The decision logic module translates information about the state of the driver and the automation from the respective modules into a Mediator action requests. In this process the module detects events that require Mediator actions and selects appropriate actions based on information about past, current and expected situations. These requests are then sent to either the HMI or the automation module that is responsible for the execution of the actions and monitored such that adjustments can be made if needed.
- The context module receives context information from different sources, including the automation module and in-vehicle equipment for status monitoring. The module integrates this information into specific context characteristics requested by other modules, mainly the driver module, the HMI module, and the decision logic.
Ethics and validation
As the Mediator system can affect society, also ethical considerations should be taken into account in its design. Important ethical considerations lead, for example, to requirements on system transparency, accountability and validity and the autonomy of humans.
To evaluate how effectively the Mediator system is achieving the objectives proposed throughout this document, a set of key performance indicators is defined. This initial list of indicators can be used in the validation of the system.
The work presented in this document leads to an overview of a set of functional requirements for the Mediator system. These functional requirements are used to guide design and development of the Mediator system, subject to the constraints given by development time, budget, and available prototype platforms.
MEDIATOR has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 814735.