Guidelines safe HMI design

Samenvatting (EN)

This document, one of the final deliverables of the 4-year MEDIATOR project, formulates general recommendations on HMI design for semi-automated and highly automated vehicles, to obtain safe interactions between the driver and the driving system. HMI is one of mediator’s main components, next to Automation State, Driver State, and Decision Logic which is the central component that mediates in between, based on who is most fit to drive. In this project, the HMI is composed of all components that interact with the driver visually, auditory, and haptically.

Framing of the project

Knowledge gaps

At the beginning of the MEDIATOR project, knowledge gaps have been identified and prioritized for further research. Five knowledge gaps have been designated as primary, which are Transfer of control, Transparency and Information Overload, Keeping the Driver in the Loop, Conflict Negotiation, and OEM Design Space. Additionally, secondary knowledge gaps that could be investigated within the primary studies are Intuitive Learning, Long term effects i.e., Skill Degradation and Complacency, and Human Driver Characteristics.

HMI Design Guidelines

To frame the HMI design process, five initial HMI design guidelines have been defined:

Embrace a holistic approach entails two main principles. Firstly, the Mediator HMI facilitates and manages all interaction components between human and vehicle for both primary, driving-related tasks as well as for most secondary tasks like climate control or entertainment. Secondly, mode awareness is elicited by the entire (holistic) experience of the driver by ambient lighting.
Design a generic transfer ritual entails the principle, that all interaction between the Mediator system and the driver are constructed in a single ritual i.e., a way of doing something in which the same actions are done in the same way every time, to facilitate quick intuitive learning and minimizing the risk of bias. While the sequence of the ritual up to e.g., a control transfer is always the same, its elements i.e., three signals and two time-intervals in between, are variable in their timing, intensity, and modality.
Design for learned affordances assures that HMI design is compatible with current and future standards for HMIs for ADS and in line with users’ intuitive expectations. A learned affordance relates to an existing knowledge and experiences and therefore suggest an object’s function and how it should be used. Learned affordances (standardisation) are essential to overcome issues related to learning new (driving) skills, process the complexity of information and reduce cognitive response time.
Design for user acceptance surpasses the common assumption in autonomous driving research and design projects, is that a driver’s suitability to control the vehicle is being determined by the system. While the HMI plays a crucial role in driving perception (mode awareness) and driving behaviour, its success depends on its ability to establish trust, provide comfort, and facilitate driver autonomy, all of which are interdependent.
Design for industry acceptance is important because the automotive industry is structured by, and built on, so-called brand identities. Hence, the importance of expressing brand identity through design is fundamental. Therefore, design space which allows to adapt HMI design to a specific industry brand identity is deemed crucial for industry acceptance.

Mediator autonomous driving modes

Although the SAE distinguishes six driving modes, in Mediator we have recognised three driving modes i.e., Continuous Mediation (CM; comparable to SAE L2), Stand-by (SB; comparable to SAE L3) and Time to Sleep (TtS; comparable to SAE L4). From a human factors’ perspective, and with respect to a transfer period which will include ‘conventional’ vehicles, in the Mediator HMI design these have been transformed into Manual, Assisted and Piloted driving.

The main feature of the HMI is its ambient lighting to evoke mode awareness, in which Assisted driving is expressed through high-luminosity amber, and Piloted driving though lower-luminosity purple. Those colours are applied consistently in HMI design in e.g., time-budget representation on the display and the countdown towards a mode transfer by led strips in the steering wheel.

Methods

The HMI is developed in a research-by-design process with several design iterations, in which the design is merely a means to research. Versions of the HMI have been tested in VR studies, mockups, simulator testing and on-road trials. Testing was done by ten original use-cases with derivatives for specific ODDs, representing as much as possible the infinite number of possible use-cases. The design process and studies were furthermore driven and supported by research into existing HMIs, literature research and expert opinions, and stakeholder workshops.

Transition of control

Well-designed transitions between different levels of automation are of utmost importance in establishing a safe interaction between the driver and the system (Lu, Happee, Cabrall, Kyriakidis, & de Winter, 2016). HMI concepts have been developed that deal with preparing the driver for an upcoming takeover, informing the driver on the upcoming takeover, the timing of the takeover as well as on the urgency of the takeover. All types of transfers are based on the Mediator template ritual. While its components are fixed, the values of each component vary.

An effective way of communicating transfers to the driver in the Mediator HMI appeared to be LEDs in the steering wheel accompanied by haptic and other audio-visual information. An appreciated aspect was a visual countdown of the LEDs, indicating the time left before a required takeover.

The following recommendations have been derived to address this knowledge gap:

An HMI should have a basic ritual for all changes and transfers. The template is fixed but the values of each component vary.
Fitness of the driver, and of the driving system, should be continuously communicated.
Personal adaptation of warning-timings in takeovers should be possible.
Latencies between a signal and driver response should be optimized in relation to urgency and human reaction times / expectations.
The intentions of the vehicle and the reasons for a takeover during automated driving should be clearly communicated to the driver.
When a transfer to manual control is required, an HMI should support the driver in preparing for takeover. For example, after a ‘wake-up call’ the driver should remain attentive before the actual takeover finds place.
In case the driver must regain control and the urgency level is high, the takeover request should be communicated by means of multi-modal intrusive signals.
Apply a visual countdown instead of a constant or single-frequency signal, to indicate a takeover procedure.
A visual countdown, through led or otherwise, must be positioned in the primary sightline of the driver i.e., on the steering wheel.
Use colour codes with a dynamic pattern for request messages such as ‘automation available’, ‘activating automation’, and ‘automated driving is activated’ (see also Section 3.2).
Use distinct colour codes to convey the vehicle mode or level related information (see also Section 3.2).
A planned takeover, signalled by a visual countdown, must be supported by additional signals, such as seat-vibration, and textual or auditory messages to guide users through the actual take over.

Transparency & information overload

A great risk of mode confusion is that drivers misjudge their own tasks and responsibilities (Tinga, Cleij, Jansen, van der Kint, & van Nes, 2022). One way to establish mode awareness is to make the system transparent i.e., a system that provides sufficient and clear information to the driver about the functioning of the system. On the other hand, drivers should not experience an information overload that reduces driver comfort and decreases the ability to perceive and process new information. An HMI should be designed in such a way that this delicate trade-off between transparency and information-overload is optimally balanced in all situations.

Several Mediator HMI concepts have been developed to establish this delicate trade-off. Based on the finding that drivers prefer information on the automation status above information on the desired driving task the concept of ambient lighting has been developed. Indicating the current mode, the time left to next mode and the anticipated time in the next mode (i.e., time budget) addressed the user’s desire for anticipatory information.

Findings addressing this knowledge gaps resulted in the following recommendations:

The HMI must communicate the current driving-mode continuously and in a holistic way. This can be achieved, for example, through ambient lighting (see 3.2).
The HMI should communicate the time left in current mode/time to next mode continuously while clearly signifying the current mode. This can, for example, be attained through communicating the time in a number, or, through a LED bar depleting over time with decreasing time in the mode.
When the current mode will change to another mode the HMI should communicate the reason for this change in advance. This requirement can be attained by for example using icons for an event that will occur in the environment, for example indicating that roadworks ahead, or that the car will leave the city.
The HMI should nudge and/or inform the driver about what to do. Especially drivers that have not much experience with the driving system should get explicit information (for example, icons or spoken text) next to non-intrusive implicit information (for example, ambient lighting).
Design ambient awareness with different colour codes to continuously inform the driver in a non-intrusive way of the current automation level.
Ambient lighting should, especially with inexperienced drivers, be accompanied by other types of information (for example visual/auditive information) on automation levels/modes.
Continuously communicate the time budgets in the current mode as well as, the upcoming driving mode.
The symbols and colours of the different driving levels shown in the time-budget widget should be consistently used throughout the ambient communication of the different driving levels (such as ambient lighting).

Keeping the driver in the loop

Partial automated driving requires the driver to continuously monitor the driving situation. Next to mode awareness a major challenge is keeping vigilance in monitoring. Vigilance deterioration i.e., driver unfitness can be caused by task overload (active fatigue), task underload (passive fatigue), sleepiness as well as distraction. In the MEDIATOR project corrective as well as preventive HMI concepts have been applied that address fatigue (task underload & sleepiness) as well as distraction.

In one Mediator evaluation simulator study a non-driving related trivia task was used as a preventive measure for fatigue due to underload during assisted driving. In this simulator study, an auditory and visual Trivia game was used to prevent task underload (passive fatigue). The invitation to play the game appeared on the infotainment display when fatigue was monitored or suspected, respectively as corrective, or preventive measure. Although, fatigue-related differences were not found between the Trivia- and non-Trivia groups, the Trivia game appeared to support maintaining situational awareness.

In three further (two on-road and one simulator) Mediator evaluation studies HMI warning concepts in case of distraction or fatigue have been tested, based on the Mediator generic transfer ritual.

If fatigue was detected, a degraded fitness message was shown on the displays and an audio alert was triggered. In the on-road trial also a cushion in the seat was inflated to give the participant a more upright position as well as a gentle vibration in the seatbelt. Warnings were escalated if needed, visually, auditory and with seatbelt vibration. A similar procedure was carried out if distraction was detected, but with a corresponding distraction message instead of a degraded fitness message. In case a suitable (higher) level of automation was available in case of distraction, the Mediator advertised the concerning driving mode. Results indicated that the suggested mode switch with alert messages somewhat reduced task-related fatigue but not sleep-related fatigue. Next to some criticism towards the specific working of HMI components, the results also indicated that the Mediator HMI, including distraction warnings, resulted in less distraction in assisted driving compared to a baseline HMI, with no distraction warnings.

A haptic seatbelt (pull) was introduced in the Mediator HMI because haptic feedback can effectively redirect visual attention to time-critical events or important information. The part of the seatbelt that touches the lower part of the torso contains a seatbelt pull force that is activated in case of an emergency/unplanned takeovers. Evaluations showed that the haptic feedback of the seatbelt is effective.

The following design recommendations have been derived for this knowledge gap.

The availability of an optional and conditional NDRT in the HMI is recommended.
Alert messages should be designed according to a generic (see Chapter 2) escalation ritual.
Alert messages should consist of multi-modal signals, such as acoustic signals, text messages, and seatbelt vibration and visual signals.
Alert messages should be accompanied by suggestions or instructions of actions to be taken by the driver.
Apply haptic seatbelt feedback in case of emergency or unplanned takeovers as well as in case of distraction. The force of the seatbelt feedback should be adjusted to the weight and length.

Negotiation conflicts

While acknowledging driver autonomy as a key element of comfort, interrelated with trust, and therefore crucial for user acceptance, it surfaces the need to negotiate disagreements between the driver and the automation system over whom should take control. Based on human negotiation styles the implementation of the negotiation ritual in the HMI encompasses an interaction flow and a force feedback mode shifter.

The negotiation routine is evoked in the standard ritual, whenever a driver does not comply with the, driving mode, that is by advocated the system. Depending on the reason for the advocated driving mode, which may be comfort driven or safety driven, respectively seductive negotiation is deployed or a somewhat more rigorous persuasive negotiation. In case of the latter, and upon driver persistence to choose a driving mode that is not advocated, the system may initiate counter measures like e.g., limited vehicle performance in acceleration or speed.

A crucial HMI hardware component in Mediator’s human machine negotiation is the Force Feedback Shifter. A conventional automatic gear shifter, in compliance with learned affordances, is expanded with Mediator’s two automated driving modes, Assisted and Piloted. Its force feedback mechanism resists mode selections that are discouraged by the system, and blocks mode selections that are not available.

The following design recommendations have been derived for this knowledge gap:

The need for, or the value of, driver autonomy is generally confirmed.
Countermeasures that reward a driver’s regular compliance with DL advertised autonomous mode like parking benefits, work better than negative countermeasures, such as reducing a vehicle’s performance in terms of speed and acceleration.
The reason for a by DL advised autonomous mode must be communicated to the driver. This contributes to the negotiation outcome and calibrates trust.

OEM design space

For industry acceptance, diversification in brand identity i.e., brand specific design of the human-product interaction, and manifestation of the HMI system (behaviour, look and feel) are crucial. In the MEDIATOR design process this means that we aimed to identify design space i.e., applicable value ranges and variation in visual, auditory, and tactile design, rather than single values. For this knowledge gap, unfortunately, the initial research plan could not be completely fulfilled because of extensive COVID-19 restrictions at the key partner. Further research is foreseen.

However, from the regular interaction, and particularly from the stakeholder workshops we suspect that design space can be facilitated as Mediator’s HMI has been built in various compositions and designs of its components, be it within the design guidelines and with respect to e.g., the designated HMI ambient light colours. The leading question was, what is required to ensure safety?

The following general design recommendations have been derived for this knowledge gap:

In the development of legislation or design recommendations, it is important to acknowledge the OEM design space, so that brand identity can be conveyed.
The transfer of control ritual must be similar across vehicles.
In case of mode awareness through ambient colours, colour coding must be similar across vehicles and brands. Colours cannot have another meaning from one vehicle to the next.
Information on autonomous driving modes in the context of mode awareness, learning and information overload, must be standardised across all brands.
It is important to acknowledge that brand experience includes, next to visual, auditory, and haptic experience, also a brand appropriate HMI behaviour through e.g., vehicle dynamics.

Conclusions

Five HMI design guidelines have been defined to form the basis of designing an HMI that establishes a safe interaction between a semi-automated driving system and a human driver. Based on the evaluations of the developed Mediator HMI concepts, main conclusions and recommendations per guideline are described.

Embrace a holistic approach

The safety of a partially automated driving system is based on the driver being continuously aware of the current driving mode and the related responsibilities of the driver, as well as the system’s capabilities and limitations. Holistically communicating driving modes should enable continuous awareness in a non-intrusive way and without needing much information-processing capacity. Holistic communication of the driving modes can, for example, be established by ambient lighting as was done in the MEDIATOR project.

Design a generic transfer ritual

Predictability of a system is essential in establishing an interaction that is trusted, reliable, and comprehensible. Therefore, within the MEDIATOR project, a standard ritual was developed based on which HMI information flows to address the different use-cases were designed. The results showed that the Mediator ritual was generally appreciated and understood by users providing a strong indication that a fixed ritual is valuable.

Design for learned affordances

Relate concepts and activities to existing knowledge and experience of people saves mental energy and therefore increases chances that people may perform these new activities safely. Within the MEDIATOR project the HMI concepts were, as much as possible, based on existing affordances, such as concrete icons, the application of the gear shifter in transferring between modes, a seatbelt pull to indicate urgency, and escalating sound patterns in case of an urgent takeover request.

Design for user acceptance

Driver autonomy is deemed crucial for user acceptance of partially autonomous vehicles. Specifically, in the case of negotiations, information on the reason for a proposed autonomous mode by the DL, is important to evoke trust and persuade the driver to follow the recommendation.

Design for industry acceptance

The validity and importance to construct legislation and guidelines such that the design space to express a brand identity is being confirmed. At the same time, it became clear that there are limitations, specifically if it comes to mode transfers and mode awareness across vehicles and across brands.

Limitations & recommendations for future research

Further research that has derived from the Mediator studies concerns:

The determination of optimised timing intervals in transfer rituals, that are dynamic because of real-time changes to its parameters and because they are likely to be subject to individual driver preferences.
While we have surpassed our intended studies on driver-automation negotiations, the interesting outcome implies that more research into the exact working of such a negotiation in the HMI requires more design concepts and more studies.
Although the need for design space with respect to automotive branding has been acknowledged, more studies as they intended before the pandemic, are still due, to determine more specifically the limits of how to balance cognitive response with brand experience.
With respect to the knowledge gap Human Driver Characteristics, the common finding across MEDIATOR studies that adaptation to individual driver preferences is important, further research will provide inside into the determination of personal preferences and implementation of an adaptive HMI.
From the common finding throughout the MEDIATOR studies is that for completely new functionalities, some form of driver education or instruction is needed, further research is needed into the design of HMIs that have instructive or education functionality.
Research into the long-term effects of Mediator like systems, through longer and larger studies, must provide knowledge for designing HMIs that can benefit from those positive, or mitigate negative effects.
The holistic approach towards HMI design, in principle, leads to more valuable research results because it allows to test and validate all elements in their realistic context. However, collecting quantitative data requires much larger studies in all their aspects e.g., sample and duration. Further larger studies are expected to validate quantitative studies thoroughly.

For further reading

The document structure is similar to the headings in this Executive Summary. HMI design guidelines for specific knowledge gaps are listed per chapter. General design guidelines are listed in the Conclusions chapter.

MEDIATOR has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 814735.