Sound and vision: how can auditory displays support supervision of automated driving?

Higher levels of vehicle automation allow for less constant attention on the driving task, enabling drivers to engage in non-driving related activities (NDRAs). Shifts between levels of automation require self-regulation of NDRA engagement to remain sufficiently attentive for future changes in automation level. A visual HMI can provide information about current and upcoming changes in automation level but requires visual attention, compromising the convenience of automation. Auditory displays have been used to provide continuous information during monitoring tasks, but research into how sound can aid automation supervision in the driving context is limited. This online video study explored how auditory displays can augment a visual HMI of a self-driving vehicle while participants were engaged in an NDRA, comparing sounds that convey information about current and (time to) future system states through changes in volume, inter-pulse interval, harmonic series, and pitch. Adding sound improved perceived direction of change and remaining time until changes in automation level. Despite best efforts in sound design, sounds in the vehicle were initially perceived as negative even when they indicated something positive such as an upcoming increase in automation level. This and other implications for HMI research and design are discussed further.