Understanding communication behaviour to provide intent-based personalisation

The launch of our new premium hearing aid Oticon Intent™ marked a significant step forward in understanding the user’s listening needs and communication intentions in order to seamlessly provide the brain with individualised hearing support in any situation.

Our journey to develop this groundbreaking sensor-driven BrainHearing™ technology started when we asked ourselves a key question: How can we understand the user's listening intentions, and use this understanding to provide individualised hearing support to people with hearing loss?

We wanted to know: What are the key features in human communication that reveal listening needs and intentions? How do people behave in real life when communicating becomes challenging? Do people adopt some strategies that help them cope with environmental complexity? And how can a hearing aid pick up on all this and provide personalised support to the user in any situation?

All these questions led us to a much deeper understanding of how head and body movements, together with information on the surrounding environment, are key to capturing listening intentions, as well as revealing communication challenges and coping strategies (for a recent review, see Higgins et al., 2023). By considering cues such as body and head movements and acoustic surroundings, we can understand how the user is engaging with the world. These were the foundational insights that led to the development of the world’s first hearing aid with user-intent sensors: Oticon Intent.

This work also brought us closer to some of the key researchers within communication behaviour, such as Dr Lauren V. Hadley. This is why we invited Dr Hadley to the Oticon headquarters to talk about “Communication behaviour of older adults during face-to-face conversations,” as part of the BrainHearing™ Network webinar series.

Figure 1 Steps for verbal communication as defined by Kiessling et al. (2003): Step 1: Hearing, Step 2: Listening, Step 3: Comprehending, and Step 4: Communicating.

A dive into Dr Hadley’s talk “Communication behaviour of older adults during face-to-face conversations”

In this webinar, with attendees from all over the world, Dr Hadley shared her advances in conversation research. In particular, Dr Hadley addressed the behaviours that people show when struggling in a conversation, the meaning of those behaviours, and the cognitive mechanisms that may underlie them.

Conversation and behaviour: strategies for when communication gets challenging

“Understanding what makes a conversation challenging depends on how we define a successful conversation,” explained Dr Hadley. What are the factors that define a successful conversation? In a recent study from Dr Hadley’s lab (Nicoras et al., 2023), older adults with and without a hearing loss were asked to identify what factors are important to make a conversation successful. Seven factors were identified. The three most important factors were the following:

1. Being able to listen easily
2. Being spoken to in a helpful way
3. Being engaged and accepted

In other words, in terms of listening experience, what was particularly relevant for older people was to feel engaged in the conversation without having to strain to hear. A challenging conversation can occur when any of the factors of success are not fulfilled, for example, in the presence of a noisy environment or a listener limitation (e.g., hearing loss).

Previous studies have identified that people with normal hearing can use different strategies to help themselves or their conversation partner when the conversation is challenging (see Table 1 for a summary).

Table 1 Strategies that can be used when the conversation is challenging. ¹Zollinger & Brumm, 2011; ²Smiljanic and Bradlow, 2009; ³Hazan & Pettinato, 2014; ⁴Grange et al., 2018; ⁵Munhall et al., 2004; ⁶Grant and Seitz, 2000

However, these strategies were predominantly reported in studies where the talker and the listener were studied in isolation. “How do these behaviours occur and interact in real face-to-face communication?”  wondered Dr Hadley while proceeding to present her studies on conversation behaviour (Hadley et al., 2019, 2021). In these studies, Dr Hadley asked older adults to hold a conversation while sitting in groups of two or three people in the lab and while wearing a head-movement crown and eye trackers. The background noise level was systematically adjusted to see the effect of noise on people's behaviour. Dr Hadley found that as the noise level increased, people used a variety of coping strategies, but only to a small extent and not enough to fully compensate for the additional challenge.

In particular, she found that as the noise increased, participants:

• spoke louder
• talked for less time (shorter utterances)
• moved their head closer by a couple of centimetres
• looked increasingly more at their partner’s mouth and less at their partner’s eyes
• but did not use head orientation optimally, i.e.:
  • participants remained oriented straight toward the other conversation partner in a two-person conversation
  • participants pointed their heads approximately in the middle of the other two conversation partners in a three-person conversation, likely due to social considerations

Overall, participants seemed to use beneficial strategies, but not optimally. “It seems that participants might be missing out on some strategies that could help them,” concluded Dr Hadley.

Effects of hearing loss

Dr Hadley explained that people with a hearing loss have often been found to either withdraw from a conversation (Jaworski & Stephens, 1998) or to dominate the conversation (reducing the need to listen; Sørensen et al., 2020). Additionally, turn-taking is less smooth and more variable when the conversational partners have a hearing loss (Petersen et al., 2022).

In an ongoing study, Dr Hadley and colleagues have been evaluating the behaviour and dynamics of four people having a conversation, where two participants had normal hearing and two participants had a hearing loss. Early analyses indicate that participants with hearing loss speak for less time, move closer to the talkers, and spend more time oriented directly toward the talker - hence, they do not use head orientation optimally. As these behaviours are consistent in response to both noise and hearing loss, Dr Hadley wondered, “Could these behaviours play a role as a social cue to others?”, besides being beneficial to increase audibility. Indeed, if the conversation partners can identify these behaviours as a sign of difficulty, they could provide better support and more accommodation.

In a follow-up study, Dr Hadley's student Raluca Nicoras realised that observers were remarkably able to identify which participants had a hearing loss in a group conversation, particularly when the participants were unaided. Dr Hadley concluded that “hearing loss appears to lead to observable changes in behaviour that others can pick up on and use to better accommodate.”

Conversation and cognition

When having a conversation, people need to listen (see Figure 1, Step 2), comprehend (Figure 1, Step 3), prepare a response, and produce a response (Figure 1, Step 4). Since turn-taking in a conversation (the pause between one person stopping their talking and the next person responding) is very quick, people must be predicting to some extent what the other conversation partner is going to say and when they are going to finish. Dr Hadley wondered, “Is this sort of prediction affected by noise level or hearing loss and can we measure it in a conversation?” Hadley and Culling (2022) found that increasing the noise level led to more variability and uncertainty in predicting a turn in a conversation. Additionally, people with hearing loss report more difficulty catching the start of a conversational turn and were slower to predict the end of sentences, as found in Hadley’s recent eye-tracking study, possibly due to reduced availability of cognitive resources. This reduced prediction can make it harder for people with a hearing loss to follow and jump into conversations.

Clinical implications

People with hearing loss use some beneficial strategies when facing a challenging conversation, though not to their full potential, and also show some behaviours that may be suboptimal. In particular, they tend to face the current talker rather than using head orientation optimally to improve audibility (“Turn an ear to hear”, Grange et al., 2018). Advice from hearing care professionals to explore head orientations when the listening situation gets challenging could provide an audibility benefit. However, body language and other indications of difficulty following a conversation could in fact help the conversation partner to subtly and spontaneously accommodate the person with a hearing loss.

Hearing loss affects not only audibility, but also the availability of beneficial cognitive processes. This may be due to the loss itself or to the additional effort and cognitive resources expended on listening. Dr Hadley emphasised that “finding ways to help listeners stay on the front foot and potentially use prediction effectively may lead to benefits in social engagement.”

Key takeaways

Here are some key takeaways from Dr Hadley’s talk:

• When the listening situation gets challenging, people can adopt different strategies to improve their listening experience. However, not all strategies are exploited optimally in face-to-face conversation.
• Hearing loss leads to observable changes in behaviour that others can pick up on and use to better accommodate the person with hearing loss. ‘Appearing impaired’ can provide an accommodation benefit during a conversation.
• Hearing loss affects not only audibility, but also the availability of other beneficial cognitive processes.

References

Grange, J. et al. (2018). Turn an ear to hear: How hearing-impaired listeners can exploit head orientation to enhance their speech intelligibility in noisy social settings. In Proceedings of the International Symposium on Auditory and Audiological Research 9 - 16.

Grant, K. W., & Seitz, P. F. (2000). The use of visible speech cues for improving auditory detection of spoken sentences. The Journal of the Acoustical Society of America, 108(3), 1197-1208.

Hadley, L. V., Brimijoin, W. O., & Whitmer, W. M. (2019). Speech, movement, and gaze behaviours during dyadic conversation in noise. Scientific reports, 9(1), 1-8.

Hadley, L. V., Whitmer, W. M., Brimijoin, W. O., & Naylor, G. (2021). Conversation in small groups: Speaking and listening strategies depend on the complexities of the environment and group. Psychonomic Bulletin & Review, 28(2), 632-640.

Hadley, L. V., & Culling, J. F. (2022). Timing of head turns to upcoming talkers in triadic conversation: Evidence for prediction of turn ends and interruptions. Frontiers in Psychology, 13, 1061582.

Hazan, V., & Pettinato, M. (2014). The emergence of rhythmic strategies for clarifying speech: variation of syllable rate and pausing in adults, children and teenagers. In Proceedings of the 10th international seminar on speech production (pp. 178-181).

Jaworski, A., & Stephens, D. (1998). Self‐reports on silence as a face‐saving strategy by people with hearing impairment. International Journal of Applied Linguistics, 8(1), 61-80.

Munhall, K. G., Jones, J. A., Callan, D. E., Kuratate, T., & Vatikiotis-Bateson, E. (2004). Visual prosody and speech intelligibility: Head movement improves auditory speech perception. Psychological science, 15(2), 133-137.

Nicoras, R., Gotowiec, S., Hadley, L. V., Smeds, K., & Naylor, G. (2023). Conversation success in one-to-one and group conversation: a group concept mapping study of adults with normal and impaired hearing. International Journal of Audiology, 62(9), 868-876.

Petersen, E. B., MacDonald, E. N., & Josefine Munch Sørensen, A. (2022). The effects of hearing-aid amplification and noise on conversational dynamics between normal-hearing and hearing-impaired talkers. Trends in Hearing, 26, 23312165221103340.

Sørensen, A. J. M., MacDonald, E. N., & Lunner, T. (2020). Timing of turn taking between normal-hearing and hearing impaired interlocutors. Proceedings of the International Symposium on Auditory and Audiological Research, 7(August), 37 - 44.

Smiljanić, R., & Bradlow, A. R. (2009). Speaking and hearing clearly: Talker and listener factors in speaking style changes. Language and linguistics compass, 3(1), 236-264.

Zollinger, S. A., & Brumm, H. (2011). The Lombard effect. Current Biology, 21(16), R614-R615.