In this study, we explicitly code and study the social, referential, and pragmatic features of gaze in human-human spontaneous dyadic interaction, providing novel observations that can be executed in a machine in order to improve multimodal human-agent dialogue. Gaze is an important non-verbal social signal that contains attentional cues about where to look and provides information about others' intentions and future actions. In this work, various types of gaze behaviour are annotated in detail along with speech to explore the meaning of temporal patterns in gaze cues and their co-relations. Considering that 80% of the total stimuli perceived by the brain is visual, gaze behaviour is complex and challenging; hence, implementing human-human gaze cues to an avatar/robots could improve human-agent interaction and make it more natural.

Laughter and gaze have an important role in managing and coordinating social interactions. We investigate whether laughs performing distinct pragmatic functions, either related to humour or to a potentially social discomforting utterance or situation, are accompanied by different gaze patterns. Using a multimodal corpus of dyadic taste-testing interactions, we show that people tend to not look at their partner while producing laughs related to humour, whereas laughs that relate to potential social discomfort are accompanied by gaze at the partner. With respect to the non-laughing partner's gaze at the laugher we observe the opposite pattern around the laughter offset. We show that gaze contributes to the synchronisation and alignment of laughter production, analogously to previously reported results for speech turn-taking. Our study also provides empirical evidence to the debate about gaze aversion, opposing the view that social stress is the main explanatory factor.

Gaze is cardinal for interaction since it allows us to visually interact with the environment and understand the attention and intent of others. Gaze also plays a critical role in HRI tasks such as object recognition and manipulation, as a robot can use gaze to direct its attention to specific objects or areas of interest in its environment. In this study, we automate gaze estimation for various types of gaze behaviours (such as turn taking, joint attention, gaze following, gaze aversion and mutual attention) in a natural dyadic interaction using videos without wearable cameras or eye trackers that are implementable on a robot. There is no single dataset that covers all of the different gaze and scene combinations that is address in this paper. We propose a model that utilizes the manual annotation of gaze targets in a natural dialogue setting and generate simultaneous gaze prediction for both parties in the video, along with attention heatmaps that provide exclusive information of the target object-of-interest in the scene, by also providing out-of-scene gaze predictions. Our model performs better than the baseline methods that currently exist and the data that was generated is available for different categories of gaze

Much problem-solving research has investigated if and why ‘two heads are better than one’, but typically posits that if there is any process gain observed it is because of the exposure to the ideas provided by another person’s attempted solutions. This work fails to acknowledge or investigate what the interaction itself contributes to joint problem solving. Using an online version of the Alternative Uses Task, we compare situations in which people are passively exposed to what is said in a dialogue with situations in which people are actively engaged in the dialogue, thus varying the interactivity independently of the informational content that participants were exposed to. Interacting participants produce more turns overall, but they do not come up with more ideas. Interacting participants were also more likely to build on each other’s ideas and produce more complex ideas when a turn is linked to a previous idea; following leads to elaboration – but only if there is genuine interactivity. These results indicate that conversational mechanisms promote the exploration of a problem space and that merely counting the number of ideas produced would miss the importance of the interaction itself.

Disfluencies such as self-repairs, filled pauses such as ‘um’and silent pauses are pervasive in dialogue, but there is no con-sensus in the literature as to whether they reflect internal pro-duction pressures, or interactive issues – or how their effectsare manifest in dialogue. It is well-known that patients withschizophrenia have problems with language and social cogni-tive skills, yet little research has investigated how these impactinteraction. We report a study on the disfluency behaviours ofpatients with schizophrenia and their interlocutors who wereunaware of the patient’s diagnosis, compared to healthy con-trol groups. Results show that patients use fewer self-repairsthan either their partners or controls and fewer filled pauses(‘er’, ‘um’) than controls. Furthermore, the presence of the pa-tient also affects patients’ partners, who use fewer filled pausesthan controls and more unfilled pauses than both patients andcontrols. This suggests that smooth coordination of turns isproblematic in patients’ dialogues