818 research outputs found
Incremental multi-party conversational AI for people with dementia
No full textSpoken dialogue systems (SDSs, e.g. Siri and Alexa) are trained on huge corpora, helping
them accurately understand the ‘average’ user. Speech production is nuanced, however,
so some user groups fall outside the ‘average’. This thesis focuses on SDSs for people
with dementia (PwD). More naturally interactive and accessible SDSs can improve people’s autonomy at home, and in public spaces. Three challenges are tackled in this thesis,
ethical data collection, incrementality, and multi-party conversations (MPCs).
Part I details the motivations of this work, in the context of voice assistant accessibility,
with a specific focus on language technologies for people with dementia. The thesis is
then introduced in its entirety through published paper summaries, with a structure guide.
Part II focuses on data collection. An ethical framework is presented to ensure all data is
collected ethically. A data capture device is then presented to address novel challenges
introduced by COVID-19. Using the ethical framework and device, the DEICTIC corpus
was collected. It verified that, when talking to an SDS, PwD pause significantly more
often, and for significantly longer durations, than people without dementia. The corpus
also reveals that 28% of PwD’s interactions with an SDS are MPCs involving their partner. SDSs are not adapted for MPCs, so a second data collection was designed. Hospital
staff subsequently used this design with memory clinic patients and their companions.
Part III focuses on incrementality. Microsoft’s incremental speech recognition is the most
responsive, stable, accurate, and the only one that preserves disfluent material. IBM’s
services were the most suitable for MPCs. Two corpora were created and released to
explore incremental semantic parsing, together containing over 105,000 interrupted utterances paired with their underspecified meaning representation. SDSs interrupt users
if they pause too long mid-utterance, requiring them to frustratingly repeat themselves.
The use of incremental clarification requests (iCRs, e.g. “author of what?”) leads to more
naturally interactive SDSs, and improves their accessibility for PwD. Another new corpus
was created and released, containing 3,000 human elicited clarification requests. It was
used to show that some large language models (LLMs) can generate context-appropriate
iCRs, and can interpret clarification exchanges as if they were one uninterrupted turn.
Part IV tackles MPCs. The hospital corpus showed that MPCs elicit unique, complex behaviours. LLMs performed remarkably at the new task of multi-party goal tracking, when
given examples from the corpus. A multi-party SDS is required for further research, so
all the work presented in this thesis was integrated into one system, embodied by an ARI robot. It has been designed to handle MPCs with memory clinic patients and their companions, and is designed to be accessible for PwD. When PwD pause mid-utterance, the
system generates an appropriate iCR, and interprets the resulting clarification exchange.
In summary, this thesis identifies that PwD pause significantly more often, and for significantly longer durations, than people without dementia. Additionally, these interactions
are often multi-party. When mid-utterance pauses occur, interactions can be recovered
through the use of iCRs. Using the SLUICE-CR corpus, LLMs can generate effective
and human-like iCRs. They can also be used to interpret clarification exchanges, and interpret multi-party interactions. This work was integrated and deployed on a social robot
to enable conversations between the robot, memory clinic patients, and their companions
Autumn leaves : sound and the environment in artistic practice
No full textThis publication is a book that represents an innovative, international and multi-disciplinary approach to conceptualising the dynamic relationships between sound and the environment. The editorial process involved directly commissioning textual, graphic and photographic work. The vast majority of the book represents new work, produced specifically for this publication. For the purposes of tracing historical development, an article from 1974 and three older projects have been revived and recontextualised. In addition to the editorial responsibility, the researcher wrote the introduction and conducted three original interviews. The book draws work from visual, sound and performance art, acoustic science, anthropology, cultural studies, public policy, and architectural theory. Just as it is true to say that these disciplines have not previously been brought together in this way, equally, it is no exaggeration to identify the contributors as the leading international lights in the field: Chris Watson, Tim Ingold, Hildegard Westerkamp, Christina Kubisch, Alvin Lucier, David Toop. The book is published by Double Entendre, the French publisher of the premier sound arts journal, Vibro. The book is accompanied by an audio compilation published by the German record label, Gruenrekorder (Gruen 053). www.autumn-leaves.gruenrekorder.de. The researcher co-curated the compilation, selecting relevant work that illustrated the book’s themes. The book was the catalyst for a one-day symposium at the Tate Britain called The Performance of Sound (May 19th, 2006), which the researcher co-organised. The researcher was invited to speak on the book at the Audio Extranautes: Flux, Distance, Sociability symposium at the Villa Arson in Nice in December 2007. Autumn Leaves has been reviewed in the French journal Mouvement; in MCD where the reviewer reported that “this book deserves to be translated into French”; and Soundscape: The Journal of Acoustic Ecology. Soundscape 7 (1), Autumn, 2007 reprinted an interview conducted by the author from the book. Autumn Leaves, edited by CRiSAP co-director Angus Carlyle, seeks to draw together a number of different perspectives on how the environment is made audible through sound. The perspectives contained in the book are made manifest through more traditional textual analyses, interviews, image-based works (both photography and graphic illustration) and ‘artist’s pages’ (which combine different registers of information).
Among the articles included in the book are a superb deconstruction of the concept of soundscape by anthropologist Tim Ingold; an intriguing analysis of sound from an acoustic point-of-view (or point-of-audition) by Bill Davies; Steve Goodman’s dynamic opening up of city sound to a bass materialism provoked by Greg Lynn’s ‘blob’ architecture; Salome Voegelin’s evocative mapping of sci-fi aesthetics onto the project of acoustic ecology; a wonderful meditation on the heard and the unheard by David Toop; Sylvain Marquis powerfully drawing out the ‘presence’ of Phill Niblock; Rahma Khazam finding new ways of listening through an inspired conceptual conversation between art, architecture and relational aesthetics; and a re-print of Hildegard Westerkamp’s pioneering discussion of soundwalking from 1974.
Interviews include a wide-ranging discussion with Alvin Lucier about his work and working practices; an exploration of Christina Kubisch’s long-standing commitment to teasing out the complexities of the sounds that surround us; Peter Cusack providing an exciting account of his Sound of Dangerous Places project; Chris Watson talking us through his inspirational field-recording; and Max Dixon offering fresh perspectives on how the development of strategies for noise in urban environments meshes policy with research into bio-acoustics, acoustics and creative practice.
Images include Dan Holdsworth’s haunting representations of anechoic chambers through Charles Fox’s photographs of microphone arrays in the wilderness, Axel Stockburger’s ASCII art evocations of video-game space and Nicholas Gansterer’s intricate diagrams of our heard world.
What remains of the book is devoted to the artists’ pages. In these a whole host of contemporary practitioners spanning the disciplines of graphic design, music, photography, performance and visual art offer their provocative takes on sound and the environment. Here we encounter John Wynne and Tim Wainwright presenting their collaborative work in Harefield Hospital; Aki Onda pursuing his Cinemage project; Claudia Wegener finding poetry in ear- and eye-witnessing; an unpacking of the theories and technologies behind the exciting Locus Sonus audio streams; NYSAE opening up its portfolio of acoustic ecology-inspired activities; Goran Vejvoda mobilising a modular manifesto from his three decades of sound art; the Gruenrekorder label reviewing the thinking behind its 40 releases; Jem Finer show-casing his Score For A Hole in the Ground; Cathy Lane mapping her memories of the Hebrides; Zoe Irvine making an art of places out of abandoned audio tape; and Mira Choi introducing her noise-responsive graphic software.
The editorial work and its presentation has been a collaborative venture with the designer Ian Noble.
Autumn Leaves is CRiSAP's first book and is edited by CRiSAP Co-Director Angus Carlyle[/b] and published by the exciting French sound art initiative Vibro / Double Entendre. It contains work by a variety of artists including several of CRiSAP's members - Salomé Voegelin, John Wynne, Peter Cusack, Cathy Lane and David Toop
Grounding LLMs to In-prompt Instructions: Reducing Hallucinations Caused by Static Pre-training Knowledge
No full textWhen deploying LLMs in certain commercial or research settings, domain specific knowledge must be explicitly provided within the prompt. This in-prompt knowledge can conflict with an LLM’s static world knowledge learned at pre-training, causing model hallucination (see examples in Table 1). In safety-critical settings, like healthcare and finance, these hallucinations can harm vulnerable users. We have curated a QA corpus containing information that LLMs could not have seen at pre-training. Using our corpus, we have probed various LLMs, manipulating both the prompt and the knowledge representation. We have found that our ‘Jodie’ prompt consistently improves the model’s textual grounding to the given knowledge, and in-turn the overall answer accuracy. This is true in both the healthcare and finance domains – improving accuracy by up to 28% (mean: 12%). We have also identified that hierarchical and direct node-property graph structures could lead to more interpretable and controllable systems that provide a natural language interface with real-time in-domain knowledge. Our corpus will enable further work on this critical challenge
Grounding LLMs to In-prompt Instructions: Reducing Hallucinations Caused by Static Pre-training Knowledge
No full textWhen deploying LLMs in certain commercial or research settings, domain specific knowledge must be explicitly provided within the prompt. This in-prompt knowledge can conflict with an LLM’s static world knowledge learned at pre-training, causing model hallucination (see examples in Table 1). In safety-critical settings, like healthcare and finance, these hallucinations can harm vulnerable users. We have curated a QA corpus containing information that LLMs could not have seen at pre-training. Using our corpus, we have probed various LLMs, manipulating both the prompt and the knowledge representation. We have found that our ‘Jodie’ prompt consistently improves the model’s textual grounding to the given knowledge, and in-turn the overall answer accuracy. This is true in both the healthcare and finance domains – improving accuracy by up to 28% (mean: 12%). We have also identified that hierarchical and direct node-property graph structures could lead to more interpretable and controllable systems that provide a natural language interface with real-time in-domain knowledge. Our corpus will enable further work on this critical challenge
You have interrupted me again!: making voice assistants more dementia-friendly with incremental clarification
No full textIn spontaneous conversation, speakers seldom have a full plan of what they are going to say in advance: they need to conceptualise and plan incrementally as they articulate each word in turn. This often leads to long pauses mid-utterance. Listeners either wait out the pause, offer a possible completion, or respond with an incremental clarification request (iCR), intended to recover the rest of the truncated turn. The ability to generate iCRs in response to pauses is therefore important in building natural and robust everyday voice assistants (EVA) such as Amazon Alexa. This becomes crucial with people with dementia (PwDs) as a target user group since they are known to pause longer and more frequently, with current state-of-the-art EVAs interrupting them prematurely, leading to frustration and breakdown of the interaction. In this article, we first use two existing corpora of truncated utterances to establish the generation of clarification requests as an effective strategy for recovering from interruptions. We then proceed to report on, analyse, and release SLUICE-CR: a new corpus of 3,000 crowdsourced, human-produced iCRs, the first of its kind. We use this corpus to probe the incremental processing capability of a number of state-of-the-art large language models (LLMs) by evaluating (1) the quality of the model's generated iCRs in response to incomplete questions and (2) the ability of the said LLMs to respond correctly after the users response to the generated iCR. For (1), our experiments show that the ability to generate contextually appropriate iCRs only emerges at larger LLM sizes and only when prompted with example iCRs from our corpus. For (2), our results are in line with (1), that is, that larger LLMs interpret incremental clarificational exchanges more effectively. Overall, our results indicate that autoregressive language models (LMs) are, in principle, able to both understand and generate language incrementally and that LLMs can be configured to handle speech phenomena more commonly produced by PwDs, mitigating frustration with today's EVAs by improving their accessibility
Building for speech: designing the next-generation of social robots for audio interaction
No full textThere have been significant advances in robotics, conversational AI, and spoken dialogue systems (SDSs) over the past few years, but we still do not find social robots in public spaces such as train stations, shopping malls, or hospital waiting rooms. In this paper, we argue that early-stage collaboration between robot designers and SDS researchers is crucial for creating social robots that can legitimately be used in real-world environments. We draw from our experiences running experiments with social robots, and the surrounding literature, to highlight recurring issues. Robots need better speakers, a greater number of high-quality microphones, quieter motors, and quieter fans to enable human-robot spoken interaction in the wild. If a robot was designed to meet these requirements, researchers could create SDSs that are more accessible, and able to handle multi-party conversations in populated environments. Robust robot joints are also needed to limit potential harm to older adults and other more vulnerable groups. We suggest practical steps towards future real-world deployments of conversational AI systems for human-robot interaction
Understanding Disrupted Sentences Using Underspecified Abstract Meaning Representation
No full textVoice assistant accessibility is generally overlooked as today's spoken dialogue systems are trained on huge corpora to help them understand the 'average' user. This raises frustrating barriers for certain user groups as their speech shifts from the average. People with dementia pause more frequently mid-sentence for example, and people with hearing impairments may mispronounce words learned post-diagnosis. We explore whether semantic parsing can improve accessibility for people with nonstandard speech, and consequently become more robust to external disruptions like dogs barking, sirens passing, or doors slamming mid-utterance. We generate corpora of disrupted sentences paired with their underspecified Abstract Meaning Representation (AMR) graphs, and use these to train pipelines to understand and repair disruptions. Our best disruption recovery pipeline lost only 1.6% graph similarity f-score when compared to a model given the full original sentence.</p
Understanding and Answering Incomplete Questions
No full textVoice assistants interrupt people when they pause mid-question, a frustrating interaction that requires the full repetition of the entire question again. This impacts all users, but particularly people with cognitive impairments. In human-human conversation, these situations are recovered naturally as people understand the words that were uttered. In this paper we build answer pipelines which parse incomplete questions and repair them following human recovery strategies. We evaluated these pipelines on our new corpus, SLUICE. It contains 21,000 interrupted questions, from LC-QuAD 2.0 and QALD-9-plus, paired with their underspecified SPARQL queries. Compared to a system that is given the full question, our best partial understanding pipeline answered only 0.77% fewer questions. Results show that our pipeline correctly identifies what information is required to provide an answer but is not yet provided by the incomplete question. It also accurately identifies where that missing information belongs in the semantic structure of the question
You have interrupted me again!: making voice assistants more dementia-friendly with incremental clarification
No full textIn spontaneous conversation, speakers seldom have a full plan of what they are going to say in advance: they need to conceptualise and plan incrementally as they articulate each word in turn. This often leads to long pauses mid-utterance. Listeners either wait out the pause, offer a possible completion, or respond with an incremental clarification request (iCR), intended to recover the rest of the truncated turn. The ability to generate iCRs in response to pauses is therefore important in building natural and robust everyday voice assistants (EVA) such as Amazon Alexa. This becomes crucial with people with dementia (PwDs) as a target user group since they are known to pause longer and more frequently, with current state-of-the-art EVAs interrupting them prematurely, leading to frustration and breakdown of the interaction. In this article, we first use two existing corpora of truncated utterances to establish the generation of clarification requests as an effective strategy for recovering from interruptions. We then proceed to report on, analyse, and release SLUICE-CR: a new corpus of 3,000 crowdsourced, human-produced iCRs, the first of its kind. We use this corpus to probe the incremental processing capability of a number of state-of-the-art large language models (LLMs) by evaluating (1) the quality of the model's generated iCRs in response to incomplete questions and (2) the ability of the said LLMs to respond correctly after the users response to the generated iCR. For (1), our experiments show that the ability to generate contextually appropriate iCRs only emerges at larger LLM sizes and only when prompted with example iCRs from our corpus. For (2), our results are in line with (1), that is, that larger LLMs interpret incremental clarificational exchanges more effectively. Overall, our results indicate that autoregressive language models (LMs) are, in principle, able to both understand and generate language incrementally and that LLMs can be configured to handle speech phenomena more commonly produced by PwDs, mitigating frustration with today's EVAs by improving their accessibility
The forgotten first: John MacCormick's 'Dùn-Àluinn'
No full textThe first Gaelic novel, John MacCormick's Dùn-Àluinn, no an t-Oighre 'na Dhìobarach, was serialised in the People's Journal in 1910 before being published in its entirety in 1912. Within a year of the publication of Dùn-Àluinn as a novel the second Gaelic novel, Angus Robertson's An t-Ogha Mòr, appeared in print, underlining the renaissance which Gaelic literature was experiencing. Both novels, while remarked upon by contemporaries and by general studies of Gaelic literature, have been all but ignored to date, with no criticism or analysis of either having been published. The main aim of this article is to offer some general comments about MacCormick's Dùn-Àluinn and thus to open up both the novel and indeed other early twentieth-century Gaelic writers and their work to further scrutiny. Consideration will be given to the author himself, the contemporary Gaelic literary scene and finally some of the more interesting aspects of the novel itself
- …
