Amazon Alexa+

I joined Amazon Alexa at a perfect time, getting the opportunity to make a real impact to Alexa+

I switched from academia to industry for this exact reason - contributing to a product that helps millions of people!

Alexa has been reimagined from the ground up. Alexa+ is smarter, more capable, more personalized, and unlike chatbots, also takes action to help you get things done.

This latter point is the most critical in my opinion, as Alexa+ makes everyone's lives easier, but carrying out actions via voice is accessible and can improve many people's independence in their own home.

SPRING (EU-funded)

Robots have been introduced to public spaces like museums, airports, shopping centres, and hospitals. These are complex environments for social robots to move, see, and converse in.

Eight research institutes were part of the SPRING project to tackle these challenges in a hospital memory clinic.

I worked to improve the naturalness and accessibility of the conversations that people had with the robot (project paper).

My particular focus was on multi-party interaction. Today's systems expect to chat with one person at a time (e.g. Alexa), but patients will likely take a carer or family member along.

This work was featured in TIME Magazine, here.

Designing Conversational AI for People with Dementia

Speech production changes as dementia progresses, but today's voice assistants are trained on huge datasets. They therefore work very well for the 'average' user, but not for groups whose speech differs from the norm.

I collected interactions between people with dementia and Alexa devices to find out exactly what speech changes occur, and which changes cause the voice assistant to misunderstand. We know that mid-sentence pauses become more common and more pronounced, for example (thesis).

My focus was predominantly on how to tweak current voice assistants to make them more accessible for people with dementia, and more naturally interactive in general. I am a big advocate for voice assistant accessibility.

Voice Assistants for Visually Impaired People in the Kitchen

Malnutrition is commonly associated with sight impairment because it is very difficult to shop, prepare food, and eat a meal. I had the pleasure of supervising 30 MSc students with this setting in mind. We published two papers and created an assistant focused on:

Textual information is found all over food labels. It is therefore impossible for a blind or partially sighted person to know whether their food has expired, follow the cooking instructions, find nutritional info, or check the ingredients for allergies. We developed our system to answer questions like "Is this safe to eat?", or "Is the soup vegetarian?".

Unlike the fridge, sink, or oven - utensils and ingredients move around the kitchen and can be lost. Using the stationary objects as 'anchor points', we could give the user more specific location information "just to the left of the microwave" than other VQA systems.

Trust and explainability is critical in this domain. We therefore designed our system to be transparent and answer follow-up questions like "how sure are you about that?".

Misc Conversational AI

In addition to the large projects above, I have worked on many smaller scale projects including (not exhaustive):

Detecting when people are saying inappropriate things. Swearing does not always indicate offence (especially in Scotland), and seemingly innocuous terms like "sleep with" can be used in inappropriate sentences. We therefore trained various models to detect this in voice assistant interactions.

People ask questions more conversationally when speaking with a voice assistant. I analysed how these questions differ, and which differences cause problems for today's systems.

Helping Doctors Manage Patients

Before stepping into the world of conversational agents, I worked as a machine learning engineer - focused on information extraction from unstructured data into graphs.

In one project with the NHS in Scotland, I used NLP to ensure patients that required critical care after being discharged from the hospital were highlighted to the patient's doctor.

I worked on similar projects with the Scottish Government.

Page updated