My research interests lie in the field of dialogue, both human-human and human-computer. I am interested in studying how spoken language can enhance human-computer communication and open doors to new types of intelligent systems that seamlessly integrate in our daily environment. I am also interested in applications of language technology for education.
While people communicate effortlessly with each other, automatic human-computer dialogue is a challenge currently addressed by many researchers. I am interested in analysing human-human dialogue and applying some of its models to human-computer spoken interaction.
My current work focuses on handling misrecognitions in spoken dialogue. Handling speech recognition errors is an important challenge for automatic spoken dialogue systems. Current state-of-the art dialogue systems ask generic clarification questions, such as "Please repeat" or "Please rephrase" when they fail to recognize a user utterance. Clarification questions are also used by speakers in human dialogue. However, in human dialogue, speakers are more likely to ask targeted clarification questions rather than explicitly signal their non-understanding. For example, if an utterance is "Please find a XXX" (XXX refers to a misheard word), an example of a targeted clarification question is "Find what?" or "What should I find?". In this project, we investigate how human speakers handle situations with missing information, how they construct clarification questions and how they answer different types of clarification questions. We use this information to construct targeted clarification questions automatically from an user utterance that contains a speech recognition error.
To be able to ask a targeted clarification question, a system has to detect which part of an utterance is misrecognized. In our work, we address the problem of localized error detection in Automatic Speech Recognition (ASR) output. Localized error detection seeks to identify which particular words in a user's utterance have been misrecognized. We use prosodic, syntactic, semantic, and speech recognizer's confidence features from an utterance and apply machine learning methods to determine which words in an utterance are misrecognized.
I am also interested in automatic dialogue generation. In my previous project (CODA), in collaboration with Paul Piwek at the Open University in the UK, I developed a system for automatic conversion of dialogues from text. Generated dialogues present the same information as original monologues. We constructed a parallel corpus of monologues and dialologues and automatically derived rules for translating monologue text into dialogue. In collaboration with Pascal Kuyten and Helmut Prendinger at NII laboratory in Japan, we work on visualizing these dialogues using virtual characters and animation. The product of this research may be applied in an education tool or gaming environment. I am currently investigating the impact of discourse annotation on quality of automatically generated dialogue.
StudentsI supervise graduate and undergaduage students in ongoing research projects:
- Eli Pincus (Masters, Applied Mathematics, Columbia University)
- Rashmi Raman (Masters, Journalism and Computer Science joint program, Columbia University)
- Alex Liu (Undergraduate, Computer Science, Columbia University)
- Ananta Padney (Undergraduate, Computer Science, Barnard Colledge)
Previously supervised students:
- Jingbo Yaung (MS student, Engineering and Applied Science, Columbia University)
- Philipp Salletmayer(MS student, Computer Science, Graz University)
In my previous research I have built automatic spoken dialogue systems, studied the effect of the system on the user's lexical and syntactic choices. I have analysed adaptation (speakers changing their behaviour) in human-human dialogues. I have studied how dialogue context can help spoken dialogue systems in speech recognitions.
Department of Computer Science
1214 Amsterdam Avenue
450 CS Building
New York, NY 10027
Email: sstoyanchev at cs.columbia.edu