Projects

Undergraduate and master students interested in research projects, please check out our Ads for Student Research Projects in NLP.

Current Projects

Past Projects

Current Projects

Identifying Acoustic, Prosodic, and Phonetic Cues to Individual Variation in Spoken Language

A fundamental challenge for current research on speech science and technology is understanding individual variation in spoken language. Individuals have their own speaking styles, depending on many factors, including the dialect and socioeconomic background of the speaker, as well as contextual variables such as the degree of familiarity between the speaker and hearer and the register of the speaking situation, from very casual to very formal (Eskenazi 1992). Even within the same dialect or register individual variation may occur; for example, in spontaneous speech, some speakers tend to exhibit more articulation reduction (e.g., reducing or deletion of function words) than others. In this project, we are working on identifying the acoustic-prosodic and phonetic cues that might contribute to clustering speakers based on their speaking style.

Participants: Fadi Biadsy, Julia Hirschberg, William (Yang) Wang

Extracting Paraphrase Rules from FrameNet and WordNet

FrameNet organizes lexical units into semantic frames with associated frame elements which represent the core roles of that frame. Each frame also contains annotated sentences mapping grammatical function to frame element role for the sample sentences. In our research we've extracted patterns from these annotated sentences to form paraphrase rules that cover conversives (e.g. "buy" <-> "sell") as well as other meaning-preserving verb transformations and alternations such as "The rats swarmed around the room" <-> "The room was teeming with rats.".

Participants: Bob Coyne, Owen Rambow

WordsEye: Automatic Text-to-Scene Conversion

We live in a vast sea of ever-changing text with few tools available to help us visualize its meaning. The goal of this research is to bridge the gap between graphics and language by developing new theoretical models and supporting technology to create a system that automatically converts descriptive text into rendered 3D scenes representing the meaning of that text. This builds upon previous work done with Richard Sproat in the WordsEye text-to-scene system (available online at www.wordseye.com). New research directions include the lexical semantics and knowledge acquisition needed to semi-automatically construct a new scenario-based lexical resource. This resource will be used in decoding and making explicit the oblique contextual elements common in descriptive language for the purposes of graphical depiction.

http://www1.cs.columbia.edu/~coyne/coyne-research.html

Participants: Bob Coyne, Owen Rambow, Julia Hirschberg, Gino Micelli, Cecilia Schudel, Daniel Bauer, Morgan Ulinski, Richard Sproat (OHSU), Masoud Rouhizadeh (OHSU), Yilei Yang, Sam Wiseman, Jack Crawford, Kenny Harvey, Mi Zhou, Yen-Han Lin, Margit Bowler (Reed College), Victor Soto.

Prosody of Turn-Taking in Dialogue

In conversation there are implicit rules specifying whose turn it is to talk, and conventions for switching the turn from one speaker to the other. For example, interrupting the interlocutor is a (not necessarily rude) way of grabbing the turn, while formulating a question is a way of yielding it. These rules allow dialogues to develop in a coordinated manner. The goal of this project is to study and characterize those rules and conventions, in the Columbia Games Corpus and other corpora.

Participants: Julia Hirschberg; Stefan Benus (Constantine The Philosopher University); Agustín Gravano (University of Buenos Aires), Héctor Chávez, Michael Mulley, Enrique Henestroza, Lauren Wilcox.

Speaker Entrainment in Dialogue Systems

In conversation, people entrain to their partner by adopting that partner's word choice, or by adapting aspects of their speaking style, such as speaking rate or pitch range or intensity. Such synchronization is critical to the success of human-human interactions.

While lexical entrainment has been investigated experimentally in a number of studies, other types of entrainment have received less attention. In this project, we are investigating entrainment along dimensions such as intonational contour, pitch accent, phrasing rate, pitch range, intensity, laughter, turn-taking and backchanneling behaviors.

An investigation of these behaviors will support the design of better Spoken Dialogue Systems. While entrainment has been proposed as an important method for inducing users to adopt the system's lexical items, to improve recognition accuracy, few studies have examined the importance of systems entraining to their users, to promote more successful and human-like exchanges.

Participants: Julia Hirschberg; Ani Nenkova (University of Pennsylvania); Agustín Gravano (University of Buenos Aires), Enrique Henestroza, Rivka Levitan, Adele Chase, Laura Willson, Stefan Benus (Constantine the Philosopher University), Jens Edlund (KTH), Mattias Heldner (KTH)

Past Projects

AuToBI

AuToBI is a tool for the automatic analysis of Standard American English prosody. Open source and written in Java, AuToBI hypothesizes pitch accents and phrase boundaries consistent with the ToBI prosodic annotation standard. The toolkit incluides an acoustic feature extraction frontend, and a classification backend supported by the Weka machine learning toolkit.

http://eniac.cs.qc.cuny.edu/andrew/autobi/index.html

Participants: Julia Hirschberg, Andrew Rosenberg

Deception in Speech

This project consisted in examining the feasibility of automatic detection of deception in speech, using linguistic, prosodic, and other acoustic cues. We were particularly interested in how individual differences affect the behavior of deceivers, and how such differences affect the ability of individuals to detect deception.

Our study produced the first cleanly recorded, labeled corpus of deceptive speech, the Columbia-SRI-Colorado (CSC) Corpus. Our elicitation paradigm created a context in which the subject was positively motivated to deceive an interviewer (in contrast to studies in which subjects are placed in situations where they are led to lie about potentially guilt inducing behavior). We investigated deception on two levels: we considered the speaker's overall intention to deceive (or not) with respect to particular topics, and we examined individual utterances in terms of their factual content.

Our published work produced a classification system that performs substantially better than human judges at classifying deceptive and non-deceptive utterances; a study of the use of filled pauses in deceptive speech; a method of combining classifiers using different feature sets; and a perception study showing that the personality of a listener affects his or her ability to distinguish deceptive from non-deceptive speech.

Participants: Julia Hirschberg, Frank Enos, Stefan Benus, Jennifer Venditti-Ramprashad, Sarah Friedman, Sarah Gilman, Jared Kennedy, Max Shevyakov, Wayne Thorsen, Alan Yeung, and collaborators from SRI/ICSI and from the University of Colorado at Boulder.

Emotion in Speech

The crux of this research involved characterizing acoustic and prosodic cues to human emotion, evaluating subjective judgments of human emotion, as well as exploring when and why certain emotions become confusable. We conducted on-line surveys designed to collect subjective judgments of both emotional speech as well as emotional faces. We observed that machine learning techniques applied to the prediction of human emotion given acoustic and prosodic information of the sound tokens yields a prediction rate of 75%-80%. We also found that our subjects systematically differed on how they perceived emotion in terms of valency (positive or negative affect). Furthermore, automatic emotion classification increases if we model these two groups independently of one another.

Participants: Julia Hirschberg, Jennifer Venditti-Ramprashad, Jackson Liscombe, Sarah Gilman, Daniel Vassilev, Agustín Gravano.

Detecting and Responding to Emotion in Intelligent Tutorial Systems

A tutor uses cues from the student to determine whether information has been successfully learned or not. These cues may be explicit or implicit. The first goal of this study is to examine cues to student emotions − such as frustration and uncertainty − in the context of speech-enabled intelligent tutorial systems. Such cues include lexical, prosodic, quality of voice, and contextual information. The second goal of this study is to evaluate the most appropriate strategies for responding to (negative) emotional states once they are detected. The ultimate goal is to increase the enjoyment and learning of Intelligent Tutoring Systems users.

Participants: Julia Hirschberg, Jennifer Venditti-Ramprashad, Jackson Liscombe, Jeansun Lee (Columbia University); Diane Litman, Katherine Forbes, Scott Silliman (University of Pittsburgh).

Charismatic Speech

People at an instinctual level are drawn to certain public speakers. What about it makes their speech charismatic? Our research is looking at acoustic and lexical features from public addresses to locate the source of the charisma. Though the work so far has been in American English, parallel work in Arabic may shed light on potential cultural biases in the perception of charisma.

Participants: Julia Hirschberg, Wisam Dakka, Andrew Rosenberg, Fadi Biadsy, Aron Wahl, Judd Sheinholtz, Sveltlana Stenchikova.

Speech Summarization

Speech Summarization consists of summarizing spoken data - broadcast news, telephone conversation, meetings, lectures. We are mainly focusing on summarization of broadcast news. Our speech summarization research mainly consists of three different aspects, which are i) Summarization ii) Information Extraction iii) User Interface. Summarization aspect consists of extracting significant segments of speech and concatenating them to provide a coherent summary of the given story in broadcast news. Information Extraction aspect consists of extracting named entities, headlines, interviews, different types of speakers. The last part consists of developing an user-interface that allows us to combine summary of broadcast news and other extracted information in a coherent and user-friendly speech browser.

Participants: Julia Hirschberg, Sameer Maskey, Michel Galley, Martin Jansche, Jeansun Lee, Irina Likhtina, Aaron Roth, Lauren Wilcox.

Affirmative Cue Words in Dialogue

In speech, single affirmative cue words such as okay, right and yes are often used with different functions, including acknowledgment (meaning "I believe/agree with what you said"), backchannel (indicating "I'm still here" or "I hear you and please continue"), and beginning of a new discourse segment (as in "okay, now I will talk about..."). In this project, we analyze how such functions are conveyed and perceived, and explore how they can be automatically predicted with Machine Learning algorithms.

Participants: Julia Hirschberg (Columbia University); Stefan Benus (Constantine The Philosopher University); Agustín Gravano (University of Buenos Aires), Lauren Wilcox, Héctor Chávez, Ilia Vovsha (Columbia University); Shira Mitchell (Harvard University).

Intonational Overload: Uses of the Downstepped Contours

Intonational contours are overloaded, conveying different meanings in different contexts. We are studying potential uses of the downstepped contours (especially H* !H* L- L%) in Standard American English, both in read and spontaneous speech. We are investigating speakers' use of these contours in conveying discourse topic structure and in signaling given vs. new information, and the relationship between these two functions. We designed and collected the Columbia Games Corpus especifically for this project.

Participants: Julia Hirschberg (Columbia University); Agustín Gravano (University of Buenos Aires); Gregory Ward, Elisa Sneed (Northwestern University); Stefan Benus (Constantine The Philosopher University), Ani Nenkova, Michael Mulley.

Characterizing Laughter in Dialogue

Laughter can serve many different purposes in human communication and occurs in many different forms. This project involved studying the acoustic characteristics of laughter and the functions different types of laughter may serve in human dialogue and in spoken dialogue systems.

Participants: Brianne Calandra, Rolf Carlson (KTH, Sweden).

webmaster - ecooperx[at]xcs.columbia.edu

last updated - 09.15.2011

HTML 4.01