CS 4995/6998: Topics in Speech Processing: Computational Approaches to Emotional Speech, Fall 2011

Time: W 6:10pm-8:00pm
Place: CSB 453 (CS Conference Room)

Professor Julia Hirschberg (Office Hours: Tu 3:30-4:30, Th 2:30-3:30)
julia@cs.columbia.edu, 212-939-7114

Teaching Assistant TBD

Announcements | Academic Integrity | Description
Readings | Resources | Requirements | Syllabus


This course introduces students to research on emotional speech and its applications in call centers, tutoring systems, and medical diagnosis. We will explore state-of-the-art work on recognizing and producing classic emotions automatically. Emotions such as anger, happiness, sadness, and uncertainty are important to recognize in online dialogue systems and to produce in computer games. We will also study the recognition of other types of speaker state, including deceptive and charismatic speech, and the uses of acoustic and prosodic information in medical domains, for the diagnosis of mental and physical conditions. Classes will be lecture and discussion with an emphasis on group participation. There are no prerequisites for the course and no exams.


Students will be expected to complete all required reading assignments before the class for which they are assigned.  They will prepare weekly discussion questions based on assigned readings and submit them by noon on the day before class.  Prepare 3 questions for each of the required papers for the week.  Each student will lead one class discussion on an optional paper of their choice.  They will also prepare a term Project either alone or in teams for presentation during the finals period assigned for the course. There will be no midterm or final exam. 

Question submission procedures are here.

Information on paper presentations is here.

See the syllabus for project proposal, interim report, and final report requirements.

NB: The Columbia Speech Lab is available for use in projects on a signup basis.


Class participation: 30%

Class presentation: 10%

Project Proposal: 10%

Interim Report: 10%

Full project submission: 30%

Project presentation: 10%

Academic Integrity

Copying or paraphrasing someone's work (code included), or permitting your own work to be copied or paraphrased, even if only in part, is not allowed, and will result in an automatic grade of 0 for the entire assignment or exam in which the copying or paraphrasing was done. Your grade should reflect your own work. If you believe you are going to have trouble completing an assignment, please talk to the instructor or TA in advance of the due date.


Required readings are available on line from links in the syllabus below.




Date Topic Readings Due Dates
Week 1 (9/7)
Introduction Emotional Synthetic Speech; The Love Detector

Week 2 (9/14)
Evidence of Emotion: Theory, Experiments and Corpora
Cornelius00; Schereretal 03

*Wiltingetal06; *Enos&Hirschberg06

Week 3 (9/21)
Cues to Emotional Language; Introduction to Acoustic/Prosodic Features;Introduction to Praat Liscombeetal05; *Kumaretal06;

Praat tutorial; Acoustics of Speech; Speech sounds

Download Praat to your laptop and bring headphones if you have them
Week 4 (9/28) Techniques for Emotion Classification
Cowie00; Feeltrace; Whissel's Dictionary of Affect; *Satohetal07; *Nisimuraelal06;

Project Ideas due
Week 5 (10/5)
Emotion in IVR Systems:  Anger and Frustration

Devillers&Vidrascu06; Angetal02; *Laukkaetal11 (Wilkey); *Gupta&Rajput07

Week 6 (10/12)
Emotion in Tutoring Systems: Confidence and Confusion
Aietal06; Liscombeetal05b
Project Proposal due
Week 7 (10/19)
Emotion in Meetings:  Hot Spots and Laughter
Wrede&Shriberg03; Laskowski&Burger07 (Graham); *Jurafskyetal09 (Willson)

Week 8 (10/26)
Producing Emotional Speech
Cahn90; Eideetal04;
*Beskow&Nordenberg06 (Sabatini); *Yanushevskayaetal05 (Kuo);
cf Synface demo, Mary TTS, JHU workshops

Week 9 (11/2)
Cultural Differences
Ekman&Friesen03 (Amde); *Shahidetal07 (Park)
Interim Report due
Week 10 (11/9) Lexical Cues to Emotion: Sentiment Analysis Hatzivassiloglouetal97(Satsangi); Pangetal02;*Turney02; *Cohnetal04; Agarwaletal11(Kripa); DAL  
Week 11 (11/16)
Non-Linguistic Cues to Emotion: Face, Body, Brain
Johnstoneetal06 (Kisch); Ekman video;
Ekman bio; cf FACS

Week 12 (Tue, 11/22)
Deception Detection
Hirschbergetal05; *Eriksson&Lacerda07 (Cooper); YouDecide;*Hancocketal08 (Cabral); DePaulo03

Week 13 (11/30)
Emotion Detection in Medical Diagnosis
Hirschbergetal10; Levitetal01; Ramirez-Esparzaetal08; *Kalioubyetal06; *Rudeetal04;  *Snowdenetal; *Luoetal06 (Archibong)

Week 14 (12/7)
Modeling Other Speaker States:  Sarcasm, Charisma; "Good news, bad news"
Teppermanetal06 (Shin); Davidovetal10 (Gainer); *Biadsyetal08 (Ulinski); Paralinguistic Challenge *2009, *2010, 2011; Swerts&Hirschberg10

Finals week (12/21)

Projects due

Project presentations



Julia Hirshberg Portrait

Julia Hirschberg
Professor, Computer Science

Columbia University
Department of Computer Science
1214 Amsterdam Avenue
M/C 0401
450 CS Building
New York, NY 10027

email: julia@cs.columbia.edu
phone: (212) 939-7114

Download CV

Columbia University Department of Computer Science / Fu Foundation School of Engineering & Applied Science
450 Computer Science Building / 1214 Amsterdam Avenue, Mailcode: 0401 / New York, New York 10027-7003
Tel: 1.212.939.7000 / Fax: 1.212.666.0140