COMS 6998:
Advanced Topics in Spoken Language Processing
Instructors: Julia Hirschberg
Time: Tu 4: 10-6:00 (Fall 2024)
Location: Mudd 1127
Prerequisite: COMS
4705 or another speech or NLP class and experience in Machine Learning
Description: This class will introduce students to spoken language
processing: basic concepts, analysis approaches, and
applications. Applications include Text-to-Speech Synthesis,
dialogue systems, and analysis of entrainment, empathy, personality, emotion,
humor and sarcasm, deception and trust, radicalization and charisma, all using
text and speech information and some visual features as well.
Required readings:
Jurafsky & Martin 2023
(3rd edition draft) chapters
These and other readings are linked from this syllabus for
each class.
Suggested:
Keith
Johnson. Acoustic & Auditory Phonetics (3rd edition). Wiley. 2011.
Resources:
A list of resources can be found here.
Office Hours
Julia Hirschberg: W 2-3:30/4pm
Lin Ai: F 2-4pm
Debasmita Bhattacharya: Th 9:30-11:30am
Harsh /srvastavaL M 9-11am
Grade Breakdown
5% attendance
20% weekly
posts
20% HW1
25% HW2
30% HW3
Also please note our late policies:
For weekly posts:
Monday deadline 11:59pm; 1 late day allowed but 1 point lost
For homework: 3 late days allowed but 5 points lost for each
late day
Academic Integrity
The SEAS academic integrity policy is found here.
The CS academic integrity policy is found here.
Syllabus
Note: Schedule and readings are
subject to change. Readings labeled with
* are optional.
Date
|
Topic
|
Readings
|
Assignments
|
Week 1: 9/3
|
Introduction
to Speech Processing (Lin Ai, Harsh Srivastava); Julia
Hirschberg
|
|
|
Week 2: 9/10
|
From
Sounds to Language
|
Jurafsky & Martin Chapter 28 (Chapters 1-3)
|
|
Week 3: 9/17
|
Acoustics
of Speech
|
Jurafsky & Martin Chapter 28 (sections 4-6)
|
|
Week 4: 9/24
|
Tools
for Speech Analysis
|
*Praat Tutorial (just use for reference)
Watch
all these Praat video tutorials here
(1-7)
*Also some video tutorials on acoustics of speech here
Download the latest version of Praat
Record your own voice saying
these sentences
Bring your laptop and headphones to class
|
HW1: Praat Recording and
Analysis (assigned)
|
Week 5: 10/1
|
Analyzing
Speech Prosody
|
ToBI Conventions
AuToBI
Prosody
and Meaning
*Guidelines
for ToBI Labeling
|
|
Week 6: 10/8
|
Text-to-Speech
Synthesis (Andrew Rosenberg)
|
Jurafsky & Martin Chapter 16 (Introduction, sections
6, 8)
*Prosody Prediction from Syntactic, Lexical, and Word
Embedding Features, *Comparing
acoustic and textual representations of previous linguistic context for
improving Text-to-Speech,
*Where do the
improvements come from in sequence-to-sequence neural tts?
The
sound of trustworthiness: Acoustic-based modulation of perceived voice
personality
|
HW1 due
|
Week 7: 10/15
|
Speech
Recognition (Bhuvana Ramadabhan)
|
Jurafsky & Martin Chapter 16 (Introduction, sections
1-5, 7-8)
Twenty-Five Years of Evolution in Speech and
Language Processing
*Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages
*Robust Speech Recognition via Large-Scale Weak
Supervision
*SLM: BRIDGE THE THIN GAP BETWEEN SPEECH AND TEXT FOUNDATION
MODELS
|
|
Week 8: 10/22
|
Spoken Dialogue Systems; Empathy in Spoken Language
(Run Chen?)
|
Jurafsky & Martin Chapters 14,
15,
27
RASwDA: Re-Aligned Switchboard Dialog Act Corpus for
Dialog Act Prediction in Conversations
*Generative
Agents: Interactive Simulacra of Human Behavior
Nora
the Empathetic Psychologist
*11
Nonverbal Ways to Express Empathy And Camaraderie With Your Team
|
HW2 assigned
|
Week 9: 10/29
|
Speech Analysis: Emotion and Sentiment Detection (Zixiaofan Yang); Emotion
Elicitation (Sara Gong)
|
Predicting
Arousal and Valence from Waveforms and Spectrograms using Deep Neural
Networks
Emotions and
Types of Emotional Responses
Eliciting
Rich Positive Emotions in Dialogue Generation
|
|
Week 10: 11/5
|
Election Day No class
|
|
|
Week 11: 11/12
|
Speech Analysis: Entrainment;
Code-Switching
(Debasmita Bhattacharya)
|
Identifying
Entrainment in Task-oriented Conversations
What
Code-Switching Strategies are Effective in Dialog Systems?
|
HW2 due
|
Week 12: 11/19
|
Speech Analysis: Personality (Michelle Levine) and
Mental State
|
Predicting
the Big 5 personality traits from digital footprints on social
media: A
meta-analysis
Multimodal
Deep Learning for Mental Disorders Prediction from Audio Speech Samples
Speech
Processing Approach for Diagnosing Dementia in an Early Stage
|
HW3 assigned
|
Week 13: 11/26
|
Multimodal
Analysis: Charismatic Speech; WordsEye (Bob Coyne)
|
What
Makes a Speaker Charismatic? Producing
and Perceiving Charismatic Speech
"Would You Buy A Car From
Me?"-- On the Likability of Telephone Voices
Extracting
Social Meaning: Identifying Interactional Style in Spoken Conversation
Multimodal
Indicators of Humor in Video
CHoRaL: Collecting
Humor Reaction Labels from Millions of Social Media Users
|
|
Week 14: 12/3
|
Speech Analysis: Deception and Trust; Intent
Detection and Radicalization (Lin Ai)
|
Acoustic-Prosodic
and Lexical Cues to Deception and Trust:
Deciphering How People Detect Lies
Multimodal
Deception Detection using Automatically Extracted Acoustic, Visual and
Lexical Features
Identifying
the Popularity and Persuasiveness of Right- and Left-leaning Group Videos on Social Media
Unveiling
the Influencers of Radical Content: A Multimodal Analysis of QAnon Videos
|
HW3 due
|