Pre-requisites for Language Generation and Summarization
Students must have received a B or better in COMS 4705 (NLP) or equivalent. The version of NLP that you took must have covered deep learning methods for tasks in NLP. Please provide information about your background by filling out the google form found Here. The google form also includes an assignment that must be completedin order to be eligible. The assignment is due by the day after the first class (Thursday, September 7th), but we will be accepting a percentage of the class early for forms completed before August 26th. Only students who fill out the form and meet the requirements will be considered for entrance into the class. Please do note request approval by email.
Course Information
| Time | W 4:10-6:00pm |
| Place | TBD |
| Professor | Kathleen McKeown |
| Office Hours | M 4:00-5:00, 722 CEPSR W 6:00-7:00, 722 CEPSR |
| kathy@cs.columbia.edu | |
| Phone | 212-939-7118 |
Weekly TA hours (EST) are listed below. TA hours will be held in the NLP Lab unless otherwise noted.
| Thursday | Melanie Subbiah | m.subbiah@columbia.edu | Th 1:15-3:15, 724 CEPSR |
| Tuesday | Nick Deas | ndeas@cs.columbia.edu | T 2:00-4:00, 7LW1 CEPSR |
Here is where to find these rooms on the 7th floor of CEPSR.
Course Description
There has been tremendous progress recently in the development of models to generate language for different purposes – few-shot classification, document summarization, creative writing, and image captioning to name a few. This success has largely come about due to rapid advances in large language models like GPT3, ChatGPT, GPT4, PALM, T5, Alpaca and Flamingo. In this class, we will explore four main topics: language generation, multimodal generation, summarization and long-format question answering. We will study large language models that have been used for these different tasks and the issues that arise with their use. For example, how can we control the output of these large language models along different dimensions? How do we evaluate the text generated by such systems? How can we develop models to produce or summarize creative texts? What are the ethical issues surrounding these kinds of models?
The class will be held in person this fall. We will have some invited speakers, as shown on the syllabus. Typically an invited speaker will present for half of the class and may present remotely.Requirements
Students who take the class will have three main assignments: 1. For each class there will be a reading assignment consisting of several research papers. Students are responsible for reading all papers. 2. Students will be part of a presentation group which will be responsible for presenting a paper and raising critiques about one or more papers in class. Following presentation, smaller discussion groups will be formed to discuss each paper. Class participation will be graded. 3. Each student will carry out a semester-long project. This project requires submission of: a. a proposal for the project near the beginning of class; b. a midterm progress report and c. a final report and code for their project as well as a short video presentation which will be made available to the class, the TAs and the instructor for viewing.
Late submission policy: You have 4 free late days to use across the 3 assignments for the course (proposal, midterm, final). Once you have used all your days, then you will lose 7% of the points per day late unless you have demonstrated a valid medical excuse.
There will be no midterm or final exam. We will use Google Cloud for the course. Instructions for setting up the cloud can be found here.
Reading
There is no text book for the class. There will be multiple research papers assigned for each class, listed next to the class in the syllabus below. Students should read the papers before class as they will need to participate in discussion of the papers.
Syllabus
This syllabus is still subject to change. Readings may change. But it will give you a good idea of what we will cover.
| Week | Class | Topic | Speaker | Reading | Assignments |
|---|---|---|---|---|---|
| 1 | Sept 6 | Introduction and Course Overview | Plan then Generate | (Sept 7) Enrollment Form | |
| 2 | Sept 13 | Large Language Models | Melanie Subbiah (Columbia) Kathy McKeown |
GPT3 Instruction Tuning Alpaca Farm Resources for open source models |
|
| 3 | Sept 20 | Controllable Language Generation | Kathy McKeown Emily Allaway (Columbia) |
Reward Gaming in Conditional Text Generation A Recipe For Arbitrary Text Style Transfer with Large Language Models Truncation Sampling as Language Model Desmoothing Neologic |
|
| 4 | Sept 27 | Creative Language Generation | Tuhin Charkrabarty (Columbia) Paper Presentations |
ByGPT5: End-to-End Style-conditioned Poetry Generation Zero-Shot Sonnet Generataion Creative Writing with an AI-Powered Writing Assistant |
|
| 5 | Oct 4 | Explanation Generation and Evaluation | Chenhao Tan (U Chicago) Paper Presentations |
Towards a Science of Human-AI Decision Making Do Models Explain Themselves? SCOTT: Self-Consistent Chain-of-Thought Unsupervised Selective Rationalization |
Semester Project Proposal |
| 6 | Oct 11 | Multimodal Generation | Amith Ananthram (Columbia)) Paper presentations |
Hierarchical3D Adapters Ambiguity with Images Scaling Laws |
|
| 7 | Oct 18 | Deployment and Ethics | Pamela Mishkin (OpenAI) Paper Presentations |
Trails of Political Biases WinoQueer Annotators with Attitude |
|
| 8 | Oct 25 | Bias | Nick Deas (Columbia) Paper Presentations |
African American Language Bias Adding Instructions during Pretraining COBRA Frames Memorization of Conspiracy Theories |
|
| 9 | Nov 1 | Summarization Introduction and Tasks | Kathy McKeown Paper Presentations |
Text-Blueprint Conversation Benchmark Neutral multi-news |
|
| 10 | Nov 8 | Controllable Summarization and Diffusion | Jacob Andreas (MIT) Paper Presentations |
HYDRASUM Comparative Opinion Summarization Controllable Diffusion |
Midterm Project Update |
| 11 | Nov 15 | Summarization Evaluation | Alex Fabbri (SalesForce) Paper Presentations |
Multi-Dimensional Summarization Eval Factual Errors SNAC |
|
| 12 | Nov 29 | Emotion Summarization and Multilingual Summarization | Jessy Li (UT Austin) Paper Presentations |
Emotion Triggers Multilingual Summarization Multilingual Not Multicultural Code-Switch Synthesis |
|
| 13 | Dec 6 | Summarization Beyond News | Kathy McKeown Paper Presentations |
SummScreen Calibration Sets |
(Dec. 13) Final Paper |
Announcements
Check EdStem for announcements and discussion. Check courseworks for your grades (only you will see them).. All questions should be posted through EdStem instead of emailing Professor McKeown or the TAs. They will monitor the discussion lists to answer questions.
Academic Integrity
Copying or paraphrasing someone's work (code included), or permitting your own work to be copied or paraphrased, even if only in part, is not allowed, and will result in an automatic grade of 0 for the entire assignment or exam in which the copying or paraphrasing was done. Your grade should reflect your own work. If you believe you are going to have trouble completing an assignment, please talk to the instructor or TA in advance of the due date.