COMS E6998-9: Software Security and Exploitation


Course Overview:


The course involves the study of emerging and unexplored topics in software security and the exploitation of security vulnerabilities. It begins with the foundations of secure programming. We will then examine language-specific security issues, vulnerabilities, exploitation techniques, operating system defenses, compiler defenses, and models of secure software development. Students will learn about the boundaries and effectiveness of techniques such as virtualization, stack and heap protections, address space randomization, session security, and other current approaches. Student projects will analyze advanced software exploitation techniques and countermeasures.





Students will be evaluated based on one exam, class participation, homework, and one major project.


Midterm Exam: 30% - Scheduled on April 4th

Project: 50% - click here for more details

Homework: 20%



Herbert Hugh Thompson, Ph.D. (bio)




Detailed Outline:


Lecture 1: Introduction

January 24, 2011


Introduction to software security

Understanding hackers, the underground, and Security in the Software Development Life Cycle (SDLC)

Looking at recent vulnerabilities and how they were discovered and exploited.

Thinking like an attacker and defending against them

A first look at the course project




Lecture 2: Software Security Design Principles

January 31, 2011


Thinking like an attacker

Gateway data

Security design principles




Lecture 3: Design Principles Continued and Input Validation

February 7, 2011


Security Design Principles Cont.

Input Validation

Buffer Overflows




Lecture 4: Video lecture on 3rd Party Trust


Note: Watch the video of the Friday keynote from RSA Conference here.



Lecture 5: Buffer Overflows In-Depth

February 21, 2011


Buffer overflow mechanics

Stack vs. Heap overflows


Note: Homework 1 can be found here


Lecture 6: Buffer Overflow Defenses and other Input Validation Issues

February 28, 2010


More on buffer overflow defenses

Command injection

SQL injection





Lecture 7: More Vulnerabilities; Data Security and Cryptography; Fuzzing

March 14, 2010


Cross site scripting, more command injection

Data security







Lecture 8: Fail Secure; DoS Defenses; Evaluating 3rd Party components

March 21, 2010


Fail Secure

DoS Prevention

Evaluating Components for Security




Note: Exam is on April 4th. It will cover all material discussed in class as well as the following reading list:

·         “Smashing the stack for fun and profit” by Aleph One. (originally in Phrack Vol. 7, Issue 49)

·         “Beyond Stack Smashing: Recent Advances in Exploiting Buffer Overruns” by Jonathan Pincus and Brandon Baker (IEEE Security & Privacy 2 (4): 20–27)

·         “Buffer Underruns, DEP, ASLR and improving the Exploitation Prevention Mechanisms (XPMs) on the Windows platform” by David Litchfield

·         “On the effectiveness of address-space randomization” by Hovav Shacham , Matthew Page , Ben Pfaff , Eu-Jin Goh , Nagendra Modadugu , and Dan Boneh, Proceedings of the 11th ACM conference on Computer and communications security, October 25-29, 2004, Washington DC, USA



Lecture 9: Code/Binary Analysis for Security Vulnerabilities


Note: No class on March 28th – Video lecture will be posted after the exam on April 4th.



Lecture 10: Secure Development Methodologies and Exam


Secure development methodologies


Note: Midterm exam for the second half of the class