Lecture 0: Introduction

COMS E6998 Formal Verification of System Software
Fall 2018
Ronghui Gu

1 Formal Verification

Goal: Prove the absence of program errors in a system

1.1 Why should we care?

Software systems run everywhere and even control our lives
We have to trust them but they are not trustworthy due to program errors
- Toyota’s killer firmware (stack overflow in the assembly module)
- Software Errors Cost Economy $312 Billion Annually. – Cambridge[2013]
- $600M lost in 12/2017
- TheDao (decentralized autonomous organization) attack (double-spend attack), 3.6M ETH (around $700M) lost
```
void withdraw(account) {
    uint balance = bank.balance[account];
    if (balance <= 0) {
      return;
    }
    bank.fund -= balance;
    receiverHandlePayment (account, balance);
    bank.balance[account] = 0;
}
```

1.2 Challenges

Goal: prove (or verify) the absence of program errors
Easy? NO!
- Finite code often has an infinite set of behaviors:
  sort (int* a)
  Infinite number of a-inputs and outputs
- What does it even mean for sort to be correct?
A number of ways to approach the problem

2. Methods to Approach the Goal

2.1 Testing?

Can we use testing to prove correctness/incorrectness?
No! Need an infinite test set

Program testing can be used to show the presence of bugs, but never to show their absence. – Edsger Dijkstra

2.2 An alternative approach: formal verification …

Formal verification is an act of proving/disproving the correctness of systems HW/SW.
Correctness: means with respect to the specification/intention
Techniques for improving safety and security of computer systems
- Applicable to both software and hardware
- We focus on software

2.2.1 Algorithmic Verification?

Can we come up with an algorithm to prove correctness/incorrectness for any program automatically?
Create a program av that inputs another program p and after finite time outputs false if p has a bug for some input, or true otherwise.

  bool av (program p) {
    if (p has bugs) {
      return false;
    }
    return true;
  }

NO! Because of the halting problem [Alan Turing 1936]: if av exists then there is a paradox; thus av can’t exist.
Proof.
Let “bug”=infinite loop=infloop(). If p is a program then let 'p(p) be the program that when run, it will execute p on input p. Assume av exists. Create:
paradox(p:Prog) = if av(’p(p)) then infloop() else true
What does paradox(paradox) do?
1. If paradox(paradox) loops forever then av(’paradox(paradox))=false, thus paradox(paradox) returns true. Contradiction
2. If paradox(paradox) returns true, i.e., av(’paradox(paradox))=true, thus paradox(paradox) loops forever. Contradiction
A non-constructive proof

3n+1 conjecture:

  void collatz (int i) {
    while (i > 1) {
      if (i is even)
        i = i / 2;
      else
        i = 3 * i + 1;
    }
  }

2.2.2 Deductive Verification?

Can we have a mathematical proof system to prove correctness/incorrectness for all programs?
Can we create a system L of logical axioms and rules, such that for any program p we can prove either
- p has a bug for some input
- p has no bug for any input
NO! If such a system L exists then we can create a fully automatic verification algorithm (simply systematically explore all logical derivations and eventually, in finite time, derive “p has a bug” or “p has no bug”.)
Hilbert’s program in early 1920s
- A formulation of all mathematics; in other words all mathematical statements should be written in a precise formal language, and manipulated according to well defined rules.
- Completeness: a proof that all true mathematical statements can be proved in the formalism.
- Consistency: a proof that no contradiction can be obtained in the formalism of mathematics.
- Decidability: there should be an algorithm for deciding the truth or falsity of any mathematical statement.
- …
NO! Kurt Gödel proved in 1931 （incompleteness theorems）that no such logical system exists.

“Any consistent formal system F within which a certain amount of elementary arithmetic can be carried out is incomplete; i.e., there are statements of the language of F which can neither be proved nor disproved in F.”

3. Holy Grail of Formal Verification

Soundness: If the verification method reports no failure, then the program under examination has no bug Completeness: If the verification method reports a failure, then the program under examination has a bug Termination: The verification method will terminate, giving back an answer.

Pick TWO. Having all three is theoretically impossible.
Usually, algorithmic verification systems pick soundness and completeness, while deductive verification systems pick soundness and termination.

4. Methods for Formal Verification

However not all is lost! Sound and terminating systems can prove the correctness of virtually every program we would care about.

The scientific community continuously pushes the limits of these systems to extreme levels!

4.1 Algorithmic verification

create a model of the program in a decidable framework (finite state system, pushdown system)
usually: semi-automatic (semi-manual) model creation
automated model verification

4.1.1 Model Checking

A model checker is a program that checks if a (transition) system satisfies a (temporal) property.

10: while (true) {
11:   wait(turn == 0);
      // critical section critical section
12:   work(); turn = 1;
13: }
// concurrently with
20: while (true) {
21:   wait(turn == 1);
      // critical section critical section
22:   work(); turn = 0;
23: }

Example property: in all the reachable states (configurations) of the system, the two processes are never in the critical section at the same time.
- (Finite State) Program
- State Transition Graph
- Reachability
State exploration problem

4.1.2 Symbolic Execution

Evaluate the program on symbolic input values and use an automated theorem prover to check whether there are corresponding concrete input values that make the program fail.

foo (x) {
  if (x > 0)
    x = x + 1;
  else
    x = 1 - x;
  x = 8 / x;
}

True branch:
- x0 > 0
- x1 = x0 + 1
- x1 = 0
False branch:
- x0 <= 0
- x1 = 1 - x0
- x1 = 0
Satisfiability Modulo Theories (SMT) solver, e.g., Z3

4.1.3 Abstract Interpretation

A theory of sound approximation of the semantics of computer programs.
It can be viewed as a partial execution of a computer program which gains information about its semantics without performing all the calculations.

foo (x) { // x:(-$, +$)
  if (x > 0) // x:(0, +$)
    x = x + 1; // x:(1, +$)
  else // x:(-$, 0]
    x = 1 - x; // x:[1, +$)
  x = 8 / x; // x:[1, +$)
}

What if the program is as below?

foo (x) { // x:(-$, +$)
  if (x > 0) // x:(0, +$)
    x = 2 * x + 1; // x:(1, +$)
  else // x:(-$, 0]
    x = 1 - 2 * x; // x:[1, +$)
  x = 8 / (x % 2); // x:[1, +$)
}

A better approximation:

foo (x) { // x:(-$, +$), x%2: {0, 1}
  if (x > 0) // x:(0, +$), x%2: {0, 1}
    x = 2 * x + 1; // x:(1, +$), x%2: {1}
  else // x:(-$, 0], x%2: {0, 1}
    x = 1 - 2 * x; // x:[1, +$), x%2: {1}
  x = 8 / (x % 2); // x:[1, +$), x%2: {1}
}

model checking, symbolic execution, abstract interpretation
- create a model of the program in a decidable framework (finite state system, pushdown system)
- usually: semi-automatic (semi-manual) model creation
- automated model verification

4.2 Deductive verification

create a correctness proof of the program in a logic (with axioms and logical rules)
usually: semi-automatic (semi-manual) specification construction and proof construction
automated proof checking

4.2.1 Hoare Logic

Weakening Rule

$\frac{P\rightarrow P'\quad \{P'\}\ C\ \{Q'\}\quad Q\rightarrow Q'} {\{P\}\ C\ \{Q\}}\text{WEAK}$

Assignment Rule

$\frac{} {\{A[E/x]\}\ x:=E\ \{A\}}\text{ASG}$

Sequential Rule

$\frac{\{P\}\ C_1\ \{A\} \quad \{A\}\ C_2\ \{Q\}} {\{P\}\ C_1; C_2\ \{Q\}}\text{SEQ}$

Conditional Branch Rule

$\frac{\{P \wedge B\}\ C_1\ \{Q\} \quad \{P \wedge \neg B\}\ C_2\ \{Q\}} {\{P\}\ \textbf{if } B \textbf{ then } C_1 \textbf{ else } C_2\ \{Q\}}\text{IF}$

Partial Correctness of Loops Rule

$\frac{\{I \wedge B\}\ C\ \{I\}} {\{I\}\ \textbf{while } B\ C\ \{I \wedge \neg B\}}\text{While}$

// { T }                  (PRE)
foo (x) {
  if (x > 0) {
    // { T ∧ x > 0 }     (IF)
    // { x + 1 > 1 }      (WEAK)
    x = x + 1;
    // { x > 1 }          (ASG)
    // { x >= 1 }         (WEAK)
  }
  else {
    // { T ∧ ~(x > 0) }  (IF)
    // { 1 - x >= 1 }     (WEAK)
    x = 1 - x;
    // { x >= 1 }         (ASG)
  }
  // { x >= 1 }           (IF)
  x = 8 / x;
}

4.3 Success stories from the mid-1990s

Paris metro line 14 (1998, combination)
Flight control software of A380 (2005, abstract interpretation)
seL4. verified sequential micro-kernel (2009, separation logic)
CompCert, a verified C compiler (2005 ~ present, simulation, Coq)
MS windows drivers (2010, model checking)
FSCQ, a verified file system, (2015, program logic, Coq)
CertiKOS, a verified concurrent OS kernel (2016 ~ present, simulation, Coq)
Push-button verification, (2016, symbolic execution)