30 August 2013
Recently, there's been a lot of snickering over the NSA's claimed inability to search its own emails. Surely, "If anybody is going to have the money to engage in evaluation of digital information, it's the NSA for heaven's sake". Could it be true? Is it because the NSA's email system really "a little antiquated and archaic"? I suspect that it is true, and that what we're seeing is a reflection of a high-security architecture.
The NSA is a high-security organization that has real enemies, both external and internal. (Even before the Snowden leaks, it planned to re-investigate at least 4,000 people because of an insider threat.) It makes all sorts of sense to protect information even internally, to minimize the damage in event of a penetration. There are at least two possible ways that this can be done.
The first is simple: have many internal email servers, one for each major security "compartment". Searching emails would then require searching every such server—but for security reasons, no single employee should have access to more than a very few such servers. That means that a global search requires contacting a large (and possibly unknown) number of mail administrators, and asking them to search. (Why "possibly unknown"? A list of all major security compartments is itself very sensitive.)
The second and better way to have this sort of high-security email is to encrypt all email, even internally. That is, have a directory of all employees' public keys; the senders' email clients look up these keys and encrypt the email before transmitting it. There is evidence that at least some parts of NSA do exactly that. Searching such emails would then require access to all private keys, a very serious security risk. It's probably possible, but very expensive.
If this email architecture is used, it is extremely likely that the NSA has escrowed copies of all private keys. Even apart from the need for investigations and FOIA responses, employees leave or die without first turning over their credentials. For that matter, the credentials themselves are likely on smart cards that can be lost, fail, etc.; there has to be some way to recover these keys. This backup copy is obviously extremely sensitive. I certainly don't know how it's done, but one plausible solution is to use a smart card per employee, all locked in a vault. Searching the emails of a few employees is relatively straight-forward: present proper credentials and authorization to the vault-keeper, remove a few smart cards, and do the necessary decryptions. Searching all emails, though, would require shuffling through tens of thousands of cards. This is a manual process, very time-consuing and expensive, and (from a security perspective) extremely risky.
There have been a number of proposals (such as this one) for searching secure email, I don't know of any support for such schemes in commercial products; furthermore, the security model for many such schemes is for search by the owner, not by the server operator. Not having the necessary search capabilities for FOIA requests may not be a sign of an antiquated system; rather, it's a sign of not having an extremely advanced system.
Now, I don't know anything real about how the NSA does its internal email. Maybe their system is, per their own FOIA officer, "a little antiquated and archaic." Or maybe it's a high-security design.