Subject: RE: [Asrg] Comments on draft-church-dnsbl-harmful-01.txt

From: "Nick Nicholas" <[email-masked]>
Date: Thu, 30 Mar 2006 09:27:13 -0800

On Thursday, March 30, 2006 at 8:49 AM Tony Finch wrote:

> DNSBLs are not only useful, they are crucial.

I'm not disagreeing with you, in fact I agree wholeheartedly.  However,
does any empirical data exist to support this assertion?  I think we
will need some solid data to support such a position in order to refute
the Church I-D.  Emprical data of this nature also would be useful for
the DNSBL BCP.

Again, I'm not arguing against you.  I just want to make the strongest
case possible to support this position.  Just making the assertion won't
be sufficient.

Regards,

Nick

--
 
Nick Nicholas
Knowledge Engineer
Habeas Inc.
650-694-3320
[email-masked]

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Daniel Feenberg <[email-masked]>
Date: Thu, 30 Mar 2006 12:36:17 -0500 (EST)

> does any empirical data exist to support this assertion?  I think we

What sort of data did you want to see? Something about the average success 
rates and false positive rates over all DNSBLs? Or an analysis of one or a 
couple? When I looked, I found that most DNSBLs blocked little spam, but 
that the best were quite effective. Is it sufficient to show that the best 
are well run, or do we have to show that all are?

Neither statistic addresses what the state of the spam situation would be 
if DNSBLs did not exist, since they are a powerfull tool discouraging MTAs 
from spewing.

---

From: Michael Thomas <[email-masked]>
Date: Thu, 30 Mar 2006 10:03:11 -0800

Daniel Feenberg wrote:
 > What sort of data did you want to see? Something about the average
 > success rates and false positive rates over all DNSBLs? Or an analysis
 > of one or a couple? When I looked, I found that most DNSBLs blocked
 > little spam, but that the best were quite effective. Is it sufficient to
 > show that the best are well run, or do we have to show that all are?

Does there not exist something like, oh say, BLreports that
judges you on false positive/false negative, coverage, timeliness,
etc?

		Mike

---

From: "william(at)elan.net" <[email-masked]>
Date: Thu, 30 Mar 2006 10:16:49 -0800 (PST)

> judges you on false positive/false negative, coverage, timeliness,

Yes and no. The problem is that finding false negatives is difficult
in automated way (need to clearly separate ham from spam manually)
especially for real-time updated data.

And of course different places receive different kinds of emails
(from different regions and different audience) and while one list
may work well for party1 it may be a disaster for party2.

-- 
William Leibzon
Elan Networks
[email-masked]

---

From: [email-masked] (Justin Mason)
Date: Thu, 30 Mar 2006 19:31:53 +0100

> judges you on false positive/false negative, coverage, timeliness,

http://wiki.apache.org/spamassassin/DnsblAccuracy082005

--j.

---

From: "Chris Lewis" <[email-masked]>
Date: Thu, 30 Mar 2006 13:44:36 -0500

> judges you on false positive/false negative, coverage, timeliness,

In a general sense, large enough to be statistically significant with
the sort of accuracy required for traditional research rigor?

No.  Unfortunately.  There are too many variables (eg: what actually
_is_ a FP in a given environment), and correlating what is _REALLY_ HAM
vs. SPAM in a large enough environment to be significant is impractical.

["Spam" and "Ham" corpuses more than a day or two old are entirely
useless, no matter how big they are.  Yesterday's post-facto analysis
(eg: training on spam) says _nothing_ about how well a particular
technique is going to work in real-time on tomorrow's spam.

Vendor: "In tests performed yesterday, our uber-fantastic spam filter
caught today 99% of spam [from a data set it was trained on from several
years ago]".

Me: "Something must be broken if it couldn't catch 100% of the spam
you've already trained it on".]

The best you can do is approximations that rely on various assumptions.

For example, in our environment with approximately a million emails per
day, approximately 75% of all spam is caught by one DNSBL, with a .01%
"would be FP if we didn't whitelist" rate.

But that carries with it various assumptions.  Eg: what the volume of
spam really is (which we get by inference from another complex metric),
and that our FP handling process catches most of the FPs.

---

From: Seth Breidbart <[email-masked]>
Date: Thu, 30 Mar 2006 14:16:16 -0500 (EST)

Daniel Feenberg <[email-masked]> wrote:

> When I looked, I found that most DNSBLs blocked little spam, but
> that the best were quite effective. Is it sufficient to show that
> the best are well run, or do we have to show that all are?

It certainly isn't the case that all are, so clearly showing that some
are is what's necessary.

Seth

---

From: Daniel Feenberg <[email-masked]>
Date: Thu, 30 Mar 2006 16:27:59 -0500 (EST)

> judges you on false positive/false negative, coverage, timeliness,

I attempted something along these lines some time ago, it is posted at 
http://www.nber.org/sys-admin/dnsbl-comparison.html but some might 
consider it out of date. It is very difficult to count false positives, 
since they are rarely reported.

We are very satisfied with the DNSBL we chose (Spamhaus) and have found 
that at any one time out of the over a 1,000 correspondents we deal with 
regularly and would notice if we blocked, usually none are blocked, 
sometimes one is. Since blocked legitimate mail is always returned to the 
sender by the source MTA, that does not represent lost mail, the sender is 
made aware of the problem and given an alternative (a web page) with an 
explanation.

There is no question that there are worse blacklists, which hardly seems 
like a reason to eliminate the good ones. Overall Spamhaus blocks less 
mail than content filters I am aware of (Spamassassin and Google Mail), 
but we judge the benefit of no lost mail to compensate.

A better study of false positives would require a large corpus of known 
good mail for a diverse set of destinations, with connecting MTA IP 
addresses. One could query the DNSBLs for those IP addresses, and 
calculate the probability that a legitimate message would be blocked. But 
I haven't found a corpus of known good mail. One source would be email 
confirmations of mailing-list signups, if anyone would like to share that 
with me. The saved mail file of an individual isn't very 
representative even if it is large.

---

From: Daniel Feenberg <[email-masked]>
Date: Thu, 30 Mar 2006 16:38:58 -0500 (EST)

> http://wiki.apache.org/spamassassin/DnsblAccuracy082005

Can you explain how to read the chart? As I understand it, of the messages 
identified by the XBL as from a spam source, 100% were classified by 
Spamassassin as spam, and of the messages Spamassassin thought were OK, 1% 
were on the XBL list. But I get a fair amount of spam with Spamassassin 
scores of 4 or less, so it wouldn't be right to call it an error rate of 
1%, would it? And what are the other 4 numbers?

Daniel Feenberg

---

From: [email-masked] (Justin Mason)
Date: Fri, 31 Mar 2006 12:36:16 +0100

> 1%, would it? And what are the other 4 numbers?

Hi Daniel -- 

No, these are not figures based on SpamAssassin classifications, so
SpamAssassin's error rate is irrelevant.

These are hand-classified messages, sorted by a human being.  So messages
classed as "spam" really are spam, for sure, and vice-versa.  We're almost
[*] certain of it ;)

To explain the fields:

OVERALL%   SPAM%     HAM%     S/O    RANK   SCORE  NAME
 17.449  24.9285   0.0113    1.000   0.97    3.90  RCVD_IN_XBL

That means that 24,9285% of the incoming spam was hit by XBL, and 0.0113%
of the incoming ham was hit (false positives).

That gives an "S/O ratio" if 1.000.  S/O is similar to bayesian
probability, or positive predictive value in medicine.  A 1.0 S/O is a
"perfect" score, meaning no false positives.  (Unfortunately the XBL
doesn't have a *real* 1.0 S/O here -- it's nearly there at 0.9995469, and
the 1.0 listing is due to rounding.)

As the page notes, http://wiki.apache.org/spamassassin/HitFrequencies has
lots more info on this accuracy-measurement format.

[*: of course, as a fair bit of the academic research recently has
noted, it's very difficult to be 100% sure, even with a human looking
at every single message.]

--j.