Subject: RE: [Asrg] Comments on draft-church-dnsbl-harmful-01.txt

From: "Nick Nicholas" <[email-masked]>
Date: Thu, 30 Mar 2006 10:15:42 -0800

On Thursday, March 30, 2006 at 9:36 AM Daniel Feenberg wrote:

> What sort of data did you want to see? Something about the 
> average success rates and false positive rates over all 
> DNSBLs? Or an analysis of one or a couple? When I looked, I 
> found that most DNSBLs blocked little spam, but that the best 
> were quite effective. Is it sufficient to show that the best 
> are well run, or do we have to show that all are?
> 
> Neither statistic addresses what the state of the spam 
> situation would be if DNSBLs did not exist, since they are a 
> powerfull tool discouraging MTAs from spewing.

It's the latter that I had in mind.  What data can be adduced to
demonstrate that we're better off with DNSBLs than without them?
Perhaps something like sharply increased CPU loads would be a useful
data point.  Can we point to incidents where server meltdowns occurred
because no DNSBLs were in place?

Exactly what kind of data might work best to address this point is
something I'm still thinking through, and I welcome the thoughts of
other ASRG participants.  One of the problems with the two suggestions
in the preceding paragraph is that those data points are essentially
anecdotal, but it may be that's the best we'll be able to come up with.

Still, I think some thought needs to be given to how we can empirically
demonstrate that "DNSBLs are an essential part of a healthy diet".  It's
evidence of this nature that's most likely to be persuasive to the IETF.

Regards,

Nick

--
 
Nick Nicholas
Knowledge Engineer
Habeas Inc.
650-694-3320
[email-masked]

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Tony Finch <[email-masked]>
Date: Thu, 30 Mar 2006 19:31:44 +0100

>because no DNSBLs were in place?

On my servers, DNSBLs deal with about 80% of ALL email (including internal
email - our mail hub handles everything). Without them we would need
probably 25 machines instead of 4. (I'm multiplying by more than 5
because we don't run internal email through SpamAssassin.)

We do get complaints when the blacklists overstep the mark. For example, I
decided to stop using the MAPS RSS because they don't expire old entries,
and we were getting false positives from address ranges that had been
re-assigned in the four or so years since the original listing. The fact
that we almost never get complaints, but we do get them when there are
problems, indicates to me that our false positive rate is acceptably low.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
DENMARK STRAIT: NORTH OR NORTHEAST 5 TO 7. SNOW SHOWERS. GOOD, BUT MODERATE
OR POOR IN SNOW SHOWERS. LIGHT OR MODERATE ICING, TEMPERATURES MS02 TO MS07.

---

From: Laird Breyer <[email-masked]>
Date: Fri, 31 Mar 2006 10:58:39 +1000

On Mar 30 2006, Tony Finch wrote:

> The fact

That doesn't mean much. Do people always complain about every mistake
that happens to them? I don't think so, people will just mentally
downgrade the value of the service and not waste their time. Some will
be vocal.

What you should be doing is taking say 10% of the emails that
 your dnsbl censors and letting those through to your users, wrapped in
an attachment with a note telling them it was rejected, but can they
confirm it's spam for QA reasons? Maybe only do that with some
volunteers, but ideally you should pick your test subjects randomly.

To make sure your volunteers actually look at those QA emails
properly, you should also wrap a few real, non-spam emails into such
QA queries without warning them (ideally the same number as dnsbl
censored ones). 

If they don't tell you your system fucked up with
_all_ the non-spam emails, then you know those users don't actually do
the QA properly. That in turn would give you an apathy level.

-- 
Laird Breyer.

---

From: "Chris Lewis" <[email-masked]>
Date: Fri, 31 Mar 2006 08:49:28 -0500

> downgrade the value of the service and not waste their time. Some will

We're a business, and the contracts are worth a _lot_.  So the incentive
to report, since the senders will get almost immediate notification, is
very high.

> the QA properly. That in turn would give you an apathy level.

Except for upper management which'll start screaming that the spam
filters are broken.

---

From: Douglas Otis <[email-masked]>
Date: Fri, 31 Mar 2006 09:18:26 -0800

On Mar 30, 2006, at 10:31 AM, Tony Finch wrote:

> For example, I decided to stop using the MAPS RSS because they  
> don't expire old entries, and we were getting false positives from  
> address ranges that had been re-assigned in the four or so years  
> since the original listing.

This is the open-relay list.  Once an open relay is reported and a  
test is made of a source appearing to be an open-relay, there is no  
subsequent retesting without a request.  Open-relays are used for  
purposes other than email, and may have sporadic availability.  Many  
owners don't care about being on an open-relay list, but see testing  
as intrusive.  This policy may need to be reviewed, as this list does  
require feedback.

-Doug

---

From: "Peter J. Holzer" <[email-masked]>
Date: Fri, 31 Mar 2006 21:18:07 +0200

On 2006-03-30 19:31:44 +0100, Tony Finch wrote:
> >data point.  Can we point to incidents where server meltdowns
> >occurred because no DNSBLs were in place?
> 
> On my servers, DNSBLs deal with about 80% of ALL email (including
> internal email - our mail hub handles everything).

That's impressive. For our servers it was little more than 5% in the
second half of 2005. Most mails (42%) were rejected because they were
addressed to non-existent recipients. 22% were accepted (I don't know
what percentage of those was then filed into the Junk folder via
MUA-side filters). The single most effective anti-spam measure we have
is to filter the hostname given in HELO/EHLO against a few known bad
strings (localhost, friend, our own IP address, etc.) at 7%. Greylisting
used to be quite effective, but is now about comparable to RBLs (but
then far fewer of our users have greylisting enabled).

	hp

-- 
   _  | Peter J. Holzer    | Ich sehe nun ein, dass Computer wenig
|_|_) | Sysadmin WSR       | geeignet sind, um sich was zu merken.
| |   | [email-masked]         |
__/   | http://www.hjp.at/ |	-- Holger Lembke in dan-am
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Douglas Otis <[email-masked]>
Date: Fri, 31 Mar 2006 11:30:57 -0800

On Mar 31, 2006, at 11:18 AM, Peter J. Holzer wrote:

>>> Perhaps something like sharply increased CPU loads would be a  
>>> useful data point.  Can we point to incidents where server  
>>> meltdowns occurred because no DNSBLs were in place?
>>
> That's impressive. For our servers it was little more than 5% in  
> the second half of 2005.

5% is about the right figure for the traditional RBL.  There are  
other lists that prove far more effective.  As an RBL list is  
effective at prohibiting a source, other sources are then uncovered.   
A greater percentage will be found using the newer types of lists.   
These newer lists look beyond just the RBL and approach the 80%  
figure that Nick indicates.  : )

-Doug

---

From: "Peter J. Holzer" <[email-masked]>
Date: Fri, 31 Mar 2006 22:27:13 +0200

> figure that Nick indicates.  : )

Any hints on which lists these are? ;-)

	hp

---

From: "Jim MacLeod" <[email-masked]>
Date: Fri, 31 Mar 2006 13:19:41 -0800

> figure that Nick indicates.  : )

What techniques are these "newer" lists using that increases their
effectiveness so much?

-Jim

---

From: Douglas Otis <[email-masked]>
Date: Fri, 31 Mar 2006 13:46:06 -0800

> What techniques are these "newer" lists using that increases their  

Sorry, the comment was from Tony and not Nick.  Each organization  
does things differently, and this is part of the battle.

I should have known better than to offer statistics.  Here is a link  
that offers some explanation already public.

http://readlist.com/lists/trapdoor.merit.edu/nanog/2/10498.html

The link contained in this message references a graph labeled as  
Kelkea, a name used to transition MAPS from an LLC to Inc, but  
identified here as one of the newer lists.  Different lists use  
different tactics.  One tactic is to identify addresses operating  
under AUPs that prohibit servers, another looks to be more timely,  
where perhaps RSS would be at the other extreme.  It does not end  
here and there are other tactics providing similar results.  The  
bottom line is that lists, for all their problems, protects requisite  
resources to run filters.

-Doug

---

From: "Walter Dnes" <[email-masked]>
Date: Fri, 31 Mar 2006 17:50:10 -0500

On Fri, Mar 31, 2006 at 09:18:07PM +0200, Peter J. Holzer wrote

> Most mails (42%) were rejected because they were addressed to
> non-existent recipients. 22% were accepted (I don't know what
> percentage of those was then filed into the Junk folder via MUA-side
> filters). The single most effective anti-spam measure we have is to
> filter the hostname given in HELO/EHLO against a few known bad strings
> (localhost, friend, our own IP address, etc.) at 7%. Greylisting
> used to be quite effective, but is now about comparable to RBLs
> (but then far fewer of our users have greylisting enabled).

  I have my personal domain MX hosted at clss.net, which has qmail
hacked to enable end-user-configured smtp-stage blocking.  It issues the
big 550; it does *NOT* do bounces.  I use a combination of whitelists,
CIDRs, and various other rules.  I put DNSBLs at the very end of the
chain.  The first hit on a rule will cause email to be accepted
(whitelist) or rejected (whatever blocking rule).  The DNSBLs see very
little of my incoming email.  This is nice because...

  1) It reduces chances of a DNSBL false-positive
  2) It's "kinder and gentler" because it minimizes the load on DNSBLs.

  For the first 30+ days of March (7 hours remaining), my combined rules
have rejected a total of 5771 delivery attempts.  The first 4 rules in
the blocking chain are...

  - Badly forged HELO = 150 (HELO'ing from waltdnes.org or clss.net)

  - No hostname = 2535 (Look Ma, no rDNS!)

  - Dynamic IP by rDNS regex = 2442 ("[0-9]+-[0-9]+-[0-9]+" or
    "[0-9]+\.[0-9]+\.[0-9]+" or "adsl" or "dhcp" or "dynamic")

  - Country by rDNS = 355

  That accounts for 5482 out of 5771 delivery attempts blocked.  Being
at the very end of the blocking chain, the DNSBLs I use only blocked 100
delivery attempts so far this month.  I'm sure that DNSBLs would get
much higher numbers if I put them at the front of the chain, or used the
option to force an email to be inspected by all rules, even if it hits
one early in the chain.  But that would simply put more load on my ISP's
MTA and on the DNSBLs.

  One of the items I'll include in my comments about the Church draft is
his strawman approach of using statistics generated from *ONLY* using
DNSBLs.  A DNSBL is merely one weapon in the arsenal of the war against
spam.

-- 
Walter Dnes <[email-masked]> In linux /sbin/init is Job #1
My musings on technology and security at http://tech_sec.blog.ca

---

From: "Chris Lewis" <[email-masked]>
Date: Fri, 31 Mar 2006 17:59:42 -0500

Jim MacLeod wrote:
> On 3/31/06, *Douglas Otis* <[email-masked]
> <mailto:[email-masked]>> wrote:
> 
> 

> What techniques are these "newer" lists using that increases their

It's worth perhaps defining what "Traditional RBL" actually means ;-)

I take it to mean DNSBLs based on some combination of "open SMTP relays"
and "dedicated spam factory" as in the original MAPS RBL.

As open relays have largely fallen out of favour in spamming (estimates
in the 1-2% range), "traditional RBL" essentially means "time-consuming
subjective evaluations of static spammers".  As in high latency, poor
timeliness, ineffective against open proxies and zombies.

DNSBLs based on that (essentially the original MAPS RBL, SBL, SPEWS to a
certain extent and others) are at best in the 8-10% range of effectiveness.

The next type are the "active probers" (testing for open SMTP relay,
open proxy, open socks, formmail etc).  These used to be capable of
effectivenesses > 50%, however, "smtp relay" is obsolete, and proxy
probing has gotten vastly more complicated with "inserted proxies" on
odd ports.  Eg: Monkeys (defunct), OPM, NJABLproxy/socks, several of the
SORBS lists, parts of DSBL, ORDB (open relay, irrelevant).  These days
effectivenesses of these (taken singly) is at _best_ in the 25-30%
range.  Open proxy lists usually have very low FPs, because the
compromised machines _rarely_ also host direct mail senders.

DULs: where the listings can either be those given to them by the ISPs
themselves, or inferred by rDNS/TTLs etc heuristics, or a combination.
Originally exclusively listing dialups (hence the term derives from
"dialup user list"), but later DHCP in any form, and now mostly tending
to mean "generic consumer space" - because that's where the bots are.
None of these lists are exclusively "ISP provided" - because not enough
ISPs are publishing lists to make the resulting BL particularly useful.
Given that they mostly use rDNS et. al. heuristics, they're horrendously
time-consuming to maintain.  Effectiveness can range up to 70% or even
more.  Moderate FPs.

"Emission signature" - the only one I'm familiar with is the CBL
(included in the Spamhaus XBL with OPM and NJABLproxy), tho I believe
that is a component of some of the SORBs lists.  These look for specific
characteristics of the email being sent.  The CBL in particular seems to
work well detecting both proxies and trojans/zombies - tho of course
effectiveness will depend highly upon the cross-sectional area of the
detectors.  FPs very low.  Effectiveness of the CBL/XBL is at least
equal to the DULs.

---

From: Laird Breyer <[email-masked]>
Date: Sat, 1 Apr 2006 09:43:10 +1000

On Mar 31 2006, Chris Lewis wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Laird Breyer wrote:
> 

That is an important point, and you're in the best position to judge
it, but people can't report what they don't know might be occurring,
whatever the incentive. So claiming incentive as an argument for the
success of dnsbls won't (and really shouldn't) sway standards committees.

Some businesses expect a lot of short communication with unknown
members of the public, while others only communicate with a smaller
number of well defined partners. 

Missing mail with the latter can be detected by the business due to
inconsistencies, and a strong incentive helps with detection in
that case.

But lost mail in the former case depends on the apathy of the
initiator, ie the member of the public sending the initial mail, and
doesn't depend _at all_ on any incentives that the business being
emailed might have. 

While I expect you are well aware of this, it still leaves you (and us
here) with the need to perform a more direct investigation to back up
the claim.

Moreover, it is simply not true also to argue that if dnsbls are likely to
be helpful in the latter case because of the incentive argument and a
low complaint rate, this therefore should imply that dnsbls are likely to
be helpful in the former case too.

> Except for upper management which'll start screaming that the spam

Of course, you really do want to weigh the business benefits of ignorance
versus testing with respect to your clients. There's a risk to the
reputation of your service, which is unreasonable for us to expect you
to take just like that.

-- 
Laird Breyer.

---

From: "Chris Lewis" <[email-masked]>
Date: Fri, 31 Mar 2006 19:38:13 -0500

> it, but people can't report what they don't know might be occurring,

That's the point - they _do_ know it's occuring.

> success of dnsbls won't (and really shouldn't) sway standards committees.

You're missing the intent.  The intent is to show that our FP statistics
have pretty strong validity, full stop.  In that, it should sway those
wanting such statistics as proof/disproof of the success of DNSBLs.

Even if you don't accept the transferance of such numbers to other
environments, it still demonstrates proof of the effectiveness of DNSBLs
in at least some environments.

> be helpful in the former case too.

But it is true in terms of demonstrating the FP level of a given DNSBL -
that's going to at least have some implications for "lower value"
scenarios that don't have notifications.

Certainly better than a hand-picked insignificantly small sample.

It simply isn't possible to generate statistical accuracy in this field.
 Spam/Ham collections large enough to be useful can't be generated
timely enough to give accurate measures of real-time reputation systems.

I should point out that we ALSO have full quarantine of all blocked
email that the recipients can examine.  There's an insignificantly small
number of FPs reported that way.

---

From: Laird Breyer <[email-masked]>
Date: Sat, 1 Apr 2006 12:16:16 +1000

> wanting such statistics as proof/disproof of the success of DNSBLs.

But in terms of arguments, a generic success is canceled out by a generic
failure. You need to qualify your success claim with a domain of applicability
which covers your users but is narrow enough to not cover cases where
others might legitimately claim failure. 

The author of the draft isn't the only one who claims to have been
bitten by dnsbls, there are plenty of rants on the net, e.g.
http://paulgraham.com/spamhausblacklist.html

While that doesn't make their claims of failure statistically
significant, it means that your counterclaim of strong success of
dnsbls will necessarily be treated very skeptically, unless you can
explain where in the email usage pattern your results apply, ie how
can your claim be true and also the church claim be true simultaneously?

What's a plausible distinguishing feature? Quality of blacklist, 
type of user, ... ?

> in at least some environments.

I think the environment matters. I accept your claim, and I accept other
people's claims. Ideally I want to reconcile the opposing viewpoints.

> timely enough to give accurate measures of real-time reputation systems.

I'm not convinced it can't be. The NIST TREC people
(http://plg.uwaterloo.ca/~gvcormac/spam/) are interested in this very
kind of problem, so it's at least "on the radar", whether it will
ultimately be successful or not.

-- 
Laird Breyer.

---

From: Douglas Otis <[email-masked]>
Date: Fri, 31 Mar 2006 18:47:07 -0800

On Mar 31, 2006, at 2:50 PM, Walter Dnes wrote:
>
> One of the items I'll include in my comments about the Church draft  
> is his strawman approach of using statistics generated from *ONLY*  
> using DNSBLs.  A DNSBL is merely one weapon in the arsenal of the  
> war against spam.

These alternatives assume dependence upon EHLO verification and rDNS  
would be good enough.  EHLO verification and histories based upon the  
DNS would be a powerful alternative, without expecting rDNS  
administration.

Assume block-lists are used and achieve 43% blocking.

Here are some maawg statistics:
http://www.maawg.org/about/FINAL_4Q2005_Metrics_Report.pdf

For an example, an institution operates a mail server and receives an  
average of 50,000 valid emails/day.  Assume a blocked connection on  
average would otherwise attempt to deliver 3 messages with an average  
size of 2KB, and that an average valid email is 6 KB.  In other  
words, each blocked connection represents one valid message.  For  
each valid email there will be on average 1.7 blocked connections and  
2.2 bad emails.

With these assumptions, the numbers breakdown as follows:
   50,000 ham (desired 35 msg/minute)
  110,000 spam (undesired 76 msg/minute)
   85,000 blocked

      300 MB/day of ham
      225 MB/day of spam
      525 MB/day total ham/spam (49 Kb/s)
      522 MB/day blocked
     1047 MB/day ham/spam/blocked (97 Kb/s)

In other words, at 43% blocking the network traffic is halved.

Assume the blocking rate is 75% with a steady counter effort. (They  
try harder when blocked.)

   50,000 ham  (35 msg/minute)
   48,000 spam (33 msg/minute)
  146,000 blocked

      300 MB/day of ham
      100 MB/day of spam
      400 MB/day total ham/spam (37 Kb/s)
      900 MB/day blocked
     1700 MB/day ham/spam/blocked (157 Kb/s)

If the institution was a school on a DSL line, networks could be  
saturated with just incoming email traffic.  The overhead handling  
spam and virus filtering is also sizable in terms of hardware, and  
where block-lists provide real value.  There is also the storage  
costs when emails are archived.  A reduction in the numbers of emails  
stored, especially when reduced by factors of 2 or 3, adds up.  Who  
wants to archive spam, or have it consume allocated space in the junk  
folder?

-Doug