Subject: [Asrg] About that Church draft

From: John L <[email-masked]>
Date: Tue, 4 Apr 2006 21:17:54 -0400 (EDT)

We sure have a lot of raw material.  Can I have a volunteer editor or two 
to moosh the good stuff into one reasonably coherent document explaining 
what we think would need to be changed for it to be good?

Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Information Superhighwayman wanna-be, http://iecc.com/johnl, Mayor
"I dropped the toothpaste", said Tom, crestfallenly.

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Laird Breyer <[email-masked]>
Date: Wed, 5 Apr 2006 15:05:33 +1000

> what we think would need to be changed for it to be good?

I've just read through the threads again and done a summary of all the
main relevant points everyone discussed on the list (see attachment). 

If someone with the time wants to rearrange the paragraphs and flesh
them out into a final document, please pipe up and do so. I tried to
relate every point explicitly to the Church draft. Apologies if I
missed someone's contribution.

-- 
Laird Breyer.

---

From: "Chris Lewis" <[email-masked]>
Date: Wed, 05 Apr 2006 03:07:20 -0400

> main relevant points everyone discussed on the list (see attachment).

I think you did a rather excellent job, and summarized it all pretty well.

I'll discuss this with Thomas, and see if he wants to volunteer to do a
final draft, but, frankly, I think it's pretty good as-is.  I
particularly like how the various threads are summarized.

Almost ;-)  I'll add a couple of points:

> Flaw: Church's experiment is applicable at best to one type of email user.
> Chris Lewis's experience strongly contradicts the Church conclusion, and is
> based on very different business user email patterns.

Agreed, Church's environment and mine are _highly_ different.  However,
given our relatively permissive policy in terms of personal use of
email, we're probably not "extremely business pattern" either.  Much
"stranger to stranger" communications as well as communications with
non-corporate/non-business-related environments.

[You'd be amazed at how much business goes on with random
freemail/consumer ISP accounts for example.]

[This is more of an amplification of what you summarized - I added a
real world example of high variability]

Flaw: Church didn't describe his environment in terms of user
characteristics, nor population, and is likely to be a very small group.
Experience demonstrates that the "spam experience" varies wildly even
within a single domain.  In the Nortel environment (65,000 users),
roughly 50% get virtually no spam at all (<1/day), where as others are
more than 500/day.  Similarly, the types of spam, and which techniques
work best, vary wildly between users.

Hence: to get a broader more accurate general picture, you need a very
large user population and high volume of email.

> Regarding 2) and 3), it's important for users of DNSBLs to apply diligence
> and caution. A single DNSBL verdict should never be used on its own, as
> is implied in 5).

I disagree. I don't think you'll get general agreement on the last
sentence.  I'm going to back this up and make a more generalized comment
that I think should be reasonably acceptable to all:

We have no business dictating what individual site policy on email
acceptance policy should be.  What we assert is that site administrators
need to assess what's available (whether it be DNSBLs or any other
filtering technique) against their needs, and choose/implement
appropriately - due diligence.

If a site determines that it has no reason to communicate with, for
example, a given country, or sites based on specific listing criteria,
there is no reason why the site couldn't use a "single verdict" choice
based on an appropriate DNSBL or other filtering method.  On the other
hand, some DNSBLs are of a nature or the needs are such where scoring
methods will be more appropriate.

All filtering techniques have such trade offs. Just about all techniques
used "single verdict" can be just as prone, if not more so than many
DNSBLs, to miss-trigger[+].  We should not be singling out DNSBLs,
because other commonly used filters often have similar (or worse)
failure rates.

Site administrators must accept that it is _their_ responsibility to
_know_ what they're getting no matter what it is, know what their needs
are, apply the former appropriately to satisfy the latter, and continue
to monitor the system in terms of performance.  Success or failure of a
filtering system is always ultimately the administrator's responsibility.

[+] To raise an obvious one: "bad word" filters.  It's surprising how
many 4 and 5 letter "crude Anglo-Saxonisms" have found their way into
place names and common terminology.  We occasionally use "bad word"
filtering, but only the ones that we believe very unlikely to appear in
non-spam communications.  Unfortunately, on the last one I
miscalculated, and tripped over the name of a provincial park in
Newfoundland (almost caused some people to miss their reservations),
which also happens to be the "proper" term for part of a skydiver's
equipment...  We won't go into earlier experiences, for example, when a
very large ISP blocked the word "breast" - and entirely disrupted
several breast cancer support groups.  But, apparently "bad word"
filters have become a lot more common these days.  These things can be
hideously inaccurate.

---

From: Daniel Feenberg <[email-masked]>
Date: Wed, 5 Apr 2006 07:32:51 -0400 (EDT)

> missed someone's contribution.

Should the draft be so compact? There isn't room to include a paragraph or 
two from each of the most coherent messages? Some of the posters were
quite eloquent.

The draft seems to accept Church's notion that if a spam message
comes from 123.123.123.123.dial-up.pool.example.com and that is added
to a DNSBL, the DNSBL operator has an obligation to detect the 
reassignment of that IP address to another customer next week, and remove
the address. Is that the sense of the group? I would argue that a 
perfectly legitimate DNSBL could block dynamic addresses, even when those
addresses were temporarily assigned to legitimate MTAs.

Daniel Feenberg

---

From: Jose Marcio Martins da Cruz <[email-masked]>
Date: Wed, 05 Apr 2006 13:55:08 +0200

Chris Lewis wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 

...

> freemail/consumer ISP accounts for example.]

Yess...

> appropriately - due diligence.

Example :

This is the result of one day of mail filtering with mail-abuse. 
127.1.0.2 means the dial-up and dynamic addresses.

  -- 127.1.0.1        :   2301
  -- 127.1.0.2        :  30080
  -- 127.1.0.3        :    158
  -- 127.1.0.4        :     60
  -- 127.1.0.5        :      1
  -- 127.1.0.6        :      3
  -- 127.1.0.8        :    101
  -- 127.1.0.9        :      1
  -- 127.1.0.10       :     47

Data above is the equivalent of "If I use mail-abuse, I block everything 
from dial-up and dynamic IP addresses".

Some kind of organisations may say : "I have not to do with dial-up 
users - so I block them". On the other hand, a commercial organisation 
whose clients usually send their orders by mail coming from this kind of 
addresses will probably accept messages from this kind of addresses.

Not all blacklists results can be seen the same way, so saying what and 
how to do with blacklists is better placed in a BCP, not an RFC.

-- 
   ---------------------------------------------------------------
   Jose Marcio MARTINS DA CRUZ           Tel. :(33) 01.40.51.93.41
   Ecole des Mines de Paris              http://j-chkmail.ensmp.fr
   60, bd Saint Michel                http://www.ensmp.fr/~martins
   75272 - PARIS CEDEX 06      mailto:[email-masked]
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Laird Breyer <[email-masked]>
Date: Thu, 6 Apr 2006 09:59:01 +1000

> two from each of the most coherent messages? Some of the posters were

Yes! By all means expand on it. My intention was to bring up all the
main points equally so as to not lose people's comments, while skipping
the digressions and non technical parts.  We shouldn't lose the
momentum now.

Several people have given complete answers independently and this summary
is an aide-mémoire rather than a final document. When you read it, you
see that several paragraphs overlap, so can be rearranged to be more cohesive.

The web accessible archive doesn't carry all the messages, otherwise
I would have quoted URLs. Instead, you can search the Message-IDs in
your mailbox to obtain the relevant threads. 

> addresses were temporarily assigned to legitimate MTAs.

Please change this draft as appropriate. The attachment is there to be
edited and reposted (the ASRG policy is to show all working in the
mailing list I think). I am not the best person to edit the final
document, as my experience is in statistical filtering.

-- 
Laird Breyer.

---

From: Laird Breyer <[email-masked]>
Date: Thu, 6 Apr 2006 10:23:45 +1000

> I think you did a rather excellent job, and summarized it all pretty well.

Thanks. But do edit the document, I don't lay any claims to it. There
are details in the threads and in other people's comprehensive comments
which could be added.

> methods will be more appropriate.

It's a good general principle, but does it apply to big sites such as
Yahoo or Hotmail? I think that's one of the issues implied in the
Church draft we haven't explored yet. 

-- 
Laird Breyer.

---

From: "Chris Lewis" <[email-masked]>
Date: Mon, 24 Apr 2006 16:05:02 -0400

> main relevant points everyone discussed on the list (see attachment).

Nobody else has stepped up to the plate it seems.  How do we go about
publishing this John?  Is what Laird (and my subsequent comments) wrote
good enough as is?

> ------------------------------------------------------------------------
> 
> Summary of the ASRG discussion on draft-church-dnsbl-harmful-01.txt as
> of 04-05-2006.
> 
> This is a review of the draft-church-dnsbl-harmful-01.txt document as discussed
> on the ASRG mailing list http://news.gmane.org/gmane.ietf.asrg/
> 
> Note: Some of the posts are missing from the archive, such as Dan Oetting's
> message-id <[email-masked]>. I was going
> to refer to archive urls but ended up just giving Message-IDs.  
> 
> The church document has the following structure.
> 
> 1) An informal description of IP DNS blacklists.
> 2) A criticism of the policies and goals of DNS blacklist operators.
> 3) A claim of damage to third party mail transport.
> 4) A claim that botnets cause DNS blacklists to be out of date immediately.
> 5) An experiment on the accuracy of filtering using DNS blacklists.
> 6) A suggestion for the improvement of IP recognition
> 7) A moral argument of responsibility by DNS blacklist operators.
> 8) A suggestion to deprecate parts of the internet mail system to newer technologies.
> 
> Taken together, the author comes to the conclusion that DNS blacklists should
> not be used.
> 
> The ASRG discussion centered on several of these points.
> 
> Claims 2) and 3) are sensitive issues fraught with political aspects,
> and are therefore not appropriate for technical criticism, although
> they do represent some valid concerns that are often brought up.
> 
> Dan Oetting's thread deals with 4). There is agreement that botnets are
> a problem, but there is some research into the problem.
> 
> Jim MacLeod <http://article.gmane.org/gmane.ietf.asrg/11516> thread makes
> these points:
> 
> Regarding 4) IP assignments to particular computers tend to change
> slowly enough (on the order of a month) to make IP lists
> valuable. [reference?}
> 
> Regarding 2), diversity is a strength in anti-spam.
> 
> Regarding 5), the experiment is flawed due to its size.
> 
> Daniel Feenberg <[email-masked]>
> mentions the increased legitimacy of rejecting mail at the SMTP
> transaction protocol level rather than silently discarding it for
> filtering reasons. Unlike content filtering, DNSBLs can be used at the
> transaction time.
> 
> Steve Atkins <[email-masked]>
> suggests there are significant general problems with DNSBLs, such as being
> misused as reputation systems, technical flaws in their usage,
> etc. It may be (Chris Lewis
> <[email-masked]>) that the real problem is that
> people responsible for choosing DNSBLs aren't necessarily exercising
> due diligence and knowing what they're getting.
> 
> Tony Finch disagrees with Steve Atkins, calling DNSBLs crucial. 
> Nick Nicholas asked for solid evidence, thereby being relevant to 5).
> 
> There is a big thread on testing philosophy of DNSBLs. Everyone agrees
> that 5) in Church's paper is flawed. The flaws are as follows:
> 
> Flaw: The Church sample size (1,374 msgs) is much too small to be
> meaningful.  Chris Lewis <[email-masked]>
> disagrees with Church's conclusions based on 65,000 users and 600,000
> emails per day at Nortel.
> 
> Flaw: the comparison of filters in Church's table is
> inapplicable. Steve Atkins
> <[email-masked]> points out that
> pretending that a DNSBL filter was used for comparison purposes, as is
> done by Church, is meaningless when it actually wasn't, because filtering
> during the SMTP transaction gives feedback (Daniel Feenberg's point)
> and therefore changes the spammer's behaviour.
> 
> Flaw: Church doesn't explicitly define his terms, such as just what he
> considers spam. In turn, this means derived definitions of FP, FN, TP
> etc. are not fully mapped out. Laird Breyer and Chris Lewis discuss
> the consequences of using different definitions, resulting in
> contradictory claims.
> 
> Flaw: Church's table data suggests rejecting DNSBLs individually on
> accuracy grounds. However, this is flawed in that DNSBLs are often
> used in conjunction with other filters. The table doesn't address
> these other possibilities at all.
> 
> Several people point out that DNSBLs are meant to be used as part of a mix of
> techniques, ie that the whole is bigger than the sum of its parts.
>  
> Peter J. Holzer <[email-masked]> offers some statistics
> on the most effective elements of his mix, others chime in on that thread.
> 
> Walter Dnes <[email-masked]> addresses 2) and 3),
> which are out of scope of the technical criticism. He points out that
> 5) is flawed because it compares a DNSBL-only solution against an
> unnamed, unknown custom method, and lists several other flaws
> above. This is not realistic since DNSBLs are used in conjuction with
> other methods. Walter also points out that 8) is ridiculous.
> 
> Thomas Choi <[email-masked]> summarizes various issues
> that appeared during the discussion. 
> 
> Regarding 5), spam filtering statistics always depend on type of email
> mix being received. This varies depending upon the type of user
> (business or private individual) and the type of social interactions
> (frequent mail with regular correspondents or regular short exchanges
> with strangers), etc.
> 
> Regarding 8), Thomas points out that mail technologies arise out of
> many different needs of users, and what is optimal for one person or
> entity in terms of usability, time, expertise, processing power etc. is not
> necessarily optimal for another.
> 
> 
> 
> ------------------------------------------------------------------------
>

---

From: John Levine <[email-masked]>
Date: 24 Apr 2006 21:07:48 -0000

>publishing this John?  Is what Laird (and my subsequent comments) wrote

I sent Laird's summary to the RFC Editor.  They're smart, they know
what it means.

R's,
John

---

From: "Chris Lewis" <[email-masked]>
Date: Mon, 24 Apr 2006 18:53:05 -0400

> I sent Laird's summary to the RFC Editor.  They're smart, they know

Excellent.

With my followon comments? ;-)

[Hopefully they shouldn't be necessary.]