Subject: Re: [Asrg] Comments on draft-church-dnsbl-harmful-01.txt

From: "Chris Lewis" <[email-masked]>
Date: Sat, 01 Apr 2006 15:52:28 -0500

Laird Breyer wrote:
> On Mar 31 2006, Chris Lewis wrote:
> 
> 
>>>whatever the incentive. So claiming incentive as an argument for the
>>>success of dnsbls won't (and really shouldn't) sway standards committees.
>>
>>You're missing the intent.  The intent is to show that our FP statistics
>>have pretty strong validity, full stop.  In that, it should sway those
>>wanting such statistics as proof/disproof of the success of DNSBLs.
> 
> 
> But in terms of arguments, a generic success is canceled out by a generic
> failure.  You need to qualify your success claim with a domain of applicability

I'm a lot closer to showing a generic success, than Church is of even a
limited scope failure.

He's reporting for a spam filter responsible for handling at most a
handful of users.  I have 65,000, and handle 500 times as much email per
_day_ as his (I believe) multiple day dataset.  Sites handling 1000
times as much email as I do are ALSO using DNSBLs.

I also have far more stringent requirements regarding FPs than most,
because a single lost email could cost us millions (of my employer's) money.

We do this for a living.  I've been living in this dataset for over 10
years.  As have some of the bigger environments, who've also found
DNSBLs to be successful.

The fact that we even have to answer to such nonsense, instead of
summarily throwing that in the trash is an insult to professionals.

But we do it anyway, because we're professionals.

> which covers your users but is narrow enough to not cover cases where
> others might legitimately claim failure.

I don't presuppose to cover vanity or hobby domains.  Which is pretty
clearly what Church is talking about.

> The author of the draft isn't the only one who claims to have been
> bitten by dnsbls, there are plenty of rants on the net, e.g.
> http://paulgraham.com/spamhausblacklist.html

Paul chose to deliberately misrepresent the situation for political
ends.  As did Moveon.  As did Gilmore.  And at least _two_ of those
listings were legitimate spam blocks. There are similarly plenty of
issues about other types of filtering goofing up.

Come guys, _no_ filter is perfect.  You can't simply pick the horror
story about the filtering technique you don't like, and ignore the rest.

> While that doesn't make their claims of failure statistically
> significant, it means that your counterclaim of strong success of
> dnsbls will necessarily be treated very skeptically, unless you can
> explain where in the email usage pattern your results apply, ie how
> can your claim be true and also the church claim be true simultaneously?

Church has a personal axe to grind, his usage patterns/user community
are unknown, his "preferred" filter is only described in terms of
unverifiable handwaving - no numbers, and is of a size that can best be
described as statistically irrelevant.

That should be enough.

> What's a plausible distinguishing feature? Quality of blacklist, 
> type of user, ... ?

Yes.  And others.

>>It simply isn't possible to generate statistical accuracy in this field.
>> Spam/Ham collections large enough to be useful can't be generated
>>timely enough to give accurate measures of real-time reputation systems.

> I'm not convinced it can't be. 

Okay, suggest how a spam/ham collection can be used to measure the
effectiveness of the following techniques that are, or can be, used in
an anti-spam solution:

1) grey listing
2) sender/sender domain verification
3) Challenge/response
4) SPF and DKIM
5) PKI
6) CSV
7) Non-existant users
8) DCC or other distributed checksumming methodologies.

Similarly DNSBLs.

> The NIST TREC people
> (http://plg.uwaterloo.ca/~gvcormac/spam/) are interested in this very
> kind of problem, so it's at least "on the radar", whether it will
> ultimately be successful or not.

They've been at that for rather a long time.  They should be doing a
reasonably good job for a specific class of filtering technology.

But it's not _spam_ filtering technology.

_Anything_ that uses "adjudicators" to determine whether an email is
spam or ham is clearly missing an important clue: spam isn't about
content, it's about _consent_.  It is a behaviour (sending without
consent), NOT what is in the email.

DNSBLs that can, for example, detect compromised machines or illegal
methods being used to send the email have a better handle on intent,
than an "adjudicator" could possibly have.

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: "Chris Lewis" <[email-masked]>
Date: Sat, 01 Apr 2006 19:33:11 -0500

Justin Mason wrote:
> Chris said:
> The effectiveness of all of these, except possibly for greylisting and
> C/R, can be measured accurately. We are doing it in SpamAssassin ;) --
> here's how.
> 
> First, you accept every message, and record the "real-time" data
> points regarding how the message was listed against those services,
> and/or how it *would* have been rejected at SMTP transaction time (if
> at all). However, you don't reject, you accept everything.
> 
> Then, later, provide a way for hand-sorting to take place, and compare
> the results of the hand-sorting with what the various other techniques
> would have done with those messages.
> 
> Hey presto, you've now got a way to compare accuracy and effectiveness
> of those techniques!  Simple as that.

Do you see any provision for that with _any_ of the spam/ham
collections?  Can you do that with a pre-existing spam/ham collection if
the technique you were trying to test _wasn't_ being collected at the
time the spam/ham collection was being made?

This points up my very point: in order to do a proper sampling of the
effectiveness of a technique is that you do it in real time, at the time
the emails were sent.  In other words, on real mail streams.

---

From: [email-masked] (Justin Mason)
Date: Sun, 02 Apr 2006 20:20:50 +0100

> the emails were sent.  In other words, on real mail streams.

Ah, if you're talking specifically about *pre-existing* spam/ham
collections, then it's correct to say that they won't help, no.   To
measure network test efficacy, you need to create the spam/ham collection
in real-time, during the test.

--j.

---

From: Steve Atkins <[email-masked]>
Date: Sun, 2 Apr 2006 13:42:04 -0700

>>> at all). However, you don't reject, you accept everything.

Not rejecting the mail alters the behavior of the sender. That skews
the data from this approach. Doesn't mean it's not a useful metric, but
it's not a real measurement, and can't be used to give good estimates
of real-world performance (at least, not without quite a lot of  
additional
data, anyway).

It's still a good approach, as long as you remember that different
categories of email delivery will respond differently to changes in
recipient behaviour. Often dramatically so.

> in real-time, during the test.

And to measure actual efficacy you need to actually reject mail,
if what you're measuring is behaviour if you reject mail. (If the
behaviour you're trying to model is "what happens if we bulk folder
or devnull mail detected by a filter?" then that's not a problem,
but one of the usually mentioned advantages of DNSBLs is that
you don't need to accept the mail).

Cheers,
   Steve

---

From: Laird Breyer <[email-masked]>
Date: Mon, 3 Apr 2006 09:46:35 +1000

> of real-world performance (at least, not without quite a lot of  

During the SMTP transaction, the server might reject the data from the
client, and would do so on _some_ grounds (eg IP address). Strictly
speaking, _these_ grounds are what needs to be logged and verified by
a human somewhere down the line.

As such, it's not necessary to accept the message at all and no sender
skewing takes place. In a formal test, a human adjudicator may well
decide that a single logged IP address + dnsbl response and without an
accepted message body are insufficient grounds for summary rejection,
and count it as an FP. That's still a valid test though, because the
human observer takes the final responsibility, and can explain his
reasoning if necessary.

For example, the criterion might be consent (as in CAN-SPAM) and the 
adjudicator might be a lawyer, and a single IP listed on a dnsbl might
not constitute proof of consent or dissent under the regulations. 

> you don't need to accept the mail).

True, but in that case it is doubly important to not amalgamate
accuracy testing with efficacy testing.  These aspects should be
clearly separated in arguments intended for non-experts.

-- 
Laird Breyer.

---

From: Laird Breyer <[email-masked]>
Date: Mon, 3 Apr 2006 10:51:24 +1000

> times as much email as I do are ALSO using DNSBLs.

Right, this is a useful point against the draft.

> But we do it anyway, because we're professionals.

Yes, and you're appreciated for it. But if the answer is easy enough
to dismiss it won't stop the next person from deeming dnsbls harmful
yet again. There's a lot of folklore and crap on the net, and that's
the environment in which opinions are formed. The ASRG doesn't have
the luxury of dismissing ideas from the (valuable) experience of its
members only.

> clearly what Church is talking about.

Right, this is a useful point against the draft.

> issues about other types of filtering goofing up.

Which doesn't help in rejecting the draft. I know I brought it up,
but it's best to stay clear of this kind of argument.

> his usage patterns/user community

Yes, I suggest backing that point up with the numbers you mentioned
at the top.

> consent), NOT what is in the email.

It still needs a human adjudicator. In this case, a human who
interprets the CAN-SPAM regulations and decides whether (the legal
definition of) consent did apply.

> than an "adjudicator" could possibly have.

Being an automated method of filtering, it depends on a set of criteria
and conditions stored in the program logic and database. 
These criteria and conditions, if given in full to a human adjudicator,
could be judged by said human just as well with authority.  

Now on to your list, I answered it first but it's based on the above

> 8) DCC or other distributed checksumming methodologies.

As Steve Atkins mentioned later in the thread, there's efficacy and 
accuracy, two different aspects. I don't think there's anybody who would
dispute efficacy - rejecting at the source is clearly more efficient.

For accuracy testing, any method has to involve a human adjudicator
somewhere downstream. Moreover, whatever the method being tested, all
relevant information (ie information used for that particular decision) 
must be logged. 

If a machine can take a decision, then a human who is given the same
information (no more no less) must be able to take a decision too. The
human has the last word, and that's the basis for accuracy testing.

This basis is approriate whether you take spam as "whatever I don't want"
or whether it's UCE under CAN-SPAM etc. In the latter case, the last word
would be interpretation of the law by humans in a court, in principle. 

Now if you have a record of each transaction, and if you build a
system which submits a sample of these records to intended recipients
for verification, (and your recipients are willing to help test the
system, etc) then I think you have all the elements you need for testing.

For example, when a message is rejected via dnsbl, your record
contains the dnsbl query and a statement of rejection. That record can
be shown to the intended recipient to be ok'd. If the recipient
disagrees with the machine, then that's a FP or FN.

For non-existant users, the decision is vacuous, ie anything the server 
does is appropriate. For PKI, there is no need to ask a human to verify
the decryption with pencil and paper, obviously. Etc.

-- 
Laird Breyer.

---

From: Steve Atkins <[email-masked]>
Date: Sun, 2 Apr 2006 17:59:41 -0700

> human observer takes the final responsibility, and can explain his

If you reject the message during testing where you would have accepted
the message in operation then you may change the delivery policy of the
sender to that recipient for all future messages. That invalidates  
any data
that includes any future delivery attempts from the same sender (and  
that
sender might well be a spammer coming from any of thousands of open
proxies sending hundreds of different messages).

That means that any test where your rejection behavior differs from your
operational rejection behaviour will see different delivery policies,  
and
hence different numbers and origins of delivery attempts. That means  
that
you _cannot_ use this approach of testing multiple rejection criteria  
against
a single delivery stream and get entirely valid data (except for the  
case where
you accept all the mail, both in testing and operation).

That doesn't mean it can't provide useful data, but it will not give  
the same
answers as if you really rejected the mail in real time as you would in
practice.

> clearly separated in arguments intended for non-experts.

Neither accuracy nor efficacy are accurately measurable against
multiple different filtering policies in parallel in this sort of  
test. By
correcting for sender behaviour (which I've never seen anyone
claiming these sorts of numbers do[1]) you could certainly improve
the quality of the results, but they'd still be skewed relative to using
the same approach during real mail delivery.

It's a metric. It's not a bad metric, but don't mistake it for even an
accurate measure of results on the one set of mail tested.

Cheers,
   Steve

---

From: Laird Breyer <[email-masked]>
Date: Mon, 3 Apr 2006 12:01:12 +1000

On Apr 02 2006, Steve Atkins wrote:
> 
> If you reject the message during testing where you would have
> accepted the message in operation then [...]
> invalidates any data that includes any future delivery attempts 

Yes, we agree on that. Another way of saying this is that predictive
simulation isn't truly possible because the internal state of the
spammer's software is unknowable. Thus comparing two filtering methods
by _simulation_ , ie creating hypothetical worlds where each method is
used on its own, only makes sense in some cases.

It makes sense when the simulation is valid, eg for testing local
content filtering inside a mail client. It isn't valid for dnsbl
comparisons at the SMTP level because as you say the internal state of
the spammer software is unknown.

But predictive simulation as a way of comparing two filters (or one
filter, and the absence thereof) is not the only way to do accuracy
testing. I was saying that humans who adjudicate ex post facto the
machine decision rules (rather than the messages themselves, which may
or may not be available) can validly count misclassifications after
the fact.

This gives a valid measure of accuracy for the effect of a single
filter, that can be summarized in terms of FP and FN. In turn, FP and
FN can be compared at an abstract level.

However, you are exactly right in pointing out that multiple filters
cannot be therefore compared on the same datastream when predictive
simulation isn't possible.

But even that is only part of the story. If you take two spam datastreams
and wait long enough, you'll get some identical messages in both streams.
If you prune these datastreams to keep only those identical messages, you
can compare two filters on the same data, even though they operate on
separate streams. 

That's not much use in general, but suppose you're comparing two
dnsbls, and suppose these dnsbls identify spam messages by requiring
at least two identical message submissions. In that case, comparing
the intersection of two datastreams makes a lot more sense, because
those dnsbls only advise on multiply sent spams.

-- 
Laird Breyer.

---

From: Seth Breidbart <[email-masked]>
Date: Sun, 2 Apr 2006 22:22:57 -0400 (EDT)

> definition of) consent did apply.

If the adjudicator comes in after the fact, it is _impossible_ to
tell.

Example: [email-masked] subscribed to a non-confirming newsletter,
and typoed the email address.  When he noticed he wasn't getting it,
he subscribed again, typing the address correctly this time.

Panix gets two copies.  One to me, one to him.

One of them is spam.

Unless you get Dave Hayes to write the adjudicator, it won't always be
right.

Seth

---

From: der Mouse <[email-masked]>
Date: Sun, 2 Apr 2006 22:32:35 -0400 (EDT)

> accuracy, two different aspects.  I don't think there's anybody who
> would dispute efficacy - rejecting at the source is clearly more
> efficient.

That last word makes it appear you are confusing efficacy
("effectiveness") with efficiency ("efficientness").

As for whether rejecting at the source is more efficient, that depends
on what you are measuring efficiency with respect to.  Network traffic?
CPU cycles?  Implementor time?  Maintainer time?  It makes a big
difference.

/~\ The ASCII				der Mouse
\ / Ribbon Campaign
 X  Against HTML	       [email-masked]
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

---

From: "Chris Lewis" <[email-masked]>
Date: Sun, 02 Apr 2006 22:43:39 -0400

Laird Breyer wrote:

Thomas is composing a reply to the paper, and I've taken note of some of
your comments for inclusion.

> definition of) consent did apply.

First of all, CAN-SPAM doesn't define what is spam, and what isn't. It
simply defines a subset of spam that is legally actionable under US law.
The definition of spam doesn't depend on local ordinances.

Analysis under CAN-SPAM determinations simply isn't relevant, for
example, in our jurisdiction.  Spam is spam, whether it's in Canada, or
the US or Australia.

Secondly, it simply isn't possible to determine to that level of
certainty "consent" in anything more than a tiny fraction of spam.
Spammers simply don't cooperate.  Indeed, many of those who appear to
cooperate lie.

[I'll omit the story about proving that all of the "consents" Richter
used to justify email to ~700 of our users were fraudulent. I hesitate
to suggest that adjudicators would know what questions to ask, and how
to determine that they were fake.  Let alone being able to do it at all
in a large or broad enough sample to be statistically relevant.]

> could be judged by said human just as well with authority.  

And these adjudicators would be able to make such judgements correctly?
Unlikely.  And furthermore, even if they were not 100% accurate, ONLY
the overall result matters.

An IP HELO'ing as your server is unlikely to be judged "proof positive"
of compromise.  It isn't.  But testing shows that the number of
desirable emails mistakenly blocked by such a heuristic are essentially
non-existant.

We don't judge testing of drugs on whether "adjudicators" understand how
the drugs work, we judge drug efficacy on _results_ in trials.

> dispute efficacy - rejecting at the source is clearly more efficient.

Efficacy has nothing to do with the success/failure of DNSBLs in a
general sense.  Nor is rejecting at source a necessary consequence of
DNSBLs.  We use DNSBLs, do rejections, but still get to quarantine the
whole email.

More importantly, Steve (and I) pointed out that these techniques simply
can't be tested to the level you might want to see, because they have to
be done with real mailstreams in realtime, and with, for example, grey
listing, it's simply NOT possible to judge in your terms whether the
email was spam or not.  Applying the heuristic changes the sender
behavior, and you don't have the email to judge.

Similarly, with DNSBLs, we actually see spammers noticing that one
sender was blocked, and trying a chain of source IPs until one gets through.

The situation is simple: there's a variety of techniques that simply
cannot be tested to this level of academic certainty with "recipient" or
"recipient's judge" techniques, because applying them changes the
behaviour, and secondly, the recipient _cannot_ have anything to judge
it with.

> relevant information (ie information used for that particular decision) 

We already have the best possible scenario: the human _upstream_ who
noticed his email didn't get through, and then contacts the
administrators to get the problem fixed.

Letting the email get through in some sense (whether by quarantine or by
tagging mechanisms) simply doesn't work in the large scale. Given the
volumes that people get and the quantities they'd have to wade through,
they're simply not going to give an accurate picture - FPs will be missed.

Instead, we know that for the most part, only valid senders get notified
of rejections.  They had a reason to send it in the first place.  The
user at the other end doesn't know that there was something valid to
look for.

> human has the last word, and that's the basis for accuracy testing.

This particular line of reasoning is to do with the notion that the
machine may be mistaken that the source IP is listed in the DNSBL under
evaluation.  This is the least of our concerns.

> would be interpretation of the law by humans in a court, in principle. 

I could just imagine the industry being crippled in that fashion - you
can only tell how well your filters work by getting all the spammers in
court.  If that's the criteria for success, that criteria will be ignored.

> system, etc) then I think you have all the elements you need for testing.

I contend that sender notification is a more accurate and practical
metric.  Which is what we do.

Still, we do have exactly what you suggest.  Our quarantine - each time
someone forwards a message to themselves out of the quarantine (hence
suggestive of a FP), is recorded and sent to us for evaluation, where we
can tell what the original blocking reason was.  I don't recall _ever_
seeing one about the XBL or SBL.

> the decryption with pencil and paper, obviously. Etc.

What does showing the exact reason to the end user have to do with
whether it's a FP of the dnsbl?  A valid email being blocked is a FP,
period.  Counting it as a FP against the DNSBL as appropriate, need not
be done by the user.

Secondly, there's no reason to ask a human or adjudicator to verify that
a rejection based on a given DNSBL is really listed in the DNSBL.  Given
the volumes we're doing, errors of that nature would be immediately
catastrophically obvious.

Thirdly, even I'm not that sanguine about "non-existant users". Which is
why we simply don't count emails to non-existant users.  Otherwise, I'd
include the CBL against our spamtrap - our spamtrap does something like
3-6 million emails/day.  The CBL kills 80% of it, and in at least some
sense, that has no FPs by definition.  The numbers are useful, but you
cannot reliably determine _real_ FPs from it.

---

From: "Chris Lewis" <[email-masked]>
Date: Sun, 02 Apr 2006 23:10:12 -0400

> ("effectiveness") with efficiency ("efficientness").

Obviously, I blindly followed that misuse, to whit:
- -------------------------------

Efficacy has nothing to do with the success/failure of DNSBLs in a
general sense.  Nor is rejecting at source a necessary consequence of
DNSBLs.  We use DNSBLs, do rejections, but still get to quarantine the
whole email.
- -----------------------------
What I intended to say is that the DNSBL's efficiency (if that) isn't
relevant to success/failure _unless_ it impairs its useability.  And as
such, that's a per-instance issue, not generic to the technique.

---

From: Laird Breyer <[email-masked]>
Date: Mon, 3 Apr 2006 13:36:07 +1000

> Panix gets two copies.  One to me, one to him.

Correct. The formal test must specify the adjudicator. It doesn't
matter what the cause of the spam is (ie a benign typo here), only the
measurable result. If you're the adjudicator, then it's spam. If
sethbr is the adjudicator, then it's not spam. If you both are, then
the formal test must specify how to deal with ties. If the law is the
test then there are two deliveries, and one of them increments the
spam count.

We can always argue about testing criteria, but as long as they are
clearly spelled out to the experts, there is value in such tests.

-- 
Laird Breyer.

---

From: Laird Breyer <[email-masked]>
Date: Mon, 3 Apr 2006 13:45:03 +1000

> ("effectiveness") with efficiency ("efficientness").

Ah, you're right, I am using the wrong word :(, I meant efficiency. Thanks.

-- 
Laird Breyer.

---

From: Laird Breyer <[email-masked]>
Date: Mon, 3 Apr 2006 15:16:08 +1000

> example, in our jurisdiction.  Spam is spam, whether it's in Canada, or

Only when using your definition as anything that is sent without the
receiver's consent. The merits of various definitions have been
discussed to death on this list in the past, there's no need to rehash
them.  I mentioned CAN-SPAM as I got the impression you used consent
in a more than informal way.

To stay on topic, do you accept that with your definition, the only
authority which can reliably decide consent (and therefore spamminess)
is the receiver?  Moreover, insofar as the automated filtering systems
that are in place are concerned, these act as proxies for the
recipient only.

> the overall result matters.

How do you define the overall result if you don't take
recipients authority as final. I should say that the adjudicators I mention
are assumed to be actual recipients in this case.

> the drugs work, we judge drug efficacy on _results_ in trials.

Correct, but _results_ are well defined in terms of observable
biological effects. If spam is defined in terms of recipient consent,
then observation requires asking the recipients if their consent is
given in each test case.

> DNSBLs.  We use DNSBLs, do rejections, but still get to quarantine the

I apologize for using efficacy incorrectly, I meant efficiency in the
sense of machine workload. 

However, if you accept the email in quarantine and silently reject
after the fact (correct me if I'm wrong), then I do not see that
Steve's point applies. You can test the effectiveness of your choice
of dnsbls without interacting with the spammer who sends the
message. In particular, you can do what Justin Mason outlined.

> behavior, and you don't have the email to judge.

What you do have is all the information that the mail server has when
it makes the rejection choice. Namely X number of attempts from the
same host in Y time. If consent can only be authoritatively judged by
the recipient, then this recipient should be asked to judge based on
precisely that information (during a QA test, for verification). But
see the next paragraph. 

> behaviour, and secondly, the recipient _cannot_ have anything to judge

No behaviour is changed by auditing the accuracy of a system after the
fact. But it doesn't come for free, logging relevant information is
necessary.

The recipient may well find it difficult to judge based on so little
information and without seeing the never accepted mail, whereas you
have much more experience with mail systems in general and for you the
greylisting record is quite sufficient, without the need to see the
mail at all.

I think that points to the fact that it makes more practical sense for
_you_ to judge consent than for the intended recipients, in these
cases.  But that requires changing the meaning of spam slightly to
include you as an authoritative proxy for the recipient in difficult
cases. If so, then your comments (notifying the sender etc)
makes a lot more sense to me.

I don't want to make this reply too long, so I won't do a point by
point reply. I think we agree that your points apply to regular
frequent business communication, so there's no need to criticize them
again from a different perspective.

-- 
Laird Breyer.

---

From: John Levine <[email-masked]>
Date: 3 Apr 2006 05:50:20 -0000

>is the receiver?

Not really.  I get spam complaints all the time for mail from lists
that I know perfectly well that they signed up for and confirmed.
"Oh, I don't want that any more."  For the ones that aren't totally
redacted, my setup turns them into unsubs so they don't get any more
mail for that particular list, but I don't think it's fair to count
mail as spam if it depends on reading the recipient's mind in
real-time.

Testing spam filters is a really, really hard problem, for all the
reasons that people have mentioned, basically that you can't test
without perturbing the system you're testing, and you can't capture
enough state to rerun the same test more than once.  So you have to
estimate based on complaint rates from bounces and the like.

R's,
John

---

From: "Walter Dnes" <[email-masked]>
Date: Mon, 3 Apr 2006 05:42:23 -0400

On Mon, Apr 03, 2006 at 10:51:24AM +1000, Laird Breyer wrote

> the luxury of dismissing ideas from the (valuable) experience of its

  Maybe the only permanent answer is to...

  - *not* deny that there may be *SOME* harm from DNSBLs

  - but show that there is *MORE* harm in *NOT* using them

  In Chris' case...

  - how many messages do his sytems reject per day?

  - how much additional DASD farm would be required to store the
    messages that wouldn't be rejected?

  - how much additional CPU power would be required to crunch those
    no-longer-rejected messages through Church's pet filters?

  - how much additional air-conditioned and fully-powered computer rack
    space would Chris' employer have to rent or buy+maintain to house
    the additional storage+servers required in the 2 previous points?

  - how much would it cost to pay for additional salary and benefits,
    and office-cubicle space, and desktop PCs that would be required for
    the extra employees to maintain+operate the extra storage+servers
    identified above?

  And furthermore, remember that in the case of a false-positive...

  - a DNSBL will send a 5XX reject to a legitimate sender's MTA, which
    will notify the sender of the reject.

  - a content-filter will bury the email in a "spam folder" with
    thousands of real spam, where it'll probably never be found.  The
    sender will believe that the intended recipient has received the
    message and ignored it, while the intended recipient will believe
    that the sender hasn't sent the message.

-- 
Walter Dnes <[email-masked]> In linux /sbin/init is Job #1
My musings on technology and security at http://tech_sec.blog.ca

---

From: [email-masked] (Justin Mason)
Date: Mon, 03 Apr 2006 12:33:04 +0100

> estimate based on complaint rates from bounces and the like.

The Spam Heisenberg Uncertainty Principle. ;)

by the way this is one reason why I, personally, prefer to accept spam in
full, instead of rejecting during the SMTP transaction, greylisting, or
tarpitting.   It's more practical to test and instrument that way, and
thereby get a solid idea of FP rates, instead of guesstimation.

--j.

---

From: Michael Thomas <[email-masked]>
Date: Mon, 03 Apr 2006 08:08:05 -0700

>   - but show that there is *MORE* harm in *NOT* using them

It doesn't seem to me that boosterism is really the right answer
though. DNSBL's are what they are. What would be nice to know as
an outsider is _what_ they are, how they roughly work, what their
upside is, and what their downside is. That way you can make the
risk/reward tradeoff which is invariably context sensitive.

>     that the sender hasn't sent the message.

That depends a whole lot on the receiving end's policy, doesn't it?
Spamassassin for example can use DNSBL's, the result of which is
effectively silent discard. And it would be a fairly trivial exercise
to make Spamassassin cause a 5xx to be generated for > 5.0 SA scores.

		Mike

---

From: "Chris Lewis" <[email-masked]>
Date: Mon, 03 Apr 2006 12:10:33 -0400

> risk/reward tradeoff which is invariably context sensitive.

I think as a nutshell, from the IETF perspective, Church's draft would
have to:

   a) become a DNSBLs _may_ be harmful if you don't...
   b) _know_ what the filters you're using actually do (at least
     in terms of results).

As such, it could be a kernel for a more general "filtering BCP" paper.
But I don't think Church is willing to write one of those ;-)

[I've written a sketch for one, but nothing beyond that.  There was a
mailing list created by Crocker (I think) to carry that onwards to
something useful, but that mailing list was DOA.]

>>     will notify the sender of the reject.

It doesn't have to.  But it can (and does in perhaps most implementations).

>>     that the sender hasn't sent the message.

It doesn't have to.  But it can (and does in many implementations).

> to make Spamassassin cause a 5xx to be generated for > 5.0 SA scores.

You have to be careful in your generalizations.  Virtually _no_
filtering (aside from things like grey-listing) forces you to adopt one
form of disposition or other.

In our installation: DNSBLs reject inline _AND_ quarantine[+] (which
means it still takes the session up to the end of DATA, and then 550's
that).

Similarly, our content filters, including SpamAssassin, ALSO reject
inline and quarantine (550's after DATA).  There is no "post-accept"
filtering of any consequence in our environment (except for the
infrastructure A-V to catch the viruses the front end filters miss, and
a few users using their own client-end filters).

[+] Quarantine is conceptually equivalent to spam folders, but
implemented differently.

---

From: "Chris Lewis" <[email-masked]>
Date: Mon, 03 Apr 2006 12:12:31 -0400

> The Spam Heisenberg Uncertainty Principle. ;)

Perhaps not surprisingly, I was thinking about Heisenberg too.

Or, at least, the "Cockroach Uncertainty Principle".  Spammers, like
cockroaches, behave differently when you shine a light on them.

> thereby get a solid idea of FP rates, instead of guesstimation.

Which is why we do that too ;-)

---

From: Seth Breidbart <[email-masked]>
Date: Mon, 3 Apr 2006 12:21:44 -0400 (EDT)

>    that the sender hasn't sent the message.

Actually, a content-filter COULD generate a 5XX; of course, that
requires even more CPU capacity, because the filter has to run in
almost real time.

But that would be a useful metric: "Not using XBL would cost $COMPANY
$BIGNUM per year, and $COMPANY receives X% the amount of email AOL
does . . ."  leading to an estimate that "Not using DNSBLs would cost
those who now use them $HUGENUM per year".

Seth

---

From: "Chris Lewis" <[email-masked]>
Date: Mon, 03 Apr 2006 12:45:31 -0400

> mail as spam if it depends on reading the recipient's mind in

Similarly, we get complaints from receivers that we blocked certain
emails that we _know_ are spam.  Like the individual who insisted that
it was necessary for him to receive the "nortel.com abuse team" messages
(aka Bagle).  Or the people who complain about us blocking important
email from their bank... that are in reality phishes.  Or the people
who've fallen for Nigerian money laundering, "change for a fraudulent
money order" scams.

[As far as we know, we managed to catch the only one of the latter here
just before the victim wrote the cheque.]

In fact, receiver adjudication would likely _lower_ the FP rate. On
balance, I believe it's more likely that a user doesn't recognize
something he really did ask for (thru some twisty chain of branding,
lowering the FP rate) than "recognize" something he didn't ask for
(raising the FP rate).

In other words, receiver adjudication would bias the evaluation
_towards_ DNSBLs (lower FPs), not away (higher FPs).  Which, if you must
accept bias (and are evaluating in a conservative approach), is the
_wrong_ direction.

Thus, receiver adjudication has an error rate, further as I'll mention
below, likely _higher_ than the measures we're trying to test.

I'll point out that in most environments who're running multiple
techniques, FN rates of individual techniques is not relevant, except in
evaluating incremental implementation cost of that technique (cost vs.
benefits).  So what if SBL-XBL only catches, say, 80% of our spam flow?
Our other techniques fill in that gap.  Our other techniques put
together can't reach even 80%.

> estimate based on complaint rates from bounces and the like.

Exactly.

Except in samples that are too small to be statistically significant,
it's not possible to use "receiver adjudication" in an effective
fashion.  Not because we can't do that, but because the error rate in
"receiver adjudication" is going to be at least an order of magnitude
higher than some of the methods we're trying to test, and in fact,
likely to be strongly biased in _favour_ of of the filtering technique
(for reasons outlined above).

Most of the time the receiver is correct, no question.  Which is why we
don't deny more than an infinitessimal fraction of FP/FN complaints.

But human nature being what it is, when faced with hundreds of emails in
a folder to assess, a miniscule error rate of 1% isn't unreasonable to
expect.

Yet, we already know that the CBL FP rate on our stream is approximately
2 orders of magnitude _smaller_ (and we whitelisted them, so there's no
ongoing issues), and the SBL FP rate is usually not measurable, because
we don't see any for days at a time (out of 300,000-600,000 filtered per
day).

You can't measure the accuracy of something to .01% if the instrument
you're measuring it with is only 1% accurate.  Further, we'd find a 1%
FP rate on our filtering _totally_ unacceptable (that's 3000+/day). I'd
be looking for another job in another field if it was remotely that bad.

---

From: Laird Breyer <[email-masked]>
Date: Tue, 4 Apr 2006 09:58:41 +1000

> mail as spam if it depends on reading the recipient's mind in

Isn't that perilously close to saying that if they decided something
in the past, they're not allowed to change their mind?

My position is that there's absolutely nothing wrong with the "spam is
what I say is spam" definition. It's workable, provided you keep spam
filtering at the ends of the network, ie the desktop. 

The issue is feedback. You don't want the spam complaints because you
don't consider yourself these people's spam filter. Chris actually 
is in some sense his customers' spamfilter, and he goes out of his way
to obtain feedback. A trainable desktop filter, whatever the way it 
works internally, is designed to allow quick feedback to make the "spam is
what I say is spam" definition workable.

I also have no problem admitting that for group filtering during SMTP
transport, the "spam is what I say is spam" definition isn't great.
For one, it's hard to reconcile two users who respectively want and
not want a particular type of message. Another definition such as 
"did you consent?" is possibly better. 

We know there are plenty of definitions. It's important to agree on one
in any one discussion.

> estimate based on complaint rates from bounces and the like.

Sure is, or nobody would make a big deal about it. I don't agree that
perturbing the system is necessary, however. In the end, testing is
about making the most of the information that's actually available. If
what you have is a dnsbl verdict rather than a message body (say),
then you use that and don't throw up your arms and say "why can't I
have a message body?".

But again, it's important that all discussants use the same
definition, etc.  The devil is in the details.

-- 
Laird Breyer.

---

From: Seth Breidbart <[email-masked]>
Date: Mon, 3 Apr 2006 20:23:36 -0400 (EDT)

> in the past, they're not allowed to change their mind?

No.  If someone asked for something, they're certainly entitled to
change their mind and request (and enforce) not getting it.  They're
not entitled to change their mind and decide they never asked for it.

> My position is that there's absolutely nothing wrong with the "spam
> is what I say is spam" definition. It's workable, provided you keep
> spam filtering at the ends of the network, ie the desktop.

You can't actually _filter_ according to that rule.

> The issue is feedback. You don't want the spam complaints because
> you don't consider yourself these people's spam filter.

It has nothing to do with spamming.  Unsubscribes and spam complaints
are not the same thing.  Spam complaints about solicited email are
bogus, incorrect, and wrong.

[testing]

> I don't agree that perturbing the system is necessary, however.

Describe how you'd test greylisting without perturbing the system.

Seth

---

From: Douglas Otis <[email-masked]>
Date: Mon, 3 Apr 2006 17:27:41 -0700

On Apr 3, 2006, at 4:58 PM, Laird Breyer wrote:

>>> To stay on topic, do you accept that with your definition, the  
>>> only authority which can reliably decide consent (and therefore  
>>> spamminess) is the receiver?
>>
>> Not really.  I get spam complaints all the time for mail from  
>> lists that I know perfectly well that they signed up for and  
>> confirmed. "Oh, I don't want that any more."  For the ones that  
>> aren't totally redacted, my setup turns them into unsubs so they  
>> don't get any more mail for that particular list, but I don't  
>> think it's fair to count mail as spam if it depends on reading the  
>> recipient's mind in real-time.
>
> Isn't that perilously close to saying that if they decided  
> something in the past, they're not allowed to change their mind?

This could be a case where the subscriber declares the list's email  
as spam even though they failed to take steps to unsubscribe.  There  
might also be a problem created by lists that fail to confirm a  
subscription as well.  The lack of a confirmation step would be a bad  
practice.  Some have subscribed victims to lists and then send their  
spam through the list when this confirmation step is missing.

> spam filtering at the ends of the network, ie the desktop.

There is no reason networks must endure abuse.  At least refusal at  
the SMTP session provides valuable feedback to the sender.  Nothing  
should force providers to adopt dangerous and unworkable "opt-out"  
methods either.  Fair rules can be established that _always_ consider  
bulk unsolicited messages as abuse.  This is a _simpler_ definition  
that can be implemented in a fashion that protects resources.  There  
is no reason to waste the time of resources of billions of people to  
achieve some nonsensical ideal.

-Doug

---

From: Laird Breyer <[email-masked]>
Date: Tue, 4 Apr 2006 10:41:13 +1000

> lowering the FP rate) than "recognize" something he didn't ask for

That's your judgement talking, not the user. You're saying that you
can't let users full reign because that's like herding cats, it would
mess with your aggregate server statistics and thereby impact the
business. It's a very reasonable position from your point of view.

A guy like Church doesn't accept your point of view because he has
different priorities. I find it hard to agree with you because my
"natural" definition puts the user first. I can see that a clueless
user is well advised to let you make decisions for him, but not
someone who has used mail for a long time.

> below, likely _higher_ than the measures we're trying to test.

Only from your point of view. From the point of view of the user, the
errors you mention aren't FPs, they are one instance of FN (he doesn't
recognize something he asked for, ie that your system delivers to his
inbox) and one instance of FP (he does recognize something he didn't
ask for, ie that your system doesn't deliver).

> be looking for another job in another field if it was remotely that bad.

It's not difficult to set a fixed FP rate on a classifier and trade
off FN to achieve it. Every filter can achieve < 0.01% FP. In the
trivial extreme, no filtering at all achieves 0% FP and 100% FN.

What I just want to point out is that your filtering costs (ie the
estimated cost of an FP and the estimated cost of an FN) are not the
same as for an individual user. They make sense for you, no question,
but it's another potential source of valid disagreement worth laying
on the table. 

One way to route around this problem is to use ROC curves, which give a
global picture that doesn't depend on any one person's cost estimates.

-- 
Laird Breyer.

---

From: Laird Breyer <[email-masked]>
Date: Tue, 4 Apr 2006 10:52:50 +1000

>     that the sender hasn't sent the message.

That's merely an artifact of current implementations. The key to this
problem is ranking.

Do you remember the internet before Google? The top search engine
around 97 was Altavista, and it was at the top because it had a large
index and other features, but it didn't rank pages so it was necessary
to find sites by wading through many hits, like wading through an
unordered spam folder.

Google was a success because it was the first to offer usable ranking
of search hits, turning a deep search list into a shallow search list.
The same applies to the problem of finding false positives in a spam
folder.  

-- Laird Breyer.

---

From: Laird Breyer <[email-masked]>
Date: Tue, 4 Apr 2006 11:23:24 +1000

> not entitled to change their mind and decide they never asked for it.

That's because you're using the "consent" version of the spam definition.
You're not using the "spam is what I don't want" definition. Reread 
John's paragraph, he wrote "Oh, I don't want that any more".

> You can't actually _filter_ according to that rule.

As I said, it's a matter of obtaining timely feedback.  But you are
entirely right in saying that some systems, especially large
multi-user heavy traffic systems today can't possibly do this well.

I'm not pushing the definition on anybody, I'm defending it as valid and
freely admit that it's best applied to filtering at the MUA level.

> Describe how you'd test greylisting without perturbing the system.

I already have outlined it several times now. The greylisting system
logs all the events which enter into the final decision. When a mail
is rejected, the record of evidence which triggered that rejection 
(call it R) is kept. 

Later, during QA, some volunteer users are sent a fraction of
rejection records R for mail messages that were intended for them, to
be approved. When the definition is "spam is what I say is spam",
these volunteers are the final arbiters of mail intended for them,
whether other people think they could do a better job or not.

Finally, you measure the number of records R which the volunteers say
are incorrectly rejected by the greylisting system. In this way, no
heisenberg effect occurs.

If you like, you can also implement a tool which automatically derives
other information from the record R, such as reverse IP ->country
mapping, or inferring the sender domain, to make it easier for the
volunteers to arbitrate. In principle, a volunteer could implement
such a tool himself, or just go on WHOIS etc. Since the recipient is
the final arbiter, he can do anything he likes.

Again, if you're secretly (or not so secretly) using a different definition
of what spam is, then you'll find this bogus.

-- 
Laird Breyer.

---

From: "Chris Lewis" <[email-masked]>
Date: Mon, 03 Apr 2006 22:01:13 -0400

> John's paragraph, he wrote "Oh, I don't want that any more".

I don't think John is calling that spam, is he?  I don't either.

I won't block on that basis.  I'll just help the user stop getting more.

> are incorrectly rejected by the greylisting system.

Sorry, that notion is nonsensical with greylisting.  Are you aware of
how greylisting works?

greylisting works by tempfailing the connection of _every_ "new" unique
sender/recipient/source IP triple (or some variant), then letting the
email through if the sending server retries.  The notion being that most
spam senders cannot retry.

[There's various timeout values involved w.r.t. minimum retry interval,
how long you remember a successful passthru etc.]

It never permanently rejects.  The only notion of "incorrectly rejected"
there could be is if the implementation is broken and not implementing
greylisting.  In which case, you're not testing greylisting, you're
testing something else.

---

From: "Chris Lewis" <[email-masked]>
Date: Mon, 03 Apr 2006 22:16:57 -0400

> someone who has used mail for a long time.

The user is first.  I simply recognize that a user's accuracy rate on
determining FPs is _lower_ than the automated systems are.

You can't measure a voltage to 6 digits of accuracy if your equipment
only does three.

> trivial extreme, no filtering at all achieves 0% FP and 100% FN.

But you're no longer testing what you were trying to test, are you?

> global picture that doesn't depend on any one person's cost estimates.

ROC - "Return on cost" curves?  Whassat?

---

From: Seth Breidbart <[email-masked]>
Date: Mon, 3 Apr 2006 22:34:26 -0400 (EDT)

>> Describe how you'd test greylisting without perturbing the system.

That was a trick question.  You provably can't.

> is rejected, the record of evidence which triggered that rejection

Consider some spamware: based on whether it gets 4XX or 5XX, early or
late in the transaction, it decides how to continue (4XX resends the
same thing later, 5XX early sends from another IP address, 5XX late
changes hashbusters and sends from the same IP address, a second fail
changes hashbusters again and sends from a different IP address, where
to send other spams going to the same mailserver but to different
addresses from, . . .)

Now, whatever you do affects what you see later.  Therefore you
_cannot_ do a test without perturbing the system.

Seth

---

From: "David Nicol" <[email-masked]>
Date: Tue, 4 Apr 2006 00:53:07 -0500

> > mail as spam if it depends on reading the recipient's mind in

that's brilliant.  Imagine a shared-training filter (like dspam) that will
respect and follow header-listed unsubscribe instructions instead of
touching the beysian weights when someone marks a well-behaved
list posting as spam.

Hmm.  No, it would be very bad, as if a spam were to appear on a mailing
list everyone who reported it would get unsubscribed.

Maybe the MUA should have a "unsubscribe" button when it recognizes that
I am reading a mailing list message.

---

From: "Thomas Choi" <[email-masked]>
Date: Tue, 04 Apr 2006 04:07:15 -0400

Here's my reply to Church's paper. :)

**************************

Church’s paper entitled "DNS Blacklists Considered Harmful" discusses 
some legitimate concerns with using DNS-blacklists. One such concern 
with DNS-blacklists is the lack of control that an e-mail administrator 
has over such lists. In particular, the author mentions that "it may 
take a significant amount of time for the server operator's request for 
removal from the blacklist to be answered.". With e-mail being a 
business-critical means for communicating, it's essential that the 
e-mail administrator have the ability to make immediate changes to their 
filters when needed. Since DNS-blacklists are operated and maintained by 
third party sources, and not the administrator him/herself, the author's 
concern does seem to be legitimate. In such a case, a question becomes, 
what can the administrator do when a third party blacklist is blocking 
legitimate mail that needs to get through right away? To answer such a 
question, one should note that it's possible for a spam filter using a 
DNS-blacklist to not block messages just because a sender’s source IP 
address is listed in said list. For example, by using the spamassassin 
tool, a score is associated to each DNS-blacklist rule and by lowering 
said score, one could limit the disruption caused by false positives 
from that list. Alternatively, one could simply setup their own 
'whitelist' of IP addresses that they can use to ensure that mail from 
specific sources will always get through.

Another concern that the author raised in his paper was with regards to 
the actual practices of the operators of such blacklists. Specifically, 
the author argues that "a far more serious problem is that of arbitrary 
policies instituted and decisions made by blacklist operators.". In 
response to the authors’s concern, it should be noted that each list has 
their own policies and as a result, some lists may have more 
questionable policies than others. So before adding any DNS-blacklists 
to their anti-spam solution, it is highly recommended that e-mail 
administrators verify the reliability and integrity of the policies of 
the lists they are interested in and select the one(s) that works best 
for them. That being said, it should be noted that there are in fact 
some high quality lists that are being widely used today by corporations 
and Internet service providers as part of their filtering solution.

A final concern was raised after the author conducted an analysis of the 
mail received and filtered on his mail server. In this analysis, the 
author compared the effectiveness of a content-based filter against five 
different blacklists on a corpus of 1,374 messages. When presenting the 
results, the author noted that "only one of the blacklists (SBL-XBL) 
managed to correctly flag more than 50% of incoming spam and all 
blacklists rejected some of the incoming nonspam messages.". In 
comparison, the content-based filter "correctly filtered 97% of the 
spam, without any false positives.". Although the test results clearly 
showed that the content-filters performed better than the 
DNS-blacklists, it should be noted that sample size was quite small and 
that there was no mention of the number of users nor the variety of spam 
that was used in the experiment. With such a small sample size, it’s 
unlikely that the experiment was tested with the many different variants 
of spams (i.e. scam spam, junk mail, non-commercial, phishes, viruses 
etc.) that larger entities would likely see. For example, one such 
entity is Nortel which has over 65 000 distinct user accounts and 
receives an average of 600k e-mails per day. Nortel uses a hybrid of 
filtering mechanisms comprising content rules and the CBL DNS-blacklist, 
which blocks approximately 300k e-mails per day and includes all of the 
variants described above. It should be noted that the blacklist alone 
accounts for approximately 80% of all of their blocked e-mails with a 
false positive rate of 0.01%. With other large entities reporting 
similar DNS-blacklist effectiveness results, it’s not surprising that 
many large-scale environments are currently using blacklists as part of 
their own anti-spam solution.

Towards the end of the paper, the author suggests that technological 
advances will likely reduce the need for blocking mail at the server 
level. Such advances would include mail client spam filtering software, 
which can learn from previous spam. The author goes on to say that such 
software is "far more effective than simple server-based blacklists, 
particularly since they can adjust for the types of spam received by 
individual users.". In response, it should be mentioned that in order to 
effectively use such a filter, the end user is required to train the 
filter so that it can distinguish between both good and bad mail. With 
the limited amount of spare time in people's lives, it wouldn't be 
surprising if most users would balk at the idea of having to train their 
mail filter software to detect spam and non-spam. Instead, they would 
probably prefer that spam be blocked at the gateway instead of being 
filtered at the client. Furthermore, it should be noted that the author 
acknowledges in the problem section of his paper that approximately 70% 
of all e-mail is spam. With such an incredibly high volume of spam, does 
it really makes sense to waste processing power of the user's desktop so 
that its anti-virus scanner to scan extra message for viruses and does 
it really make sense to waste extra bandwidth needed to deliver all mail 
to the end user. Since e-mail virus infections are costly to any 
corporate entity, most administrators would probably agree that it's 
highly desirable to have the gateway filters block bad messages before 
they get to the end user.

In conclusion, Church’s paper brings up some legitimate concerns with 
the use of DNS-blacklists. Each of these concerns were analyzed and 
addressed in this report to show that DNS-blacklists can in fact be 
useful when the administrator selects the right one for their particular 
environment. In other words, when an e-mail administrator verifies the 
reliability and integrity of any list they wish to use and selects the 
one that works best for them, DNS-blacklists can be useful.

---

From: Daniel Feenberg <[email-masked]>
Date: Tue, 4 Apr 2006 07:49:45 -0400 (EDT)

On Tue, 4 Apr 2006, David Nicol wrote:

...>
> I am reading a mailing list message.

There is already a standard "List-Unsubscribe" header described by 
RFC2369, so an MUA could do this. It would be a great advance. The syntax
of the field is not rigid (it could be a mailto or a URL, but the MUA 
could treat it as html and whatever it was, it would be.

I do get a lot of list mail with this header - it isn't DOA.

Daniel Feenberg
feenberg isat nber dotte org

---

From: "Peter J. Holzer" <[email-masked]>
Date: Tue, 4 Apr 2006 18:37:59 +0200

On 2006-04-03 22:01:13 -0400, Chris Lewis wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Laird Breyer wrote:
[...]
> greylisting.  In which case, you're not testing greylisting, you're

I am testing whether greylisting has the intended effect. Sure, the MTA
implementing greylisting never issues as 5xx response. But the effect of
one or more 4xx responses may be the same - either because the client
MTA is broken (doesn't retry) or because of the client's queuing setup
and the servers greylisting parameters don't match. 

So if I have records from the greylisting MTA, I can in theory look at
every delivery attempt and classify it as:

1) Delivery was accepted and message was not spam (TN)

2) Delivery was accepted and message was spam (FN)

3) Delivery was temporarily rejected, but a later delivery of the same
   message was accepted. This puts it into one of the two categories
   above, but adds extra info (number of delivery attempts, delay from
   first delivery attempt to acceptance).

4) Delivery was temporily rejected and no later delivery attempt was
   successful. Message was spam (TP).

5) Delivery was temporily rejected and no later delivery attempt was
   successful. Message was not spam (FP).

In practice this is difficult. Normally, greylisting is done at the RCPT
command, so there isn't much information available (EHLO, MAIL, RCPT,
IP-address of client, maybe size of the message). So it is even
difficult to determine whether two delivery attempts are for the same
message, and even more difficult to determine whether the message was
spam. Still, for small samples, where a human can go over the records
and apply real-life knowledge, it should be possible to get a (somewhat
fuzzy) categorization which is better than that determined by
greylisting itself.

(As an aside, we provide our users with a webbased view of the log files
for their own mail addresses, so they can do similar checks - some of
them do and report FPs due to greylisting to us)

Greylisting could be delayed until after the message is received. This
would make provide more information for auditing, but it alters the
algorithm enough that it cannot be considered the same algorithm any
more (so you cannot do this "just for testing", you have to do it in
production, too).

	hp

-- 
   _  | Peter J. Holzer    | Ich sehe nun ein, dass Computer wenig
|_|_) | Sysadmin WSR       | geeignet sind, um sich was zu merken.
| |   | [email-masked]         |
__/   | http://www.hjp.at/ |	-- Holger Lembke in dan-am
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Laird Breyer <[email-masked]>
Date: Wed, 5 Apr 2006 09:34:24 +1000

> determining FPs is _lower_ than the automated systems are.

Again, I think it's due to a disagreement as to what spam is.  You
claim a user's accuracy is lower than an automated system's at
detecting consent-spam. I claim the user's accuracy is necessarily
higher than a third party at detecting what-I-want-now-spam. We could
go around in circles for a long time with this, but it makes more
sense to recognize that we're talking about slightly different ideas
of spam, which implicitly means your FPs aren't the same FPs I'm
talking about etc. 

> You can't measure a voltage to 6 digits of accuracy if your equipment

I agree, and I'm not proposing that. A typical statistical hypothesis
test has the option of being inconclusive. But here we're not even 
measuring the same FPs so it's moot. 

A ROC curve is a "Receiver Operating Characteristic". Imagine you have
a spam filter sytem and a panoply of subsystems and parameter values
to choose from. For each possible setting, you measure the (FP%,FN%)
pair on some data.  Then you plot these points in a square of side
length 100%. This gives a picture of the overall quality of the spam
filter as you insert or remove the subsystems and parameters.

It allows you to compare different spam filters over various operating
ranges, and gives insight into the accuracy tradeoffs you make when
you replace one subsystem with another.

This tutorial may be a little technical, but the important things are
the pictures.
http://www.cs.bris.ac.uk/~flach/ICML04tutorial/ROCtutorialPartI.pdf

This is a readable introduction with a lot of detail.
http://home.comcast.net/~tom.fawcett/public_html/papers/ROC101.pdf

-- 
Laird Breyer.

---

From: Laird Breyer <[email-masked]>
Date: Wed, 5 Apr 2006 09:44:20 +1000

> _cannot_ do a test without perturbing the system.

I don't think we're talking about the same thing. When I'm talking
about measuring, I'm not talking about intervening directly, making a
choice about 4XX or 5XX on the spot.

First, your greylisting system does what it normally does. The spammer
reacts and life goes on.  A week later (say), you go through the logs
and you verify that the system is not losing mail you think it should
accept. Think of it as debugging, QA, whatever. That's the accuracy
test.

You summarize this QA with a FP/FN rate, which counts the number of
mistakes/bugs of each type. There you are, an FP/FN rate. You can now
compare these numbers with FP/FNs obtained from another spamfilter on
comparable data.

-- 
Laird Breyer.

---

From: "David Nicol" <[email-masked]>
Date: Tue, 4 Apr 2006 19:04:48 -0500

> http://home.comcast.net/~tom.fawcett/public_html/papers/ROC101.pdf

Which brings us back to tagging.  In the tagging vision, the end-user gets
to adjust their own sliders and weights based on verifiable analyses done
by upstream expert systems.

--
David L Nicol
Should the bike shed have bunks?  Or maybe cots?

---

From: Laird Breyer <[email-masked]>
Date: Wed, 5 Apr 2006 10:12:00 +1000

> I won't block on that basis.  I'll just help the user stop getting more.

He's not, neither are you. I'm using that definition, as do
other people who develop Bayesian content filters. John was giving an 
example why he thinks that definition is unworkable. However, it gets
used every day by lots of people and filters. Seth mentioned deciding
they never asked for it. 

> spam senders cannot retry.

Yes, I am aware of that. I'm just trying (perhaps wrongly) to keep it
high level instead of discussing rejection modes at the SMTP
transaction level, or the socket connection level, etc. I think the
important aspect for testing is that the server doesn't always want to
relay the message body that the client wants to submit. So the mail is
effectively rejected. When (if) the client tries again, who's to say
that mail is the same one?

If you were developing a greylisting system and you were debugging it,
you'd log a lot of information. That's what I believe can be packaged
for QA testers in this scenario.

-- 
Laird Breyer.

---

From: Laird Breyer <[email-masked]>
Date: Wed, 5 Apr 2006 10:21:07 +1000

On Apr 04 2006, Peter J. Holzer wrote:

(greylisting testing)

You've explained the point I was trying to make much better than I just
have. 

-- 
Laird Breyer.

---

From: John Levine <[email-masked]>
Date: 5 Apr 2006 01:07:45 -0000

>in the past, they're not allowed to change their mind?

Of course not.  You can change your mind all you want, but you can't
expect me to know that you have done so until you tell me.  So if the
user unsubs and the mail keeps coming, then it's spam, but if he asked
for it and hasn't asked for it to stop, it's legit.

>perturbing the system is necessary, however.

Then I have to say that you're not thinking very clearly about the way
that spam filters work.  As soon as you do or don't deliver a message,
or delay it by greylisting, or query a DNSBL, or any of a zillion
other things that filters do, you've perturbed the system.

> In the end, testing is about making the most of the information
> that's actually available.

Well, sure, but it's also about knowing whether your synthetic tests
have any relevance to the real system you're trying to model.

R's,
John

---

From: Laird Breyer <[email-masked]>
Date: Wed, 5 Apr 2006 12:09:14 +1000

> for it and hasn't asked for it to stop, it's legit.

What if he likes to read the list messages on a single topic, but
doesn't want those messages that deal with another topic? 

Or take the ASRG list itself. For argument's sake, I might be only
interested in micropayment-type systems, and don't want to see any other
threads such as announcements on IETF drafts. When signing up, did I
know that I was going to get IETF announcements? Am I wrong to claim
the list sends me some spam?  Did the list description on the ASRG homepage
change at all from the time several years ago when I subscribed and now? 
Consent is not that simple either. If I unsubscribe, I lose the micropayment
threads, if I stay subscribed I get spam announcements :)

> other things that filters do, you've perturbed the system.

That's the spamfilter operating, not the QA test of how well the
spamfilter is judged to be working.

It is obvious that some spamfilters can change spammer behaviour
directly. We've been discussing (at least I think we have) whether the
act of measuring the operation of a spamfilter itself changes the
spammer behaviour. And my claim is that the act of measuring the
operation of the spamfilter need not affect spammer behaviour
directly.

There's also a subtlety with predicting the effect of a putative,
nonexistent spamfilter if its operation directly changes spammer
behaviour, but that's another thing again.

> have any relevance to the real system you're trying to model.

If you can obtain FP and FN numbers, then I believe you have a
valuable summary of accuracy for that system, but it won't tell you
about other aspects such as cpu load etc. One thing at a time please.

-- 
Laird Breyer.

---

From: John Levine <[email-masked]>
Date: 5 Apr 2006 15:03:11 -0000

>>  if he asked

Then he should change his subscription, but that doesn't make the mail
spam.

>Or take the ASRG list itself. For argument's sake, ...

Same answer, deal with it any way you want, but mail you asked for
isn't spam.  I agree that if you signed up for one thing and they send
you something completely different, which sometimes happens, that's
bad, but splitting hairs isn't going to get anyone anywhere.

> And my claim is that the act of measuring the operation of the
> spamfilter need not affect spammer behaviour directly.

That's fine if you don't want to test filters that reject, greylist,
delay, or otherwise do anything that senders detect.  For the other
99% of spam filters, you've got a problem.

>about other aspects such as cpu load etc.

Nor will it let you compare that system to others since the FP and FN
numbers you get on different inputs are not necessarily comparable.

R's,
John

---

From: Laird Breyer <[email-masked]>
Date: Thu, 6 Apr 2006 10:45:09 +1000

> bad, but splitting hairs isn't going to get anyone anywhere.

Agreed, I'm not looking to impose a single definition of spam, just 
point out that consent isn't a panacea.

> 99% of spam filters, you've got a problem.

I still don't agree, but I'm happy to label the topic as "non-trivial".

> numbers you get on different inputs are not necessarily comparable.

That's a minor issue. The basis of filter evaluation is that you analyze
performance in the past to judge performance in the future. If you
truly believe that your own email stream in the near future is
incomparable to your email in the recent past, then you can just pick
all your filters by flipping a coin.

Email streams of different people can well be incomparable, of course,
as can mail of one individual at different periods in time, etc.  Even
that has a limited variation however. For example, the TREC 2005 tests 
showed a fair amount of consistency over several test corpora. The
variation is not so great as to make FP and FN comparisons on
different corpora meaningless. 

-- 
Laird Breyer.