Subject: [Asrg] greylisting with whitelist of good mailservers

From: William Leibzon <[email-masked]>
Date: Sun, 29 Jan 2006 05:53:37 -0800 (PST)

I've been experimenting with technique that involves combination of
mail server ip address whitelist with greylisting. The idea is by
default to greylist, but once good email is received, the ip address
of mail system is added to special whitelist (I also record name
reported in EHLO but do not use right now) and afterward greylisting
is no longer done and email is processed immediately. Note that messages
from both whitelisted and non-whitelisted mailservers still go through
my normal spamfilters, so this is purely greylisting-related whitelist.

This techniques appear to be extremely useful against viruses and 
zombies (4 hour greylisting is used which is fairly long, shorter
periods may not be as effective), while still allowing mail from all
common sources to be received immediately.

One of the problems is however determining what is good email. Right
now I'm doing it more or less manually, which makes it more difficult
and slow to build list of good mail servers. I'm wondering if there
are any good automated methods people use when they need to separate
"absolutely non-spam"?

What I'm thinking however is that for this technique (since spam-
filtering is still being done) it should ok to just take email if
spamassasin score is low (say < 1.0). I'm concerned however that
this could open a door for spammers to test how "good" my system
thinks their email is and adjust the content. Am I being too
paranoid?

Anyway comments on the technique and on questions are welcome.

---
William Leibzon
   mailto: [email-masked]
Anti-Spam and Email Security Research Worksite:
   http://www.elan.net/~william/emailsecurity/
Whois & DNS Network Investigation Tools:
   http://www.completewhois.com

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: John Levine <[email-masked]>
Date: 29 Jan 2006 15:50:07 -0000

>is no longer done and email is processed immediately.

That's close to the usual way to do greylisting.  See my CEAS paper
from last year.  My greylister doesn't make any attempt to look at the
contents of mail, just remembers the envelope, and if it sees a retry
with the same envelope in a reasonable time window, it succeds and
whitelists the IP.

The original greylist implementation greylists every (IP,sender,recip)
triplet separately, but that never made any sense to me and I always
assumed it is an implementation bug.  Once you know that an IP's
client retries, you've learned all you're going to learn from
greylisting and there's no point in delaying any more.

>One of the problems is however determining what is good email.

I don't even try.  What's the advantage?  You're already doing another
pass to figure out what's good and what's not.  If a host sends only
spam, it's worth blacklisting, but I don't see any connection between
that and whether it retries.

> I'm concerned however that this could open a door for spammers to
>test how "good" my system thinks their email is and adjust the
>content. Am I being too paranoid?

Probably.  I doubt they do per-recipient-host tuning for anyone
smaller than Earthlink.

R's,
John

---

From: William Leibzon <[email-masked]>
Date: Sun, 29 Jan 2006 19:27:20 -0800 (PST)

> greylisting and there's no point in delaying any more.

Actually I do store (IP,sender,recip) tripple for greylisting database.
The same server trying to deliver email from the same envelope sender
to different recipient would mean separate greylisting delay. This is
done so that bulk system which sends email with same MAIL FROM but to
different recipients on my system would keep retrying rather then being
able to delivery at once on one of the next runs.

But my whitelisting database is only IP address, so if any emails from
the system is non-spam, it ends up being whitelisted. This is possibly
not that good because in addition to the noted possibility allowing to
probe my content filter (which is probably indeed every rare), this
also easy enough to bypass.

What I'm thinking for one of the next stages is change this whitelisting
ip database into (IP,score) where score is updated and is medium of the
scores of the emails that came from the system before - i.e. it basicly
is a real-time updated reputation system. This reputation score is already 
of interest for some other uses (maybe even worth making it public), but 
for greylisting its use would be that amount of delay would depend on
this score (i.e. very low score - no delay; medium score - 30 minutes 
delay; high score - 6-24 hours delay; very high score - ip is entirely 
blacklisted). I'm also thinking this kind of reputation score would
have to degrade with time - i.e. if no emails come from the ip, the score 
will decrade and so even blacklisted systems would ultimately get back 
into grey area.

Any comments on this?

I intend to write proper software for this reputation system (perl or 
python scripts  utilizing postgres for backend) next month. If there
is an interst, I can release it. This would be written for use with 
spamassassin but greylisting component is obviously separate.

---
William Leibzon

---

From: John Levine <[email-masked]>
Date: 30 Jan 2006 03:16:40 -0000

>to different recipient would mean separate greylisting delay.

Same here, the greylist entry is a triple, but the whitelist is just
the IP.

>the system is non-spam, it ends up being whitelisted.

In my system, if any host retries with the same envelope, it now
bypasses the greylister but still has to run the gauntlet of all the
other mail.  This means that only people writing from hitherto unseen
IP addresses get legit mail delayed by greylisting, which is rare.

>What I'm thinking for one of the next stages is change this
>whitelisting ip database into (IP,score) where score is updated and
>is medium of the scores of the emails that came from the system
>before - i.e. it basicly is a real-time updated reputation system.

Seems reasonable except I don't see what it has to do with greylisting.

R's,
John

---

From: "Peter J. Holzer" <[email-masked]>
Date: Mon, 30 Jan 2006 10:57:50 +0100

> greylisting and there's no point in delaying any more.

This assumes that an IP address identifies an SMTP client.

Many DSL and cable providers use dynamic IP addresses with short leases
(8 to 24 hours seems to be rather common). So if you know that the SMTP
client on a certain IP address retried yesterday, you don't know that
this IP address is still in use be the same host today. (And yes, I know
people who run an MTA for outgoing mail on a host with a dynamic IP address).

Even if the IP address is static, a host running a regular MTA may also
be infested with a worm or spambot.

	hp

-- 
   _  | Peter J. Holzer    | Ich sehe nun ein, dass Computer wenig
|_|_) | Sysadmin WSR       | geeignet sind, um sich was zu merken.
| |   | [email-masked]         |
__/   | http://www.hjp.at/ |	-- Holger Lembke in dan-am
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg