Subject: [Asrg] pre-rfc thought balloon: ESMTP DATAFIRST

From: "David Nicol" <[email-masked]>
Date: Mon, 5 Jun 2006 16:29:04 -0500

This e-mail is a presentation of an idea for comments prior to drafting the idea
into a formal request for comments document.  The idea presented herein is
within the realm of e-mail practice and is therefore on-topic for the anti-spam
research group, although the idea would not directly reduce spam, as the
idea if put into wide practice would simplify the operation of large
mailing lists
on both the sending and receiving end.

The idea if implemented would ease content-based filtering at the server as
messages to multiple recipients would no longer require separate complete
transactions for each recipient when sending to servers that implement complex
per-recipient acceptance policies.

The idea is a new ESMTP extension.

The new ESMTP extension is tentatively called "DATAFIRST"

When a server identifies itself as supporting the DATAFIRST extension,
the SMTP transaction may be re-ordered from MAIL FROM, RCPT TO, DATA
to MAIL FROM, DATA, RCPT TO

with the full range of acceptance/delivery status response codes
available at RCPT TO
time rather than at the end of the DATA command.

-- 
David L Nicol

_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: Claus Assmann <[email-masked]>
Date: Mon, 5 Jun 2006 14:40:13 -0700

On Mon, Jun 05, 2006, David Nicol wrote:

> The idea if implemented would ease content-based filtering at the
> server as messages to multiple recipients would no longer require
> separate complete transactions for each recipient when sending to
> servers that implement complex per-recipient acceptance policies.

> When a server identifies itself as supporting the DATAFIRST
> extension, the SMTP transaction may be re-ordered from MAIL FROM,
> RCPT TO, DATA to MAIL FROM, DATA, RCPT TO
> available at RCPT TO time rather than at the end of the DATA
> command.

Why not support the RCPT status model of LMTP? That way you get the
recipients at the start of the transaction and don't need to transfer
any messsage if there are no valid recipients and you can give a
per-RCPT status after the message has been sent:

RFC 2033:

4.2.  The DATA command
...
   The change is that after the final ".", the server returns one reply
   for each previously successful RCPT command in the mail transaction, 
   in the order that the RCPT commands were issued.  ....

---

From: "B. Johannessen" <[email-masked]>
Date: Mon, 05 Jun 2006 23:39:22 +0200

> to MAIL FROM, DATA, RCPT TO

So you'll have to accept the data even though there may be no valid 
recipient?

I'd think an extension where the responses to RCPT was delayed until 
after DATA would be much more useful! That way you could reject DATA if 
there has been no valid recipients, while keeping the ability to do 
per-recipient content scanning.

	Bob

---

From: Douglas Otis <[email-masked]>
Date: Mon, 5 Jun 2006 14:45:53 -0700

> to MAIL FROM, DATA, RCPT TO

This would the commit servers into accepting data without being  
provided any clue the sender knows who would be available to receive  
the information.  A better system would seem to be a call-back mode  
for the data.

A new ESMTP extension DATA-CALLBACK.

The current DATA phase becomes a the header only portion of the  
message.  A new header also provides a token used by the MUA to  
retrieve the rest of the data.  The recipient could check the DKIM  
signatures before deciding whether the signing domain had a suitable  
reputation before even requesting the information.  The bulk of the  
message would be held by the sender rather than by the intermediary  
for the recipient.

-Doug

---

From: "Jim MacLeod" <[email-masked]>
Date: Mon, 5 Jun 2006 15:05:50 -0700

Forgive my ignorant skepticism, but what problem would this solve?  The
blurb you wrote sounds like it's a mechanism to bypass policy restrictions,
which doesn't sound like a good thing.

If I read it correctly, the problem sounds like "Messages to lots of
recipients have to be sent to each recipient individually if the server
won't accept the message to some recipients."  Is the solution that, by
sending the DATA first, the receiving server could separately 250/5xx for
each recipient, based on a complex decision involving the content?

I wonder what's next, an ESMTP extension replacing DATA with a URI that
could be retrieved ala ETRN?

-Jim

On 6/5/06, David Nicol <[email-masked] > wrote:
>
> This e-mail is a presentation of an idea for comments[...]

---

From: "william(at)elan.net" <[email-masked]>
Date: Mon, 5 Jun 2006 15:19:25 -0700 (PDT)

On Mon, 5 Jun 2006, Douglas Otis wrote:

> This would the commit servers into accepting data without being provided any 
> clue the sender knows who would be available to receive the information.  A 
> better system would seem to be a call-back mode for the data.
>
> The current DATA phase becomes a the header only portion of the message.  A 
> new header also provides a token used by the MUA to retrieve the rest of the 
> data.  The recipient could check the DKIM signatures before deciding whether 
> the signing domain had a suitable reputation before even requesting the 
> information.  The bulk of the message would be held by the sender rather 
> than by the intermediary for the recipient.

The above is main reason I worked on META Signatures. I specifically 
wanted signature in email header that could reference email data
(which may not necessarily be present as part of the message) and where 
such signature can still be confirmed. EDigest (to be renamed 
Transmission-Digest in the next draft release) implements this all by 
allowing "u" as URL reference to actual digest data context and META
obviously only signs header data fields. I believe I mentioned this on 
ASRG once before and published couple drafts 1-2 years ago. Some of the 
things I envisioned was doing im2000-like system entirely within context 
of SMTP with header becoming email service notification  carrying 
information on where actual data can be retrieved as well as carrying 
authentication/accreditation (stamps-like) data within the signature
(or within signed data header). The forwarding in this system also works 
perfectly well and does not make forwarder responsible for the message 
(because forwarder would simply send the header and let the final 
recipient retrieve the message from the original location) and similarly 
for mail lists (which could choose to forward the message adding reference 
to additional mime context for mail list footer). The support for data
retrieval was supposed to be carried by means of SMTP extensions with 
server that has to send it further but next hop does not support it then 
has to retrieve the data, however if at later hop server supports 
retrieval it could just drop the body again (and this all has no
problems as far as signature confirmation!).

DKIM however can not be used for this - this is the signature that
does not separate header from email body properly and forces the
signature to be for entirely data segment rather then based on
actual context part(s) and only allows to verify the signature
once data has been retrieved.

If somebody is interested I have whole bunch of notes on all this - 
however considering what I've seen I do not see that internet
community (and more specifically email industry) would be willing
to implement all necessary extensions and I think new email protocol
is better way to go to really fix SMTP rather then several additional 
patches which are hard to fit within current infrastructure.

-- 
William Leibzon
Elan Networks
[email-masked]

---

From: "Mark E. Mallett" <[email-masked]>
Date: Mon, 5 Jun 2006 18:32:34 -0400

> each recipient, based on a complex decision involving the content?

I think that is indeed the goal (not bypassing policy restrictions, but
enforcing them individually, without having to 4xx all but one
recipient).  My reaction was also that the LMTP approach is better: be
able to adjust or confirm, after DATA, any successful RCPT-TO that
happened before DATA.  I'm sure this has been suggested before, but
dunno why it hasn't gotten any start.

mm

---

From: Douglas Otis <[email-masked]>
Date: Mon, 5 Jun 2006 15:38:44 -0700

On Jun 5, 2006, at 3:19 PM, william(at)elan.net wrote:
>> provided any clue the sender knows who would be available to  
>> receive the information.  A better system would seem to be a call- 
>> back mode for the data.
>>
>> signatures before deciding whether the signing domain had a  
>> suitable reputation before even requesting the information.  The  
>> bulk of the message would be held by the sender rather than by the  
>> intermediary for the recipient.
>
>
> actual context part(s) and only allows to verify the signature once  
> data has been retrieved.

Actually, with the inclusion of the bh= header, the body can be  
checked separately with DKIM.  The just the hash of the body is signed.

> to implement all necessary extensions and I think new email  
> protocol is better way to go to really fix SMTP rather then several  
> additional patches which are hard to fit within current  
> infrastructure.

If email becomes more reputation based, a call-back mechanism should  
eliminate much of the risks associated with auto-executed malware.   
It would also isolate the ESP from being responsible for filtering  
this part of the message, which is why this was seen as relating to a  
filtering overhead concern.  This mechanism could also support point  
to point TLS protections, keeping email confidential.  The data must  
still match that of a sha-256 hash and does not require the message  
to be encrypted by the application.

-Doug

---

From: "David Nicol" <[email-masked]>
Date: Mon, 5 Jun 2006 18:20:20 -0500

> dunno why it hasn't gotten any start.

I had lunch last week with some guys who send "responsible e-mail advertising"
and got a glimpse of the situation from their points of view.  If you had a list
of, say, every potential coca-cola drinker at AOL, switching to a DATAFIRST
sequence would allow you to put the ad in first and then chase it with the whole
mailing list, easing your conflicts with the recipient postmasters who currently
have to deal with multiple open connections from you.  The bulk of the
transferred
data is the vast list of recipients, perhaps millions.  Currently as
we all know a
bulk sender must work within recipient limits, and per-recipient policies do not
work at all with multiple recipients.  LTMP's approach works but it is
still half-way
for a large list, as the list has to be cached on both sides while the
per-reipienct
policy is checked.  With a short message and a long list and
per-recipient policy,
DATAFIRST is the correct order.

The concern about needing to receive the data provisionally while waiting for
valid recipients is a non-issue.  The DOS potential is no greater than current
practice, and the SIZE extension would take care of announcing that you
will only take so much DATA before closing.  DATAFIRST should have its
own size restriction that is separate from the SIZE size restriction, that's
a good point.

-- 
David L Nicol
"fans of liza minelli should always be
disconnected immediately" -- Matthew Henry

---

From: Daniel Feenberg <[email-masked]>
Date: Mon, 5 Jun 2006 19:30:48 -0400 (EDT)

On Mon, 5 Jun 2006, Jim MacLeod wrote:

...
> won't accept the message to some recipients."  Is the solution that, by

I don't think many MTAs send such messages, since 99.44% of them would be
to forged addresses. Instead they just discard mail whose content appears 
to be spammish. In some cases that judgement is wrong, and inefficiency or 
ill-feeling can result. So operators of MTAs using content analyzers would 
presumably be interested in a mechanism for rejecting mail during the SMTP 
command phase on a recipient by recipient basis. Sending the data first 
seems to be an easy way to enable this, although it has been pointed out 
that the information could be returned after the data phase.

Presumably only MTAs with content based spam scanners, and among those 
only ones with individual databases, would offer the DATAFIRST option to 
connecting MTAs. And among connecting MTAs, only those who cared if 
messages were delivered would act on the offer. It isn't obvious that 
anyone would be harmed. There is plenty of experience to suggest that 
individual databases work much better than system-wide databases.

Nor does this option imply any additional wasted bandwidth over the 
current widely practiced method of accepting the message and then 
discarding it.

> could be retrieved ala ETRN?

That suggestion is frequently made, and this seems to be a way to avoid 
such things.

Daniel Feenberg
NBER

---

From: Douglas Otis <[email-masked]>
Date: Mon, 5 Jun 2006 16:58:56 -0700

On Jun 5, 2006, at 4:20 PM, David Nicol wrote:

> I had lunch last week with some guys who send "responsible e-mail  
> advertising" and got a glimpse of the situation from their points  
> of view.  If you had a list of, say, every potential coca-cola  
> drinker at AOL, switching to a DATAFIRST sequence would allow you  
> to put the ad in first and then chase it with the whole mailing  
> list, easing your conflicts with the recipient postmasters who  
> currently
> have to deal with multiple open connections from you.  The bulk of  
> the transferred data is the vast list of recipients, perhaps millions.

Unsolicited email based upon a list of potential customers does not  
sound like "responsible e-mail advertising."  Who gets the message  
must not be based upon exposing an individual's profile of filter  
settings, whether or not an ESP reaps some monetary advantage.   
Dynamically negotiating a long list of recipients based upon content  
would also be a privacy concern, and would be counter productive at  
abating abuse.  Unsolicited email must not be allowed under any  
efficacy guise.

-Doug

---

From: John Levine <[email-masked]>
Date: 6 Jun 2006 00:21:16 -0000

>servers that implement complex per-recipient acceptance policies.

I wouldn't bother, for several reasons:

One is that you're optimizing a rare case and pessimizing an unusual
case.  I get a lot of mail that I reject at RCPT TO, little mail to
multiple recipients, and no mail at all to multiple recipients who
need to do different things with it based on the content.  If one
thinks it's spam and the other doesn't, that invariably means one of
them has a misadjusted spam filter.

The next is that for reasons unrelated to bandwidth, there are good
reasons to send list mail one copy at a time. On the lists I run,
plain old text lists like this one, each copy of each message is sent
out with a unique bounce address that makes list management vastly
easier by encoding the list, the message number, and the recipient.
I'm also thinking of embedding per-user unsubscribe links, since it's
quite common for people to get stuck when they subscribe at address A,
don't keep the welcome message that tells how to unsub, then try to
unsub from new address B.

The last is that despite the persistent folklore, the amount of
bandwidth that mail uses is trivial compared to web and P2P, and it is
a foolish economy to try and reduce it even more.  The experiments
I've seen say that for normal MTAs on normal network links, individual
deliveries get list mail done in less elapsed time due to the
increased parallelism.  (In the past 30 years, the bandwidth of
networks has vastly increased, but the end to end time hasn't changed
much, what with that pesky speed of light.)

So there are certainly problems to be fixed in e-mail, but this isn't
one of them.

R's,
John

PS: If you really want to do this bad idea, leave the MAIL FROM at the
beginning, and have the extension return a list of codes at the end of
data, one per rcpt address.  Then at least you're no worse off than
you are now.

---

From: "Jim MacLeod" <[email-masked]>
Date: Mon, 5 Jun 2006 17:21:25 -0700

Uh-oh.

DATAFIRST seems like a great tool for a DHA (directory harvesting attack):
here's an innocent-seeming email, and here's the million possible recipients
for it...

How about a compromise, something like a NOACK?  After the DATA, the
receiving server replies only that it did/did not receive the message,
without any information to the sender whether any given recipient will
receive the message.  Probably should add a message type that says "message
received, but none of the recipients are valid", so the sender won't send to
that domain any more.  Think about multicast IP (IGMP, PIM, DVMRP, etc) as a
model.

- The upside: less traffic, since the message is only sent once (by
"responsible" senders).

- The compromise: those senders don't get feedback on whether the addresses
are valid.  For mailing list maintainers, there's nothing to allow them to
prune their lists.  However, for spammers, there's no feedback for DHA
purposes.

- The downside: senders who really really want that address-level feedback
will still send individually.

Perhaps this "downside" would still be a good thing, as senders who still
choose to send individually will suffer reputation harm.

It's also possible that extensions of this kind could encourage more spam
messages to be sent, since receiving MTAs which adopt the extension could
increase the recipient limits.  OTOH, receiving MTAs might just quietly
discard the messages with the same limits as before.  This would make for an
interesting model in game theory.

-Jim

On 6/5/06, David Nicol <[email-masked]> wrote:
> ...If you had a list
> mailing list...

---

From: "Jim MacLeod" <[email-masked]>
Date: Mon, 5 Jun 2006 17:30:26 -0700

> That suggestion is frequently made, and this seems to be a way to avoid

(Please forgive me if I'm making a newbie suggestion.  I realize that this
is not The Universal Answer To The Spam Problem, If Only Everyone Would
Adopt It, but it seems like an interesting area for research.)

...and that model is bad why?  Obviously, if implemented with FTP-like
back-channel connections, it would be a nightmare, but if implemented with a
well-known port and a bit of auth, I don't see too many problems.

Benefits:
- it pushes the email burden back onto the sender - for real, not like many
anti-spam companies claim.
- it makes the sender easier to track.
- it would seem to reduce the spam-zombie problem by requiring that inbound
connections to the mail sender be available through the firewall.
- it gives the receiving MTA the option not to retrieve the email message.

-Jim

---

From: "Mark E. Mallett" <[email-masked]>
Date: Mon, 5 Jun 2006 20:59:47 -0400

On Mon, Jun 05, 2006 at 06:20:20PM -0500, David Nicol wrote:
> advertising"

I dunno.  I somehow doubt that receivers will want to keep a delivery
transaction open while accepting and processing a large enough recipient
list that would make caching that list a problem for either side.
The statement that "I have so many recipients to give you all at once
that it's a burden to me to keep track of them" does not strike me
as a friendly one from the point of view of the receiver.

[And do senders really want to send so much non-personalized bulk in
this day and age?]

The LMTP-style solution adds value without taking any away, whereas a
mere rearrangement of commands does take functionality away.  Among the
things lost are:

 - a point at which you can revoke/abort the session, despite any
   previously accepted RCPT-TOs (there are various policy reasons for
   which this is common behaviour).  You could of course add a new
   RCPT-TO response that means the same thing, or an "all done" command
   phase, but this would again mean caching the list on both sides;

 - being able to reject the DATA because you've decided against it
   based on the recipient list; you mention later that there's no
   benefit in this, but I have a hard time agreeing with that.

> practice,

The percentage of transactions turned away pre-DATA purely because of
bad RCPT-TO (or a decision made during RCPT-TO processing) is large.
This rearrangement of commands could conveivably turn that to zero; even
if not widely adopted, it could be easily used for mischievious purposes.

> and the SIZE extension would take care of announcing that you

This merely parameterizes the problem, whereas LMTP-style deals with it
directly.

mm

---

From: Steve Atkins <[email-masked]>
Date: Mon, 5 Jun 2006 18:11:57 -0700

On Jun 5, 2006, at 4:20 PM, David Nicol wrote:

> and got a glimpse of the situation from their points of view.   
> [...] The bulk of the
> transferred data is the vast list of recipients, perhaps millions.

That's pretty rare behaviour within the reputable bulk email
industry. VERP and per-recipient customization make one
email per recipient the method of choice for most of
the B2C mailers higher in the ecosystem than spammers.

(It is less unusual in the discussion list side of things, but
that's not really where this is aimed at).

Cheers,
   Steve

---

From: "David Nicol" <[email-masked]>
Date: Tue, 6 Jun 2006 00:05:10 -0500

> that's not really where this is aimed at).

thank you.  DATAFIRST would certainly break VERP and mass customization,
short of making the early data some kind of template to be filled by information
accompanying the recipient address, like a label printing language,
which seems far too clever.

---

From: Steve Atkins <[email-masked]>
Date: Mon, 5 Jun 2006 22:22:51 -0700

On Jun 5, 2006, at 5:30 PM, Jim MacLeod wrote:

> (Please forgive me if I'm making a newbie suggestion.  I realize  
> that this is not The Universal Answer To The Spam Problem, If Only  
> Everyone Would Adopt It, but it seems like an interesting area for  
> research.)
>
> ...and that model is bad why?  Obviously, if implemented with FTP- 
> like back-channel connections, it would be a nightmare, but if  
> implemented with a well-known port and a bit of auth, I don't see  
> too many problems.
>
> Benefits:
> - it pushes the email burden back onto the sender - for real, not  
> like many anti-spam companies claim.
> - it would seem to reduce the spam-zombie problem by requiring that  
> inbound connections to the mail sender be available through the  
> firewall.

Did you take a look at RFC 1521 or RFC 2017?

Cheers,
   Steve

---

From: "william(at)elan.net" <[email-masked]>
Date: Tue, 6 Jun 2006 00:26:15 -0700 (PDT)

On Mon, 5 Jun 2006, Douglas Otis wrote:

> Actually, with the inclusion of the bh= header, the body can be checked 
> separately with DKIM.  The just the hash of the body is signed.

Better then nothing at all but this is still rather limiting in
actually referencing entire body data. Current email system allows
to reference external mime parts and email servers add new mime parts
as well.

>> however considering what I've seen I do not see that internet community 
>> (and more specifically email industry) would be willing to implement all 
>> necessary extensions and I think new email protocol is better way to go to 
>> really fix SMTP rather then several additional patches which are hard to 
>> fit within current infrastructure.
>

It is not about malware - there are actually ways to get rid of it that
are fairly effective even now (most malware is easy to identify based
on common signatures).

The reality is that im2000 is generally alot better approach for email
infrastructure where you do not expect that every email would be
delivered.... The advantages are that it can offer:
  1. More effective per-ip and per-domain filters which are similar
     to current URI filters but here URL is that of data store.
     SPF-like verification based on URL location starts to work
     with no forwarding problems for multiple identities.
  2. Zombies are no longer as effective as its necessary for such
     systems to be able to receive incoming connections and not
     only establish outgoing connections (as these systems must
     also be accessible through DNS).
  3. Integrated email tracking - sender knows if email has been
     picked up or not (possibly done all the way to MUA as IM2000
     wanted, but I consider it to be an option and not default) and
     sender can also cancel non-picked up message.
  4. Better multi-party delivery that is done on per-site basis
     (this is related to current thread - on per-site common emails
     to multiple recipients can have shared referenced body part(s)
     and those can be retrieved by the site once and can be reused).
  5. Email messages that have multiple alternative content (i.e.
     both HTML and text) is less wasteful on bandwidth as end-user
     can have setting that only certain types of content is to be
     retrieved. Similar works for messages with content in multiple
     languages (end-site may even communicate what language ise
     prefers like its done in HTTP). Also if end-site has policies
     that certain attachments are not allowed those attachments
     need not be retrieved.
  6. Other advanced security choices available once communication
     between end-site and sender is possible...

-- 
William Leibzon
Elan Networks
[email-masked]

---

From: Danny Angus <[email-masked]>
Date: Tue, 6 Jun 2006 09:17:37 +0100

> message would be held by the sender rather than by the intermediary

A suggestion has been made before (here I think defintely elsewhere) that a
temporary error response might be issued after RCPT (indeed after any
command) and before DATA which would allow the recipient MTA to carry out
asynchronous verification of the sender MTA and any other checks deemed
necessary without the cost of recieveing the message content.

In the existing protocol this will be followed eventually by a re-try which
can then be accepted or rejected permanently based upon the new knowedge we
have gained about the sender address and sending system. An enhancement to
the error codes could include a retry interval, hinting to the sender the
optimum wait for the best balance between futile retries and undue delay.

An alternative would be that the recipient MTA could issue an ETRN to the
sender to hint to the sender that it could send any outstanding messages to
this recipient.

An enhancement of (or replacement for) ETRN might make it mean that the
sender should send any mail _from_ a particular _sender_ which is due to be
routed through this host, as this is possibly a more logical grouping if we
assume that the command is going to be used to "unblock" temporary
restrictions on individual senders.

d.

***************************************************************************
The information in this e-mail is confidential and for use by the addressee(s) only. If you are not the intended recipient (or responsible for delivery of the message to the intended recipient) please notify us immediately on 0141 306 2050 and delete the message from your computer. You may not copy or forward it or use or disclose its contents to any other person. As Internet communications are capable of data corruption Student Loans Company Limited does not accept any  responsibility for changes made to this message after it was sent. For this reason it may be inappropriate to rely on advice or opinions contained in an e-mail without obtaining written confirmation of it. Neither Student Loans Company Limited or the sender accepts any liability or responsibility for viruses as it is your responsibility to scan attachments (if any). Opinions and views expressed in this e-mail are those of the sender and may not reflect the opinions and views of The Student Loans Company Limited.

This footnote also confirms that this email message has been swept for the presence of computer viruses.

**************************************************************************

---

From: der Mouse <[email-masked]>
Date: Tue, 6 Jun 2006 10:30:15 -0400 (EDT)

>> retrieve the rest of the data.

I like this, for normal mail transport.  I don't see it as being of any
particular anti-spam use, though; spam will simply be sent without
using it, the way it is today.  (Given all the usual deployment issues,
it'll be a long time before anyone but the most lunatic fringe can
afford to reject mail not sent using it.)

> A suggestion has been made before (here I think defintely elsewhere)
> that a temporary error response might be issued after RCPT (indeed
> after any command) and before DATA which would allow the recipient
> MTA to carry out asynchronous verification of the sender MTA and any
> other checks deemed necessary without the cost of recieveing the
> message content.

Except for the bit about putting a minimum delay on the retry, this
amounts to greylisting.  The hardest part here is telling whether a new
SMTP session is a retry of a previously-rejected one or a new message.

/~\ The ASCII				der Mouse
\ / Ribbon Campaign
 X  Against HTML	       [email-masked]
/ \ Email!	     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

---

From: Douglas Otis <[email-masked]>
Date: Tue, 06 Jun 2006 07:41:41 -0700

> to reference external mime parts and email servers add new mime parts

I see that as a plus.  The entire body would be retained at the signing
domain and accessed with a hash value (see below).

> are fairly effective even now (most malware is easy to identify based

But the filtering for the malware depends upon the DKIM signing domain.
The reputation of the DKIM signing domain must be good before attempting
to access the data section.  The other benefit of this approach is it
will not require any DSNs.  There is no need to bounce a message that is
never transferred. 

>      with no forwarding problems for multiple identities.

This is not a secure scheme.  SPF "like" verification has a rather
serious packet amplification problem as well.  Unlike IM2000, the entire
headers are included to allow verification of the DKIM signature.
Worrying about the Subject line becoming the spam is a secondary concern
at best, while also tossing out an ability to authenticate the message
when the subject line is part of the signature.  Any summary scheme for
the subject line could be part of the MUA.  Reputation prevents spam. 

>      also be accessible through DNS).

Zombies have no difficulty establishing their own (highly transitory)
DNS infrastructure.  That too offers little protection.

>      sender can also cancel non-picked up message.

This could work on the suggestion being made as well.

>      and those can be retrieved by the site once and can be reused).

The concept in mind was to leverage efforts being made by the T10 group
on object storage to encompass the entire message body. 

http://dl.alphaworks.ibm.com/technologies/osdsim/osdsim2.pdf
http://www.emc.com/products/networked/cas/index.jsp
http://www.lustre.org/

This product allows object storage where expiry of the data can be
included within the API, for example.  Information retrieval would be a
hash of the information together with the hash of the hash and the file
instance time stamp, as an analogy.

>      that certain attachments are not allowed those attachments

In most cases the bandwidth saved would be rather small for a fairly
high amount of complexity.  As the store remains at the sender, any type
of added authentication remains possible as separate elements are
retrieved (such as external MIME elements via RFC2017).  

>      between end-site and sender is possible...

Security would be part of any scheme.  DKIM is a good starting point.

-Doug

---

From: "william(at)elan.net" <[email-masked]>
Date: Tue, 6 Jun 2006 09:13:00 -0700 (PDT)

> But the filtering for the malware depends upon the DKIM signing domain.

Its not about DKIM or other signature scheme, you can identify malware 
right now quite effectively with existing tools and signatures I meant
are not crypto but virus pattern recognition signatures (which only need
to be tested against .exe and similar attachments). I'm also fairly
certain 99.999...% of email users have learned enough directly or
through friends to not open email attachments that need to be executed.
We'd have to wait for some research measurements, but I'd not be surprised 
if virus infection rate through email is significantly on decline.

> will not require any DSNs.  There is no need to bounce a message that is 

Yes and that is the main point of im2000 - don't get what you dont want.

With crypto signatures the overhead to calculate hash is actually a lot
smaller then that required to do RSA, so in reality the crypto effort
itself would be almost the same. But anything that can be done with 
non-crypto session authentication is going to have less overhead where
as cryptographic public-key signatures offer most advantages when two 
end-nodes are not directly talking to each other.

> This is not a secure scheme.  SPF "like" verification has a rather

SPF-like includes CSV is you prefer to hear that. And this has nothing
to do about your opinion on possible dns overhead with SPF which has
not shown up to be a problem, in fact this does not directly has to
do with SPF either. The scheme I mention verifies not the incoming
server connection but location of the message store based on store 
address (for which in fact RMX or CSV would be better as far as their
design but the reality is that people will use SPF anyway including
for this). Very low overhead, effective and no forwarding side-effects.

> DNS infrastructure.  That too offers little protection.

You're missing the point. Go work for ISP and you'll know that its a
lot easier to filter incoming ports end-users then it is to filter 
outgoing traffic for those users (a lot more complaints for the later).

Necessity for DNS infrastructure is secondary and serves as extra barrier
for spammers to overcome which BTW makes identifying zombies easier as 
you only need to monitor certain domains on regular basis (i.e. every 10
minutes for those "transitory" domains).

> on object storage to encompass the entire message body.

That is OT to email infrastructure design and in any case no such 
complexity is going to be there for most who use email.

> high amount of complexity.

Its the other way around actually especially as far as attachments go.

> Security would be part of any scheme.  DKIM is a good starting point.

Encumbered, oversimplified and limited by design - a lot worth then
it would have been, have it been a full IETF design effect like MASS
was supposed to provide for (which is how IETF usually operates,
i.e. designing protocol in open basis at WG). It is only now being
fixed and those fixes can not fully change some of the limitations
deliberately put in the WG charter.

---

What I said is valid for any im2000-like scheme and its just a matter
of how much of the advantages the sheme can provide (i.e. if it can
fully take advantage of retrieval architecture or has limitations
that only allow some features to be useful).

But in any case I'm not going to bother on this thread against only
Doug as otherwise it can go on forever with same arguments and no
benefits to anybody.

-- 
William Leibzon
Elan Networks
[email-masked]

---

From: Tony Finch <[email-masked]>
Date: Tue, 06 Jun 2006 18:17:40 +0100

>A new ESMTP extension DATA-CALLBACK.

You should use BURL for this.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
FISHER GERMAN BIGHT: WEST OR NORTHWEST 4 OR 5, OCCASIONALLY 6 IN NORTHEAST
FISHER. FAIR. MAINLY GOOD.

---

From: "Peter J. Holzer" <[email-masked]>
Date: Tue, 6 Jun 2006 19:39:04 +0200

> them has a misadjusted spam filter.

I agree that the case is rare, but nevertheless I think it is an
important case (I have implemented a plugin for qpsmtpd to allow
per-recipient content-filters - an ugly hack, but it seems to work).

There are cases where it isn't a misadjusted spam-filter (a mail
crossposted to a closed and an open mailing-list comes to mind), and
even where it is, one user's misadjusted spam filter shouldn't prevent a
different user from getting legitimate mail.

> reasons to send list mail one copy at a time.

I agree with that, but messages with lots of recipients are often not
sent through mailing-list software. People create "lists" in their
personal address books and simply send a mail to umpteen recipients.

> a foolish economy to try and reduce it even more.

Last time I looked at our site the smtp traffic was comparable to http
and quite ahead of the p2p protocols I know. I was quite surprised. I'll
have to measure that again soon.

But I agree that bandwidth isn't much of a problem. The problem is that
messages with dozens or hundreds recipients are transferred in a single
transaction and that the SMTP protocol currently doesn't offer any any
method to return per-recipient results after receiving the content.
Current solutions (like my cf_wrapper mentioned above) rely on temporary
failures, which may cause long delays if there are many recipients. 

So I think that some extension should be implemented. Like some others I
would prefer something based on the LMTP model, though. (But maybe
that's just because I already know LMTP)

	hp

-- 
   _  | Peter J. Holzer    | Ich sehe nun ein, dass Computer wenig
|_|_) | Sysadmin WSR       | geeignet sind, um sich was zu merken.
| |   | [email-masked]         |
__/   | http://www.hjp.at/ |	-- Holger Lembke in dan-am
_______________________________________________
Asrg mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/asrg

---

From: "william(at)elan.net" <[email-masked]>
Date: Tue, 6 Jun 2006 10:58:33 -0700 (PDT)

On Tue, 6 Jun 2006, Peter J. Holzer wrote:

> So I think that some extension should be implemented. Like some others 
> I would prefer something based on the LMTP model, though. (But maybe

What should really be done is new extension with command that replaces
DATA and allows parts of the data to be sent separate (i.e. message 
header) and allow per-recipient result codes when this new command
is used. I'll ping Hector as he has latest draft on it but it is not
quite what I wanted either.

-- 
William Leibzon
Elan Networks
[email-masked]

---

From: "Peter J. Holzer" <[email-masked]>
Date: Tue, 6 Jun 2006 20:09:58 +0200

On 2006-06-05 20:59:47 -0400, Mark E. Mallett wrote:
> mere rearrangement of commands does take functionality away.

I agree that an LMTP-style solution has merit. There are however, some
details of LMTP which make it (IMHO) unsuitable for deployment on the
internet (in fact, RFC 2033 warns of this, too).

1) It requires a different greeting from the client (LHLO instead of
   EHLO). A client cannot detect an LMTP server from the response to
   EHLO. This doesn't fit into the ESMTP extension model.

2) It changes the meaning of DATA and BDAT. In the ESMTP extension model
   there is no way for the client to tell the server which extension it
   supports, so the server cannot change the meaning of standard
   commands.

Therefore I propose an ESMTP extension which does the following:

The server announces that it supports the extension with a keyword in
the reply to EHLO. I propose MULTI-RCPT-DATA.

The client may then use the command MDAT instead of DATA. MDAT behaves
the same as DATA in LMTP. 

Open to question:

 * Is it worthwhile to specify an analog of BDAT? BDAT doesn't seem to
   be widely implemented.

 * While we are specifying a replacement for DATA: Would it be
   worthwhile to split the DATA into header and body? 

   	hp

---

From: Douglas Otis <[email-masked]>
Date: Tue, 6 Jun 2006 11:22:02 -0700

On Jun 6, 2006, at 9:13 AM, william(at)elan.net wrote:
> Its not about DKIM or other signature scheme, you can identify  
> malware right now quite effectively with existing tools and  
> signatures I meant are not crypto but virus pattern recognition  
> signatures (which only need to be tested against .exe and similar  
> attachments). I'm also fairly certain 99.999...% of email users  
> have learned enough directly or through friends to not open email  
> attachments that need to be executed. We'd have to wait for some  
> research measurements, but I'd not be surprised if virus infection  
> rate through email is significantly on decline.

When the data is stored at the sender, no intermediary is able to  
"filter" this information based upon signatures.  Many of today's  
viruses are also polymorphic and change rapidly.  Client side  
filtering and precautions are required using this mode of delivery,  
placing a greater onus on the signing domain and the recipient.   
Parents still expect that their child will not damage their  
computer.  Parental controls will need to limit access to specific  
trusted domains.  DKIM verification clearly identifies who is being  
trusted.  Business communications still place message security as a  
high priority which can also use the trust model.

>> The reputation of the DKIM signing domain must be good before  
>> attempting to access the data section.  The other benefit of this  
>> approach is it will not require any DSNs.  There is no need to  
>> bounce a message that is never transferred.
>
> With crypto signatures the overhead to calculate hash is actually a  
> lot smaller then that required to do RSA, so in reality the crypto  
> effort itself would be almost the same. But anything that can be  
> done with non-crypto session authentication is going to have less  
> overhead where as cryptographic public-key signatures offer most  
> advantages when two end-nodes are not directly talking to each other.

Using the existing email infrastructure has several advantages.  A  
transition to a store at the signing domain is appealing, but using  
SMTP for notification is not point to point.  A few milliseconds per  
message signature checks does not appear to be an impediment.

> SPF-like includes CSV is you prefer to hear that. And this has  
> nothing to do about your opinion on possible dns overhead with SPF  
> which has not shown up to be a problem, in fact this does not  
> directly has to do with SPF either. The scheme I mention verifies  
> not the incoming
> address (for which in fact RMX or CSV would be better as far as  
> their design but the reality is that people will use SPF anyway  
> including for this). Very low overhead, effective and no forwarding  
> side-effects.

When this is a transition using SMTP for notification, leveraging off  
of a suitable authentication scheme remains an important  
consideration.  While SPF advocates often ignore possible packet  
amplification exploits, major networks have been brought to their  
knees by an DDoS exploit running at about 60x amplification.  As it  
is now, SPF could easily achieve 300x while also thwarting ongoing  
efforts aimed at preventing the current exploit of the day.  As more  
applications attempt to utilize SPF, this figure gets even more  
frightening. : 0

Of course, a URL could be required to use a host name which would  
exceed the protections of an SPF IP address authorization scheme, but  
without the serious risks.  Within the underlying transfer protocol,  
simply require the use of host names.  : )

> You're missing the point. Go work for ISP and you'll know that its  
> a lot easier to filter incoming ports end-users then it is to  
> filter outgoing traffic for those users (a lot more complaints for  
> the later).

There is still the issue of notification.  Subsequent to sending the  
notification, obtaining the message body is the means to reach past  
firewall protections.

> Necessity for DNS infrastructure is secondary and serves as extra  
> barrier for spammers to overcome which BTW makes identifying  
> zombies easier as you only need to monitor certain domains on  
> regular basis (i.e. every 10 minutes for those "transitory" domains).

Agreed, which is why SPF is such a bad idea.  SPF never assures the  
controlling domain is identified, making this scheme worse than  
worthless in many cases.  A reputation check should be made prior to  
accessing the message content, and when verifying the DKIM signature.

> complexity is going to be there for most who use email.

When attempting to provide supporting infrastructure, these types of  
object storage systems removes a major amount of the required  
development.  The number of messages and maintenance overhead  
overwhelm a traditional file system or database.  Why not take  
advantage of existing tools?  When distribution of such a messaging  
system is not queue based, the alternative must be considered when  
done at a massive scale.  Object storage has been engineered to  
address many of these issues, such as preventing replicate storage  
base upon content, handing automated archival and removal, etc.

> Its the other way around actually especially as far as attachments go.

But an attachment would not be part of the display selection as you  
suggested.  The attachment could simply be a MIME extension as  
suggested.

> deliberately put in the WG charter.

Cheer up.  You have good ideas.  There is something to be said for a  
simplified approach however.

-Doug

---

From: Douglas Otis <[email-masked]>
Date: Tue, 6 Jun 2006 11:24:51 -0700

> You should use BURL for this.

An IMAP server could leverage this extension.  I had imagined the  
split would happen at the body, but the exchanged reference could  
repeat the entire header and message.

-Doug

---

From: Tony Finch <[email-masked]>
Date: Tue, 06 Jun 2006 20:15:25 +0100

>   supports, so the server cannot change the meaning of standard

Wrong: the client indicates that it wants to use an extension either by
issuing a command defined by the extension or by using the extension's
MAIL or RCPT parameters.

For an LMTP-like extension it would be appropriate to define a MAIL
extension parameter which changes the semantics of the subsequent
DATA command (including variants like BDAT and BURL).

In addition to that I think it would be more tasteful to change LMTP
slightly so that it follows the SMTP one-response-per-command model
more closely. It would also be nice to make per-recipient confirmations
optional if the server configuration doesn't require them (e.g. all
recipients have the same security preferences). That is:

* The SMTP service extension is called "recipient checking after data"

* The EHLO keyword associated with the extension is RECHECK, which has
no parameters.

* There is one new parameter for the MAIL verb, "RECHECK", and the
maximum length of the MAIL command is extended by 8 characters.

* There are no new parameters for the RCPT verb.

* There is one new verb, "RECHECK".

The extension is used as follows:

When the client starts a transaction with MAIL FROM:... RECHECK, the
server MAY respond to some or all of the transaction's RCPT commands
with a "350 Recipient pending" reply. (Other replies to RCPT have the
same semantics as usual.)

If the message data is not accepted by the server (a 4xx or 5xx reply),
the client MUST issue a RSET command. The server SHALL respond to any
pipelined RECHECK commands with a "503 bad sequence of commands" reply.
After the message data has been accepted by the server (a 250 reply),
the transaction is not complete until the client has re-checked all the
pending recipient addresses and then issued a RSET command. The client
MUST issue a RSET command even if there are no pending recipients.

The client re-checks a recipient address by issuing the command
  "RECHECK" SP ("<Postmaster@" domain ">" / "<Postmaster>" / Forward-Path)
                    [SP Rcpt-parameters] CRLF
The parameters MUST be the same as in an earlier RCPT TO: command. If
they do not match, the server SHALL respond with a "503 Bad sequence of
commands" reply. If the server responded to the earlier RCPT command with
a 2xx, 4xx, or 5xx reply, it SHALL respond to the RECHECK command with the
same reply. If the server responded to the earlier RCPT command with a
350 reply, it SHALL respond with a 2xx, 4xx, or 5xx reply as appropriate.

The client MUST re-check all pending recipients and MAY re-check other
recipients. It MAY issue the RECHECK commands in any order. If the
client does not re-check every pending recipient before resetting the
transaction, the server MUST NOT deliver the message to the unchecked
pending recipients.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
RATTRAY HEAD TO BERWICK ON TWEED: SOUTH 3 OR 4 BECOMING VARIABLE 3 OR LESS.
SHOWERS LATER. MODERATE OR GOOD, OCCASIONALLY POOR LATER. SMOOTH OR SLIGHT.

---

From: Tony Finch <[email-masked]>
Date: Tue, 6 Jun 2006 20:23:41 +0100

On Tue, 6 Jun 2006, Douglas Otis wrote:
> An IMAP server could leverage this extension.  I had imagined the split would
> happen at the body, but the exchanged reference could repeat the entire header
> and message.

BURL isn't necessarily tied to IMAP, and the URL it carries doesn't
necessarily have to refer to the whole of the message data: you can use
BDAT chunks to transfer other fragments of the message, and they do not
have to share any higher-level boundaries.

Or (as has already meen mentioned) you could use MIME external-body parts.

However, in both cases postmasters are unlikely to want to use these
extensions because they make content security scanning harder, not easier.
(MIME external bodies are already frequently blocked.)

I really don't see any advantages to the IM2000 idea. Here's Jim MacLeod's
list of IM2000 advantages:

- it pushes the email burden back onto the sender - for real, not like
many anti-spam companies claim.

What burden? Spamware generates the junk email on the fly.

Moreso than the web site URLs in existing spam?

- it would seem to reduce the spam-zombie problem by requiring that
inbound connections to the mail sender be available through the firewall.

Greylisting is already effective.

We already have that option: 550 no thanks, sonny-jim.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
FORTIES CROMARTY FORTH TYNE DOGGER: SOUTH OR SOUTHWEST BECOMING VARIABLE 3 OR
4, OCCASIONALLY 5 AT FIRST. SHOWERS. MODERATE OR GOOD, WITH FOG PATCHES LATER.

---

From: "David Nicol" <[email-masked]>
Date: Tue, 6 Jun 2006 14:44:28 -0500

Okay, how does this grab you:

The new extension is called "BULK" and it introduces a new keyword,  "TEMPLATE"
which, when invoked, accepts a template, which lasts until a RSET is received.
When there is an active template, DATA sections contain key/value pairs for
filling the template.  The envelope return path, for VERPing, can be changed
with a MAIL FROM without resetting the template or data.  RCPT TO causes
a local delivery attempt and per-recipient result using the current template and
data.

in bulk mode, we don't care about the ordering of MAIL FROM and TEMPLATE
and DATA but they all must precede RCPT TO, and they all persist between
recipient specification.

For pipelining purposes, the receiving end can stack up as many of these
commands as it cares to, with the sending end forcing processing with a FLUSH
command or a RSET.

in the absence of a TEMPLATE command appearing, it's business as usual.

High-volume B2C peers might be given special treatment -- like getting allowed
to send more messages from one IP than otherwise would be allowed --
when using the BULK delivery method.

---

From: "David Nicol" <[email-masked]>
Date: Tue, 6 Jun 2006 15:02:07 -0500

On 6/6/06, Douglas Otis <[email-masked]> wrote:
>  Why not take
> advantage of existing tools?

if I understand you correctly, we should not take advantage of existing
tools due to the possibility of packet amplification exploits.

---

From: "David Nicol" <[email-masked]>
Date: Tue, 6 Jun 2006 15:52:53 -0500

point by point responses below.

On 6/6/06, George Schlossnagle <[email-masked]> wrote:
> Why should a reciever expend the resources to provide templatization of
> senders' mail?

receivers coddle B2C senders in order to keep them better behaved.  Less
bandwidth is used by templated mass-customized mail than would be by
fully expanded mail.  Rate limits can be imposed (and exceptions sold.)  The
receivers offer templatization as a carrot to encourage B2C bulk senders
migrating to the new method.

> Also, if I were sending such mail, I would be concerned
> that it leaves my control in a non-final state - seems a huge liability
> issue on both sides of the transaction.

well-defined templatization can be considered a form of compression.  Would
you be concerned about losing control if you knew that your message was
going to pass through gzip and then, later, gunzip?  (as an aside I think there
are compressed MIME types but no ZIP SMTP extension defined at this
time, without doing any research on that point, although I expect TLS might
offer some compression along with the encryption)

by leaving out the variable replacement placeholders, whatever they will be --
I'm not envisioning full-blown TAL here -- a TEMPLATE with no DATA
would amount to the previous proposal, with the addition of the possibility
of changing the envelope return address if so desired.

By filling most of your templates at the sending side and taking hash
sigs, you could
even fill your [HASH_SIGNATURE] slot with the result.

---

From: George Schlossnagle <[email-masked]>
Date: Tue, 06 Jun 2006 16:13:29 -0400

Why should a reciever expend the resources to provide templatization of 
senders' mail?  Also, if I were sending such mail, I would be concerned 
that it leaves my control in a non-final state - seems a huge liability 
issue on both sides of the transaction.

George

---

From: George Schlossnagle <[email-masked]>
Date: Tue, 06 Jun 2006 17:11:13 -0400

> migrating to the new method.

I don't buy this at all.  I don't know of any receivers that I would 
classify as 'coddling' large b2c senders.  In general receivers don't 
need to be offering carrots to large b2c senders, the stick seems to 
work fine; there's been mainly stick (or future threat of stick) around 
setting up DK/DKIM signing and adoption of that has been working well.

> offer some compression along with the encryption)

The differences in regards to content creation responsibility, interop, 
liability for cretaed content etc. between doing templatization and 
doing compression are hopefully obvious.

George

---

From: Douglas Otis <[email-masked]>
Date: Tue, 6 Jun 2006 15:11:13 -0700

On Jun 6, 2006, at 1:02 PM, David Nicol wrote:
>> Why not take advantage of existing tools?
>
> if I understand you correctly, we should not take advantage of  
> existing tools due to the possibility of packet amplification  
> exploits.

The recommendation for object storage was recognizing a transmitting  
domain needs to allow random access to a sizable array of data  
objects.  This is not a simple problem to scale.  The advantages of  
such a solution would be improved security, provide optional privacy  
without directly encrypting the message, eliminate DSNs, and a reduce  
network bandwidth when tighter size limits are placed upon header  
only notification while 80% of the email is spam.  Consider the SPF  
warning added to enforce that made by IESG, in addition to being  
categorized as experimental.

-Doug

---

From: Seth Breidbart <[email-masked]>
Date: Tue, 6 Jun 2006 18:43:12 -0400 (EDT)

"David Nicol" <[email-masked]> wrote:

> The new extension is called "BULK" and it introduces a new keyword,
> "TEMPLATE" which, when invoked, accepts a template, which lasts
> until a RSET is received.

What problem is this intended to solve?

It saves bandwidth, which isn't expensive.

If spam processing can be done on the template, it saves CPU (compared
to multiple transmissions), but templates just provide another way to
obfuscate spam.

It seems to me the issue is deferring the response to RCPT TO until
after the message body is received; that's independent of any other
change.

Seth

---

From: "David Nicol" <[email-masked]>
Date: Tue, 6 Jun 2006 17:56:49 -0500

> categorized as experimental.

a http server with directory indexing turned off and files given hard-to-guess
long names and a policy of tarpitting peers that repeatedly attempt to
access nonexistent files can scale using standard load-balancing techniques,
for what that's worth.

-- 
David L Nicol

---

From: John Levine <[email-masked]>
Date: 6 Jun 2006 23:12:28 -0000

>migrating to the new method.

Since your original assumption, that saving bandwidth is important, is
false, the whole plan kind of falls apart doesn't it?

There are plenty of ideas floating around about carrots and sticks that
receivers can use to persuade senders to behave better, but offloading
template expansion is so far off the radar that metaphors fail me.

R's,
John

---

From: "David Nicol" <[email-masked]>
Date: Tue, 6 Jun 2006 18:17:21 -0500

> setting up DK/DKIM signing and adoption of that has been working well.

Sorry, I guess I got the verb tense wrong there. That present-tense
assertion was meant to be taken subjunctively.  "The proposal would make
sense in a possible future in which receivers wish to manage B2C senders
with carrots as well as sticks" is closer to what I meant.  By giving B2C
senders their own special protocol, they could be effectively rate-limited,
and billed.  B2C e-mail is big business, and AFAIK the stick approach is
just making all involved frustrated and unhappy.  With a carrot approach,
a receiving domain would have less of their incoming bandwidth taken up
by B2C traffic and make the B2C people carry their weight when they want
to use more.

Maybe the BULK extension would start with some sender accounting identification,
before issuing templates.

> doing compression are hopefully obvious.

As are the similarities.  Are you seriously arguing that "templatization is a
form of compression" is a false statement?  Have you ever written a driver
for an industrial label printer?

Really, about liability -- does anyone have any links to case law concerning
templatization and common-carrier issues?  A lot of web pages anymore are
created with dynamic client-side tools, and there is no confusion
that, for instance,
the images presented by google maps are fully the responsibility of google and
not my responsibility because the image was knit together by their
program running
on a computer plugged into my wall, or that a the responsibility for a
cross-site-scripting
attack lies with the perpetrator of the attack rather than with the victim.

Thanks very much for the feedback on my off-the-cuff ideas; the amount of typing
I've done on this thread today (large) is far out of proportion to the
weight of my
convictions on the issues (small).

---

From: Douglas Otis <[email-masked]>
Date: Tue, 6 Jun 2006 17:16:49 -0700

On Jun 6, 2006, at 3:56 PM, David Nicol wrote:

> a http server with directory indexing turned off and files given  
> hard-to-guess long names and a policy of tarpitting peers that  
> repeatedly attempt to access nonexistent files can scale using  
> standard load-balancing techniques, for what that's worth.

Consider the numbers involved.  While a header only message may be a  
bit quicker to send, retaining these messages for the next 4 weeks to  
permit MUA access to the data dramatically increases the number of  
messages that are made accessible.  A large ESP might send 100,000  
messages per minute.  A transmit-side retention scheme running at  
that scale would then need access to 4 G objects.  This is something  
a typical Unix filesystem can handle.  On the other hand, a content  
addressable system split between headers and body should dramatically  
reduce the amount of storage needed to archive this outbound data by  
eliminating all redundant copies.  This scheme also needs to age and  
archive this information at a rate of 100,000 per minute.  While  
special applications could be created to replicate these features,  
these would be normal features of this type of CAS storage system.   
Little to no extra work is needed to do all the right things.  While  
this may seem like added complexity, this approach should  
significantly reduce the amount of information retained while still  
ensuring complete records.  A reduction of copies also helps with  
filtering.

By having the unique token used to access the information, there is  
also an accurate record as to when the message was received.  The  
recipient would have a SHA256 hash of the body to assure accurate  
delivery as well, that can also be retried when needed.  When used  
for commerce, this would provide better transparency while also being  
able to ensure greater privacy when needed.  This seems to offer some  
real benefits for the typical corporate enterprise.

-Doug

---

From: "william(at)elan.net" <[email-masked]>
Date: Wed, 7 Jun 2006 12:28:05 -0700 (PDT)

I'm not a fan of what you proposed with separate keyword just to
check status. I'd much rather have this as separate DATA commands
if done at all. If not done as part of DATA (which I do not see
as being worth the effort), then the way to go is to add an extension
with RCPT parameters, so for example:

250-helo example.com, pleased to meet you
...
250-RCPTSTATUS
250 HELP
MAIL FROM:<[email-masked]>
250 <[email-masked]> - sender ok
RCPT TO:<[email-masked]> STATUS
250 <[email-masked]> - recipient ok
RCPT TO:<[email-masked]> STATUS
250 <[email-masked]> - recipient ok
RCPT TO:<[email-masked]>
250 <[email-masked]> - recipient ok
DATA
[header and email body goes here]
[email-masked] ok
[email-masked] mailbox full try again later *
250 OK

[* I don't remember correct code for this and don't have time to lookup]

The last line is the one for those addresses not specificaly mentioned 
before, i.e. those addresses not listed with STATUS at RCTP line.

However as I noted before, I think doing it with new DATA command
which has this new type of status by default like LMTP would provide
more benefits to the userbase to make worth such deployment.

---

From: Tony Finch <[email-masked]>
Date: Wed, 07 Jun 2006 22:16:38 +0100

"william(at)elan.net" <[email-masked]> wrote:
>if done at all.

That doesn't work because there isn't an extension mechanism for
adding parameters to the DATA command, so it means your extension
must duplicate all the other DATA-like commands, i.e. DATA, BDAT,
BURL, and you must revise your spec whenever a new DATA-like
command is introduced. This is not a good long-term strategy.

Your suggestion of a STATUS parameter on the RCPT command is not
bad, but the problem is that the client doesn't know which recipients
have differing content-filter options, so it doesn't know which RCPT
commands do or do not require the parameter.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
NORWEGIAN BASIN: SOUTH VEERING SOUTHWEST 5 TO 7, OCCASIONALLY GALE 8 LATER IN
FAR NORTHWEST, DECREASING 4 IN FAR SOUTH. RAIN THEN SHOWERS. MODERATE OR POOR,
BECOMING MAINLY GOOD.

---

From: "william(at)elan.net" <[email-masked]>
Date: Wed, 7 Jun 2006 14:55:09 -0700 (PDT)

> command is introduced. This is not a good long-term strategy.

We're only talking about SMTP (as in multi-network email transport),
not LMTP which has different set of commands for data and already has
ways to deal with this. As I noted I personally feel that DATA command 
needs to be extended for SMTP use like it was done for other protocols 
derived from SMTP.

BTW, that reminded me (including of the conversation just a few months 
ago) that I do see a need to add parameters to new DATA command, in fact 
in the way I saw it the first such parameter would be what data it is 
(i.e. header of body part).

I also have to note that nothing really prevents us from actually adding 
parameters to DATA command and extending it this way (well maybe except 
RFC ABNF, but we know how that worked from RFC821 to 2821) within context 
of SMTP extension framework. You could say that if server advertises in 
EHLO certain extension (like say "EXTENDEDDATA'), it means it the server
understands extended DATA syntax and client could choose to do:

EHLO example.com
250-mail.example.org, please to meet you
250-EXTENDEDDATA
250 HELP
MAIL FROM:<[email-masked]>
250 <[email-masked]> sender ok
RCPT TO:<[email-masked]>
250 <[email-masked]> recipient ok
RCPT TO:<[email-masked]>
250 <[email-masked]> recipient ok
DATA RCPTSTATUS PART=HEAD
354 Enter email HEAD part and end with '.' line
From: [email-masked]
To: [email-masked]
Cc: [email-masked]
.
350-<[email-masked]> unable to deliver at this time, queue not available
250 <[email-masked]> message part accepted, please send the rest
DATA RCTPSTATUS PART=BODY ENDDATA
354 Enter email BODY part and end with '.' line
Hi There, you seem to have a broken link on your homepage.
.
250 <[email-masked]> message accepted for delivery
QUIT

But I also quite easily see that maybe new command (EDATA) would be better
to make it clear that its basically an extended command (like we did when
moving from HELO to EHLO).

> commands do or do not require the parameter.

Some smtp clients may have means of finding which users have different 
'content-filter options', plus I  see use and non-use of STATUS for RCTP 
in this case as being based on recipient domains (i.e. if you send it
to secondary or to primary mx for a domain and other parameters), but
I'd suspect that most who want this will just add STATUS to all RCTP 
commands - its not a big deal or a problem.

-- 
William Leibzon
Elan Networks
[email-masked]

---

From: Tony Finch <[email-masked]>
Date: Wed, 07 Jun 2006 23:04:45 +0100

"william(at)elan.net" <[email-masked]> wrote:
>ways to deal with this.

No, SMTP and LMTP have exactly the same commands and extensions.

>DATA RCPTSTATUS PART=HEAD

You can already send a message in multiple chunks using BDAT.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
LUNDY FASTNET IRISH SEA SHANNON: VARIABLE 3 OR 4, BECOMING SOUTHEAST 4 OR 5 IN
FASTNET AND SHANNON. MAINLY FAIR. MODERATE OR GOOD WITH FOG PATCHES.

---

From: "william(at)elan.net" <[email-masked]>
Date: Wed, 7 Jun 2006 15:45:53 -0700 (PDT)

> No, SMTP and LMTP have exactly the same commands and extensions.

Perhaps I should have been more clear that its not just LMTP.
RFC4468 says that BURL 'extends the message submission protocol
on port 587 and MUST NOT be advertised by regulsr server on port 25'

But ok, perhaps RCTPSTATUS should not go with DATA and stay as
RCTP keyword if we want all existing DATA commands supported.

> You can already send a message in multiple chunks using BDAT.

I don't see people supporting CHUNKING and I'd really like to see
more clear separation of what separation of what these chunks are.

An alternative to what I wrote is what some others prefer as in
specialized HEAD command (which is in fact what HTTP did for such
an extension).

-- 
William Leibzon
Elan Networks
[email-masked]

---

From: mathew <[email-masked]>
Date: Fri, 09 Jun 2006 09:42:33 -0500

william(at)elan.net wrote:
> delivered...

I'll just throw in my own writeup of IM2000 benefits, which is at 
<URL:http://meta.ath0.com/articles/2006/01/22/spam2>

> afford to reject mail not sent using it.)

True, but you can use lack of IM2000 as another indicator of likely 
spamminess.

Douglas Otis wrote:
> "filter" this information based upon signatures.

That's not true. There are HTTP proxies that filter for malware.

> simply require the use of host names.  : )

If it were me, I'd require HTTPS and an SSL certificate. Anyone wanting 
to run their own mail server can afford $30 a year.

> What burden? Spamware generates the junk email on the fly.

The bandwidth burden for starters. And I suspect it's not easy to set up 
a web server that can take a few million hits just after 9am on a Monday 
morning, still remain anonymous, and change its domain name, IP address 
and SSL certificate each spam run.

> Moreso than the web site URLs in existing spam?

Plenty of spam doesn't have web URLs.

mathew