Subject: Re: [EAI] Re: for8

From: John C Klensin <[email-masked]>
Date: Fri, 19 Jan 2007 20:18:24 -0500

--On Friday, 19 January, 2007 21:00 +0100 Frank Ellermann
<[email-masked]> wrote:

> Harald Alvestrand wrote:
> 
>   [<for> inherits <mailbox>]
>>> And smtpext-02 in fact updates <mailbox> (or rather its 2821
>>> <Mailbox> counterpart), what am I missing here ?
> 
>> The fact that downgraded UTF8SMTP messages will expect to be
>> RFC2822 (no extensions) compatible.
> 
> That limits the "for8" problem to removing the optional <for>
> clause in the downgrade procedure.

But, as Harald mentions, we've got a fairly firm rule against a
given server tampering with trace fields inserted by other
servers.  That rule is based on long and hard experience.  One
could declare this to be the exception and go back and remove
earlier "for" fields, but, speaking personally, I'm not
convinced that the value of the "for" field and the importance
of this case are sufficient to justify breaking that rule.

It also introduces the requirement that systems be able to parse
earlier trace fields, which we have never before required.
There is actually a reason for that -- while trace fields
inserted by SMTP servers have a predictable format and syntax,
it is not possible to know much about fields inserted by
gateways and it is even less feasible to be able to predict the
exact format of trace fields inserted before a message crosses
the gateway.

Again, we could make a whole series of assumptions and the
depart from those rules, but it is both harder and riskier than
it might appear on first glance.

>> And the fact that altering trace headers is considered a
>> Very Bad Thing.
> 
> Forbidding <for> everywhere in UTF8SMTP is also very bad.  One
> of the UTF8SMTP axioms is that downgrading will rarely happen.
> 
> I've my doubts about this axiom.  But if we use it for some
> decisions we can as well use it consistently whenever it helps:
> "remove the optional <for> if necessary" is no rocket science.

Actually, given all of the gateway constraints, it might require
magic, which is often much harder than rocket science.  See
above.

    john

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 20 Jan 2007 04:13:44 +0100

John C Klensin wrote:

> as Harald mentions, we've got a fairly firm rule against a
> servers.  That rule is based on long and hard experience.

A downgrading MTA is expected to parse the complete header
for its job, stripping offending <for> clauses while it's at
it would be a rather simple part of this task.

> It also introduces the requirement that systems be able to
> parse earlier trace fields, which we have never before required.

I understand your reasons, but downgrading MTAs are forced to
parse the complete header anyway when they try to create an
ASCII message/rfc822 surrogate of the message/utf8smtp header.

Frank

---

From: Randall Gellens <[email-masked]>
Date: Mon, 22 Jan 2007 12:41:36 -0800

>  of this case are sufficient to justify breaking that rule.

It seems we have two unpleasant choices:

1: Prohibit "for" clauses when the address is non-ASCII.
2: Alter trace headers when downgrading.

The basic direction of the group, as I understand it, assumes that 
mail systems will fairly rapidly be upgraded to handle UTF8, and 
tries to avoid creating short-term kludges which will inevitably live 
on long after everyone has upgraded.

We may be better off with choice 2.  Choice 1 means that as systems 
are upgraded, "for" clauses vanish.  I've found these clauses to be 
very helpful when debugging things.
-- 
Randall Gellens
Opinions are personal;    facts are suspect;    I speak for myself only
-------------- Randomly-selected tag: ---------------
Democracy is a form of government that substitutes election by the
incompetent many for appointment by the corrupt few.
                                                      --G. B. Shaw

---

From: John C Klensin <[email-masked]>
Date: Mon, 22 Jan 2007 20:07:59 -0500

--On Monday, 22 January, 2007 12:41 -0800 Randall Gellens
<[email-masked]> wrote:

>>  But, as Harald mentions, we've got a fairly firm rule
>>  against a given server tampering with trace fields inserted
>>  by other servers.  That rule is based on long and hard
>>  experience.  One could declare this to be the exception and
>>  go back and remove earlier "for" fields, but, speaking
>>  personally, I'm not convinced that the value of the "for"
>>  field and the importance of this case are sufficient to
>>  justify breaking that rule.
> 

Yes, I think that summarizes the problem.

> The basic direction of the group, as I understand it, assumes
> that mail systems will fairly rapidly be upgraded to handle
> UTF8, and tries to avoid creating short-term kludges which
> will inevitably live on long after everyone has upgraded.
> 
> We may be better off with choice 2.  Choice 1 means that as
> systems are upgraded, "for" clauses vanish.  I've found these
> clauses to be very helpful when debugging things.

Yes, although less useful than they were before security issues
and the text of 2821 discouraged using them except under very
restricted conditions.

That conclusion makes me very, very nervous --as does every step
we take that moves downgrading closer to requiring the category
of activities that 2821 permits only for gateways-- but I can
live with it if that is the group's decision.  Since I am
working on the I-D for 2821bis concurrently with the EAI effort
--and with a target date of getting 2821bis-01 posted really
soon now-- if there is text there that I need to fine-tune,
suggestions would be welcome.

Even then, the concerns that drive the nervousness suggest that,
if we are going to alter earlier trace fields on downgrading,

(i) It might be better to just drop "for" clauses that contain
non-ASCII material rather than figure out some creative way to
encode them.... possibly replacing them with a comment whose
semantics are closer to "non-ASCII for clause dropped on
downgrading" than to an attempted encoding of the prior address.

(ii) we might want to require the MTA that does the downgrading
to insert a very specific trace header that explains the damage
it did.  Any comments that contain encodings of prior addresses
should probably be on that trace header, not retrofitted into
headers that did not contain information in that form.

Or maybe not... it could be that I'm unhappy enough about this
to not be thinking clearly enough.

     john

---

From: "Yao Jiankang" <[email-masked]>
Date: Tue, 23 Jan 2007 11:06:16 +0800

----- Original Message ----- 
From: "John C Klensin" <[email-masked]>
To: "Randall Gellens" <[email-masked]>; "Frank Ellermann" <[email-masked]>; <[email-masked]>
Sent: Tuesday, January 23, 2007 9:07 AM
Subject: Re: [EAI] Re: for8

> downgrading" than to an attempted encoding of the prior address.

Figuring out some creative way to encode them is not good. It seems that dropping them is also unpleasant choice.
Is it possible to use ASCII escapes for Unicode characters to represent it if non-ASCII material is included in the "for" clauses?

YAO Jiankang
CNNIC

---

From: John C Klensin <[email-masked]>
Date: Tue, 23 Jan 2007 07:59:01 -0500

--On Tuesday, 23 January, 2007 11:06 +0800 Yao Jiankang
<[email-masked]> wrote:

> Figuring out some creative way to encode them is not good. It
> seems that dropping them is also unpleasant choice. Is it
> possible to use ASCII escapes for Unicode characters to
> represent it if non-ASCII material is included in the "for"
> clauses?

ASCII escapes for Unicode characters are nothing more than a
different "creative way to encode..."

    john

---

From: Randall Gellens <[email-masked]>
Date: Tue, 23 Jan 2007 10:27:54 -0800

At 8:07 PM -0500 1/22/07, John C Klensin wrote:

>  live with it if that is the group's decision.

There is one good thing about this approach: as downgrading becomes 
less common, the unpleasant behavior also becomes less common.  In an 
environment where UTF8SMTP messages either successfully go through or 
are bounced and resent as all-ASCII, there won't be any of this 
mucking around with messages.

>  downgrading" than to an attempted encoding of the prior address.

I agree: deleting the "for" clause is safer than encoding just the 
"for" clause.  It might be good to both delete the "for" clause and 
put the entire header inside a downgrade-* header (as per downgraded 
draft).

>  headers that did not contain information in that form.

I agree -- this is an excellent requirement and should help debug the 
inevitable problems.

-------------- Randomly-selected tag: ---------------
Premature optimization is the root of all evil
                              --C. A. R. Hoare

---

From: Chris Newman <[email-masked]>
Date: Thu, 25 Jan 2007 21:27:30 -0800

> headers that did not contain information in that form.

The robustness gained by never altering Received headers greatly exceeds the 
value of the "for" clause (for is a convenience -- the relevant MTA logs are 
the only place where the canonical data can be found).  However, regardless of 
what the specification says, we can't usefully stop email originators from 
putting a UTF-8 address in the "for" clause of Received.  A robust MTA has to 
deal with that case anyway.  So I'd say three things about this issue:

1. SHOULD NOT generate for clause with UTF-8 address: forcing systems to alter
    received headers is bad
2. MUST remove a Received for clause with a UTF-8 address on downgrade: to avoid
    breaking systems further down the line
3. MUST NOT include an encoded version of the UTF-8 address in the
    existing Received header but MAY include it in a Downgraded-details trace
    field to minimize risk of damage to Received headers.

                - Chris

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 26 Jan 2007 13:32:43 +0100

Chris Newman wrote:

> 1. SHOULD NOT generate for clause with UTF-8 address: forcing systems
>    to alter received headers is bad

You could as well say "sending UTF8SMTP to MTAs not supporting it forces
them to try some 'downgrade' magic, and therefore the senders SHOULD NOT
use UTF8SMTP in the first place".   But that's obviously not the idea of
this experiment...

> 2. MUST remove a Received for clause with a UTF-8 address on downgrade:
>    to avoid breaking systems further down the line

...yes.  Downgrade is such a complex procedure that adding this minor
detail as MUST won't make it worse.

>    existing Received header but MAY include it in a Downgraded-details
>    trace field to minimize risk of damage to Received headers.

The "MUST NOT" part is already covered by your second point, the draft
could still state why that probably won't work:  The encoding would have
to guarantee that it can't be confused (syntactically) with a real ASCII
addresss.  In theory that's possible, e.g. Google Groups use "..." in a
munged local part.  In practice it fails because spammers and some MTAs
don't care about any IETF fine print declaring adjacent dots as illegal.

Frank

---

From: John C Klensin <[email-masked]>
Date: Fri, 26 Jan 2007 08:35:47 -0500

--On Thursday, 25 January, 2007 21:27 -0800 Chris Newman
<[email-masked]> wrote:

>> (ii) we might want to require the MTA that does the
>> downgrading to insert a very specific trace header that
>> explains the damage it did.  Any comments that contain
>> encodings of prior addresses should probably be on that trace
>> header, not retrofitted into headers that did not contain
>> information in that form.
> 
> The robustness gained by never altering Received headers
> greatly exceeds the value of the "for" clause (for is a
> convenience -- the relevant MTA logs are the only place where
> the canonical data can be found).  However, regardless of what
> the specification says, we can't usefully stop email
> originators from putting a UTF-8 address in the "for" clause
> of Received.  A robust MTA has to deal with that case anyway.
> So I'd say three things about this issue:
> 

This seems about right to me.

FWIW, if I recall, the text in 2821 was rather carefully written
to permit additional trace header types to be added.  So, while
it is hard to extend Received and its subfields, adding a
Downgraded-details (or equivalent) trace field is fairly easy
from a syntax and definitional standpoint.   The bad news is
that I believe that there are a lot of spam filters out there
that treat messages in which anything but Received headers
appears between the first Received header and the last one as
suspicious.  The contents of those filters are completely
outside the EAI scope or the 821/2821/ 2821bis specs, but they
may be significant in practice.

To paraphrase a comment I made in another note, adding new
headers (at the specification level) is never free, or even
cheap.

    john

---

From: John C Klensin <[email-masked]>
Date: Fri, 26 Jan 2007 09:02:03 -0500

--On Friday, 26 January, 2007 13:32 +0100 Frank Ellermann
<[email-masked]> wrote:

> Chris Newman wrote:
> 
>> 1. SHOULD NOT generate for clause with UTF-8 address: forcing
>> systems to alter received headers is bad
> 
> You could as well say "sending UTF8SMTP to MTAs not supporting
> it forces them to try some 'downgrade' magic, and therefore
> the senders SHOULD NOT use UTF8SMTP in the first place".   But
> that's obviously not the idea of this experiment...

Frank, I don't understand this comment.  I can't even parse it
completely, i.e., I don't know whether "them" refers to the
sending or receiving server.   

If it means what I think it might, you are suggesting that, if
one cannot have "for" with a UTF-8 address, then the whole idea
of UTF8SMTP is useless.  I don't believe that.  I suspect you
don't either.

Another interpretation is that downgrading is hopeless and, if
one encounters a legacy server while trying to transport
UTF8SMTP mail, one should simply deal with the message as
undeliverable.  I've made not secret that I've got some growing
sympathy for that position, but I think you are among the vast
majority who believe that downgrading is a good (and necessary)
mechanism, so you presumably don't mean that either.

If the comment is hyperbole, hyperbole does not help us make
progress here.

>...

    john

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 26 Jan 2007 18:20:48 +0100

>> that's obviously not the idea of this experiment...

[...]
> I don't know whether "them" refers to the sending or receiving
> server.

I meant "them" downgrade gateways.

> Another interpretation is that downgrading is hopeless

Now "hopeless" is a bit strong, let's say "difficult" with a
chance to screw up something (not limited to DKIM signatures).

> I think you are among the vast majority who believe that
> downgrading is a good (and necessary) mechanism, so you
> presumably don't mean that either.

That's why I think a "SHOULD NOT use for8" like a "SHOULD NOT
use UTF8SMTP" makes no sense:  Either the downgrade works as
expected (1), or the mail is rejected (2), or the complete
route is UTF8SMTP-ready (3), or for a few remaining cases the
downgrade somehow fails miserably (4).

For (1), (2), and (3) there's no "for8" problem justifying a
SHOULD NOT.  And in the hopefully rare (4) cases it's anyway
hopeless, the presence or absence of a "for8" won't change
the outcome.  If a downgrade fails only because the required
removal of "for8" didn't work, then I propose to fix the MTA,
instead of a "SHOULD NOT use for8" for everybody in (1)..(4).

Frank

---

From: Tony Finch <[email-masked]>
Date: Fri, 26 Jan 2007 19:14:36 +0000

On Fri, 26 Jan 2007, John C Klensin wrote:
> to permit additional trace header types to be added.

However the syntax for trace fields in 2822 is not extensible.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
LUNDY FASTNET IRISH SEA: NORTHWESTERLY 4 OR 5. MODERATE OR SLIGHT. SHOWERS AT
FIRST THEN MAINLY FAIR. MODERATE OR GOOD.

---

From: Frank Ellermann <[email-masked]>
Date: Fri, 26 Jan 2007 21:23:07 +0100

> However the syntax for trace fields in 2822 is not extensible.

RFC 2822 refers to RFC 2821 for a "full discussion".  RFC 2822 also
says that they are strictly informational for its own purposes (the
Internet Message Format), e.g. in RFC 3834 that's not the case.

And there is an additional trace header field defined in RFC 4408.
IMO ugly like hell, but registered as 3864 "permanent header field".

DKIM-base defines the second new trace header field since RFC 822.
In the 2822upd-00 draft it's clearer that defining new trace header
fields is allowed.

Frank

---

From: John C Klensin <[email-masked]>
Date: Fri, 26 Jan 2007 15:37:31 -0500

--On Friday, 26 January, 2007 19:14 +0000 Tony Finch
<[email-masked]> wrote:

>> FWIW, if I recall, the text in 2821 was rather carefully
>> written to permit additional trace header types to be added.
> 

Shame on Pete :-(.  If that is the case, belongs on the list of
things in 2822 that need fixing to align it with 2821.

    john

---

From: John C Klensin <[email-masked]>
Date: Fri, 26 Jan 2007 17:28:16 -0500

--On Friday, 26 January, 2007 21:23 +0100 Frank Ellermann
<[email-masked]> wrote:

> RFC 2822 refers to RFC 2821 for a "full discussion".  RFC 2822
> also says that they are strictly informational for its own
> purposes (the Internet Message Format), e.g. in RFC 3834
> that's not the case.
> 
> And there is an additional trace header field defined in RFC
> 4408. IMO ugly like hell, but registered as 3864 "permanent
> header field".
> 
> DKIM-base defines the second new trace header field since RFC
> 822. In the 2822upd-00 draft it's clearer that defining new
> trace header fields is allowed.

Frank, thanks for this clarification, but let me try to put one
thing in perspective.   Registration according to RFC 3864, even
registration as "permanent" doesn't imply any special status
other than "someone else is using this name, please pick
something else for your next bright idea".  Status and authority
come from the documents or protocols to which 3864 and the
registry refer, not from being listed.  So, for example, a
header registered by 4408 (an experimental document) implies a
name that probably should not be reused, but doesn't imply that
anyone needs to recognize or support that header unless they are
part of the experiment.

In any event, the conclusion I infer from your comments is
clearly correct: we do make new trace headers, but we don't do
it very often.

best,
   john

---

From: Frank Ellermann <[email-masked]>
Date: Sat, 27 Jan 2007 00:08:41 +0100

John C Klensin wrote:

> Registration according to RFC 3864, even registration as 
> "permanent" doesn't imply any special status other than 
> "someone else is using this name, please pick something
> else for your next bright idea".  Status and authority
> the registry refer, not from being listed.

Yes, the registration template notes the status (in the case
of RFC 4408 now "experimental", the proposal was "standard")
and the change controller (now "IETF", updated following the
the status changes).

> doesn't imply that anyone needs to recognize or support
> that header unless they are part of the experiment.

Yes, but of course I'm interested in such details for users
wishing to participate in both experiments (EAI and SPF):

Downgrading a Received-SPF header field is not as simple as
"remove any 'for8'".  The envelope-from= value is important
(as far as this trace header field is important at all).

> we do make new trace headers, but we don't do it very often.

2006-1982=24 ;-)

Frank

---

From: Tony Finch <[email-masked]>
Date: Sun, 28 Jan 2007 20:57:50 +0000

On Fri, 26 Jan 2007, John C Klensin wrote:
> On Friday, 26 January, 2007 21:23 +0100 Frank Ellermann <[email-masked]> wrote:
> > Tony Finch wrote:
> >
> > RFC 2822 refers to RFC 2821 for a "full discussion".

I believe that's because 2821 has rather more details about Received: and
Return-Path: than 2822 does. However 2822's header ABNF and the
accompanying text is quite explicit about what can be a trace field.

> clearly correct: we do make new trace headers, but we don't do

Right. As well as the ones Frank noted, 2822 also defines new trace fields
(because it says that Resent- fields are trace fields which isn't the case
in 822). So 2822bis needs to make them properly extensible.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
ROCKALL MALIN: NORTHWEST 4 OR 5 BACKING WEST 5 TO 7. MODERATE OR ROUGH.
OCCASIONAL RAIN. MODERATE OR GOOD.

---

From: John C Klensin <[email-masked]>
Date: Sun, 28 Jan 2007 19:56:59 -0500

--On Sunday, 28 January, 2007 20:57 +0000 Tony Finch
<[email-masked]> wrote:

> Right. As well as the ones Frank noted, 2822 also defines new
> trace fields (because it says that Resent- fields are trace
> fields which isn't the case in 822). So 2822bis needs to make
> them properly extensible.

That leaves us something else we need to straighten out sooner
or later.  While it isn't explicit, the 2821 definition of
"trace field" is "something inserted by the MTA to trace
progress or handling of a message".  To the extent to which
Resent-* is supplied by an MUA as part of some MUA-level
process, they aren't trace fields in the SMTP/2821 sense.

This is obviously not a problem for this WG, but Pete and I
probably need to talk.

    john

---

From: Tony Finch <[email-masked]>
Date: Mon, 29 Jan 2007 10:44:11 +0000

> process, they aren't trace fields in the SMTP/2821 sense.

However they do trace progress or handling of a message, so interleaving
Resent- fields with Received: fields is the right thing. 2822 allows
multiple resendings to be expressed unambiguously whereas 822 does not.

Tony.
-- 
f.a.n.finch  <[email-masked]>  http://dotat.at/
VIKING NORTH UTSIRE SOUTH UTSIRE FORTIES: NORTHWEST 5 TO 7, OCCASIONALLY GALE
8 AT FIRST, BACKING WEST 4 FOR A TIME. VERY ROUGH DECREASING ROUGH. OCCASIONAL
RAIN. GOOD.

---

From: John C Klensin <[email-masked]>
Date: Mon, 29 Jan 2007 07:16:05 -0500

--On Monday, 29 January, 2007 10:44 +0000 Tony Finch
<[email-masked]> wrote:

> However they do trace progress or handling of a message, so
> interleaving Resent- fields with Received: fields is the right
> thing. 2822 allows multiple resendings to be expressed
> unambiguously whereas 822 does not.

Understood.  That is part of what needs to be clarified, since
2821 sort of assumes that, when an MUA gets and handles a
message, the Received and Return-path fields will be removed
before handoff back to an MTA or MSA.  I really don't see this
as a problem, just need to be sure that the relevant text in the
two documents is aligned.

   john

---

From: Frank Ellermann <[email-masked]>
Date: Mon, 29 Jan 2007 16:05:17 +0100

> This is obviously not a problem for this WG, but Pete and I

There are more minor issues for a future 2822bis:  a pending
erratum, and some USEFOR folks think that the Message-ID 
syntax has to be cleaned up to reflect common practice and
what's stated in two drafts (and several older RFCs).

Frank