Subject: Re: [EAI] Consensus Result on Issue #1 - Parameter on MAIL

From: John C Klensin <[email-masked]>
Date: Wed, 02 Feb 2011 11:03:42 -0500

Hi.

Because I really don't have any strong feeling about this issue
other than:

	(i) When 5336bis is revised to reflect the parameter, it
	must be absolutely clear and precise about what the
	parameter means.  That implies specifying exactly both
	when a client is expected (or required) to supply the
	parameter and how a server is expected to interpret its
	presence (or absence).
	
	(ii) We need to keep this as simple as possible (but, in
	Wirth's words, no simpler).   I think that an
	arrangement involving either multiple parameters or a
	single parameter with multiple alternate values should
	require a lot of justification.  While we may be looking
	at the issue differently, I think that aligns me with
	those who oppose lots of "finicky rules that
	implementers will ignore".

All comments below are strictly personal opinions.

One observation to try to find out whether we are really one the
same page, then a suggestion to see if we can converge.  

(1) --On Monday, January 31, 2011 20:59 +0000 Shawn Steele
<[email-masked]> wrote:

> Message routing shouldn't be
> dependent on information that isn't within the message.

Interesting observation because we do it all the time and it is
the norm rather than the exception.   Note that we route
messages based on addresses in the envelope and that there is no
requirement whatsoever that those addresses appear in the
headers.  For the very common case of distribution from mailing
lists, the forward-pointing envelope (delivery) addresses rarely
appear in any header field.  Or did you mean something else?

(2) Over the years, there have been many occasions with email
specifications on which we could agree on the right thing to do
even if we could not agree on the reasoning or theory to justify
that choice.   I think what I'm hearing (again, personal opinion
and proposal, not a consensus declaration) is that some text
along the lines of the following might be acceptable to most of
us as requiring the parameter where it is required but not
trying to impose a requirement for high-quality client behavior:

	If the envelope or message being sent requires the
	capabilities of these EAI extensions, the client MUST
	supply the UTF8SMTPbis parameter with the MAIL command.

	If the client is aware that the envelope or message
	being sent does not require any of the EAI extension
	capabilities, it SHOULD NOT supply the UTF8SMTPbis
	parameter with the MAIL command.  This specification
	does not require that the client inspect the message or
	otherwise go to extraordinary lengths to assure itself
	whether these EAI capabilities are required for the
	particular message.  However, in mixed or transitional
	environments in which EAI is not completely deployed,
	more precision on the part of the client (sender) may
	prevent unnecessary message rejections.  

The exact text should be determined to match the style of the
editors, but please note that the text above talks about "EAI
capabilities" and does not use any terminology specifically
associated with coded character sets, encodings, or language.

Is there any chance that would work for people?  And, if not,
what do you think needs to be changed and why.

     john

---

From: "Charles Lindsey" <[email-masked]>
Date: Wed, 02 Feb 2011 16:25:34 -0000

On Mon, 31 Jan 2011 20:59:51 -0000, Shawn Steele  
<[email-masked]> wrote:

>> Yes it DOES matter. Because the message would then be forwarded ONLY  
>> over
>> routes that advertise UTF8SMTPbis, and there might not exist such a  
>> route
>> to the intended destination, in which case it would bounce.
>
> I disagree.  If the message could be sent in ASCII and reach the  
> destination, then I have no problem with that message reaching the  
> destination if it happened to be encoded in UTF-8 but only used the  
> lower 7 bits.  In fact, the only way to tell the difference is if  
> someone explicitly included additional information, not in the message  
> itself, that the message was "supposed" to be UTF-8 or ASCII.  Message  
> routing shouldn't be dependent on information that isn't within the  
> message.

But that is not how our drafts are written. If the message claims (rightly  
or wrongly) to be a UTF8SMTPbis message, then MTAs MUST NOT forward it to  
any server that does not advertise UTF8SMTPbis in its EHLO response. If an  
MTA can find some alternative server that is believed to route it to its  
final destination via some all-UFT8SMTPbis path, then good luck to it.  
Otherwise it gets bounced. So our protocol DOES specify that the presence  
of this parameter may well affect the routeing.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: John Levine <[email-masked]>
Date: 3 Feb 2011 02:32:46 -0000

>	prevent unnecessary message rejections.  

OK by me, reflects both good practice and the limits of what we can
demand that coders do.

R's,
John

---

From: Dave CROCKER <[email-masked]>
Date: Wed, 02 Feb 2011 19:08:32 -0800

> 	prevent unnecessary message rejections.

So, the goal is to guarantee that no environment can ever be certain of 
operating in a simplified, "pure" UTF-8 mode?

As phrased, this language requires permanent support for legacy ASCII mode.

d/

-- 

   Dave Crocker
   Brandenburg InternetWorking
   bbiw.net

---

From: Troy Starr <[email-masked]>
Date: Wed, 2 Feb 2011 19:29:50 -0800

Hi John -

> 	prevent unnecessary message rejections.

I agree with the MUST in the first paragraph and the SHOULD NOT in the second paragraph.

In the first sentence of the second paragraph, I don't think we want "or" since you would only omit the UTF8SMTPbis parameter if neither the envelope nor the message required EAI capabilities.  So I would use a neither/nor combination, or just switch the "or" to "and" to make it clear that both conditions must be satisfied to omit the UTF8SMTPbis parameter.

I'm a little iffy about "This specification does not require that the client inspect the message or otherwise go to extraordinary lengths to assure itself whether these EAI capabilities are required for the particular message."  This sentence might suggest that I don't need to be sure that this is an EAI message, and therefore I don't need to add the UTF8SMTPbis parameter to the MAIL command.  Even though we had the MUST in the first paragraph, I think this could create some ambiguity.  I think we could make this clearer by stating that the client isn't required to verify that this isn't an EAI message.  Something along the lines of:

This specification does not require that the client inspect the message or otherwise go to extraordinary lengths to assure itself that EAI capabilities are not required for the particular message.

- Troy Starr

---

From: John Levine <[email-masked]>
Date: 3 Feb 2011 04:47:35 -0000

>As phrased, this language requires permanent support for legacy ASCII mode.

I'm having trouble understanding the objection here.  In the far off
day when everyone does EAI, since ASCII messages are a subset of UTF-8
messages, a server that does not care about the distinction between
them can ignore the flag and treat all messages as EAI, and a client
that doesn't care can set the flag on everything.

Unless you're proposing a server "send EAI or else" flag, I don't see
what you want to change.

R's,
John

---

From: Claudio Allocchio <[email-masked]>
Date: Thu, 3 Feb 2011 09:22:46 +0100 (CET)

>> 	supply the UTF8SMTPbis parameter with the MAIL command.

+1

>> 	parameter with the MAIL command.

+1 to keep alive an ASCII only legacy world forever.

"MAY NOT" if we wish the UTF-8 environment gradually become the legacy 
situation. It is not a nit, it is a phylosophical assesment.

>>      This specification

mmmhhhh... it is an attempt to explain why, but... do we really to phrase 
it like this? I share aother comments doubts here.

------------------------------------------------------------------------------
Claudio Allocchio             G   A   R   R          [email-masked]
                         Senior Technical Officer
tel: +39 040 3758523      Italian Academic and       G=Claudio; S=Allocchio;
fax: +39 040 3758565        Research Network         P=garr; A=garr; C=it;

            PGP Key: http://www.cert.garr.it/PGP/keys.php3#ca

---

From: "Charles Lindsey" <[email-masked]>
Date: Thu, 03 Feb 2011 10:24:08 -0000

> As phrased, this language requires permanent support for legacy ASCII  

Then I suggest s/it SHOULD NOT supply .../
       it MAY omit the UTF8SMTPbis parameter with the MAIL command.
       Indeed it is advisable to do so for reasons yadda yadda yadda/

Where the "Indeed ..." sentence might be better as a NOTE, and the "yadda  
yadda yadda" should cover the "extraordinary lengths" and "mixed or  
transitional environments" stuff, suitably reworded.

---

From: "Yao Jiankang" <[email-masked]>
Date: Thu, 3 Feb 2011 18:40:52 +0800

----- Original Message ----- 
From: "John C Klensin" <[email-masked]>
To: <[email-masked]>
Sent: Thursday, February 03, 2011 12:03 AM
Subject: Re: [EAI] Consensus Result on Issue #1 - Parameter on MAIL

> parameter means. 

+1

> Wirth's words, no simpler).
>   

+1

I strongly support the two principles above.

Jiankang Yao

---

From: John C Klensin <[email-masked]>
Date: Thu, 03 Feb 2011 08:31:33 -0500

--On Thursday, February 03, 2011 04:47 +0000 John Levine
<[email-masked]> wrote:

>> So, the goal is to guarantee that no environment can ever be
>> certain of  operating in a simplified, "pure" UTF-8 mode?
>> 
 
> I'm having trouble understanding the objection here.  In the
> far off day when everyone does EAI, since ASCII messages are a
> subset of UTF-8 messages, a server that does not care about
> the distinction between them can ignore the flag and treat all
> messages as EAI, and a client that doesn't care can set the
> flag on everything.

Or, put differently, a server that does not care can ignore the
flag, making it completely irrelevant what the client does.  So,
if the client happens to know and care, it is careful about
sending the flag and, if it has been updated to know that no
server cares any more, it just sends it always.

That said, we've still got clients and servers who are operating
in pure 821 mode, i.e., sending only HELO and accepting only
HELO.  A few of the clients aren't even spammers or bots.
That has been 18 years and, while I think we are in the tail of
that curve, we aren't going to see the end of the asymptote any
time soon.  Remembering that clients have to support pure-ASCII
mode (and, for the older cases, pure-821 mode) to have any hope
or getting mail to such servers and that servers have to support
it (independent of whatever "only spammers are that retarded"
scores they assign) in order to have any hope of receiving mail
from such clients, I just can't visualize the point at which it
will be possible to remove all code for ASCII-only situations.

> Unless you're proposing a server "send EAI or else" flag, I
> don't see what you want to change.

Indeed.
    john

---

From: John C Klensin <[email-masked]>
Date: Thu, 03 Feb 2011 08:54:19 -0500

--On Thursday, February 03, 2011 09:22 +0100 Claudio Allocchio
<[email-masked]> wrote:

>...
> "MAY NOT" if we wish the UTF-8 environment gradually become
> the legacy situation. It is not a nit, it is a phylosophical
> assesment.

I don't think it is a nit, but I was trying to summarize what I
saw on the list (I may not have gotten that right).  I do see
this as an operational issue: especially in the short-term
environment in which there are going to be lots more
non-EAI-capable servers around than EAI-capable ones, being
careful about that parameter will make the difference between
whether ASCII-only (i.e., 5321/5322 conforming) messages are
delivered or rejected.   It seems to me to be clearly
advantageous to have more, rather than fewer messages delivered
and to avoid unnecessary rejections.

Remember that messages that are claimed to require EAI MUST NOT
be sent unless the server advertises the capability.  So, if a
relay receives a message that appears to require EAI (parameter
sent to it on the MAIL command) and the next hop doesn't
advertise the capability, that relay must either 

	- reject the message as stated to require EAI
	functionality but having no host to which it can be
	delivered
	
	- scan the envelope and message to determine whether it
	can safely drop the flag
	
	- risk violating 5321/5322 by sending a message that
	potentially requires the EAI extensions without getting
	a server OK for using those extensions.

Whatever me might think of the third, we cannot specify it as an
acceptable option.  The choice between the first two seems to me
to be an implementation or operational choice, not one that we
should try to specify in a standard (but I can easily imagine a
"how sure do you want to be before sending the parameter" knob
for locally tuning an implementation).

In environments in which clients know that all, or substantially
all, servers support EAI, MAY may be more appropriate but, if
their users communicate outside that environment, they will
still probably want to do the checking and be careful about what
functions they claim they require.

My own personal sense is that we are better off with SHOULD NOT
now.  When these specs are revised or updated in the future, the
state of the network and EAI deployment should be assessed and
the language changed to MAY (and the explanation retuned) as
needed.  While I think it would confuse more than it would help,
one could also add an explanation of how the operational reading
of "SHOULD NOT" might evolve with increasing EIA deployment.

> mmmhhhh... it is an attempt to explain why, but... do we
> really to phrase it like this? I share aother comments doubts
> here.

FWIW, it was intended less as an explanation of why than as an
relatively brief explanation of when one would want to ignore
the SHOULD.    I was remembering an earlier comment from Dave
asking why one would not want to send the parameter and
therefore why the statement would not be "EAI-capable clients
talking with EAI-capable servers MUST always send the
parameter".   He and I are in agreement that SHOULD statements
should (sic) almost always be accompanied of an explanation of
when it was reasonable to make an exception, so I was trying to
supply one.

    john

p.s. I did not intend my language to be inserted into the
document, so nit-picking it (again, I don't consider the
SHOULD/MAY distinction to be a nit) is probably not useful.  If
we can agree on the intent, let's see what the authors come up
with.

---

From: Claudio Allocchio <[email-masked]>
Date: Thu, 3 Feb 2011 15:12:35 +0100 (CET)

> of "SHOULD NOT" might evolve with increasing EIA deployment.

yes, SHOULD NOT is better for current situation, and MAY NOT will become 
more appropriate with time. And we can always change it during the 
document updates.

> when it was reasonable to make an exception, so I was trying to

ok. But the language at first read did not seem to convey that (at least 
to me...). that was my objection.

---

From: John C Klensin <[email-masked]>
Date: Thu, 03 Feb 2011 11:07:14 -0500

--On Thursday, February 03, 2011 15:12 +0100 Claudio Allocchio
<[email-masked]> wrote:

>...
>> My own personal sense is that we are better off with SHOULD
>> NOT now.  When these specs are revised or updated in the
>> future, the state of the network and EAI deployment should be
>> assessed and the language changed to MAY (and the explanation
>> retuned) as needed.  While I think it would confuse more than
>> it would help, one could also add an explanation of how the
>> operational reading of "SHOULD NOT" might evolve with
>> increasing EIA deployment.
> 
> yes, SHOULD NOT is better for current situation, and MAY NOT
> will become more appropriate with time. And we can always
> change it during the document updates.

I don't know how others in the WG --especially those who are
likely to have to do the work-- feel about this but my
preference is that we try to get a complete and competent
protocol spec out as quickly as we can now, focusing on what
people should do in the short term/ early transition phrase as
appropriate.  We should then move to revise as soon as we have
implementations and deployment experience.  In other words, I
think we will need to revise in a few years or less anyway and
should not spend energy now trying to make sure we have all the
language exactly correct.

Again, that is mostly personal opinion, but it is more
consistent with the objective to get things done quickly than a
lot of fine-tuning.

>> when it was reasonable to make an exception, so I was trying
>> to supply one.
> 
> ok. But the language at first read did not seem to convey that
> (at least to me...). that was my objection.

On second reading, I agree that might not be clear enough.

Let's see if we can reach agreement on intent (so far, Dave's
objection is the only one I remember seeing and both John L and
I have wondered about its implications) and then have the
authors/editors propose specific language... which we can then
tune to the extent needed.

best,
   john

---

From: Bill McQuillan <[email-masked]>
Date: Thu, 3 Feb 2011 09:42:28 -0800

> 	a server OK for using those extensions.

It seems to me that this raises another concern for the maximum reliability
of message propagation. The fact that, while the message itself will not
change with respect to whether it requires EAI of not, The *envelope* may
as it progresses from hop to hop.

For instance, a mailing list message with ASCII only content is to be
distributed to a number of subscribers, some of which have EAI addresses. A
simple-minded mailer would send MAIL FROM: with the EAI flag set and then a
bunch of RCPT TO: commands with a few containing EAI-requiring addresses.
Note that as the message is forwarded to subsequent MTAs, eventually some
of the message copies will have only recipients with ASCII addresses. Who
is responsible for noting that the EAI flag is no longer needed?

Maybe the correct way to handle this is to have the original mailer
segregate the EAI-requiring recipients into a separate transaction. Is it
appropriate for this spec to mention this advice?

-- 
Bill McQuillan <[email-masked]>

---

From: Dave CROCKER <[email-masked]>
Date: Thu, 03 Feb 2011 12:40:37 -0800

Folks,

> If the envelope or message being sent requires the capabilities of these EAI
> extensions

Algorithmically, what does this language mean?  How will an implementation know 
that it "requires" that it must invoke the extension?  As used here, "requires" 
is actually not a clear and precise meaning.  The challenge in using this style 
of specification language is that it implies calculations and criteria that are 
not explicitly stated. It even hints at knowing 'intent'.

By comparison, note the simplification of saying

      If the envelope or message contain UTF-8 text, then
      this extension MUST be used.

To the extent that the criterion listed in that alternative text is not
sufficient, then add the other concrete, direct and simple criteria that are
needed.  For example:

      If the envelope or message contain UTF-8 text or if the client
      anticipates the need for UTF-8 support in command replies, then
      this extension MUST be used.[1]

> If the client is aware that the envelope or message being sent does not
> require any of the EAI extension capabilities, it SHOULD NOT supply the
> UTF8SMTPbis parameter with the MAIL command.

Effectively, this permanently excludes 0x7f and below from ever being handled as
UTF-8, except by going outside of this specification.

As I understand the goal of this working group, the simplifications sought by
the working group are to permit direct handling of UTF-8.  That is, to make
UTF-8 'native' to email.  By saying that 0x7f and below are not to be handled as
native UTF-8, this text requires ALL implementation to forever support both
ASCII and UTF-8 mechanisms, rather than ever being able to support only UTF-8.
Much of the benefit of 'pure' handling of UTF-8 is thereby lost.

In fact, for places wishing to operate in a pure-UTF-8 mode, this encourages
violating the specification.

The issue is not some 'far off day' when no one is using ASCII and a pure-UTF-8
mode is possible, but the present day, where gateways can handle the
translation, if sites wish to operate that way.[2]  The limitations of such a
mode are real, but so are the benefits.  The current proposed language places
such an operation outside the scope of the specification.  Doing an important
infrastructure specification in a way that encourages its violation is typically
counter-productive.[3]

If indeed folk expect to have ASCII support present in all systems forever, then
it would have been far easier and safer to have Unicode support be defined as a
UTF-4 (or somesuch equivalent) /overlay/ to the existing ASCII infrastructure,
since it avoids ever having to worry about damage from 'raw' UTF-8 escaping into
the ASCII world.

Some folk seem to be postulating a future time when this extension will be
deprecated as if that will be easy to do, with a quick and easy transition.
Another thought that has been expressed is for later changing the SHOULD NOT to
something milder or specifying some varying interpretation of the SHOULD NOT.

We've already seen notes that postulate some presence of ASCII forever.  It is
also clear that specifications which invite ambiguity -- that is, variable
interpretation -- invite non-interoperability.  So it would be useful to folks
who are relying on later changes to explain the likelihood, timing and effort
for these changes.  I do not recall any historical precedent upon which we can
base any predictions, other than, probably... never.  At the least, my guess is
that predictions need to be in terms of decades.

Alternative language seems as if it will supply adequate guidance, without
prohibiting a pure-UTF8 environment:

      If the envelope and message contain only ASCII, then
      the client MAY decline to use the UTF8SMTPbis extension.

Ned Freed cited benefits in staying in ASCII-only mode where possible.  Adding
language that cites those benefits would be helpful, here.  So:

           NOTE: Because there is no guarantee that a next-hop SMTP server will
                 support the UTF8SMTPbis extension, use of the extension always
                 carries a risk of transmission failure.  Hence there is a
                 distinct advantage for ASCII-only messages to be sent without
                 using this extension.

d/

[1] It occurs to me that it might be helpful to define an SMTP response -- or at
least some proposed text to include in ASCII responses -- along the lines of "I
have UTF-8 data to return to you, but you did not provide a UTF8SMTPbis
parameter that permits me to include it"...

[2] Note that messages might "escape" their pure environments without going
through a proper gateway.  However by using this extension (forever), the damage
from that anomoly is small; or rather it is isolated to the environment with the
configuration option error that permitted the escape, rather than imposing a
UTF-8 message on a site which is unable to deal with it.

[3] When considering who will have the implement this extension, in order to
achieve global success, please remember that it is not merely some of the folk
in this working group.  It is every person who writes SMTP code around the
world, which includes a rather large number of programmers who are not present
for this discussion and will not know of this group's implied interpretations.

-- 

   Dave Crocker

---

From: Dave CROCKER <[email-masked]>
Date: Thu, 03 Feb 2011 13:16:01 -0800

On 2/3/2011 5:31 AM, John C Klensin wrote:
> flag, making it completely irrelevant what the client does.

I do not understand what that sentence means.  Please clarify.

What flag is the server to 'ignore' and what is the difference in behavior.  For 
that matter, if the server has advertised UTF8SMTPbis, how is it legal for the 
server to then fail to conform to the extension when requested to?

d/

-- 

   Dave Crocker

---

From: Dave CROCKER <[email-masked]>
Date: Thu, 03 Feb 2011 13:19:32 -0800

> language exactly correct.

Sorry.  I had completely missed the "few years or less" reference.

In terms of IETF and Internet timeframes, the expectation of having to make 
changes within a few years qualifies the work as "experimental".

d/

ps. There is a cliche "I did not have time to do the work correctly, but I will 
have plenty of time to fix it later.".  The cost of that model tends to be quite 
high.

-- 

   Dave Crocker

---

From: [email-masked]
Date: Thu, 03 Feb 2011 18:13:37 -0800 (PST)

> as it progresses from hop to hop.

And so can the header. List expanders routinely modify headers in various ways,
and this will include EAI material. (Purists don't like this of course, but
like it or not it's how things are done.) So a list expander may turn a non-EAI
message into an EAI message, or vice-versa, either through envelope
modification, header modification, or both.

> is responsible for noting that the EAI flag is no longer needed?

The answer to that is ... nobody. Removing the EAI flag from a message that no
longer needs it should be legal, but not required. What does need to be
required, however, is setting the EAI flag when a non-EAI message is converted
to an EAI one.

> appropriate for this spec to mention this advice?

As I tried to explain a couple of days back, I think the answer to this is
"no". The problem with trying to explain this particular tactic is it gets
really ugly really fast. For example, consider the case of a client attempting
to submit a message to a group of recipients, some EAI and some not, where
nothing in the message content requires EAI. The transaction part isn't the
problem - clients alreayd have to be able to split groups of recipients in
order to cope with recipient limits. No, the problem is what to put into the
header - if you list the EAI recipients in a recipient field, you've converted
the message to EAI irrespective of the type of recipient. So can you avoid
putting in the recipients? Well, that becomes a complex question of intent, of
what information the client has available, and even what sequence of operations
the client uses to constrcut the message (which varies wildly from one client
to another).

Issues with list expansion may be a little simpler, but nevertheless have the
same general flavor.

So I think the correct approach is to clearly define what the EAI flag means
and when it has to be set, and a suggestion as to when it should not be set.
The rest is going to be a matter of client quality. And if past experience in 
MIME is any indication, what we actually say about this behond the actual hard
restrictions is going to matter a lot less than we'd probably like to believe.

				Ned

---

From: Claudio Allocchio <[email-masked]>
Date: Fri, 4 Feb 2011 09:36:14 +0100 (CET)

> changes within a few years qualifies the work as "experimental".

and this is not experimental, however...
>
> d/
>
> ps. There is a cliche "I did not have time to do the work correctly, but I 
> will have plenty of time to fix it later.".  The cost of that model tends to 
> be quite high.

... however, the difference here, IMHO, is that we are trying to drive (or 
guess) deployment trend of UTF-8 mail world, describing how the 
transistion should work into a standard specification.

Do we agree that the goal is to maximise mail messages successful 
deliveries?

is so:

  - Shall we go for the "quick deploymnet scenario"? then it is a MAY.
  - otherwise it's a SHOULD.

In any case I do not think we can guess it... even most major sw 
developers are here on this list... there are so many other 
implementations around, which will just "read" what we tell them and 
hopefully do it right.

PS: sorry for the "MAY NOT", it' of course wrong text. :-)

> _______________________________________________
> IMA mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ima
>

---

From: John C Klensin <[email-masked]>
Date: Fri, 04 Feb 2011 08:06:37 -0500

--On Thursday, February 03, 2011 13:19 -0800 Dave CROCKER
<[email-masked]> wrote:

>> should not spend energy now trying to make sure we have all
>> the language exactly correct.
> 
> 
> In terms of IETF and Internet timeframes, the expectation of
> having to make changes within a few years qualifies the work
> as "experimental".

> ps. There is a cliche "I did not have time to do the work
> correctly, but I will have plenty of time to fix it later.".
> The cost of that model tends to be quite high.

Dave,

First, we have been through "experimental".  Complete specs were
published, implementations were produced, tests were performed,
we learned some things from the experiments, and work now should
be focused on the needed changes/ improvements from those
original documents.  The long-term participants in the WG are at
least somewhat justified in taking the position that we should
not revisit _any_ of the issues that were discussed at length in
the first round of work (the one that produced the experimental
specs) and that were not shown to be problematic in testing
unless significant new arguments are introduced.  

Second and perhaps more important, please try to read what I
(and others) are actually writing rather than picking up on a
few words and going on the attack.  I am trying to avoid
speculating on your motivations, but I know that it is causing
significant delay to progress in the WG without, as far as I can
tell, improving protocol quality.  As examples, consider the
text you quoted above:

(i) I said "complete and competent protocol spec".  That has
nothing to do with "no time to do the work correctly" or
"experimental".  It is actually a higher standard than the "no
known technical omissions" requirement for Proposed Standard.
When combined with the history of publishing experimental
documents, having independent implementations, performing
interoperability tests on them, and learning from those tests,
it is a reasonable expectation that the protocols will be _much_
more developed --and that we will be much more sure of them--
than the IETF norm for Proposed Standards.

(ii)  Similarly, if you read all of "revise in a few years or
less anyway and should not spend energy now trying to make sure
we have all the language exactly correct" completely and in
context, it will be clear that I was not talking about the
quality or completeness of the protocol, but only about how very
fine details of the language in which the specifications are
written are sorted out.  

I also note that the intent for Proposed Standards, as described
in RFC 2026, is that documents remain at that level for between
six months and two years.   While the "update or expire"
provision of that document has proven ineffective, the
expectation of review and revision within that period is
completely consistent with my suggestion of publication,
implementation, review, and revision as necessary.  It is hard
to reconcile it with your apparent belief that any spec that is
expected to be revised within a few years is necessarily either
experimental or unstable/incomplete.

regards,
    john

---

From: "Charles Lindsey" <[email-masked]>
Date: Fri, 04 Feb 2011 13:59:41 -0000

>                  distinct advantage for ASCII-only messages to be sent  
Nice wording! Would work well together with some of the suggested "MAY"  
language.

---

From: Claudio Allocchio <[email-masked]>
Date: Sat, 5 Feb 2011 15:09:18 +0100 (CET)

Hi,

On Fri, 4 Feb 2011, John C Klensin wrote:

<cut>

> unless significant new arguments are introduced.

It seems you (Dave and John) are talking somhow about different thigns 
here. The experimental period has done the necessary experimentation in 
terms of technically working specification, but I do not know if it really 
"hit" too many times the interaction with the legacy email systems. As an 
example, I've many many correspondents which live in UTF-8 languages area 
(and indeed Italian is also UTF-8 to be exact!), I receive hundreds of 
emails per day... and I never saw one trying to implements the EAI specs.
Dave is advocating the effects of interactions between the EAI 
specifications and legacy systems, and this is a different, but also 
important, story, which was probably not addressed in the experimental 
period, or at least did not make enough challegnes.

We all are trying to fix also this second interaction, before moving to 
standards tracks the specification, and I personally believe we have 
identified a good path to do this.

We have identified that (ad least IMHO) that in order to be safe, UTF-8 is 
just not a simple harmless extension to legacy e-mail, but it is somehow a 
separte scenario which needs to coexist and correctly interact with legacy 
email. There is the concept of downgrading, and soft-gatewaying in this. 
This is why there is some re-focusing on some aspects, which we are 
discussing in the WG (see issue #1!).

> > Second and perhaps more important, please try to read 
what I > (and others) are actually writing rather than picking up on a

> experimental or unstable/incomplete.

as an opinion to all the above points... we are a bit diverging from the 
main topi...

- giving it is anyhow a "lusly separated" email system with downgrading 
and gatewaying;
- giving it also needs further specs to be added for this
- giving that we try to optimise the delivery of messages, in this mixed 
environment

then, let's be very pragmatic, and choose some way forward NOW. We are not 
talking about "wrong specs" which needs correction (but also in this case, 
we can just re-issue a Proposes Standard, instead of moving to Draft), we 
are guessing about deploymnent trends, there is no way to be 100% sure 
about it.

---

From: [email-masked]
Date: Sat, 05 Feb 2011 08:27:25 -0800 (PST)

> "hit" too many times the interaction with the legacy email systems.

That's an excellent point. I'll add that even if there was some interaction,
somwhere, the email world isn't exactly homogeneuous. Different geographies,
communities, and type of organizations do things very differently, and have
very different approaches to solving problems that come up.

When MIME first deployed, we were fooled any number of times into thinking we'd
seen all the variations out there.

> As an

FWIW, neither did I. And there was bound to be at least some leakkage had
the experiment truly been that widespread.

> period, or at least did not make enough challegnes.

Exactly. And let's not forget that the experiment was testing a much more
radical approach. It would have been foolish had the more radical aspects *not*
been the focus, as opposed to more mundane deployment considerations.

> identified a good path to do this.

I agree.

> > than the IETF norm for Proposed Standards.

Yes, but when the goals are as ambitious as they are here ... even this level
of robustness may be insufficient. EAI versus some brand new protocol in a
completely new niche is really an apples to oranges comparison.

> > fine details of the language in which the specifications are

I personally am much more concerned about the interaction of EAI with legacy
systems. Document wording... not so much. As I've said, I think the original
documents got some of this stuff wrong, and because of the nature of the
experimental deployment those issues weren't seen for what they were at the
time. As a result these specifications - and I'm talking about the actual
content, not the prose - are well designed and developed overall but have a few
fairly significant glitches here and there.

> as an opinion to all the above points... we are a bit diverging from the

Agreed.

+1

				Ned

---

From: [email-masked]
Date: Sat, 05 Feb 2011 14:50:24 -0800 (PST)

> more appropriate with time. And we can always change it during the

I also prefer SHOULD NOT.

				Ned

---

From: Dave CROCKER <[email-masked]>
Date: Sun, 06 Feb 2011 11:37:00 -0800

> I also prefer SHOULD NOT.

One view that has been expressed is that it does not really matter whether the 
specification says MAY or SHOULD, since implementers will do whatever they want. 
  That raises a basic question about pursuing standardization at all, but 
certainly calls to question having the specification say /anything/ about this 
specific issue.

A MAY essential means that the action will not break anything.  A SHOULD says 
that an exception requires knowing exactly when and why the action will not 
break anything.

The statements of preference, here, seem to hinge on near-term, versus 
long-term, expectations.

Whichever normative term is chosen, I strongly suggest that the text contain an 
explanation of a) the concern that motivates the choice, and b) the expected 
conditions under which an exception is likely to be acceptable.  That is, when 
is it likely to be safe to ignore the normative directive.

For reference, here are the long-term systems implication of the two choices:

MAY (This repeats the one I sent earlier)

  +-----------+                                      +-----------+
  | MUA-ASCII |                                      | MUA-ASCII |
  +----+------+                                      +-----------+
       |                                                  ^
       v                                                  |
  +-----------+    +-----------+    +-----------+    +----+------+
  | MSA-ASCII +--->| MTA-ASCII +--->| MTA-ASCII +--->| MDA-ASCII |
  +-----------+ ^  +-----------+ ^  +-----------+ ^  +-----------+
                |                |                |
                | Gateway        |                |
                |                |                |
  +-----------+ V  +-----------+ V  +-----------+ V  +-----------+
  | MSA-UTF8  |--->| MTA-UTF8  |--->| MTA-UTF8  |--->| MDA-UTF8
  +-----------+    +-----------+    +-----------+    +----+------+
       ^                                                  |
       |                                                  V
  +-----------+                                      +-----------+
  | MUA-UTF8  |                                      | MUA-UTF8  |
  +-----------+                                      +-----------+

SHOULD NOT:

  +-----------+                                      +-----------+
  | MUA-ASCII |                                      | MUA-ASCII |
  +----+------+                                      +-----------+
       |                                                  ^
       v                                                  |
  +-----------+    +-----------+    +-----------+    +----+------+
  | MSA-ASCII +--->| MTA-ASCII +--->| MTA-ASCII +--->| MDA-ASCII |
  +-----------+    +-----------+ ^  +-----------+    +-----------+
                                 |
                                 |ASCII/ASCII
                                 |
  +-----------+    +-----------+ V  +-----------+    +-----------+
  | MSA-UTF8  |--->| MTA-UTF8  |--->| MTA-UTF8  |--->| MDA-UTF8  |
  |     ----  |    |     ----  |    |     ----  |    |     ----  |
  |     ASCII |    |     ASCII |    |     ASCII |    |     ASCII |
  +-----------+    +-----------+    +-----------+    +----+------+
       ^                                                  |
       |                                                  V
  +-----------+                                      +-----------+
  | MUA-UTF8  |                                      | MUA-UTF8  |
  |     ----  |                                      |     ----  |
  |     ASCII |                                      |     ASCII |
  +-----------+                                      +-----------+

That is, anything in ASCII stays in ASCII, everywhere and forever, within the 
scope of this specification.

d/
-- 

   Dave Crocker

---

From: Dave CROCKER <[email-masked]>
Date: Sun, 06 Feb 2011 11:45:43 -0800

...
...
> First, we have been through "experimental".

That is why your current expectation of making changes in only a few years is so 
surprising.

Note that the criterion I cited was only the premise of instability that you 
offered.

It does not matter what the previous classification was, nor does it matter what 
the IETF formal rule on time is.  Propagation of changes to the email 
infrastructure takes quite a few years.  So an expected timeframe for making 
significant changes to a critical specification in less that 10 years invites 
the view that the specification is unstable.

I cited IETF and Internet cultures concerning change.  If you have evidence that 
making significant normative changes 'within a few years' of getting published 
does not constitute instability, please provide it.

> fine details of the language in which the specifications are

So, your view is that the difference between a specification's containing a 
normative SHOULD NOT versus MAY is merely fine-tuning?

d/

-- 

   Dave Crocker

---

From: John Levine <[email-masked]>
Date: 6 Feb 2011 20:46:24 -0000

>Whichever normative term is chosen, I strongly suggest that the text
>contain an explanation of a) the concern that motivates the choice,
>and b) the expected conditions under which an exception is likely to
>be acceptable.  That is, when is it likely to be safe to ignore the
>normative directive.

Could you send text, please?  I get the impression that you expect
something very bad to happen if clients and servers are sloppy about
what they label EAI, but I personally don't understand what the
badness is.

We abandoned all the complicated downgrade stuff, so I think everyone
understands that whenever you send a message with the EAI flag set,
there's some chance it won't be delivered unless you happen to know
that there's an EAI path to all the recipients.

Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. http://jl.ly

---

From: Dave CROCKER <[email-masked]>
Date: Sun, 06 Feb 2011 13:45:04 -0800

> what they label EAI, but I personally don't understand what the

As you know from the DKIM working group, I make it a point to offer candidate 
text, when I raise an issue and think I understand the details and preferences 
well enough.

This time, I can see the issue, but don't believe I understand what is in other 
participants' minds about the issue.  So I'm asking those who believe they do to 
offer text.

> that there's an EAI path to all the recipients.

"everyone understands"?  really?

you mean the guy who hasn't been a member of the working group, has no prior 
history with EAI and probably does not have much experience with this domain -- 
in other words, an average programmer or system administrator, in a typical 
implementation scenario -- and has the task of writing software or deploying it 
is inherently going to understand the implications?

IETF specs are not written for working group participants.  They are supposed to 
be written for non-participants.

d/

-- 

   Dave Crocker