Subject: Re: [EAI] Re: downgrading target header list

From: John C Klensin <[email-masked]>
Date: Fri, 13 Apr 2007 07:42:37 -0400

--On Thursday, 12 April, 2007 22:57 +0200 Harald Tveit
Alvestrand <[email-masked]> wrote:

> 
> 
> --On 12. april 2007 10:00 -0400 John C Klensin
> <[email-masked]> wrote:
> 
>>> IMO we need a new technique that can completely ignore the
>>> structure or not of the header field, and the existence of
>>> anything within the header field using a similar technique.
>> 
>> I agree.  Unfortunately, we have so many different ways of
>> encoding things floating around that there is a great deal of
>> reluctance (at least by me) for creating a new one.  Worse, we
>> have to optimize for two different types of MUA endpoints.
>> One is a legacy MUA that has no knowledge at all of this
>> stuff.  It is going to have to display the escaped form and
>> hope that users can make sense of it.  It was concern for
>> those sorts of MUAs that brought us Quoted-Printable which
>> many of us consider to have been largely a failure.   The
>> other is an MUA that knows all about the internationalized
>> forms but that is on the far side of something that had to
>> downgrade.  It probably will want to decode (or "upgrade")
>> later.
> 
> The work in "scenarios" has singularly failed to establish
> convincing more-than-transient scenarios where this
> functionality is needed. So (my personal technical opinion) we
> have very strong reasons to optimize for the first case.
> 
> That means that the *result* of the transformation needs to
> either be ignorable or decodable by MUAs that don't support
> UTF8SMTP.
> 
> In practice, I think that leaves us with "remove or
> encoded-word".

Sadly, I agree.   "Sadly", because I think it really implies
that, if a downgrading system doesn't understand --explicitly
recognize-- a header that contains non-ASCII information-- it
probably needs to drop it.   Anything else gets us involved with
guessing what the information there is actually about and about
what the receiving MUA may or may not know. 

A different way to look at this is that, while losing
information is unattractive, it is better than risking changing
material in a way that appears to retain information but
actually alters it.  If we try to preserve every possible bit in
the expectation of later upgrading, we are likely to increase
the number of situations in which messages are rejected or
returned rather than downgraded.

   john

_______________________________________________
IMA mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/ima

---

From: Harald Alvestrand <[email-masked]>
Date: Fri, 13 Apr 2007 16:17:23 +0200

> returned rather than downgraded.
Another theory, which may be suitable for an experiment, even though we 
would be hesitant to deploy such a thing in the running Internet:

Assume that any legal UTF-8 character in an unknown header is placed 
there in accordance to this specification.

Assume further that this specification says that you can ONLY place 
UTF-8 characters in places where either encoded-words or RFC 2231 
encoded quoted-strings are allowed. (Question: Can we manage to say that?)

If those two things are both true, we can safely use the encoded-word 
and RFC 2231 technique on strings containing such characters - on the 
naivistic theory that anything else is a protocol error, so they deserve 
what they get.

Then we run the experiment, and observe one of three outcomes:

- Our assumptions are true all the time (not likely)
- Our assumptions are false, and false so often that we need to reconsider
- Our assumptions are occasionally false, but the pain this causes is 
tolerable

We MIGHT get lucky....

---

From: John C Klensin <[email-masked]>
Date: Fri, 13 Apr 2007 11:55:08 -0400

--On Friday, 13 April, 2007 16:17 +0200 Harald Alvestrand
<[email-masked]> wrote:

>...
> Another theory, which may be suitable for an experiment, even
> though we would be hesitant to deploy such a thing in the
> running Internet:

I'd rather keep our "experiment" plausible, i.e., confine
ourselves to things we think we would be willing to deploy
barring surprises or other empirical corrections.   For your
suggestion, I fear that, even if we "get lucky", actually
incorporating the idea into a real  specification and deploying
it widely would, modulo one observation below, require proving a
universal negative.   My reason for that belief is that, just as
we had to deploy the SMTP extension model and MIME against a
backdrop of "just send 8" implementations, I think there is
every reason to believe that there are implementations out there
that use non-ASCII characters in locally-defined header fields
and, worse, that those non-ASCII characters are (as Soobok and
others have suggested in a different context), going to be coded
in local/national character sets rather than Unicode/UTF-8 in
many cases.

> Assume that any legal UTF-8 character in an unknown header is
> placed there in accordance to this specification.
> 
> Assume further that this specification says that you can ONLY
> place UTF-8 characters in places where either encoded-words or
> RFC 2231 encoded quoted-strings are allowed. (Question: Can we
> manage to say that?)
> 
> If those two things are both true, we can safely use the
> encoded-word and RFC 2231 technique on strings containing such
> characters - on the naivistic theory that anything else is a
> protocol error, so they deserve what they get.

Let me suggest a different model that we could use if you wanted
to go down this path.   Today (i.e., under 2821/2822), the
appearance of any non-ASCII character -- anything else with the
high bit turned on or any seven-bit (leading zero bit) object
intended to be interpreted as other than ASCII-- in a header is
a protocol violation.  If we take the position that anyone who
violates those base protocols deserves whatever bad things
happen to them and so do any senders or receivers who are
victims of their software, then we might be able to make a clean
downgrading model by specifying, in the UTF8Headers document,
_exactly_ what headers can contain such characters and where.
That description would need to either forbid UTF8 information in
any header not listed or would need to describe very precise
rules for unknown headers (where "unknown" == "not listed").  

If we were going to do that, the distinction between
"structured" and "unstructured" headers probably isn't good
enough.  We would either need to confine ourselves --and
permission to include 8bit information in headers-- to things
that we could enumerate or we would need to get very clever.
And hand waving, such as assuming that things in parentheses in
unknown headers were comments, would not be nearly good enough.

Whether it is worth the effort or not is an open question, but
it would at least constitute the basis for a clean experiment
that we could use.   And part of the experimental question would
be "which header fields did we not list that really need UTF8
data"?

> - Our assumptions are occasionally false, but the pain this
> causes is tolerable

You will almost certainly not be able to competently evaluate
the last case without an _exhaustive_ test.  And exhaustive
tests are essentially impossible.

> We MIGHT get lucky....

We are more likely to delude ourselves into believing that we
have gotten lucky in the absence of sufficient counterexamples,
which would be very dangerous indeed.

     john

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 16 Apr 2007 22:03:01 +0100

> what the receiving MUA may or may not know.

Do you really mean "drop it"? I thought our whole strategy was that you  
always downgraded, and if that was not possible you bounced/rejected. I  
don't think you can get away with omitting a part of some message that was  
probably put there for good reason. So if it eventually arrives at some  
recipient then every header that was present in the original should be  
present in some form in what he receives, even if not upgraded/upgradeable  
with 100% perfection.
> actually alters it.

We would only care if the alteration made it semantically different. It is  
already accepted that changes of content transfer encoding and changes of  
folding have no semantic effect (except maybe to DKIN signings). Likewise,  
different but equivalent ways of doing RFC 2047 encoding do not change the  
semantics (which is always the sequence of encoded characters as restored  
to and displayed in their original charsets). And it is also considered  
acceptable to display the encoded form if the required charsets cannot be  
displayed locally.

Within those limitations, the Downgraded header, as currently proposed,  
meets the requirements of conveying the original semantic meaning. The  
worst that can happen is that some legacy agent which was supposed to take  
some specific action upon seeing that header will not take it when  
presented with an equivalent Downgraded header. But we have taken care of  
what to do in the case of all presently known headers, and as regards  
headers not yet invented the world has to accept that there will always be  
older agents around that will not instigate whatever action that new  
header was supposed to bring about. Humans will always see such Downgraded  
headers properly displayed, if they care to look, which is all you can  
ever assume will happen with any newly-invented header.

-- 
Charles H. Lindsey ---------At Home, doing my own thing------------------------
Tel: +44 161 436 6131                       
   Web: http://www.cs.man.ac.uk/~chl
Email: [email-masked]      Snail: 5 Clerewood Ave, CHEADLE, SK8 3JU, U.K.
PGP: 2C15F1A9      Fingerprint: 73 6D C2 51 93 A0 01 E7 65 E8 64 7E 14 A4 AB A5

---

From: "Charles Lindsey" <[email-masked]>
Date: Mon, 16 Apr 2007 22:11:43 +0100

> encoded quoted-strings are allowed. (Question: Can we manage to say  

Sadly, we can't say that. It may work for RFC 2231, but RFC 2047 provides  
no way to say it even for the present ASCII headers. Which is why cuurent  
MUAs routinely take shortcuts, such as treating all unrecognized headers  
(and even many recognized ones) as "unstructured". It works well enough  
that nobody really notices the odd technical glitch, and if we can get  
downgrading to work as well as that, we will have little to complain about.