Subject: Re: [EAI] (Dave's detail suggestion )Re: The scenario favoring"uMailbox"?

From: "Jiankang YAO" <[email-masked]>
Date: Thu, 12 May 2011 11:59:27 +0800

Based on Dave's comments and suggestions,

I propose the following changes to rfc5336bis:

1 ***********************************************************************         
the abnf of mailbox is proposed below:

-----------------------------------------------------------------------------

 Mailbox = Local-part "@" ( Domain / address-literal )
              ; Inherit Mailbox Definition from RFC 5321, Section 4.1.2
              
           Local-part = Dot-string / Quoted-string
              ; Inherit Local-part Definition from RFC 5321, Section 4.1.2

           Dot-string     = Atom *("."  Atom)
              ; Inherit Dot-string Definition from RFC 5321, Section 4.1.2

          Quoted-string  = <Defined in Section 4.1.2 of RFC 5321>

           Atom           = 1*atext
                ; Inherit Atom Definition from RFC 5321, Section 4.1.2

          atext   =/  UTF8-non-ascii
             ; extend the defintion of atext in RFC5321, section 4.1.2 
            
           UTF8-non-ascii = <See  Section 4.1 of RFC5335bis>

           Domain         = sub-domain *("." sub-domain)
                ; Inherit Domain Definition from RFC 5321, Section 4.1.2

            sub-domain     = / U-label
              ; extend the defintion of sub-domain in RFC5321, section 4.1.2 

            U-label   = <see Section 2.3.2.1 of RFC5890 >
-----------------------------------------------------------------------------------

2 ******************************************

The section 3.7.3  is supposed to be updated to the following

--------------------------------------------------------------------

3.7.3 Trace Information

   For the trace information [RFC5321], this memo updates  the return path line [RFC5321] and the time stamp line formally defined as follows:

   Return-path-line = "Return-Path:" FWS Reverse-path CRLF
    ; Inherit Return-path-line defintion from Section 4.4 of RFC 5321
    ; Reverse-path may include the internationalized domain name with the U-label form 

  Time-stamp-line = "Received:" FWS Stamp CRLF
    ; Replaces Time-stamp-line in Section 4.4 of RFC 5321
    ; Stamp may include the 'For' clause where the internationalized domain name with the U-label form may be used.

Except in the 'For' clause and 'Reverse-path' value where
   internationalized domain name with the U-label form MAY be used,
   internationalized domain names in Received fields MUST be transmitted
   in the form of A-labels.  The protocol value of the WITH clause when
   this extension is used is one of the UTF8SMTPbis values specified in
   the "IANA Considerations" section of this document.

----------------------------------------------------------------------------------------

3 ********************************************************************

the  VERIFY (VRFY) and EXPAND (EXPN) command syntaxes are changed to:

----------------------------------------------------------------------------

     vrfy = "VRFY" SP String
     [ SP "UTF8SMTPbis" ] CRLF
     ; String may include UTF-8 characters

   expn = "EXPN" SP String
     [ SP "UTF8SMTPbis" ] CRLF
     ; String may include UTF-8 characters

  ----------------------------------------------------------------------------------

any comments are welcome.

Thanks a lot.

Jiankang Yao

----- Original Message ----- 
From: "Jiankang YAO" <[email-masked]>
To: <[email-masked]>
Cc: <[email-masked]>
Sent: Wednesday, April 27, 2011 8:51 AM
Subject: [EAI] (Dave's detail suggestion )Re: The scenario favoring"uMailbox"?

> thanks a lot for Dave's detail ABNF suggestion.
> Hope to see more comments to it from the WG.
> 
> Jiankang Yao
> 
> ----- Original Message ----- 
> From: "Dave CROCKER" <[email-masked]>
> Cc: <[email-masked]>
> Sent: Tuesday, April 26, 2011 11:49 PM
> Subject: Re: [EAI] The scenario favoring "uMailbox"?
> 
> 
>> Folks,
>> 
>> 
>> On 4/25/2011 10:36 PM, Jiankang YAO wrote:
>>> I think that the core issue left for the core rfcs is ABNF syntax.
>>>
>>> I hope that we can have a clear conclusion or solution for ABNF syntax soon.
>> 
>> I have looked over the ABNF in rfc5335bis and this message contains a proposal 
>> for a much simpler approach.
>> 
>> The current rfc5335bis draft specifies extensive ABNF, including additions of 
>> "u" rules, in Sections:
>> 
>>    3.3.  Extended Mailbox Address Syntax
>> 
>>    3.7.3.  Trace Information
>> 
>>    3.7.4.2.  VRFY and EXPN Commands and the UTF8SMTPbis Parameter
>> 
>> These sections have a large number of ABNF rules that add UTF-8 support 
>> selectively and in the "higher-level" rules, rather than the underlying, basic 
>> rules.
>> 
>> To follow the different and simpler approach that the EAI working group has now 
>> approved for rfc5335bis, I propose adding UTF-8 support to underlying (basic) 
>> rules.
>> 
>> Here are the only ABNF changes that appear to be needed.
>> 
>> From rfc5335bis:
>> 
>>    UTF8-enhancement  =   UTF8-2 / UTF8-3 / UTF8-4
>>                        ; incorporate from rfc5335bis, Section 2.1
>> 
>>    VCHAR   =/  UTF8-non-ascii
>>                        ; incorporate from rfc5335bis, Section 2.1
>> 
>>    ctext   =/  UTF8-enhancement
>>                        ; incorporate from rfc5335bis, Section 2.1
>> 
>>    atext   =/  UTF8-enhancement
>>                        ; incorporate from rfc5335bis, Section 2.1
>> 
>>    qtext   =/  UTF8-enhancement
>>                        ; incorporate from rfc5335bis, Section 2.1
>> 
>>    text    =/  UTF8-enhancement
>>                   ; note that this upgrades the body to UTF-8
>>                        ; incorporate from rfc5335bis, Section 2.1
>> 
>>    {{ how to add IDN to this? }}
>>    domain  =   dot-atom / domain-literal / obs-domain
>>                        ; incorporate from rfc5335bis, Section 2.1
>> 
>> Add:
>> 
>>    quoted-pairSMTP  =/ %d92 UTF8-enhancement
>> 
>>    qtextSMTP        =/  UTF8-enhancement
>> 
>> 
>> 
>> All other ABNF rules in rfc5335bis would be removed.
>> 
>> 
>> In terms of the effect of doing this, in rfc5335bis, here is an example:
>> 
>> > 3.3.  Extended Mailbox Address Syntax
>> >
>> >    uMailbox = uLocal-part "@" ( uDomain / address-literal )
>> >     ; Replace Mailbox in RFC 5321, Section 4.1.2
>> 
>> Use existing <Mailbox> definition from RFC 5321, with the <atext> enhancement 
>> from above.  This covers UTF-8 in Local-part.
>> 
>> IDN handles UTF-8 in the Domain. (Note that the specific details for specifying 
>> this in rfc5335bis is not fully resolved; it needs to be resolved the same way 
>> for that specification and for this one.)
>> 
>> Therefore uLocal-part and uDomain are not needed; none of their supporting rules 
>> are needed.
>> 
>> Similar simplifications take place for the other higher-level construct.
>> 
>> d/
>> -- 
>> 
>>   Dave Crocker
>>   Brandenburg InternetWorking
>>   bbiw.net
>> _______________________________________________
>> IMA mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/ima
> _______________________________________________
> IMA mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ima

---

From: Dave CROCKER <[email-masked]>
Date: Mon, 23 May 2011 20:03:44 -0700

> I propose the following changes to rfc5336bis:

It is troubling that there have been no responses to your proposal, for over one 
month.  This should raise a question about whether there is sufficient community 
interest in this work.

If the details from one specification are repeated in another, this creates an 
opportunity for error.  Mixing the repeated specification in with new error also 
makes it more difficult to find the rules that are new.

It is better to cite the older details than to repeat them.

The argument that I have heard to justify the repetition is saving the reader 
from having to refer back to the earlier specification.  This justification is 
based on the assumption that the reader is not going to be required to be 
familiar with the earlier specification for other reasons.

In the case of EAI, the reader is required to be familiar with RFC 5321.  So the 
repetition of some rules does not really save the reader any interesting amount 
of effort, in my opinion.

So I suggest:

  ----------

    Imported from RFC 5321, Section 4.1.2:

      Mailbox

      Local-part

      Dot-string

      Quoted-string

      Domain

      Atom

    Imported from RFC 5335bis, Section 4.1:

      UTF8-non-ascii

    Imported from RFC 5890, Section 2.3.2.1:

      U-label

    Extended from RFC 5321, Section 4.1.2:

      atext          =/  UTF8-non-ascii

      sub-domain      = / U-label

    ----------

However, note that none of the first set of rules (from RFC 5321, Section 4.1.2) 
is actually used elsewhere in this specification.  Hence, it does not make much 
sense to import or repeat them.

I believe that what is meant, when referring to them, is to help the reader 
understand that <atext> and <sub-domain> are used by these other rules.  I 
believe it will be more helpful to have text that explains this, rather than to 
cite rules that are not used in this document.

If this section is going to re-define specific, existing components, from RFC 
5321, then it should only define the parts that are different or new.  Again, it 
should not repeat the full specification.

>      ; Reverse-path may include the internationalized domain name with the U-label form

(I believe Reverse-path is defined in Section 4.1.2.)

This is not additional specification, because the existing definition's use of 
Mailbox merely uses the new and enhanced definition of sub-domain, from above. 
Hence, what might be needed is merely some text that points out this fact to the 
reader, rather than anything that tries to look like a specification.

>      ; Replaces Time-stamp-line in Section 4.4 of RFC 5321

)

 > with the U-label form may be used.

This is not additional specification, because the existing definition's use of 
Mailbox merely uses the new and enhanced definition of sub-domain, from above. 
Hence, what might be needed is merely some text that points out this fact to the 
reader, rather than anything that tries to look like a specification.

>>>     qtextSMTP        =/  UTF8-enhancement

I see that most of the enhanced rules that I suggested are not included in your 
proposal.  Please explain.

d/
-- 

   Dave Crocker

---

From: "Jiankang YAO" <[email-masked]>
Date: Tue, 24 May 2011 15:30:15 +0800

Dear Dave,

   thanks a lot for your kind response.
  More comments inline below.

Jiankang Yao

----- Original Message ----- 
From: "Dave CROCKER" <[email-masked]>
To: "Jiankang YAO" <[email-masked]>
Cc: <[email-masked]>
Sent: Tuesday, May 24, 2011 11:03 AM
Subject: Re: [EAI] (Dave's detail suggestion )Re: The scenario favoring"uMailbox"?

> month. 

My last message sent is on 12 May. There has around 2 weeks.

>
> This should raise a question about whether there is sufficient community 
> interest in this work.
> 

I have a different view about it.

The WG decided to adopt the ABNF style same or similar to your suggestion.
So I think that most WG members are waiting for your "authoritative" response.

On the other hand, the WG have reached many conclusions or results, and solved the key issues.
The current editing work is more or less just "editing or refining".
I remember that Shawn said that the implementation work of EAI will keeep same or similar in spite of that how we change the ABNF style.

> of effort, in my opinion.

My understanding is that even if the readers are familiar with RFC 5321, they can not repeat the actual ABNF definition in mind.

If the question is "who can write down  RFC 5321's ABNF definition of Mailbox, Quoted-string or Dot-string without looking up into RFC5321", I believe that few will raise the hand. At least, for me, I have to look it up in RFC5321.

So sometimes, we have to do some repetion work for easy reading.

If we are afraid of creating an opportunity for error, we can take more time to check whether our repetion is correct or not.

if we refuse to repeat some definition, I am afraid of creating an opportunity for error or confusing too.

For example, in RFC5336bis, mailbox will be mentioned  or referenced a lot of times.
If we just tell the readers that pls look mailbox definition in RFC5321, the reader may just read the mailbox definition of RFC5321 and its support rules in RFC5321, and reach a wrong conclusion that mailbox of rfc5336bis ===mailbox of rfc5321, but in fact,     mailbox of rfc5336bis includes more characters while mailbox of rfc5321 includes less characters.

any comments about it ?

> proposal.  Please explain.

In this email, I just focus on RFC5336bis. Many enhanced rules such as  VCHAR or ctext or qtext are not used in RFC5336bis, while they are used in RFC5335bis.

---

From: "John Levine" <[email-masked]>
Date: 24 May 2011 13:31:36 -0000

Sorry, I thought I'd responded to this a long time ago.

In this case, I have to agree with Dave.  If a newer RFC uses
definitions from an older one, it should always refer to the older
one, not attempt to copy the definitions.  There are too many ways
that the copies get out of sync, and it's useful to document via the
reference that the two definitions are required to be the same.

The entire RFC archive easily fits on a $2 thumb drive.  It is not
unreasonable to expect people to have access to every RFC.

R's,
John

---

From: John C Klensin <[email-masked]>
Date: Tue, 24 May 2011 14:56:57 -0400

--On Thursday, May 12, 2011 11:59 +0800 Jiankang YAO
<[email-masked]> wrote:

>    For the trace information [RFC5321], this memo updates  the
> return path line [RFC5321] and the time stamp line formally
> defined as follows:
>...
> 
>     ; Stamp may include the 'For' clause where the
> internationalized domain name with the U-label form may be
> used.

>    internationalized domain name with the U-label form MAY be
> used,    internationalized domain names in Received fields
> MUST be transmitted    in the form of A-labels.  The protocol
> value of the WITH clause when    this extension is used is one
> of the UTF8SMTPbis values specified in    the "IANA
> Considerations" section of this document.

Speaking personally, as editor of 5321, and historically, but
not as co-chair, we discussed this at length during the initial
round of EIA and concluded that it was a bad idea.   The problem
is that trace fields are of use primarily in debugging, often of
bounced messages, and there is no guarantee that the path used
to return such messages will be EAI-compliant even if the
forward path is EAI-compliant.  Return-path is not a problem
because it is  supplied only on final delivery where the
capabilities of the delivery server are noted.   But, for
anything that might be bounced, putting something in any
"Received:" (time-stamp) field that might cause the trace
information to be trashed or the bounce message to be dropped is
a very serious step.  

This is one of those obvious tradeoffs between "works well
during the transition but is ugly and creates an unpleasant
transitional situation we won't be able to get rid of" and
"looks and acts much better in the long run, but may cause
problems in the shorter term".

I am not going to take a position on what choice the WG makes,
but suggest that the WG needs to examine the question very
carefully.

      john

p.s. It seems to me that we are having a little bit of
misunderstanding about "outside the scope of this specification"
as a statement and design model for what can and cannot be said
in a specification.  There are, IMO, two flavors of "out of
scope".  One is our tradition one and is used extensively (as a
design model) in 5321 and elsewhere.  That situation arises when
a specification deliberately does not cover a particular case or
when it prohibits some behavior.  While there are exceptions in
which we actually do specify the behavior for the error cases,
our normal position on what happens if the rule is violated is
"out of scope for the specification".   The other situation
arises with extensions and requirements that the extension
specification (5336bis in this case) inherits from the base
specification (5321 in this case).  Because every SMTP extension
specification is required to follow whatever provisions of
SMTP/5321 that it doesn't explicitly override, including correct
handling or unextended SMTP/5321 messages and the presence or
absence of UTF8SMTP doesn't explicitly change the semantics or
actions associated with data elements that are 5321-conformant,
we know for certain that every EAI-compliant server must be able
to handle an all-ASCII (envelope, headers, content) message the
way 5321 specifies handling it.  It seems to me that mentioning
that (or not) is an editorial matter, best not repeated unless
doing so is needed for clarity.  But mentioning it (or not) is
not "outside the specification" because 5336bis inherits the
5321 base spec.

     john

---

From: Dave CROCKER <[email-masked]>
Date: Tue, 24 May 2011 13:13:49 -0700

...
...
...
> forward path is EAI-compliant.

There is no "guarantee" that the /forward/ path will be EAI-complaint!

What there /is/ is a working group decision to support EAI in the message.  This 
decision is damaged when it is applied to only part of the message.

The working group originally tried to use a more complicated model for its 
specification, including support for "hybrid" environments, with some Unicode 
and some ASCII-only.

The current working group model is for an 8-bit clean environment, leaving the 
concern for 7-bit environments to separate work.  That model applies to the 
return path, as well as the forward path.

As such, it is reasonable and appropriate -- in fact I think it is necessary -- 
to have the Trace fields support the EAI enhancements.

This does not deny that that there will be barriers to 8-bit clean.  Rather, it 
simply moves the issue outside of this specification, to simplify the 
specification model and effort.  (It also ought to make specification quicker...)

d/
-- 

   Dave Crocker

---

From: Dave CROCKER <[email-masked]>
Date: Tue, 24 May 2011 16:38:07 -0700

On 5/24/2011 12:30 AM, Jiankang YAO wrote:
> From: "Dave CROCKER"<[email-masked]>
>>
> My understanding is that even if the readers are familiar with RFC 5321, they
> can not repeat the actual ABNF definition in mind.
>
> If the question is "who can write down  RFC 5321's ABNF definition of
> Mailbox, Quoted-string or Dot-string without looking up into RFC5321", I
> believe that few will raise the hand. At least, for me, I have to look it up
> in RFC5321.
>
> So sometimes, we have to do some repetition work for easy reading.

There are a number of different types of people who read these documents.  Some 
readers are very casual. They have some interest in the work, but they are not 
required to develop a deep understanding of the fine-grained details.  I believe 
you are correct that the repeated text will make their reading easier.

However IETF technical specifications are not written for this type of reader. 
(I believe our specifications tend to be more readable for such folk than are 
specifications from most other standards groups, but they are not our target 
market.)

Other readers are implementers.  It might seem to be intuitively correct that 
repetition would make the work easier for these people, too.  However, it 
probably does /not.  The reality is that they are required to consult the base 
document (RFC 5321, in this case) whether this text is repeated or not:  Doing 
implementation work for an enhancement to an existing specification requires 
consulting the original specification.

For such people, having a definition exist in two places actually can make 
things more confusing.  As I previously noted, it also makes the new 
specification larger and more complicated.

> If we are afraid of creating an opportunity for error, we can take more time
> to check whether our repetition is correct or not.

Unfortunately, simply insisting on more review does not guarantee that there 
will be no errors.  (Over in the DKIM world, we are also doing a -bis version of 
the original specification.  The original was very heavily reviewed, yet it 
turns out that its ABNF had a number of very basic and very important errors.)

As for the view that the question of whether to repeat specification text really 
is important:  It is easy to believe it does not matter very much... until one 
watches the confusion of implementers and the confusion of folk who discuss 
protocol specifications later.

As for the possibility that someone on the IESG might feel that repeating the 
text is better:  This is a good demonstration that people on the IESG can be 
wrong.  It also highlights the problem with worrying more about IESG reaction 
than about doing good engineering.  The requirement for working groups is to 
produce good engineering; this is not accomplished by worrying whether one or 
another IESG member /might/ be misguided.  If the IESG is going to create a 
policy about the details of technical specifications, they can document it and 
seek IETF approval.  Until they do that, they should not try to dictate this 
sort of detail.

As for the concern about delays:  Forgive me, but this working group has not 
followed a process that has shown much concern for moving quickly, up to this 
point.  Consequently, using urgency as an argument, at this stage, as a basis 
for ignoring a basic matter of specification should not carry much weight.

> opportunity for error or confusing too.

Please forgive my repeating:  including redundant specification encourages 
confusion divergence, confusion and error.

(There is a very old joke:  if you have one watch, you know what time it is; if 
you have two, you are never quite sure.  The same applies for repeated normative 
specifications...)

> For example, in RFC5336bis, mailbox will be mentioned  or referenced a lot of
> times. If we just tell the readers that pls look mailbox definition in
> RFC5321, the reader may just read the mailbox definition of RFC5321 and its
> support rules in RFC5321, and reach a wrong conclusion that mailbox of
> rfc5336bis ===mailbox of rfc5321, but in fact,     mailbox of rfc5336bis
> includes more characters while mailbox of rfc5321 includes less characters.

rfc5336bis needs to have text that explains the relationship between its 
specification and the specifications in rfc5321. This does not require repeating 
any of the base (rfc5321) specification.

I believe that if we are consistent in applying the logic you are using, then we 
must repeat /more/ rules from RFC 5321 than you are providing:  it is not enough 
to include Mailbox and its constituents.  In fact it would be necessary to 
repeat /all/ of rfc5321, with the EAI changes.  An implementer might need all of 
them, so we would have to provide them, if the goal is to avoid having the 
implementer consult rfc 5321...

>> In addition:
>>
>>>>> From rfc5335bis:
>>>>>
>>>>> UTF8-enhancement  =   UTF8-2 / UTF8-3 / UTF8-4 ; incorporate from
>>>>> rfc5335bis, Section 2.1
...
>> I see that most of the enhanced rules that I suggested are not included in
>> your proposal.  Please explain.
>
> In this email, I just focus on RFC5336bis. Many enhanced rules such as  VCHAR
> or ctext or qtext are not used in RFC5336bis, while they are used in
> RFC5335bis.

I often miss or misunderstand details, even when I'm trying to be careful.  So 
it is entirely possible that what follows has errors...

RFC 5321 has the construct:

>                   ; MAY be case-sensitive

<atext> is a component that forms a <Dot-string>.  Your proposal kept the 
version of <atext> that I specified.  This covers the EAI requirement for a 
<Dot-string>.

<Quoted-string> also needs to support EAI.  From my proposed syntax, <qtext> was 
intended to satisfy this requirement.  However, RFC 5321 uses a different 
construct for a quoted string:  <qtextSMTP>.

Therefore I believe that was is needed is:

    qtextSMTP  =/ UTF8-non-ascii

RFC 5321 also used a quoted-pair construct:

> quoted-pairSMTP  = %d92 %d32-126

If I am reading this correctly, it already supports UTF-8, but I'd appreciate 
hearing others confirm this.  (I am a little curious about the reason it does 
not include %d127. Since that is obviously excluded explicitly, I assume that it 
is not an error.)

>                        ; incorporate from rfc5335bis, Section 2.1

are used in RFC 5322, for general text in some fields and that they are not 
needed for RFC 5321.  That is, I believe you are correct to remove them.

Again, it would be helpful to have other working group participants review this 
and confirm that they are not needed.

d/

-- 

   Dave Crocker

---

From: Dave CROCKER <[email-masked]>
Date: Wed, 25 May 2011 10:11:00 -0700

...
> consulting the original specification.

Please forgive my responding to my own posting, but I had an additional thought 
this morning:

      Consider the "=/" ABNF construct.

Note that we are already using it in the working group specifications. 
Therefore we should consider what its use means and what it implies.

The purpose of this construct is for adding one ore more alternatives to an ABNF 
rule that has previously been specified, elsewhere.  This very much does include 
having been specified in an earlier document.

The construct permits enhancing a rule... WITHOUT HAVING TO REPEAT THE ENTIRE 
RULE.  (Please forgive my "shouting" but it is important that this point be 
considered carefully.)

If we have an ABNF construct that is intended to permit additional specification 
without repating the original rule, itself, then it makes no sense to include 
the surrounding rules "to provide background" for the reader.

d/
-- 

   Dave Crocker