Subject: [apps-discuss] Objection to processing draft-ietf-appsawg-text-markdown-* documents as WG drafts (was: Re: Benoit Claise's Discuss on draft-ietf-appsawg-text-markdown-use-cases-02: (with DISCUSS and COMMENT))

From: John C Klensin <[email-masked]>
Date: Sun, 12 Jul 2015 19:20:06 -0400

Hi.  

With "normal" IETF WGs, we assume a very high level of internal
WG review and resulting WG consensus (even if rough) before a
draft goes into IETF Last Call.  It is that assumption that
justifies a two-week Last Call and maybe being relaxed about
questions about whether or not a significant fraction of the
community actually cares, etc.   I don't have much experience
with other area WGs, but there seems to be a tacit assumption in
AppsAWG that, if a piece of work is worthwhile (in this case,
getting Markdown well documents in an obvious place and reviewed
through the IETF process) it is safe to assume that others who
_really_ care will do the heavy lifting or, indeed, all of the
lifting

At least in retrospect, that didn't happen with this pair of
documents.  The kinds of problematic text (and a few actual
errors) that are being found during and subsequent to IETF Last
Call suggests strongly that the WG process failed to do an
adequate review from several perspectives.  It occurs to me that
an IESG decision to initiate a Last Call is a decision that
should be subject to appeal.  I'm not going to do that in this
case because I'm satisfied that diligent efforts by the IESG are
turning up the issues that should have been caught and fixed
earlier.  However, I encourage the IESG (and document shepherds)
to very carefully review the criteria that are being used to
initiate IETF Last Calls on documents supposedly processed
through (and carefully reviewed by) Area or other broad-focus
WGs.  

This might even be worth some discussion during the AppsAWG
meeting on 20 July.

More specifically, the issues raised by Beniot and Sean's
response forced me into a quick read-through of these two
documents for issues I care about and with which I'm familiar
(e.g., Media Type definitions and character set and i18n issues
and terminology).  That reading strongly suggests that either no
effective review occurred in AppsAWG or that it was inadequate
enough to suggest that assigning these documents to that WG was
inappropriate.   I am not only concerned about the examples
above and below; I am concerned that, if these issues have
fallen through the cracks, there may be others in the two
documents.

For example, the first paragraph of Section 1.1 of
draft-ietf-appsawg-text-markdown includes "a linear sequence of
characters in some character set (code)".  That just isn't
acceptable terminology.  Not only does it not conform to the
recommendations of RFC 6365, but, in a slightly different
environment, it would probably be read as meaning something
entirely different from what was probably intended.  That
paragraph goes on to say "Because they are non-printing, these
characters" (referring to "line breaks, page breaks, or other
control characters) "are also hard to enter with standard
keyboards."   At least for European writing systems, that is
plain silly unless one has a keyboard that lacks an "Enter" or
"Return" function or is using a _very_ strange input method
editor (IME).   The next paragraph goes on to make a suggestion
about "overload certain characters with additional meanings". At
least for SGML (and its descendents), that is not the way what
happens is described.  I'd suggest it is even less true of
LaTex, but YMMD.  What might be intended is something like
"certain characters or character sequences are treated as
reserved delimiters, with the strings they delimit acting as
processing, identification, or formatting directions".

Continuing with this theme, the "charset" portion of Section 2
of draft-ietf-appsawg-text-markdown-06 says:

	"...will get along just fine by operating on character
	codes that lie in printable US-ASCII, blissfully
	oblivious to coded values outside of that range."

I don't know what that means in spite of being regularly
mistaken for an expert in the area.  Given that you want to be
CCS-independent (see RFC6365), I think the first part probably
refers to "graphic characters in the ASCII repertoire", but I
don't know what "blissfully oblivious..." is trying to tell me.
Is it that each Markdown processor has, or assumes, a CCS and
encoding and, if anything is encountered outside that range or
is a non-graphic character in that range, it will be ignored?
Noting that set of exclusions would ignore the character known
as SP, I suggest that any such Markdown processor would be
seriously broken.  It is more likely that the sentence is wrong.

In addition, the handwaving in the first paragraph of Security
Considerations in Section 2 of
draft-ietf-appsawg-text-markdown-06 is sufficient that an
"Internationalization Considerations" section should be required
in this document (see RFC 2277).

As a final example, Content Disposition headers are orthogonal
to Media Type.  If draft-ietf-appsawg-text-markdown is about
registration of a media  type, as the title very strongly
suggests and the abstract says explicitly, then Section 4 has no
place in it.  If it is intended as a general discussion of
Markdown as it might be transmitted through systems that might
use Content-Disposition headers, then the title and abstract
need fixing and some language should be added to the
Introduction that explains the scope of the document before it
starts explaining the use and history of Markup/Down.  It would
be appropriate --and normal practice in the IETF and
editorially-- for that actual introduction to also discuss the
relationship between this document and
draft-ietf-appsawg-text-markdown-use-cases.

The comments below are about one specific set of examples of the
above that follow up an exchange that has already occurred.
These problems can be fixed, but they need, IMO, to be
understood and taken seriously, not just responded to with a
reference to something that might be relevant.  I also suggest
that this example demonstrates that the separation into two
documents either was not wise or was not executed carefully
enough.

--On Saturday, July 11, 2015 14:09 -0700 Sean Leonard
<[email-masked]> wrote:

> On Jul 10, 2015, at 9:18 PM, John C Klensin
> <[email-masked]> wrote:
>...
>> If you have a character set problem, or, as usually turns out
>> to be the case when people start talking about "character set
>> interoperability", an i18n problem, you really need to say
>> something specific (and probably provide references), not say
>> something that sounds like "it is a problem that others have
>> spent time on and therefore we can ignore".
> 
> Okay, okay…
>...
> Therefore, proposed changes are:
> {{
> There are communities that are using Markdown for
> scholarly writing [OCCASION], for screenplays [FOUNTAIN], and
> even for mathematical formulae [MATHDOWN].

Not in my area of competence (at least without careful study),
but I assume ok.
>...
> {{
> [RFC6657] provides guidance on character set parameter
> handling. }}

This is more handwaving and is not helpful.  I strongly suspect
it doesn't belong here because it is either unnecssary or
represents information without which
draft-ietf-appsawg-text-markdown (aka [MDMTREG]) is not complete
enough to conform to the letter and spirit of RFC 6838 (BCP 13).
The information in draft-ietf-appsawg-text-markdown is, IMO,
written in a confusing way and needs a detail or two, but it
appears to be to be present.

So, asking everyone's indulgence for commenting on both of these
intertwined documents in the same note, a constructive
suggestion in the spirit of "send text":

(1) Drop the above sentence (and the reference) from
draft-ietf-appsawg-text-markdown-use-cases.  If you must say
something in the "use cases" document, point back to MDMTREG and
indicate that issues associated with charset parameters are
discussed there.   And say "charset" (in quotes if you think
that is needed) and not "character set" because they do not mean
the same thing and you are talking about the former (see
elsewhere in this note).

(2) In Section 2 of draft-ietf-appsawg-text-markdown, paragraph
about "charset", add, probably after "no default value",
"consistent with the requirements of RFC 6657 [RFC6657]".  That
should put the issue to rest.

best,
    john

---

From: Dave Crocker <[email-masked]>
Date: Sun, 12 Jul 2015 21:43:26 -0700

On 7/12/2015 4:20 PM, John C Klensin wrote:
> The kinds of problematic text (and a few actual
> adequate review from several perspectives. 

John,

I'm confused.  You appear to be saying that it is exceptional for a
working group process to turn out documents for IESG processing that are
seriously flawed.  In contrast I believe it is a periodic occurrence
across all areas and all working groups, and arguably always has been.

While crappy drafts really are problematic, I do not understand what was
supposedly exceptional in the AppsArea wg process that distinguishes it
from various other working groups around the IETF over the years,
including these days.

In any event, absent very specific citations from you, that point to
actions that need to have been done differently, there is no way to know
what problems to correct, or requirements to impose, for this wg or any
other.

d/

-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: Martin J. Dürst <[email-masked]>
Date: Mon, 13 Jul 2015 16:48:41 +0900

On 2015/07/13 08:20, John C Klensin wrote:

> entirely different from what was probably intended.

Agreed. "Character set" as used in RFC 2046 is not a term we want to use 
in 2015.

> That
> editor (IME).

For line breaks, I'd argue that they are very easy to enter on pretty 
much any keyboard, because pretty much any keyboard has an Enter key 
(and because pretty much every modern language has line- or paragraph 
breaks).

For page breaks and other control characters, I'd argue that they are 
difficult to enter with the average keyboard, in any language.

So just remove "line breaks", and the problem should be fixed.

> The next paragraph goes on to make a suggestion

I'd agree that this isn't how SGML or LaTeX would describe it, but it's 
not actually in any way wrong. In XML, depending on context, '>' means 
itself or "close tag" (or any of a few more obscure meanings). We are 
still in the introduction, after all.

> seriously broken.  It is more likely that the sentence is wrong.

I know what the sentence tries to refer to. It refers to the fact that, 
as long as the syntax you are looking at only uses 7-bit bytes values 
with ASCII semantics (including the usual control characters and space), 
a processor will work for a wide range of character encodings.

The problem is that this applies to some encodings, but not to others. 
The criterion is a certain kind of strong ASCII-compatibility, namely 
that characters in the ASCII range (including C0) are always represented 
directly as 7-bit bytes, and that these 7-bit bytes always represent the 
corresponding characters and nothing else.

Encodings that qualify are of course US-ASCII itself, straightforward 
8-bit encodings starting with iso-8859-1 and including vendor encodings 
(Windows, Mac,...), some multibyte encodings such as GB2312 and EUC-JP, 
and UTF-8 (but beware of the BOM). However, it does not include some 
other encodings, in particular not iso-2022-jp, Shift_JIS, GBK, or 
GB18030, and of course not UTF-(16|32)(LE|BE).

So the text should definitely be more careful here.

Regards,   Martin.

---

From: John C Klensin <[email-masked]>
Date: Mon, 13 Jul 2015 13:42:23 -0400

--On Sunday, July 12, 2015 21:43 -0700 Dave Crocker
<[email-masked]> wrote:

>> errors) that are being found during and subsequent to IETF
>> Last Call suggests strongly that the WG process failed to do
>> an adequate review from several perspectives. 
> 
> 
> John,
> 
> I'm confused.  You appear to be saying that it is exceptional
> for a working group process to turn out documents for IESG
> processing that are seriously flawed.  In contrast I believe
> it is a periodic occurrence across all areas and all working
> groups, and arguably always has been.

Dave,

While I agree that there are periodic occurrences of seriously
flawed documents that are independent of any particular
procedural framework, I think there is a difference that is
worth noting.  I also wouldn't describe the text-markdown
documents as "seriously flawed" or even "crappy", just poorly
written and apparently unaware of some issues and established
terminology that overlaps them.    To be a little more specific,
I'm unconvinced that there is anything wrong with the two of
them that could not be straightened out by a pass or two from a
good technical editor who understood the area of application.
However, we've been told that we cannot expect that level of
editing from the RFC Editor, which means that, if the author
doesn't do it, someone else should notice the problem and do
something about it, with relying on editing by the IESG
post-Last-Call being a particularly unfortunate approach.  I
don't think that issue is unique to any particular type of WG or
other structure either.

> While crappy drafts really are problematic, I do not
> understand what was supposedly exceptional in the AppsArea wg
> process that distinguishes it from various other working
> groups around the IETF over the years, including these days.

With the understanding that there are "in between" cases,
consider two scenarios:

(1) We have a relatively classic-style IETF WG, chartered for a
very specific topic and limited number of documents and
typically expected to terminate when those are done.

(2) We have a WG with a charter for "catch-all" work that may
extend over a rather wide range of topics, with topics and
documents being added to the workload by consensus (or, more
often, especially if the WG leadership likes the proposals,
absence of consensus objections).  

Now, as you point out, neither of those scenarios provides
perfect protection about poor review, "crappy" drafts, or even
technically-defective work (I believe the current work _not_
technically defective).  However, under the first scenario,
there are strong assumptions that everyone who is participating
in the work is, indeed, participating.  Those who get involved
typically are relatively expert in the WG's topic area or at
least care enough to become informed (or should).  Mailing lists
tend to be less noisy than the second case if only because the
range of topics is narrower.  Documents either get reviewed or
engaged and alert WG Chairs engage in a good deal of exhortation
and shepherds have some obligation to report to the IESG when
there is an appearance of inadequate or indifferent reviews.
Certainly, it doesn't always work, especially in "victory by
attrition" cases, but the odds are in its favor.  

On the other hand, when one has a WG with a charter that is both
very broad in terms of topics and presumes it will stay alive
for a long time, adding new topics as it goes along, very few
people "in the WG" are going to be expert on, or care about,
every topic.  While some people may try to carefully review
every document that shows up, at least at WG LC, reactions
similar to "glad someone is doing that, but not my problem or
area of  expertise" are almost certain (and, IMO, common and
expected).

So I think there really is a difference, at least in risk and
expectations,... and that it does not require a catalog of
"crappy drafts" to prove it.

At the same time, I think that one of the remedies for the
"crappy draft" problem would be for the IESG to be a lot more
willing, when it becomes clear that a draft needs significantly
more review and/or work, to bounce it back to the originator or
originating WG and say "this is not of acceptable quality, take
it back to the virtual drawing board and, when you resubmit its
successor, be prepared to explain how something reached us in
this state".  I think it should be reasonable to do that as soon
as "not of acceptable quality" (perhaps the same as what you
mean by "crappy") is clear, whether that is before IETF LC,
during it, or after LC is nominally completed.  That suggestion
is not specific to any particular WG organization model or even
to the difference between WG and individual submissions but it
seems to me that the IESG's trying to fix such documents after
Last Call is not only a bad idea generally but reinforces bad
behavior.  Lest I be accused of being value, consider the
difference in incentives for someone deciding to hand a draft
off for Last Call between:

(i) "I think maybe this is good enough to get by and, if it
isn't, the IESG will fix it and it will get published".

and 

(ii) "I think maybe this is good enough but, if I'm wrong and
the IESG bounces it, it will result in loss of a lot more time
and possibly reassignment of individual submissions to a WG or
aggregate/area WG documents to a short-lived, document specific
WG", losing even more time".  

Faced with the strategy suggested above, I suggest that many
more authors, shepherds, and WG Chairs would adopt the second
line of thinking and try to put a little more energy into being
sure the document really is ready.  I think that would be A Good
Thing.   YMMD.

    john

   best,
     john

---

From: Sean Leonard <[email-masked]>
Date: Tue, 14 Jul 2015 09:09:07 -0700

On 7/13/2015 12:48 AM, Martin J. Dürst wrote:
> Agreed. "Character set" as used in RFC 2046 is not a term we want to 
> use in 2015.

To be clear, the drafts got a significant amount of discussion across 
the board, which resulted in the documents submitted to the IESG.

The key objections causing disgruntlement seem to be about terminology 
in or related to the introductory matter. The introduction used to be 
significantly longer (see draft-ietf-appsawg-text-markdown-02), but was 
pared down after significant community input. If there is something 
wrong with the history or the terminology then that is fine but it's 
just an historical or factual error; it's not something that compromises 
the technical integrity of the standard.

The point of the first paragraph is to acknowledge a continuum between 
textual data and binary data (binary data having been edited out, and 
subsequently restored--see prior e-mail), and then to acknowledge the 
limitations of in-band signaling with control characters that are part 
of the (coded) character set. The raison d'être for markup languages is 
that signaling with control characters has proven to be impractical or 
nonextensible. There are many standards (ISO 2022, ISO 6429/ECMA-48, 
etc.) that deal with control characters--even a family of standards for 
word processing interchange with control characters for formatting and 
whatnot. (I forgot that particular standard--does someone know it?) And 
most people don't use them these days, because markup is just easier to 
deal with.

When I (re)wrote the introduction, I relied on the Unicode definition of 
"plain text":
/Plain Text <http://unicode.org/glossary/#plain_text>/. Computer-encoded 
text that consists/only/of a sequence of code points from a given 
standard, with no other formatting or structural information. Plain text 
interchange is commonly used between computer systems that do not share 
higher-level protocols. (See also/rich text 
<http://unicode.org/glossary/#rich_text>/.)

As well as the definitions of "text/plain" from RFC 2045/2046:

        4.1.3 <http://tools.ietf.org/html/rfc2046#section-4.1.3>. Plain
        Subtype

    The simplest and most important subtype of "text" is "plain".  This
    indicates plain text that does not contain any formatting commands or
    directives. Plain text is intended to be displayed "as-is", that is,
    no interpretation of embedded formatting commands, font attribute
    specifications, processing instructions, interpretation directives,
    or content markup should be necessary for proper display.  The
    default media type of "text/plain; charset=us-ascii" for Internet
    mail describes existing Internet practice.  That is, it is the type
    of body defined byRFC 822  <http://tools.ietf.org/html/rfc822>.

Those definitions conflate the distinction between a "character set" 
(which is, literally, a set of abstract characters, or, Unicode: 
/Character Set. <http://unicode.org/glossary/#character_set>/A 
collection of elements used to represent textual information.), and a 
"coded character set" [RFC 6365]. The sentence in the text-markdown 
draft only mirrors what is already out there. Note that RFC 6365 does 
not define character set: it only defines coded character set. So the 
objection lacks basis.

If we in the software industry are going to quibble about definitions, 
it sure would be nice for the SDOs to come together and establish a 
uniform set of definitions for these basic constructs, that are worded 
exactly the same between the different SDOs.

> So just remove "line breaks", and the problem should be fixed.

Actually line breaking is and remains a persistent issue. When "Return" 
or "Enter" is pressed on a keyboard, the keyboard generates scancodes, 
typically 5A (MAKE) and F0 FA (RELEASE). It is up to the operating 
system to interpret this, which may variously generate CR, LF, CRLF, or 
something else entirely (activating a button, or for the under-35 crowd, 
sending a text). If your *intent* on a Mac OS X machine is to generate 
the control character CR, you are in for a tough time.

We may make a general distinction between control characters and 
non-control characters (which, variously, are referred to as printable 
characters or graphic characters). Markup languages encode formatting 
information in non-control characters, by overloading the meaning of 
some non-control characters beyond the meaning assigned by the character 
set.

> I'd agree that this isn't how SGML or LaTeX would describe it, but 
> it's not actually in any way wrong. In XML, depending on context, '>' 
> means itself or "close tag" (or any of a few more obscure meanings). 
> We are still in the introduction, after all.

Yes, exactly. According to Unicode, ISO 646, RFC 20, and USASI 
X3.4-1968, ">" (code point 3/14) means "GREATER THAN SIGN". It has also 
been the policy of the U.S. Government since Lyndon B. Johnson's 
approval in 1968 (see 
http://www.presidency.ucsb.edu/ws/index.php?pid=28724 ). Any meaning 
other than "GREATER THAN SIGN", such as "close tag", is exactly that: an 
additional meaning.

> I know what the sentence tries to refer to. It refers to the fact 
> that, as long as the syntax you are looking at only uses 7-bit bytes 
> values with ASCII semantics (including the usual control characters 
> and space), a processor will work for a wide range of character 
> encodings.
>
> that characters in the ASCII range (including C0) are always 
> represented directly as 7-bit bytes, and that these 7-bit bytes always 
> represent the corresponding characters and nothing else.
>
> 8-bit encodings starting with iso-8859-1 and including vendor 
> encodings (Windows, Mac,...), some multibyte encodings such as GB2312 
> and EUC-JP, and UTF-8 (but beware of the BOM). However, it does not 
> include some other encodings, in particular not iso-2022-jp, 
> Shift_JIS, GBK, or GB18030, and of course not UTF-(16|32)(LE|BE).

Many prominent Markdown processors operate on a text stream as input, 
and output the same text stream. By "text stream", I mean that the bits 
from disk (or memory or some other source) are interpreted by a software 
library, such as Perl, Python, the C++ standard library, etc., and are 
given to the Markdown processor in code units. A JavaScript-based 
Markdown interpreter, for example, almost never operates on raw octets; 
it always takes input from some other source that is dumped into 
strings. Strings in JavaScript are (virtualized) to UCS-2 or UTF-16 
16-bit code points. It doesn't matter if the source is in ISO-2022-JP, 
GBK, ISO-8859-1, EBCDIC, or UTF-32BE. What matters is that the Markdown 
processor looks for things like "*" and when it finds things like "*" in 
certain configurations, will replace it with HTML bulleted list tags; in 
other configurations, will replace it with <em> tags; and in other 
configurations, will not change the output at all.

The common theme of Markdown processors is that Markdown operates on 
"punctuation" <http://daringfireball.net/projects/markdown/syntax>:
To this end, Markdown’s syntax is comprised entirely of punctuation 
characters, which punctuation characters have been carefully chosen so 
as to look like what they mean.

...which, by the way, is exactly what that part of the text-markdown 
draft says, right before the quoted part about being "blissfully oblivious".

Bear in mind that this discussion is being had in the context of the 
"charset" parameter, which /combines/ the abstract character set with 
the character encoding. It seems that folks are asking for separation in 
a context where separation does not exist.

> So the text should definitely be more careful here.

I suppose the text could be more careful, but when reading the whole 
paragraph, I do not see a problem. In fact, to the extent that a 
Markdown processor (or the input libraries upon which it relies) 
misinterprets content as the wrong (coded) character set compared to the 
author's intent, it is very likely that the Markdown processor will not 
fail--it will just output something crazy. Markdown processors rarely 
fail; they are not designed that way. That is why the text uses the 
phrase "blissfully oblivious".

Sean