Subject: [apps-discuss] New I-D: text/markdown Media Type - draft-seantek-text-markdown-media-type-00

From: Sean Leonard <[email-masked]>
Date: Tue, 08 Jul 2014 06:24:00 -0700

Hello Apps-Discuss:

A new Internet-Draft has been posted, proposing the text/markdown media 
type to identify Markdown content.

http://tools.ietf.org/html/draft-seantek-text-markdown-media-type-00

Markdown is a family of lightweight markup languages (aka plain text 
formatting syntaxes) designed to make it easy to read and write in plain 
text, with the intention of having tools process the text into formal 
markup languages, namely (X)HTML. This media type is specifically 
intended to identify Markdown text, i.e., text intended for the Markdown 
tool or its derivatives.

References:
http://en.wikipedia.org/wiki/Markdown
http://daringfireball.net/projects/markdown/

This is sort of a strawman proposal. I definitely got interest off-list; 
the respondents encouraged me to write up a draft-00 for this list to 
review. I look forward to broader comments and participation from the 
Internet community, especially users of Markdown and authors of Markdown 
tools.

Name:      draft-seantek-text-markdown-media-type
Revision:    00
Title:        The text/markdown Media Type
Document date:    2014-07-08
Group:        Individual Submission
Pages:        9
URL: 
http://www.ietf.org/internet-drafts/drat-seantek-text-markdown-media-type-00.txt
Status: 
https://datatracker.ietf.org/doc/draft-seantek-text-markdown-media-type/

Regards,

Sean Leonard

---

From: Erik Wilde <[email-masked]>
Date: Tue, 08 Jul 2014 16:09:38 +0200

hello sean.

> http://tools.ietf.org/html/draft-seantek-text-markdown-media-type-00

it would definitely be great to be able to identify markdown with a 
media type. i also like the idea of the "flavor" parameter, because 
afaict, there is a fair number of variations floating around.

have you considered using "profile" instead of the more specific 
"flavor" concept? http://tools.ietf.org/html/rfc6906 defines profiles, 
and they are defined to be URIs. they can be identified with 
"profile"-typed links, and the RFC also recommends to support a 
"profile" parameter on the media type.

http://tools.ietf.org/html/draft-wilde-atom-profile is a draft where i 
am trying to do this for atom. the idea was that this draft could be 
used as a template for any media that would like to start supporting 
profile as a media type parameter (the source is available at 
http://tools.ietf.org/html/draft-wilde-atom-profile).

in addition to profiles as a lightweight mechanism that works with 
links, http://tools.ietf.org/html/rfc7284 establishes a registry for 
profiles. authors of markdown flavors could choose to register the URI 
that they recommend to use as a profile URI, or they could elect to not 
register and either make the URI self-documenting, or make people google 
it and find information that way.

either way, profiles could be used for your flavor concept, thus 
avoiding the need for yet another registry. the necessary change, 
though, would be to use URIs as identifiers, instead of the strings that 
you are currently using.

cheers,

dret.

-- 
erik wilde | mailto:[email-masked]  -  tel:+1-510-2061079 |
            | UC Berkeley  -  School of Information (ISchool) |
            | http://dret.net/netdret http://twitter.com/dret |

---

From: Dave Cridland <[email-masked]>
Date: Tue, 8 Jul 2014 16:23:56 +0100

On 8 July 2014 14:24, Sean Leonard <[email-masked]> wrote:

> markup languages, namely (X)HTML. This media type is specifically intended
> to identify Markdown text, i.e., text intended for the Markdown tool or its
> derivatives.
>
>
This looks good.

text/plain itself has a "format" parameter, which includes the simple (and
default) "fixed", and a "flowed". I wonder whether a "markdown" format here
would work better (or alongside) a new media subtype? (I'm honestly not
sure).

The advantage of using Markdown directly as a format parameter is that
email clients are likely to handle it immediately; I'm pretty sure that
unexpected text subtypes are probably rejected or mishandled by the
deployed base of clients, counter to the specifications.

The disadvantage is that we'd need to consider how that'd work with quoting
styles, for example - but I'm willing to work on that, and might find some
other folk to help.

Dave.

---

From: Sean Leonard <[email-masked]>
Date: Tue, 08 Jul 2014 08:53:02 -0700

Glad to see that we're getting some traction already!

On 7/8/2014 8:23 AM, Dave Cridland wrote:
> On 8 July 2014 14:24, Sean Leonard <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     Hello Apps-Discuss:
>
>     A new Internet-Draft has been posted, proposing the text/markdown
>     media type to identify Markdown content.
>
>     Markdown is a family of lightweight markup languages (aka plain
>     text formatting syntaxes) designed to make it easy to read and
>     write in plain text, with the intention of having tools process
>     the text into formal markup languages, namely (X)HTML. This media
>     type is specifically intended to identify Markdown text, i.e.,
>     text intended for the Markdown tool or its derivatives.
>
>
> This looks good.
>
> text/plain itself has a "format" parameter, which includes the simple 
> (and default) "fixed", and a "flowed". I wonder whether a "markdown" 
> format here would work better (or alongside) a new media subtype? (I'm 
> honestly not sure).
>
> email clients are likely to handle it immediately; I'm pretty sure 
> that unexpected text subtypes are probably rejected or mishandled by 
> the deployed base of clients, counter to the specifications.
>
> The disadvantage is that we'd need to consider how that'd work with 
> quoting styles, for example - but I'm willing to work on that, and 
> might find some other folk to help.

I gave it some thought, and reviewed RFC 3676.

I can see format=markdown potentially working; however, there are some 
drawbacks. Most notably, there are security drawbacks.

Markdown allows for islands of HTML (or other markup); thus a valid 
Markdown document could be 99% HTML, full of <script> tags, CSS things, 
and other goodies. From a security perspective, Markdown needs to be 
parsed much more carefully than text/plain; format=flowed. This alone 
suggests to me that it is better to treat it as a separate type.

Realistically if you're going to use Markdown in e-mail, your e-mail 
client is better off converting your Markdown on-the-fly during compose, 
so that the e-mail client sends an e-mail with text/html. This is what 
word processors have been doing for decades: for example, Microsoft Word 
will convert "[newline] * [text]" to a bulleted list. (And since Outlook 
uses Microsoft Word as a formatting engine, that is what actually happens.)

The use case I had in mind for text/markdown was less e-mail per-se, and 
more as an annotation for storage. For example, the "data liberation" 
movement aims to export data from systems in common formats, 
particularly web apps. It therefore makes sense for exporting what a 
user actually typed into the system as text/markdown, rather than plain 
text (which is not accurate--the user is putting *, -, 1. 2. 3, and so 
on in order to show HTML in a web app). Similarly, Markdown has gotten 
very catchy for software documentation such as "readme.md" files--these 
should be identified as Markdown.

Overall I think that Markdown is closest to text/troff -- see RFC 4263.

The gist of RFC 3676 (to me--I don't have experience with the long and 
tortuous history) is to deal with the fact that MIME, various e-mail 
systems, and "historical practice" impose CRLFs at discrete intervals 
even though that is not the author's intent. As an example, this e-mail 
has a lot of paragraphs, but no intended single-line breaks; 
nevertheless, the SMTP standard (to say the least) imposes a mandatory 
CRLF after at most 998 octets. To get around this, not only does my mail 
client (Thunderbird) mark the text as format=flowed; it also encodes it 
with quoted-printable. By using quoted-printable, the source text 
actually uses very few line breaks at all--the only ones are 
<CRLF><CRLF> to mark a new paragraph. In the modern world it would make 
a lot of sense if text had a "natural" paragraph separator, and in fact 
it does (U+2029 PARAGRAPH SEPARATOR), but neither common software nor 
Internet standards employ that control character as such.

I think that the format parameter is more limited in scope to deal with 
e-mail standards issues. Flowed makes less sense if you are exchanging 
data that is not limited to MIME or SMTP line length limits. 
text/markdown has a much broader aim than the format parameter, so even 
e-mail clients would have to execute a different code path (with 
security checks) much closer to text/html than text/plain.

-Sean

---

From: Dave Cridland <[email-masked]>
Date: Tue, 8 Jul 2014 17:24:20 +0100

On 8 July 2014 16:53, Sean Leonard <[email-masked]> wrote:

>> (and default) "fixed", and a "flowed". I wonder whether a "markdown" format
>> here would work better (or alongside) a new media subtype? (I'm honestly
>> not sure).
>>
> Markdown document could be 99% HTML, full of <script> tags, CSS things, and
> other goodies. From a security perspective, Markdown needs to be parsed
> much more carefully than text/plain; format=flowed. This alone suggests to
> me that it is better to treat it as a separate type.
>
>
Yeouch. I didn't realise that one, and yes, with that in mind you're right.

Thanks, though, for looking into it.

Dave.

---

From: Dave Crocker <[email-masked]>
Date: Tue, 08 Jul 2014 09:30:41 -0700

> http://daringfireball.net/projects/markdown/

Markdown certainly has gotten good traction.  So having a media type
registered for it will probably be useful.

>From the draft, it is not clear to me what the constants are, for
markdown.  That is, across 'flavors', what will /not/ vary, beyond
simply using text?  And where is this constancy defined, either within
the document or as a normative reference.

It is possible that this really should be a single, non-extensible media
type, absent other examples to register now or in the near term.

The markdown 'standard' flavor registration entry, in the IANA
Considerations section, includes two normative references that should be
cited in the draft's normative references section.

I don't remember the RFC rules for making normative references that only
land on a web page, rather than having a 'formally' published citation.
 I seem to recall that only having a URL for the references is an issue,
even for an Informational RFC.

Purely a nit, but having the initial IANA table entry be named
'standard' implies that any other entries won't be standard.  And this
is an Informational RFC; so the word 'standard' could be confusing.  I
suggest a less encumbered term, like initial or basic or
gruber-schwartz, or somesuch...

d/

-- 
Dave Crocker
Brandenburg InternetWorking
bbiw.net

---

From: Sean Leonard <[email-masked]>
Date: Tue, 08 Jul 2014 15:33:37 -0700

On 7/8/2014 7:09 AM, Erik Wilde wrote:
> that they recommend to use as a profile URI, or they could elect to 
> not register and either make the URI self-documenting, or make people 
> google it and find information that way.
>
> though, would be to use URIs as identifiers, instead of the strings 
> that you are currently using.
>

I am interested in this profiles concept/identifier in lieu of flavors.

However, I would like more input from other tool maintainers/Markdown 
derivatives, particularly those mentioned in the Wikipedia article:
GitHub Flavored Markdown (GFM) [there are several implementations]
SourceForge Markdown [unclear if a derivative or just standard Markdown]
sundown [unclear if a derivative language or actually just standard 
Markdown]
PageDown (StackExchange Markdown)
MultiMarkdown

...there are others. Here is one list:
http://www.w3.org/community/markdown/wiki/MarkdownImplementations

It is unclear how many are a) pure implementations of Standard Markdown 
[and by "standard", I mean "the original Markdown 1.0.1 Perl script from 
2004"], b) bona-fide variations, c) something in between (i.e., pretty 
much Standard Markdown with "one extra gimmick" or some such).

I think that the ideal situation is for the tool's author/maintainer to 
declare, "I am making this tool to output such-and-such flavor of 
Markdown", whether that is the standard or his/her/their innovation. If 
we just use URIs there may be a temptation for unaffiliated third 
parties to register or use URIs without declarative commitment. Without 
the tool author/maintainer's participation, the URI may be pointing to a 
moving target. (This addresses Dave Crocker's point about what is 
considered normative or not.)

It seems okay to me if the tool author/maintainer wants to make that 
variation of Markdown a moving target--basically it becomes their 
problem. They have a community of users to deal with, so they can figure 
it out.

-Sean

---

From: Sean Leonard <[email-masked]>
Date: Tue, 08 Jul 2014 15:40:50 -0700

> cited in the draft's normative references section.
OK. Done.

> even for an Informational RFC.

Ok, that's something to research--I am not familiar with all the rules. 
In any event, the Markdown community pretty much relies on those 
webpages for the "standard" (or "original") syntax and implementation.

> gruber-schwartz, or somesuch...

It could also be called "Original". I like Original or Standard. The 
term "Standard Markdown" is taken from 
<https://help.github.com/articles/github-flavored-markdown>.

Sean

---

From: Larry Masinter <[email-masked]>
Date: Tue, 8 Jul 2014 23:12:50 +0000

A 'flavor' isn't (or shouldn't be) a 'profile'. Strictly speaking,
a profile of a file format or protocol is a restriction of the 
original, where instances following the profile also can
be interpreted correctly by any implementation of the
full format or protocol. 

My impression is that the markdown flavors are not
Subsets and add features as well  as restrict them.

I think that means a text/markdown media type will
Be less useful, because there is no 'superset of
all other flavors' because two flavors could use
the same constructs in conflicting ways.

Larry
--
http://larry.masinter.net

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 9 Jul 2014 11:43:59 +1000

Maybe a bunch of media types need to be registered then, for the "bigger" flavours? 

Doing so might encourage some convergence.

On 9 Jul 2014, at 9:12 am, Larry Masinter <[email-masked]> wrote:

> _______________________________________________
> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss

--
Mark Nottingham   https://www.mnot.net/

---

From: James M Snell <[email-masked]>
Date: Tue, 8 Jul 2014 20:04:09 -0700

The issue really isn't that different from what we have with the
suffixes +json and +xml. Just define a basic text/markdown and a
+markdown suffix. Then flavors are defined as variations on the
suffix. e.g. "text/vnd.github+markdown",
"text/vnd.stackoverflow+markdown" etc.

That said, noting the previous comment about Markdown flavors that
allow embedded HTML, using "application/" would seem to make more
sense than "text/"

- James

---

From: Mark Nottingham <[email-masked]>
Date: Wed, 9 Jul 2014 13:09:51 +1000

I thought about that, but these variants have the same problem that using the 'profile' parameter does; they're not subsets of markdown, they're incompatible variants.

---

From: Erik Wilde <[email-masked]>
Date: Wed, 09 Jul 2014 09:58:05 +0200

hello larry.

> Subsets and add features as well  as restrict them.

i am not claiming to define 'profile' in a general way, but the way RFC 
6906 defines them (http://tools.ietf.org/html/rfc6906#section-3) covers 
what larry is saying here. essentially, it says that a profile may be 
adding constraints, but it can also add extensions, as long as those 
extensions are within the bounds of the underlying extension model. if 
you can process something without knowing a profile and get to a 
reasonable result, then using profiles makes sense.

on the other hand, we have this comment from mark:

> I thought about that, but these variants have the same problem that using the 'profile' parameter does; they're not subsets of markdown, they're incompatible variants.

if the various flavors of markdown indeed are incompatible, i.e. 
processing one of them using the general rules does not create 
reasonable results, then indeed profiles would not be a good concept to use.

generally speaking, profiles are useful if there is a solid foundation 
everybody can fall back to. if there isn't, then maybe it's like mark 
suggested: admit that there is fragmentation, and register media types 
for all those variants that are incompatible.

personally speaking, i think it may be hard to find out if and how 
formats are incompatible. most of them seem to be underspecified (no 
clear escaping and extension rules), so at the very least you may end up 
having to write a bunch of test cases and see what happens.

cheers,

dret.

---

From: Erik Wilde <[email-masked]>
Date: Wed, 09 Jul 2014 10:07:21 +0200

hello sean.

> much Standard Markdown with "one extra gimmick" or some such).

i think the biggest problem here is that understanding features may not 
be that hard (and they are probably documented somewhere), but 
understanding in which way certain variants support or break extension 
points may be quite a bit harder.

i tried to better understand some flavors a while ago when starting with 
the original and then trying to understand what github allows me to do. 
i found it rather hard to fully understand the features, limitations, 
and escape mechanisms from the handful of examples the typical 
"documentation" pages provided.

> considered normative or not.)

i think you have a great goal here, but it may be hard to achieve. in 
the end, the profile URI simply signifies that there is some agreed-upon 
and well-defined way how something is defined. if there isn't really, 
because there is no document that actually defines all the rules 
comprehensively, then this is mostly a problem of the rather loose way 
how many markdown variants are "defined".

> problem. They have a community of users to deal with, so they can figure

you may end up having various people claiming to identify the same 
variant, but using a different URI for identifying it, and having subtle 
but undocumented ways in which their implementations differ. but like i 
said above, that would simply reflect the current state of affairs, and 
not make things any better or worse as they are now. and like you say 
(and i think mark made the same point), maybe at least it would raise a 
bit of awareness, and encourage people to define their language variants 
better.

cheers,

dret.

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 9 Jul 2014 10:40:11 +0200

On 09 Jul 2014, at 00:40, Sean Leonard <[email-masked]> wrote:

> It could also be called "Original". I like Original or Standard. 

Original: Gruber’s 2004 spec.  While implementations of just that are now mostly obsolete, most everyone implements the contents of this spec, but there are some differences in the details around indentation detection, nested lists, and emphasis rules (variations).

Extended: Various additional interpretations of text as markup.  E.g., smartypants, the extensions made popular by php markdown extra (definition lists, fenced code blocks, tables, …), maruku (metadata at file, block, span level) etc.  Each extension interprets what would have been text in original markdown and gives that some additional processing.  While extensions are usually designed not to hit meaningful original markdown by accident, they do.  There is no “extension model” per se, so anything proper in original markdown might become something different by adding an extension.

Standard: Original markdown plus the extensions and variations that one would expect today (e.g., those by php markdown extra).  As there is no further specification beyond Gruber 2004 that claims to be authoritative outside a single implementation, this is a bit fuzzy. A quick look at major implementations does show that there is pretty much a canonical set of extensions.  We might spend some time capturing “standard markdown” (finally writing this up would be a quite worthwhile exercise in its own right), or we might avoid a rathole and never use this term.

Flavor: A euphemism for a willful violation of the rules laid down by original markup.  E.g., github changed markdown’s soft newlines to be hard in gfm.  Flavors are typically much more incompatible to each other than variations or original to extended.

Grüße, Carsten

---

From: Graham Klyne <[email-masked]>
Date: Wed, 09 Jul 2014 11:54:01 +0100

> allow embedded HTML, using "application/" would seem to make more

FWIW, allowing embedded HTML is part of the original Markdown spec, and as such 
would not be an extension.

Yet a key goal of the original spec is to be readable as text: "The overriding 
design goal for Markdown’s formatting syntax is to make it as readable as 
possible. The idea is that a Markdown-formatted document should be publishable 
as-is, as plain text, ..." [http://daringfireball.net/projects/markdown/], so I 
think the normal case should be to promote textual display by using `text/...`.

#g
--

---

From: Graham Klyne <[email-masked]>
Date: Wed, 09 Jul 2014 11:54:10 +0100

On 09/07/2014 08:58, Erik Wilde wrote:
> personally speaking, i think it may be hard to find out if and how formats are
> incompatible. most of them seem to be underspecified (no clear escaping and
> extension rules), so at the very least you may end up having to write a bunch of
> test cases and see what happens.

FWIW, there's useful information about a number of extensions in the Pandoc 
Markdown documentation at 
http://johnmacfarlane.net/pandoc/demo/example9/pandocs-markdown.html

#g
--

---

From: Mark Baker <[email-masked]>
Date: Wed, 9 Jul 2014 08:12:13 -0400

> Maybe a bunch of media types need to be registered then, for the "bigger"
flavours?

Exactly.

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Wed, 9 Jul 2014 06:02:48 -0700

> > Maybe a bunch of media types need to be registered then, for the

Sounds like there's non-trivial interest in this work.  Is this possibly a
new APPSAWG work item, or is there a better home for it?  Or have I
misjudged our interest in pursuing the work?

If the answer is that there's real work here and we want to do it, then we
can talk about starting a Call For Adoption in Toronto.

Since it's specifically a media types question, I've asked Ned to comment
as well, though I note that Mark Baker is also a DE for media types.

-MSK, co-chairin'

---

From: Ned Freed <[email-masked]>
Date: Wed, 09 Jul 2014 10:06:18 -0700 (PDT)

> I thought about that, but these variants have the same problem that using the
> 'profile' parameter does; they're not subsets of markdown, they're incompatible
> variants.

And even the variants themselves aren't all that precisely defined. Which IMO
makes them a mismatch for the +suffix concept, even if we did something like
+markdown-variant.

				Ned

---

From: Dave Crocker <[email-masked]>
Date: Wed, 09 Jul 2014 11:53:40 -0700

> makes them a mismatch for the +suffix concept, even if we did something like

The way this thread has progressed suggests to me that the spec is
better as a single, specific media type, with no variant option.

When a next form of markdown gets traction, give it its own media type
entry.

d/

---

From: Erik Wilde <[email-masked]>
Date: Wed, 09 Jul 2014 21:01:03 +0200

> better as a single, specific media type, with no variant option.

adding a profile parameter for those variants that are truly, properly, 
extensions and could still be interpreted as the vanilla media type 
wouldn't hurt, would it? it would make it easier for well-behaving 
extensions to actually be treated as proper extensions, instead of being 
something entirely different.

cheers,

dret.

---

From: Dave Crocker <[email-masked]>
Date: Wed, 09 Jul 2014 12:05:17 -0700

> something entirely different.

Adding an option that has no immediate, obvious, significant utility
causes folks to think they've got a feature that they don't really have,
because it won't really be tested adequately during development and
deployment.

d/

---

From: Ned Freed <[email-masked]>
Date: Wed, 09 Jul 2014 13:55:47 -0700 (PDT)

> better as a single, specific media type, with no variant option.

My understanding is that there are already multiple variants in play, so
you'd need multiple media types right off as well as an effort to get
them all registered.

> When a next form of markdown gets traction, give it its own media type

I'm really not a fan of this approach, but then again, my position as
reviewer may bias me somewhat here.

				Ned

---

From: James M Snell <[email-masked]>
Date: Wed, 9 Jul 2014 15:01:16 -0700

Since the variants are already incompatible with one another there are
really only three possibilities:

1. We define a single media type with a semantic that says, "while
documents identified by this media type are related, they may be
completely incompatible from one another, so therefore you have to go
look at this other bit of information to tell what you're really
looking at" (i.e. the variant or profile parameter)
   example: application/markdown;variant=github,
application/markdown;variant=stackoverflow

or

2. We acknowledge that while each of these variants are related, their
incompatibility with one another requires them to have their own
distinct media types... effectively treating them as entirely separate
formats.
   example: application/vnd.github-md, application/vnd.stackoverflow-markdown

or

3. We go with the +markdown suffix approach, which acknowledges the
related natures of the formats, but still allows each to be handled as
if they were separate formats. (it's essentially a shortcut version of
option #1)
   example: application/vnd.github+markdown,
application/vnd.stackoverflow+markdown

We would simply have to acknowledge that +markdown variants can be
incompatible with one another. That can be specified in the definition
of the +markdown suffix.

- James

---

From: Dave Crocker <[email-masked]>
Date: Wed, 09 Jul 2014 16:09:00 -0700

> reviewer may bias me somewhat here.

As others have noted, these 'variants' are incompatible.

They might have overlap, but they aren't proper subsets.  Consequently
I'm not hearing benefit in having them share namespace, distinguished by
an associated parameter.

d/

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 10 Jul 2014 01:25:05 +0200

As Graham noted, pandoc shows us the way here:

1) define tags, one for each of the extensions and willful violations ("flavors”).
   The tags defined by pandoc could serve as a starter set here.
	http://johnmacfarlane.net/pandoc/demo/example9/pandocs-markdown.html
   Add an IANA registry with “specification required” policy.
2) mark an instance by the set of tags that apply to it (or, possibly could apply to it)
3) “Accept:” instances by the set of tags that can be processed (i.e. any subset is acceptable).

If the registry has numeric values added, the sets could be efficiently indicated as a numerically encoded bit set.
(Otherwise the tags get a bit unwieldy, e.g. gfm = pipe_tables, raw_html, tex_math_single_backslash, fenced_code_blocks, fenced_code_attributes, auto_identifiers, ascii_identifiers, backtick_code_blocks, autolink_bare_uris, intraword_underscores, strikeout, hard_line_breaks.)

So this doesn’t seem too hard to do.

Worth it?

Grüße, Carsten

---

From: Dave Crocker <[email-masked]>
Date: Wed, 09 Jul 2014 16:29:01 -0700

> So this doesn’t seem too hard to do.

The goal here is to have the correct software processing the data.

So, there is a requirement to 'dispatch' to the correct code and a
requirement to hand it useful parameters.

If parameters are used for dispatching to unrelated code, then the
question is why that wasn't better achieved with a media-type registration?

Again, my reading of the discussion here is that the 'variants' are
incompatible. I take that to mean that each uses a different code base.

If, instead, there is a common code base that gets 'tuned' by something
like a flavors parameter, then it makes sense to use a parameter.

d/

---

From: Dave Crocker <[email-masked]>
Date: Wed, 09 Jul 2014 16:31:03 -0700

On 7/9/2014 6:02 AM, Murray S. Kucherawy wrote:
> Sounds like there's non-trivial interest in this work.  Is this possibly
> a new APPSAWG work item, or is there a better home for it?  Or have I

There's obviously interest in the topic, but I haven't seen anything
that looks like specification work, other than the need to resolve a
labeling/registration framework.

d/

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 10 Jul 2014 01:42:05 +0200

> like a flavors parameter, then it makes sense to use a parameter.

Right now, pandoc is the only code base I’m aware of that is fully tunable by such a set of flags.
Other code bases usually have a number of (more coarse-grained) settings; they could grow into the direction of more fine-grained control if that is deemed desirable.
(In the end, such a direction of evolution might improve interoperability tremendously, if enough implementers care.)

Grüße, Carsten

---

From: Mark Nottingham <[email-masked]>
Date: Thu, 10 Jul 2014 09:43:30 +1000

> you'd need multiple media types right off as well as an effort to get

+1. github's variant is quite popular, as is kramdown.

I think someone needs to go and do a deeper analysis of compatibility between the formats, and then make a proposal along the lines of:

- a "main" markdown media type that also allows profiles / variants that are compatible with it
- some number of derivative, incompatible-but-widely-used media types

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 9 Jul 2014 21:25:39 -0400

I suggest that we start off with just one spec.

Chances are that if there is a defined standard that makes sense, many folk
will gravitate towards it because whatever the advantages/disadvantages of
one flavor, going to a standard is usually better.

And rather than just picking what the authors think is their favorite, I
suggest that we look at the userbase. And there Wikipedia is going to win
hands down because they have a userbase of millions editing that format.
And that is the one most other folk who don't roll their own seem to follow.

Lets not begin by proliferating. Do one, do it well and then see if there
is a need for others.

When I get time, I plan to add markdown input to my rfctool. Because most
RFCs are simply slabs of text interrupted by section and paragraph breaks
and don't even need HTML.

---

From: Sean Leonard <[email-masked]>
Date: Thu, 10 Jul 2014 02:17:08 -0700

Wow; well it looks like there is definitely solid interest in this 
proposal. Great!

> - some number of derivative, incompatible-but-widely-used media types

I have started to do just that. A big point of confusion seems to stem 
around just what exactly "Markdown" is, and what "Markdown" means. 
People who view Markdown differently will naturally gravitate to 
different conclusions about how to structure the media type(s).

Other than the users, the most important stakeholders to this Markdown 
discussion are the Markdown developers. The original author, John 
Gruber, has been absent for years. I am getting discussions going in the 
community to understand what *they* think Markdown is. Once we have a 
working definition that both/all communities can agree upon, we can have 
a more goal-oriented discussion about what media type(s) and 
parameter(s) best represent the Markdown universe.

-Sean

---

From: Karl Dubost <[email-masked]>
Date: Thu, 10 Jul 2014 19:16:37 +0900

Mark,

Le 10 juil. 2014 à 18:17, Sean Leonard <[email-masked]> a écrit :
> I have started to do just that. 

There is already work going on about that. 

John MacFarlane has created a very good tool for comparing the output of different processors.
http://johnmacfarlane.net/babelmark2/

And Ciro Santilli and I have a test suite with a report showing the discrepancies and common features. The goal was to start through a test suite and what was deployed in the wild for eventually writing a spec (not mandatory).
https://github.com/karlcow/markdown-testsuite
https://github.com/karlcow/markdown-testsuite/blob/master/markdown-spec.html

We could certainly improve the output if there is interest.

-- 
Karl Dubost 🐄
http://www.la-grange.net/karl/

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 10 Jul 2014 11:22:45 -0400

I predict that when you look at the variants they will divide up into one
set that is based on some sort of FSR or Lexer and the other being based on
pattern matching.

While these approaches are close in theory, there are very important
differences in practice. Pattern matching tends to be done without regard
for quoting or escape mechanisms and make assumptions about the well
formedness of the text.

There is a school of HTTP header 'parsing' that just scans for the text
sequences of interest. So if someone puts in a header like the following,
chaos ensues:

X-Ignore: Content-Length: 103
Content-Length: 2048

Of course scanning for "\r\nContent-Length:" would not fall into this trap.
But folk tend not to think that way.

---

From: Ned Freed <[email-masked]>
Date: Thu, 10 Jul 2014 09:15:24 -0700 (PDT)

> I'm not hearing benefit in having them share namespace, distinguished by

There's huge benefit. By requiring separate registrations, you'll effectively
guarantee that nobody is going to bother registering many if not most of the
variants. Or worse, they'll send around variants incorrectly tagged with one 
of the registered names. The process, even the far more lightweight one
associated with vendor tree registrations, is simply too onerous for most
people to bother with.

There's tons of "running code" to back up my assessment here.

More generally, having now read a bunch of material on this family of formats,
it seems there's a markedly different engineering aesthetic in play here. It's
one where things are intentionally not locked down, where random people are
encouraged to experiment and create incompatible variants, where heuristic
processing is an intrinsic part of the process, and where things not working
quite right is tolerated. This is all about ease of use, not correctness.

This is emphatically not the approach the IETF takes to engineering, where the
primary goal is interoperability. And so, predictably, there have been multiple
efforts to in effect force this work into the IETF mold. I've seen calls for
someone (who?) to perform an analysis of what's in play here (which even if
possible ignores the fact that what's in play may be very different a year from
now). There have been calls for separate registrions (see above). There have
been calls for what is effectively a capability list. There have been calls for
what amounts to a subregistry. There have even been calls for making this into
a media suffix, which would effectively require a full formal specification of
the format.

All of these suggestions are no doubt well intentioned, but IMO, at least
counterproductive if not wrongminded in this case.

It would be one thing if we were being asked to publish the format as a
standards-track RFC. That would absolutely call for doing a bunch of these
things.

But we're not being asked to do that. We're being asked to publish a 
registration of a name and set of parameters for something that already exists.
Nothing more.

Now, maybe doing this is inappropriate. Maybe this type belongs in the  vendor
tree, where I note that there's not going to be a problem registering this
type. Or maybe the registration should be run through a different standards
org like the W3C that also has access to the standards tree.

My personal opinion is that defining this type, more or less as-is, in the
standards tree is fine as long as the security considerations are beefed up
substantially. I would actually prefer the approach of making this a format
value for text/plain, but I think the security issues make that a bad idea.

				Ned

---

From: Sean Leonard <[email-masked]>
Date: Fri, 11 Jul 2014 01:29:24 -0700

I agree with most of what Ned said, which is what led me to write the 
draft in the first place.

> There's tons of "running code" to back up my assessment here.

Yes. In the real world, most people use text/x-WHATEVER and call it a 
day, for years. It makes it into products, then it makes it into 
exchange formats, then it's all over the place.

> [...] We're being asked to publish a
> tree, [...]

The security issues alone warrant making Markdown content with its own, 
separate type (or series of types). Fundamentally Markdown is much 
closer to text/html than text/plain: it is one step away from being 
converted to HTML (or some other document type that can include active 
content, such as application/pdf), and can contain "islands" of markup. 
It may be a "lightweight" markup language, but it's still a markup language.

As for the vendor tree, I ruled that out early on in the drafting. 
Section 3.2 of RFC 6838 states:

    The vendor tree is used for media types associated with publicly
    available products.  "Vendor" and "producer" are construed very
    broadly in this context and are considered equivalent.  Note that
    industry consortia as well as non-commercial entities that do not
    qualify as recognized standards-related organizations can quite
    appropriately register media types in the vendor tree.

    A registration may be placed in the vendor tree by anyone who needs
    to interchange files associated with some product or set of products.
    However, the registration properly belongs to the vendor or
    organization producing the software that employs the type being
    registered, [...]

In this case, Markdown has become a techno-cultural phenomenon (like 
"mashups" and "emoji")--a force that is no longer owned by any one 
vendor or producer. Comparable lightweight markup languages are 
controlled by known entities or specific (if loose) industry consortia, 
such as:
BBCode  = bulletin board software vendors (UBB, vBulletin, etc.)
reStructuredText = Python
Javadoc = Java / Oracle
MediaWiki syntax = MediaWiki

One reason why Markdown has gotten such wide uptake has been because of 
its laissez-faire management approach: One guy did some stuff and then 
other guys did other stuff in other languages and then it became part of 
common web toolkits and then people extended it some more in their toolkits.

Thus: standards tree may not be the ideal fit, but it seems to me to be 
the best fit of all of the options. A lot of value will be generated 
simply by reflecting what Markdown is (as a technology and as a 
community), rather than shoehorning it into a process that neither the 
technology nor the community are likely to follow.

Whatever it is, I think it is important to make any registration 
procedure as simple and axiomatic as possible.

Sean

---

From: Ned Freed <[email-masked]>
Date: Mon, 14 Jul 2014 08:23:25 -0700 (PDT)

> The goal here is to have the correct software processing the data.

That's one goal of media types, and the one associated with email. But
it isn't the only goal. The other obvious one is that of type label for
stored objects.

In the case of markdown, as previously stated, it's probably better for an
email client sending markdown with the intent of displaying it the result
immediately at the receiving end to use it as an input format only, and to
convert it to text/html prior to sending.

> requirement to hand it useful parameters.

It falls short of a requirement in this case.

> question is why that wasn't better achieved with a media-type registration?

Because 20+ years of expereience say that wan't happen. Indeed, despite having
been around for quite a while, I don't believe there's a registration for
anything identified as a markdown variant.

But let's suppose for a moment that this time things are different, that
people will rush to register all the variants because, well, I don't
know why they would, but let's suppose.

What would the likely outcome of this be? If existing handling tables for
various "groups" of media types are any indication, the likely outcome will be
that all of the types (at least the ones people bother to put in the table,
which is another issue) point at a single, common handler, which then does
whatever. The odds of a proliferation of separate handlers in order to handle
minor incompatibilites a bit better seems highly unlikely. For one thing think
about the increase in code and associated vulnerabilities this would cause.

> incompatible. I take that to mean that each uses a different code base.

Yes, the variants are, in the strict sense, incompatible. But there's a broad
swatch of features and functionality that are generally supported by all of
them. And more to the point, even when dealing with truly incompatible
handling, the nature of the format is such that something sensible, abeit
suboptimal, tends to happen even when the wrong processor is used.

But you need not take my word for it. The Babelmark2 tool lets you compare
how 20+ different tools handle the same input:

I think you'll find that while it's easy to construct inputs that produce
different outputs, creating inputs that work fine in one variant but produce
garbage in another is a lot more difficult

Indeed, I find the biggest annoyances by far to the limitations of the format
itself. Which is as it should be.

> like a flavors parameter, then it makes sense to use a parameter.

To the extent that this matters, I think this is the best outcome we
can hope for in this case.

Finally, I cannot help but once again note that this has a lot more to do with
incomtible engineering aesthetics than anything else. People here are writing 
like "incompatibility" in this context means the same thing as it does for a
protocol or a format carrying critical protocol-related information. Sure, when
protocols or the formats they depend on are incompatible, things tend to go
badly wrong. But that's not applicable here.

				Ned