Subject: [Tools-discuss] notes.ietf.org markdown rendering inconsistencies?

From: Jeffrey Haas <[email-masked]>
Date: Tue, 15 Feb 2022 08:07:36 -0500

From a thread to wgchairs.

The thread was helpful in discussing that we're running tooling that implement two slightly incompatible flavors of markdown.[1]  

My request for this thread is that ietf tooling should be consistent, regardless of what flavor we choose.  One shouldn't use the GUI for the notes tool, run an import and find that the output isn't rendered properly.

I've since used the wisdom from the thread to go back and update my minutes by adding the necessary newlines before lists.  However, that shouldn't have been necessary.

-- Jeff

[1] The thread essentially explained that expecting consistency from markdown is perhaps an inappropriate behavior.  While my gut response is to say we, the ietf, chose poorly I've been through 20 years of such light markup tooling.  Choose your pain.

> Begin forwarded message:
> 
> From: Jeffrey Haas <[email-masked]>
> Subject: notes.ietf.org markdown rendering inconsistencies?
> Date: February 14, 2022 at 4:21:21 PM EST
> To: [email-masked]
> 
> Since this might be a known issue, I figured I'd start here:
> 
> https://datatracker.ietf.org/meeting/interim-2022-idr-02/session/idr
> 
> The minutes were done in the notes site.  They display nicely there.  (See
> link via tiny icon at top of meeting.)
> 
> The minutes generated from the notes site render improperly for some of the
> lists.  This results in difficult to read minutes.  In particular, bullet
> lists after a statement with a colon.
> 
> Any suggestions on best way to generally address this?
> 
> And should the bug for this go to tools-discuss or a different alias?
> 
> -- Jeff

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 15 Feb 2022 14:11:50 +0100

Yep.  In Archived-At: <https://mailarchive.ietf.org/arch/msg/wgchairs/01LB4fYh23qUvS4h2Xb6l086SjA>, Robert Sparks wrote:

>> I am working on
>> 
>> * making it so that the datatracker can pull things out of 
>> 
>> notes.ietf.org for you, applying de-gfm automatically

That would be useful to have.

Grüße, Carsten

---

From: Robert Sparks <[email-masked]>
Date: Tue, 15 Feb 2022 10:41:03 -0600

> That would be useful to have.

It's _there_.

If you've used notes.ietf.org for a meeting (at the link the meeting 
tools generate), you'll get an [Import from notes.ietf.org] button on 
the minutes row on pages like 
https://sandbox-cf.ietf.org/meeting/interim-2022-cbor-03/session/cbor

(Pointing to the sandbox so you can play with it without affecting 
production)

That workflow fetches the note and runs de-gfm over it - and then gives 
you a preview of the result.

Jeff - did you follow this flow with your session? Right now

https://sandbox-cf.ietf.org/meeting/interim-2022-idr-02/session/29265/import/minutes

looks pretty good, but that's probably because you changed the relevant 
notes page already.

If that's the case, maybe there's more that we need to do than just run 
de-gfm?

>
> Grüße, Carsten
>
> ___________________________________________________________
> Tools-discuss mailing list - [email-masked]
> This list is for discussion, not for action requests or bug reports.
> * Report datatracker and mailarchive bugs to: [email-masked]
> * Report tools.ietf.org bugs to: [email-masked]
> * Report all other bugs or issues to: [email-masked]
> List info (including how to Unsubscribe): https://www.ietf.org/mailman/listinfo/tools-discuss

---

From: Jeffrey Haas <[email-masked]>
Date: Tue, 15 Feb 2022 11:46:30 -0500

Robert,

On Tue, Feb 15, 2022 at 10:41:03AM -0600, Robert Sparks wrote:
> tools generate), you'll get an [Import from notes.ietf.org] button
> on the minutes row on pages like

Yeah, I used that.  That's what started this thread.

Now, if there's the chance that my original import ran prior to that
software being installed, I'm happy to take this as an answer of, "Works
now!".

If it was present when I hit import, it likely still had issues.  I suggest
checking the inputs for my interim meeting.

> That workflow fetches the note and runs de-gfm over it - and then
> gives you a preview of the result.
> 
> looks pretty good, but that's probably because you changed the
> relevant notes page already.

I updated the inputs on notes to add the suggested empty line before bullet
lists, then re-used the import button.

I didn't try running import again without changes.

-- Jeff

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 15 Feb 2022 18:07:04 +0100

On 2022-02-15, at 17:46, Jeffrey Haas <[email-masked]> wrote:
> now!”.

Thank you for the additional details!

I just tried de-gfm’ing a version of the idr minutes from 2022-02-01 out of hedgedoc (in the menu, select revisions, then select a revision, and then download).   I used the bog standard tool Marked 2 to look at the result, and it looks fine.

Maybe there are further incompatibilities between datatracker’s markdown interpreter and the relatively standard markdown de-gfm spits out?

Grüße, Carsten

---

From: Tom Pusateri <[email-masked]>
Date: Tue, 15 Feb 2022 23:14:35 -0500

> On Feb 15, 2022, at 12:07 PM, Carsten Bormann <[email-masked]> wrote:
> 

The datatracker renders markdown using the markdown2 package from pypi:

https://pypi.org/project/markdown2/ <https://pypi.org/project/markdown2/>

It was selected because it implements the original markdown spec as written by John Gruber.

http://daringfireball.net/projects/markdown/ <http://daringfireball.net/projects/markdown/>

Markdown2 also supports some extensions as outlined in the docs here:

https://github.com/trentm/python-markdown2 <https://github.com/trentm/python-markdown2>

If a compatibility problem is found, issues can be opened at the above github page.

Thanks,
Tom

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 18 Feb 2022 10:48:53 -0500

Tom,

[description about what tools IETF uses elided]

> If a compatibility problem is found, issues can be opened at the above github page.

Clearly a compatibility issue was found.

It'd be helpful to know whether this thread has resulted in some sort of
work the tools team will pick up or whether you're saying it now falls to
the complainant to chase this all the way through the tools producers that
the team has chosen to leverage.

Mostly so I can know whether it's worth bothering to report problems in the
future.  :-)

-- Jeff

---

From: Tom Pusateri <[email-masked]>
Date: Fri, 18 Feb 2022 12:04:01 -0500

Jeffrey,
	Sorry, I did not intend to infer that you’re on your own. Obviously, we want all the tools to work together in a compatible way. My purpose was to inform that the datatracker was trying to follow the Markdown spec as much as possible. With Markdown, this has been difficult because the spec is under-specified.

“Embrace and Extend” has been the rule of thumb for any project using Markdown it seems so incompatibilities are everywhere.

Carsten’s note seemed to indicate he tried to reproduce the incompatibility from the original note and after running it through the de-github markdown extension tool, he was not able to reproduce it. Maybe I misread that.

So given the currently know information, can you reproduce it and explain how so I can dig into more?

Thanks,
Tom

> On Feb 18, 2022, at 10:54 AM, Jeffrey Haas <[email-masked]> wrote:
> 
> Tom,
>

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 18 Feb 2022 18:23:10 +0100

> So given the currently know information, can you reproduce it and explain how so I can dig into more?

I have a copy of the original form of the hedgedoc notes, but it is not easy to use the datatracker sandbox to try to recreate the complete situation.  But trying that further is certainly on my list…

Grüße, Carsten

---

From: Robert Sparks <[email-masked]>
Date: Fri, 18 Feb 2022 13:19:26 -0600

> I have a copy of the original form of the hedgedoc notes, but it is not easy to use the datatracker sandbox to try to recreate the complete situation.  But trying that further is certainly on my li…
And mine

---

From: Robert Sparks <[email-masked]>
Date: Fri, 18 Feb 2022 14:40:49 -0600

> Mostly so I can know whether it's worth bothering to report problems in the

Hey Jeff - that feels a little hostile, even with the emoticon - I hope 
that's just email getting in the way.

Yes, we're going to improve the notes->minutes pipeline. The issue you 
ran into is one I think we can address.

What Carsten and I have been trying to convey is that the entire problem 
space around markdown is Hard (even the browser community hasn't come 
together on what its mime-type should be), and to get the really nice 
features of notes vs what we need for archival of minutes, there will be 
compromises, sometimes in frustrating places.

RjS

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 18 Feb 2022 15:47:35 -0500

> On Feb 18, 2022, at 12:23 PM, Carsten Bormann <[email-masked]> wrote:
> 

FWIW, the only changes I did from -00 of the notes to the currently published -01 of the notes was take any bullet list preceded by a descriptive sentence ending in a colon is added a newline between the two.

So, this is hopefully easy to reproduce from the published content.

-- Jeff

---

From: Jeffrey Haas <[email-masked]>
Date: Fri, 18 Feb 2022 16:02:05 -0500

Robert,

> On Feb 18, 2022, at 3:40 PM, Robert Sparks <[email-masked]> wrote:
> 
> 
> Hey Jeff - that feels a little hostile, even with the emoticon - I hope that's just email getting in the way.

I can certainly see how it can be interpreted as hostile.  Please instead consider it as "resigned to life sucking for the one reporting the issue" after having multiple such interactions this week across different organizations.  One example included reporting a valid issue on very large commercial software from a certain large Washington State company in the US and knowing that I get to suffer likely without a fix and another on a small software system built on open source stuff that could get better but only if I'm willing to do the work myself.

I know we're here in IETF on IT issues in a weird mix of community support and a certain amount of contracted support.  So, frankly, while I don't intend to come off as pissy, I also don't have the energy to do the fight to have done this bit of work on my own.  And I seriously wanted to know if that's what I was being told yet again this week.

> 
> Yes, we're going to improve the notes->minutes pipeline. The issue you ran into is one I think we can address.

I'm glad to hear that's the case!  I'd be satisfied with "a ticket has been filed for later work" one way or the other.

> 
> What Carsten and I have been trying to convey is that the entire problem space around markdown is Hard (even the browser community hasn't come together on what its mime-type should be), and to get the really nice features of notes vs what we need for archival of minutes, there will be compromises, sometimes in frustrating places.

As discussed in the wgchairs thread, this is hardly my first trip through human-friendly markup hell.  After the initial exchange I was readily able to rectify my problem, so the key thing was addressed.  The remainder was my end of community service in making sure the discrepancy was noted so that the next party wasn't similarly impacted.

You all are doing the IETF community good service by doing this stuff.  I owe you all a libation of choice when we get the opportunity to meet in person next.  But forgive me if I'm a bit soul weary from other similar interactions  with much less happy outcomes this week.

-- Jeff

---

From: Robert Sparks <[email-masked]>
Date: Fri, 18 Feb 2022 15:17:19 -0600

Trimming -

> I'm glad to hear that's the case!  I'd be satisfied with "a ticket has been filed for later work" one way or the other.
The equivalent is happening, but I think there are several things we're 
going to touch and I'm still discovering places to touch. I'll send a 
pointer to how it will be tracked when that's concrete (I considered a 
placeholder in trac, but I think it would get obviated with other places 
to track very quickly.)
>
>
>
> You all are doing the IETF community good service by doing this stuff.  I owe you all a libation of choice when we get the opportunity to meet in person next.
I really look forward to that.
>   But forgive me if I'm a bit soul weary from other similar interactions  with much less happy outcomes this week.

Yes, I definitely understand that pain.