Subject: Re: [Cellar] [FFmpeg/FFV1] update to xml2rfc v3 (#135)

From: Michael Richardson <[email-masked]>
Date: Thu, 10 Jan 2019 11:55:54 -0500

Michael Niedermayer <[email-masked]> wrote:
    mcr>     Yes, we moved to v3, which is not perhaps widely available yet.

    > yeah, and we also wrote in the README that xml2rfc 2.5.1 is
    > sufficient. You know we write that in the README in the VERY same pull
    > req that adds the requirement of v3. Maybe iam missing something and
    > surely my statement is a bit blunt but that combination does not look
    > ok to me

    >             xml2rfc: error: no such option: --v3 Makefile:12: recipe
    > for target

Hi, I wonder if you are getting the right version of xml2rfc?
Could you have more than one installed?
Dave, any comments?

    > I think some possibly dumb question, is there a requirement to depend
    > on v3 fo IETF texts ?  I mean can we just stay compatible with v2 ? or
    > is that adding workload to us. I do not want to add some extra workload
    > on every future change, that would be bad too.

We made the decision to move to v3 because:
  a) we have math in the document which v3 does a really nice job on.
  b) because its useful for the IETF to have someone clueful trying v3.

That was on the list about a month ago.

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: Dave Rice <[email-masked]>
Date: Thu, 10 Jan 2019 12:07:13 -0500

> On Jan 10, 2019, at 11:55 AM, Michael Richardson <[email-masked]> wrote:
> 
> 

The build works for me with the mmark and xml2rfc noted in the pull request. On my system I get. With the older xml2rfc and mmark, the make process would fail.

$mmark -version
2.0.7

$xml2rfc -V
xml2rfc 2.12.3

> That was on the list about a month ago.

I reviewed the comments in https://github.com/FFmpeg/FFV1/pull/135 and here’s a summary of my considerations.

The currently head of the repo requires mmark version 1.3.4 and xml2rfc 2.5.1, while the PR requires mmark version 2.0.7, xml2rfc 2.12.3. IMHO using more recent releases is preferred but I understand the argument that recent releases may not have been well disseminated through all package repositories yet.

The makefile process doesn’t fail gracefully. There is no versioning check of the dependencies so if the Makefile of the PR is run with the older versions of mmark or xml2rfc, it will fail. I think this is an issue but independent to the PR since the issue is the same with or without the PR.

The PR adds dependencies with xmlstarlet and pdfcrop. This is a disadvantage for the PR as the make process is more complex; however the advantage is that we can use v3 and have better representations of the equations rather than the ascii art version. Also the RFC outputs that support equations can supersede the custom PDF output that we had been using so a lot of the lines we were managing that were output specific can be removed so that the source document is simplified.

Also the makefile could certainly be improved and I’d wish I could contribute better to it; however the Makefiles in the cellar repositories are my first efforts at writing Makefiles and I’m not sure how to efficiently add version checks and tests to the Makefile. At any rate, with or without the PR the issues of whether the Makefile fails gracefully are the same.

Best Regards,
Dave Rice

---

From: Michael Niedermayer <[email-masked]>
Date: Thu, 10 Jan 2019 18:28:15 +0100

Hi

> The build works for me with the mmark and xml2rfc noted in the pull request. On my system I get. With the older xml2rfc and mmark, the make process would fail.

ok, lets try again (iam a bit pretending to be a average contributor which
is who this should work for IMO)

mmark was 1.3+ on the new box i tried it on, ive upgraded it to the latest available
on any ubuntu next

mmark --version
1.3.6
This is the latest version on any ubuntu release:
https://packages.ubuntu.com/search?keywords=mmark&searchon=names
Its also the latest debian packages
https://packages.debian.org/search?keywords=mmark
so this is likely what a lot of people will try it with

but that fails still being too old

now how to semi cleanly install latest xml2rfc and mmark 2.0.7 for the next try:

virtualenv ffv1build --python=python3
. ffv1build/bin/activate
export PATH=~/ffv1build/bin:$PATH
pip install xml2rfc
cd ~/ffv1build/bin
wget https://github.com/mmarkdown/mmark/releases/download/v2.0.7/mmark_2.0.7_linux_amd64.tgz
tar -xzvf mmark_2.0.7_linux_amd64.tgz
sha256sum mmark
87b2dacfe9b68ac30c56affe031925424d2215a91016b8e55266f72c80b9fadd  mmark

and drumroll ... still fails and its not obvious what tool fails

RFC rendering has been tested with mmark version 2.0.7, xml2rfc 2.12.3, xmlstarlet 1.6.1, pdfcrop v1.38, and pdf2svg 0.2.3, please ensure these are installed and recent enough.
bash makesvg
PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek.
==> 1 page written on `context.svg_cropped.pdf'.
PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek.
==> 1 page written on `quantizationtablesets.svg_cropped.pdf'.
PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek.
==> 1 page written on `rgb1.svg_cropped.pdf'.
PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek.
==> 1 page written on `rgb2.svg_cropped.pdf'.
PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek.
==> 1 page written on `samplediff.svg_cropped.pdf'.
PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek.
==> 1 page written on `rangebinaryvalues1.svg_cropped.pdf'.
PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek.
==> 1 page written on `rangebinaryvalues2.svg_cropped.pdf'.
PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek.
==> 1 page written on `rangebinaryvalues3.svg_cropped.pdf'.
PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek.
==> 1 page written on `rangebinaryvalues4.svg_cropped.pdf'.
PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek.
==> 1 page written on `rangebinaryvalues5.svg_cropped.pdf'.
PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek.
==> 1 page written on `rangebinaryvalues6.svg_cropped.pdf'.
PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek.
==> 1 page written on `rangebinaryvalues7.svg_cropped.pdf'.
PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek.
==> 1 page written on `statetransitiontable1.svg_cropped.pdf'.
PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek.
==> 1 page written on `statetransitiontable2.svg_cropped.pdf'.
PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek.
==> 1 page written on `initialstatedelta1.svg_cropped.pdf'.
PDFCROP 1.38, 2012/11/02 - Copyright (c) 2002-2012 by Heiko Oberdiek.
==> 1 page written on `initialstatedelta2.svg_cropped.pdf'.
cat rfc_frontmatter.md "ffv1.md" rfc_backmatter.md | grep -v "^RFC:" | grep -v "^SVGC" | grep -v "{V4}" | sed "s|^RFC:||g;s|{V3}||g;s|SVGI:||g" > merged_rfchtml.md
mmark merged_rfchtml.md | xmlstarlet edit --delete '//postal|//uri' | sed 's|alt="alt"|type="svg"|g' > draft-ietf-cellar-ffv1-06.xml
xml2rfc --html --v3 draft-ietf-cellar-ffv1-06.xml -o "draft-ietf-cellar-ffv1-06.html"
Parsing file draft-ietf-cellar-ffv1-06.xml
draft-ietf-cellar-ffv1-06.xml(242): Error: Did not expect element artwork there, at /rfc/middle/section[3]/section[4]/t[2]/artwork
draft-ietf-cellar-ffv1-06.xml(250): Error: Did not expect element artwork there, at /rfc/middle/section[3]/section[5]/t[2]/artwork
draft-ietf-cellar-ffv1-06.xml(302): Error: Did not expect element artwork there, at /rfc/middle/section[3]/section[7]/section[2]/t[4]/artwork
draft-ietf-cellar-ffv1-06.xml(306): Error: Did not expect element artwork there, at /rfc/middle/section[3]/section[7]/section[2]/t[6]/artwork
draft-ietf-cellar-ffv1-06.xml(327): Error: Did not expect element artwork there, at /rfc/middle/section[3]/section[8]/t[2]/artwork
draft-ietf-cellar-ffv1-06.xml(336): Error: Did not expect element artwork there, at /rfc/middle/section[3]/section[8]/section[1]/section[1]/t[2]/artwork
draft-ietf-cellar-ffv1-06.xml(339): Error: Did not expect element artwork there, at /rfc/middle/section[3]/section[8]/section[1]/section[1]/t[3]/artwork
draft-ietf-cellar-ffv1-06.xml(342): Error: Did not expect element artwork there, at /rfc/middle/section[3]/section[8]/section[1]/section[1]/t[4]/artwork
draft-ietf-cellar-ffv1-06.xml(345): Error: Did not expect element artwork there, at /rfc/middle/section[3]/section[8]/section[1]/section[1]/t[5]/artwork
draft-ietf-cellar-ffv1-06.xml(348): Error: Did not expect element artwork there, at /rfc/middle/section[3]/section[8]/section[1]/section[1]/t[6]/artwork
draft-ietf-cellar-ffv1-06.xml(351): Error: Did not expect element artwork there, at /rfc/middle/section[3]/section[8]/section[1]/section[1]/t[7]/artwork
draft-ietf-cellar-ffv1-06.xml(354): Error: Did not expect element artwork there, at /rfc/middle/section[3]/section[8]/section[1]/section[1]/t[8]/artwork
draft-ietf-cellar-ffv1-06.xml(410): Error: Did not expect element artwork there, at /rfc/middle/section[3]/section[8]/section[1]/section[4]/t[1]/artwork
draft-ietf-cellar-ffv1-06.xml(413): Error: Did not expect element artwork there, at /rfc/middle/section[3]/section[8]/section[1]/section[4]/t[2]/artwork
draft-ietf-cellar-ffv1-06.xml(988): Error: Did not expect element artwork there, at /rfc/middle/section[4]/section[1]/section[15]/t[2]/artwork
draft-ietf-cellar-ffv1-06.xml(991): Error: Did not expect element artwork there, at /rfc/middle/section[4]/section[1]/section[15]/t[3]/artwork
draft-ietf-cellar-ffv1-06.xml(3): Error: Invalid document before running preptool.
Cannot continue, quitting now
Makefile:12: recipe for target 'draft-ietf-cellar-ffv1-06.html' failed
make: *** [draft-ietf-cellar-ffv1-06.html] Error 1

From this failure how can one know which tool needs to be upgraded ?
assuming it is a tool version issue

> The makefile process doesn’t fail gracefully. There is no versioning check of the dependencies so if the Makefile of the PR is run with the older versions of mmark or xml2rfc, it will fail. I think…

Because existing distros are more likely to have recent enough versions
prior to the pr the chance of hitting this bug should be higher after the pr
so the PR makes this worse

btw thanks for your efforts to push this forward

[...]
-- 
Michael     GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB

Those who are best at talking, realize last or never when they are wrong.