Subject: [netmod] looking for practical advice on managing YANG source in XML format RFCs

From: Michael Richardson <[email-masked]>
Date: Sun, 13 Jun 2021 21:09:52 -0400

Hi, I'm not a regular netmod reader.
I subscribe via IMAP to look into what's going on.  If I'm asking a FAQ, I
appologize as I didn't find anything in the 25K messages archived.
I saw Christian's message in 2019 about org-rfc-export.
I'm not sure if I should Reply-To: tools or netmod.

I'm looking for advice on how to manage things.
A summary of my questions:

1) how to process yang files with YYYY-DD-MM into XML.
2) how to generate yang tree files.
3) how do I get my YANG includes downloaded, and do I put them into my repo?
4) how to do this with MT Makefiles?

When we were working on RFC8995 and RFC8366, in the days before Martin
Thompson's makefile was around, I just put pyang processing into my simple
Makefile.  It's here:  https://github.com/anima-wg/anima-bootstrap/blob/master/Makefile

The perl script at: https://github.com/anima-wg/anima-bootstrap/blob/master/insert-figures
was then developed to insert stuff into the XML.  It's rather hacky in some ways.
It started it's life as just something to find the latest @YYYY-MM-DD yang
file which the Makefile created. (Kent wrote that sed line three quarters of
a decade ago, and the sed line now is old enough to trick-or-treat)
Over time, it grew the ability to insert other bits of code or diagrams.
I keep using it, even when I've moved to Markdown because it inserts the
right artwork stuff, and knows to deal with YANG stuff.  I also keep using my
Makefile, because except for the simplest uses, the MT ones are too
complicated for me to hack.

With kramdown source, I can mostly just add stuff to the MT Makefile to
generate the files I need, and then mostly (without insert-figures), I can
use the ::include mechanism in kramdown to get the right things.  But, no
magic YYYY-MM-DD stuff.

So now I am collaborating with some co-authors which want to stick to XMLv3
rather than kramdown, and whose understanding of Makefiles is poor.  I want
to stick with the normal stuff so that all the github/etc. tooling works for
everyone involved.

I'm looking for advice on how do this this?

--
Michael Richardson <[email-masked]>   . o O ( IPv6 IøT consulting )
           Sandelman Software Works Inc, Ottawa and Worldwide

---

From: Michael Richardson <[email-masked]>
Date: Sun, 13 Jun 2021 21:19:31 -0400

> 3) how do I get my YANG includes downloaded, and do I put them into my repo?

RFC8995 has been published.
It has a YANG module, ietf-voucher-request, which is derived from the RFC8366
ietf-voucher.

RFC8366 has: ietf-voucher@2018-05-09
The DT still points at:
   https://www.yangcatalog.org/yang-search/module_details.php?module=[email-masked]
but search on that page leads me to:
  https://www.yangcatalog.org/yang-search/module_details.php?module=ietf-voucher

which has the 2018-05-09 version, although I don't know how to reference it
except for search for it.

But, finally, it's even at:
   /usr/local/lib/python3.7/dist-packages/pyang-2.3.2-py3.7.egg/share/yang/modules/ietf

so including ietf-voucher should not be a problem.

It seems that the yang modules ought to go into a different package than the
pyang itself, and ideally, I'd like to be able to rsync it.  I'd further like
if pyang could download stuff and cache it like xml2rfc (and kdrfc) does.

Until that point, what should I do?  I have been just putting the .yang files
into my repository where I derive stuff from them.
How do I get the DT to be updated on YANG pointers?

---

From: Michael Richardson <[email-masked]>
Date: Sun, 13 Jun 2021 21:33:22 -0400

> 3) how do I get my YANG includes downloaded, and do I put them into my repo?

The other part of the question is how to manage the various extensions that
are occuring.  What I know about so far: (ascii art warning. fixed font needed)
https://www.yangcatalog.org/yang-search/impact_analysis/ietf-voucher@2018-05-09

RFC8355(voucher) ----> RFC8995(voucher-request)
   \                                     \
   |\                                    |\
   | \                                   | \
   | ietf-anima-constrained-voucher      |  ietf-anima-constrained-voucher
   |     [ietf-constrained-voucher]      |   [ietf-constrained-voucher-request]
   |                                     |
   \                                     \
   |\                                     \
   | \                                     \
   |  ietf-anima-brski-cloud                ietf-anima-brski-async-enroll
   |     [ietf-redirected-voucher]            [ietf-async-voucher-request]
   \
    \
     \
      richardson-anima-voucher-delegation
         [ietf-delegated-voucher]

* ietf-{redirected,delegated}-voucher are probably bad names,
and it should be ietf-voucher-{redirected,delegated}.  Just noticed that.

So far, we haven't needed to mix anything from voucher extensions into
voucher-requests extensions.

But, having a constrained-voucher which is also a delegated-voucher does make
sense.   Since the constrained-voucher users the yang-sid to serialize to
CBOR, I don't know exactly how that is going to work.

I would sure like be sure that we know to mix and match these groupings.
I suspect that one has to write a new YANG file importing all the different
groupings?

I also suspect that it might be time for RFC8366bis that just wraps these
things all up into a single document.

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 14 Jun 2021 06:36:05 +0200

> 4) how to do this with MT Makefiles?

I people could tell me what they need, we could develop a feature in kramdown-rfc to handle this.
(This would presumably also include support for YANG-SID files.)

Grüße, Carsten

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 14 Jun 2021 09:39:55 +0200

Hi Michael,

you can check my project YANG-I-D [1] for some ideas.

The most elegant way of including YANG modules in RFCs would be to treat the RFC source as a compound XML document combining xml2rfc with YANG modules in the YIN syntax - if only xml2rfc could process it.

Lada

[1] https://github.com/llhotka/YANG-I-D

Michael Richardson <[email-masked]> writes:

> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod

-- 
Ladislav Lhotka
Head, CZ.NIC Labs
PGP Key ID: 0xB8F92B08A9F76C67

---

From: Kent Watsen <[email-masked]>
Date: Mon, 14 Jun 2021 10:21:39 +0000

Hi Michael!

There isn’t a standard of any sort, but as an active author of a large number of drafts, I’ve been force to automate as much as possible:

  - validating the schema
  - validating examples
  - generating tree diagrams
  - stitching all off the above into an XML file
  - on a per-build basis 

AFAIK, my system is more complete than used by other authors, most whom seem to do it manually, with mixed results...

I author in XML v3 with macros that get expanded in the source, you may have linked my INSERT_TEXT_IN_FILE macro earlier, that ‘insert-figures’ consumes.  It’s effectively the same, with little improvements here and there. 

There’s no “tool” to do it*, each draft begins as a copy/paste of one prior, and then the Makefile, refs/validate-all.sh, and refs/gen-trees.sh files are tweaked as needed.   For examples: see the top-10 repos listed here: https://github.com/netconf-wg. 

* I was previously working on ‘xiax’ here: https://github.com/kwatsen/xiax. 

K. 

> On Jun 13, 2021, at 9:10 PM, Michael Richardson <[email-masked]> wrote:
> 
> 
> Anima mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/anima

---

From: Michael Richardson <[email-masked]>
Date: Mon, 14 Jun 2021 10:09:37 -0400

> kramdown-rfc to handle this.

One possibility is that kramdown-rfc ought to look for, and include the
latest foo-YYYY-MM-DD.yang file, when told to ::include foo.yang.
Alternatively, it could perhaps do the YYYY-MM-DD substitution itself.

    > (This would presumably also include support for YANG-SID files.)

That's mostly just a question about including the results, and it's not that
hard thing to add to the main Makefile.

But, as I said, I xml2rfc doesn't have a useful include functionality, for
when not working with kramdown.

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 14 Jun 2021 17:02:16 +0200

> Alternatively, it could perhaps do the YYYY-MM-DD substitution itself.

How about

{::include foo-????-??-??.yang}

and, if there is no such file, kramdown-rfc expands the wild card in the directory and uses the numerically latest file?

(There might also be a use-case for actually including all these files, so I’m still in thinking mode, but I think this is close.)

> for when not working with kramdown.

What? :-)

Grüße, Carsten

---

From: Michael Richardson <[email-masked]>
Date: Mon, 14 Jun 2021 11:36:53 -0400

Carsten Bormann <[email-masked]> wrote:
    >> 1) how to process yang files with YYYY-DD-MM into XML.  2) how to
    >> generate yang tree files.  3) how do I get my YANG includes
    >> downloaded, and do I put them into my repo?  4) how to do this with MT
    >> Makefiles?

    > I people could tell me what they need, we could develop a feature in
    > kramdown-rfc to handle this.  (This would presumably also include
    > support for YANG-SID files.)

But, part of the question is, how does one do this with just XML.
As wonderful as kramdown is, and it's always my preference, sometimes it is
hard to convince a design team that has started on XML to go that way.
(sunk cost fallacy, etc.)

I looked at Kent's XIAX repo documentation, and what I saw was that it might
take XML as input, paste the YANG in, but leave enough meta-data in the XML so that
the XML can be processed again.  So this is an in-place edit.  I worry that
such a workflow would be confusing to some authors who might edit in the
wrong place.

Kent, I didn't look extensively at that repo you said to clone.

As to editing in YIN, which is an XML syntax way to represent
YANG.... well... I don't mind if someone wants to store YANG that way, but I
sure don't want to edit that way.

So at this point, the best I can do is to bring my "insert-figures" hack into
a repo, rename the XML to another name, and generate draft-foobar.xml in the
top-Makefile.

-- 
]               Never tell me the odds!                 | ipv6 mesh networks [ 
]   Michael Richardson, Sandelman Software Works        | network architect  [ 
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 14 Jun 2021 17:57:03 +0200

I can’t help you with your dark desires to use XML, but I would propose the following fix for that:

— use xi:include or src= for the XML source (I forget which one works), with a static name
— simply generate a symlink from the most recent .yang to the static name in the Makefile
— use --expand for generating the submission XML.

Grüße, Carsten

---

From: Michael Richardson <[email-masked]>
Date: Mon, 14 Jun 2021 12:26:18 -0400

Carsten Bormann <[email-masked]> wrote:
    >> One possibility is that kramdown-rfc ought to look for, and include
    >> the latest foo-YYYY-MM-DD.yang file, when told to ::include foo.yang.

    cb> How about

    cb> {::include foo-????-??-??.yang}

    cb> and, if there is no such file, kramdown-rfc expands the wild card in
    cb> the directory and uses the numerically latest file?

You'd use shell globs?
I think it might be better to use PCRE.
It might be better to have this as "winclude"

    cb> (There might also be a use-case for actually including all these
    cb> files, so I’m still in thinking mode, but I think this is close.)

    >> for when not working with kramdown.

    cb> What? :-)

Yes, it causes me confusion too.
Basically, I'm trying to contribute YANG modules (as pull-requests) to drafts
for which I'm not an author, and I don't really want to have to convince them to upgrade.

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 14 Jun 2021 18:29:51 +0200

>    cb> the directory and uses the numerically latest file?

Yes.  Any need to go beyond that?

> I think it might be better to use PCRE.

Much more typing and backslash-mangling, and I don’t see the use case.

> It might be better to have this as "winclude"

So ‘loseclude’ would be including all matches?

(What does the “w” stand for?)

Grüße, Carsten

---

From: "Salz, Rich" <[email-masked]>
Date: Mon, 14 Jun 2021 16:28:41 +0000

Netmod and anima moved to bcc.

Horrible idea to use PCRE instead of globs.

---

From: Michael Richardson <[email-masked]>
Date: Mon, 14 Jun 2021 14:03:43 -0400

> Much more typing and backslash-mangling, and I don’t see the use case.

DOS/Windows people don't get shell globs, and "?" is not something they know.

    > So ‘loseclude’ would be including all matches?

hah.

    > (What does the “w” stand for?)

wildcard.

---

From: Christian Hopps <[email-masked]>
Date: Mon, 14 Jun 2021 14:38:12 -0400

I didn’t think you were actually looking for a non-xml solution, but as you mentioned in an earlier mail there’s my org-rfc setup which makes all this yang stuff pretty simple. The source of the YANG module is in the single org-mode formatted draft source, along with support for YANG validation of the module and examples, generating YANG trees, downloading YANG dependencies etc.. :)

An example use: https://github.com/choppsv1/draft-chopps-netmod-geo-location <https://github.com/choppsv1/draft-chopps-netmod-geo-location>

Thanks,
Chris.

> On Jun 14, 2021, at 10:09 AM, Michael Richardson <[email-masked]> wrote:
> 
> 
>> kramdown-rfc to handle this.
>