Subject: Re: [netmod] Where can I use <CODE BEGINS> <CODE ENDS> ? - rfcstrip

From: Balázs Lengyel <[email-masked]>
Date: Mon, 30 Mar 2020 16:10:26 +0000

Hello Kent, 

Sorry for not providing a clear pointer to my work. As you have already found it, I just include it here for reference to others: https://github.com/BalazsLengyel/rfcstrip 

I assume ggrep is the gnu grep ? Maybe we should make that a variable for grep like we have for AWK=awk

I do not see the additional ^@ at EOF.  I tested it on cygwin, which is mostly GNU compatible.

This is how my file ends

3c 2f 69 6e 73 74 61 6e 63 65 2d 64 61 74 61 2d 73 65 74 3e 00 ---- </instance-data-set>

Regards Balazs

From: Kent Watsen <[email-masked]> 
Sent: 2020. március 29., vasárnap 0:18
To: Balázs Lengyel <[email-masked]>
Cc: [email-masked]
Subject: Re: [netmod] Where can I use <CODE BEGINS> <CODE ENDS> ?

Firstly, thank you for trying to improve our tooling!

It took me some time to find, for others: https://github.com/BalazsLengyel/rfcstrip

Running `rfcstrip -a` throws errors on MacOS.  My workaround fix was to ` brew install grep` and 's/grep/ggrep/' on line 483.

All The extracted files (including the two that are NOT folded) contain a ‘^@‘ character at the EOF.   This must be something introduced by `rfcstrip` because `rfcfold` does a simple file-level copy when unfolding isn't needed.

Kent // contributor

On Mar 26, 2020, at 5:45 PM, Balázs Lengyel <balazs.lengyel=[email-masked] <mailto:balazs.lengyel=[email-masked]> > wrote:

Hello,
I created an update to rfcstrip in Github.

Revision 1.1balazs - 2020-03-26
added -a option. If it is specified, the file must be an XML2RFC
XML file.
All artworks will be extracted from it if they carry a name attribute.
Artwork unfolding will also be executed.
If the artwork contains the , <CODE BEGIN/END> markers
they are removed.

Regards Balazs

-----Original Message-----
From: Balázs Lengyel <balazs.lengyel=[email-masked] <mailto:balazs.lengyel=[email-masked]> > 
Sent: 2020. március 26., csütörtök 11:55
To: Kent Watsen <[email-masked] <mailto:[email-masked]> >; Balázs Lengyel <[email-masked] <mailto:[email-masked]> >
Cc: [email-masked] <mailto:[email-masked]> 
Subject: RE: [netmod] Where can I use <CODE BEGINS> <CODE ENDS> ?

Hello Kent,
OK, as you are strongly opposed to it, I will remove <CODE BEGINS> <CODE ENDS> in the next version.
That said, I do not really agree with you. If interested see, below.
Regards Balazs

-----Original Message-----
From: netmod <[email-masked] <mailto:[email-masked]> > On Behalf Of Kent Watsen
Sent: 2020. március 25., szerda 18:25
To: Balázs Lengyel <balazs.lengyel=[email-masked] <mailto:balazs.lengyel=[email-masked]> >
Cc: [email-masked] <mailto:[email-masked]> 
Subject: Re: [netmod] Where can I use <CODE BEGINS> <CODE ENDS> ?

Balazs,

While possible to use the markers for instance examples...

 - they’ve never been used before, AFIAK  (historical context)
BALAZS: S0 I am a revolutionary :-)  
As I understand validation of examples have always been an issue, so making that more simple by automating extraction would be a step in the good direction.
 - they're unneeded, as examples don’t contain copyright (per Lada’s comment)
BALAZS: I agree that copyright is a non-issue for these examples. However <CODE BEGINS> is needed for another reason: easy extraction, which would make checking the data simpler.
 - they're disallowed by RFC 8407 for example modules (per Benoit comment)
BALAZS: strictly speaking that is about YANG examples not instance data examples. I also don't know/understand the reasoning behind it. A separate example tag would be nice, but we don't have one.
 - as a contributor, I don’t wish to birth a new convention of using the markers for examples.  

In order to support programmatic extraction of examples from drafts, please just set the “name” attribute on the <sourcecode> or <artwork> element in the XML draft. 
BALAZS: As far as I understand sourcecode is only available in XML2RFC v3.  Name is not handled by rfcstrip or xym. (I don't have time to update the tools right now.) It would be nice to have an update to rfcstrip that checks every <artwork> and if it has a name attribute, it extracts the content into a file called "name".

2.48.2.  "name" Attribute

  A filename suitable for the contents (such as for extraction to a
  local file).  This attribute can be helpful for other kinds of tools
  (such as automated syntax checkers, which work by extracting the
  source code).  Note that the "name" attribute does not need to be
  unique for <artwork> elements in a document.  If multiple
  <sourcecode> elements have the same "name" attribute, a formatter
  might assume that the elements are all fragments of a single file,
  and such a formatter can collect those fragments for later
  processing.

Kent

On Mar 24, 2020, at 11:33 AM, Balázs Lengyel <balazs.lengyel=[email-masked] <mailto:balazs.lengyel=[email-masked]> > wrote:

Hello,
A reply from the rfc-editor on <CODE BEGINS>. It seems it is OK.
Regards Balazs

-----Original Message-----
From: Alice Russo <[email-masked] <mailto:[email-masked]> > 
Sent: 2020. március 24., kedd 16:00
To: Balázs Lengyel <[email-masked] <mailto:[email-masked]> >
Cc: RFC Editor <[email-masked] <mailto:[email-masked]> >
Subject: Re: Where can I use <CODE BEGINS> <CODE ENDS> ?

Greetings,

Please see below.

On Mar 21, 2020, at 8:30 AM, Balázs Lengyel <[email-masked] <mailto:[email-masked]> > wrote:

Hello,
Sorry if I am sending this question to the wrong person. 
We have a discussion in the NETMOD workgroup about when is it appropriate/allowed to use the <CODE BEGINS> <CODE ENDS> markings in drafts and RFCs.
We regularly use it to bracket the YANG modules defined. 
e.g.    <CODE BEGINS> file "[email-masked] <mailto:[email-masked]> " 

Just FYI, if you are using the new XML vocabulary [1], the markers can be created automatically by using markers="true"
-- for example --
<sourcecode name="[email-masked] <mailto:[email-masked]> " type="yang" markers="true">

However is it allowed to use it to bracket XML or json examples in drafts/rfcs? 
e.g.    <CODE BEGINS> file [email-masked] <mailto:[email-masked]> 

Yes, it is allowed. The markers <CODE BEGINS> and <CODE ENDS> are mentioned in the Trust Legal Provisions [2].

[1] https://protect2.fireeye.com/v1/url?k=50261255-0cacc750-502652ce-861010bc36ff-8b3716f9fa495535 <https://protect2.fireeye.com/v1/url?k=50261255-0cacc750-502652ce-861010bc36ff-8b3716f9fa495535&q=1&e=11f6fa43-0168-4d1f-8f5d-bffd46fff0d3&u=https%3A%2F%2Fwww.rfc-editor.org%2Fmaterials%2FFAQ-xml2rfcv3.html> &q=1&e=11f6fa43-0168-4d1f-8f5d-bffd46fff0d3&u=https%3A%2F%2Fwww.rfc-editor.org%2Fmaterials%2FFAQ-xml2rfcv3.html
[2] https://trustee.ietf.org/license-info/IETF-TLP-5.htm (item 4d)

Thank you.
RFC Editor/ar

It is very handy that the rfcstrip tool can extract the CODE sections from a draft which facilitates checking these by external tools like yanglint. The checking is needed both for the YANG modules and the examples.
If you as an rfc-editor could give guidance on the issue it would be appreciated. I attached a planned draft.
Best regards Balazs Lengyel 

-- 
Balazs Lengyel                    Senior Specialist                       Ericsson Hungary Ltd. 
Mobile: +36-70-330-7909              email: [email-masked] <mailto:[email-masked]> 

<draft-ietf-netmod-yang-instance-file-format-11.txt>

_______________________________________________
netmod mailing list
[email-masked] <mailto:[email-masked]> 
https://www.ietf.org/mailman/listinfo/netmod

_______________________________________________
netmod mailing list
[email-masked] <mailto:[email-masked]> 
https://www.ietf.org/mailman/listinfo/netmod
_______________________________________________
netmod mailing list
[email-masked] <mailto:[email-masked]> 
https://www.ietf.org/mailman/listinfo/netmod

---

From: Kent Watsen <[email-masked]>
Date: Mon, 30 Mar 2020 23:11:56 +0000

> On Mar 30, 2020, at 12:10 PM, Balázs Lengyel <[email-masked]> wrote:
> 
> Hello Kent, 
> Sorry for not providing a clear pointer to my work. As you have already found it, I just include it here for reference to others: https://github.com/BalazsLengyel/rfcstrip <https://github.com/BalazsLengyel/rfcstrip>

Yes, and so you don’t feel isolated, please note (Martin) that I have to use “AWK=gawk” on MacOS as well…this has nothing to do with Balazs’s recent change.

> 3c 2f 69 6e 73 74 61 6e 63 65 2d 64 61 74 61 2d 73 65 74 3e 00 ---- </instance-data-set>

I’d say to “please test on MacOS”, but it seems that Martin’s commit from today no longer contains that `grep` call: https://github.com/mbj4668/rfcstrip/blob/mbj/artwork/rfcstrip <https://github.com/mbj4668/rfcstrip/blob/mbj/artwork/rfcstrip>

BTW, Martin, running mbj/artwork on MacOS:

WITH AWK=awk:

$ ~/Code/GitHub/mbj4668/rfcstrip/rfcstrip -a draft-ietf-netmod-yang-instance-file-format-10.xml 
awk: cmd. line:13: warning: regexp escape sequence `\"' is not a known regexp operator
example1.xml.folded: 82 lines.
awk: cmd. line:13: warning: regexp escape sequence `\"' is not a known regexp operator
example2.xml: 32 lines.
awk: cmd. line:13: warning: regexp escape sequence `\"' is not a known regexp operator
example3.json: 26 lines.

WITH AWK=gawk:

$ ~/Code/GitHub/mbj4668/rfcstrip/rfcstrip -a draft-ietf-netmod-yang-instance-file-format-10.xml 
gawk: cmd. line:13: warning: regexp escape sequence `\"' is not a known regexp operator
example1.xml.folded: 82 lines.
gawk: cmd. line:13: warning: regexp escape sequence `\"' is not a known regexp operator
example2.xml: 32 lines.
gawk: cmd. line:13: warning: regexp escape sequence `\"' is not a known regexp operator
example3.json: 26 lines.

Kent // contributor

---

From: Erik Auerswald <[email-masked]>
Date: Tue, 31 Mar 2020 11:18:18 +0200

Hi,

On 30.03.20 18:10, Balázs Lengyel wrote:
 > > [...]
 > > All The extracted files (including the two that are NOT folded)
 > > contain a ‘^@‘ character at the EOF.   This must be something
 > > introduced by `rfcstrip` because `rfcfold` does a simple file-level
 > > copy when unfolding isn't needed.
> [...]

This shows a NUL byte at the end.  In "cat -A" output a NUL byte is
shown as "^@":

     $ printf -- '%s' 3c 2f 69 6e 73 74 61 6e 63 65 2d 64 61 74 61 2d \
                      73 65 74 3e 00 | xxd -r -p | cat -A
</instance-data-set>^@

Your use of "grep -z" shows that NUL bytes are used as line endings
instead of NL bytes ("Line Feed" character, '\n').

I did not look into this further since Martin seems to be working on an
alternate implementation.

Thanks,
Erik