Subject: [Cbor] More formal definition of CDDL matching rules

From: Carsten Bormann <[email-masked]>
Date: Fri, 3 Nov 2017 21:24:20 +0100

On Jul 18, 2017, at 20:43, Kevin Braun <[email-masked]> wrote:
> 
> I know the question of more formally specifying validation rules already came up.

I have started writing up the matching rules, piece by piece motivated by the syntactic elements of the language.

https://cbor-wg.github.io/cddl/matching/draft-ietf-cbor-cddl.html#rfc.appendix.B

The new text is not yet finely honed yet; at this point I’m mainly interested to hear whether the approach goes into the right direction.

My plan is that we could have a -01 with an editorially combed version of this by November 11.

Grüße, Carsten

---

From: Jim Schaad <[email-masked]>
Date: Fri, 3 Nov 2017 17:34:13 -0700

My first answer, having tried to read it and not managed it, is that this is a really bad approach.  As a consumer of CDDL, I do not care what the ABNF is and how that is implemented.  What I want to know is for a FOO operation, what are the rules that I should expect.  If this is just supplementary information that is covered in the main document it is fine.

Jim

> -----Original Message-----
> From: CBOR [mailto:[email-masked]] On Behalf Of Carsten Bormann
> Sent: Friday, November 3, 2017 1:24 PM
> To: [email-masked]
> Subject: [Cbor] More formal definition of CDDL matching rules
> 
> The new text is not yet finely honed yet; at this point I’m mainly interested to
> hear whether the approach goes into the right direction.
> 
> _______________________________________________
> CBOR mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cbor

---

From: Henk Birkholz <[email-masked]>
Date: Sat, 4 Nov 2017 03:33:30 +0100

Hi Jim,

I think there are distinct advantages to include this content as an 
appendix. There most certainly is demand for the capability to check if 
a cbor message matches a certain CDDL data definition via a general 
function. And this section will be enable readers to create 
corresponding match functions.

True, it is not of interest to every consumer of the document, but it 
has that distinct purpose and maybe we just have to highlight that 
purpose more explicitly?

Viele Grüße,

Henk

---

From: Jim Schaad <[email-masked]>
Date: Fri, 3 Nov 2017 22:01:50 -0700

> From: CBOR [mailto:[email-masked]] On Behalf Of Henk Birkholz
> Sent: Friday, November 3, 2017 7:34 PM
> To: [email-masked]
> Subject: Re: [Cbor] More formal definition of CDDL matching rules
> 
> Hi Jim,
> 
> I think there are distinct advantages to include this content as an appendix.
> There most certainly is demand for the capability to check if a cbor message
> matches a certain CDDL data definition via a general function. And this section
> will be enable readers to create corresponding match functions.
> 
> True, it is not of interest to every consumer of the document, but it has that
> distinct purpose and maybe we just have to highlight that purpose more
> explicitly?

This is where I would explicitly disagree.  The matching rules are of interest to EVERY reader of the document.  The implementation details about how to do an ABNF match is of interest only to the people who want to implement a CDDL matcher.

Jim

> > My first answer, having tried to read it and not managed it, is that this is a
> really bad approach.  As a consumer of CDDL, I do not care what the ABNF is
> and how that is implemented.  What I want to know is for a FOO operation,
> what are the rules that I should expect.  If this is just supplementary
> information that is covered in the main document it is fine.
> >
> > Jim
> >
> >
> >>> I know the question of more formally specifying validation rules
> >>> already came
> >> up.
> >>
> >> I have started writing up the matching rules, piece by piece
> >> motivated by the syntactic elements of the language.
> >>
> >> The new text is not yet finely honed yet; at this point I’m mainly
> >> interested to hear whether the approach goes into the right direction.
> >>

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 4 Nov 2017 10:24:47 +0100

Hi Jim,

thank you for the very quick feedback.

The appendix uses the ABNF just as a reminder of the structure of a CDDL specification — I couldn’t come up with a more succinct way.  But the information is in the text, not the ABNF — that is just a repeat of Appendix D.

> On Nov 4, 2017, at 01:34, Jim Schaad <[email-masked]> wrote:
> 
> My first answer, having tried to read it and not managed it, is that this is a really bad approach.  As a consumer of CDDL, I do not care what the ABNF is and how that is implemented.  

The text is not about implementation (although you can indeed implement it from the text), but about the language semantics.

(I understand it may be a bit confusing to discuss a language that is in effect notating a (tree) grammar by going along the (string) grammar of the (tree) grammar language.)

> What I want to know is for a FOO operation, what are the rules that I should expect.  If this is just supplementary information that is covered in the main document it is fine.

It should already be covered in the main document, but the exact way the matching is supposed to be done has to be inferred from the text (and we have got feedback that the inference may not be obvious).  So we thought it might be a good idea to summarize the matching rules once more in the appendix.

Grüße, Carsten

---

From: Christoph Vigano <[email-masked]>
Date: Sat, 4 Nov 2017 12:20:33 +0100

Hi Carsten,

thank you for writing an explanation of the matching rules.

> It should already be covered in the main document, but the exact way the matching is supposed to be done has to be inferred from the text (and we have got feedback that the inference may not be obv…

Would it help to have more examples in the text to further showcase the
various edge cases and specific matching rules? Or would that worsen the
readability of the text?

I think my point is that the edge cases and specific matching rules
would be better understood if also shown in a specific example, as
opposed to having to extract and infer the information from a text.

Also, is there a documentation somewhere where those edge cases are
collected, so others could help writing those samples more easily?

What do you think?

Viele Grüße
Christoph

---

From: Brian E Carpenter <[email-masked]>
Date: Sun, 5 Nov 2017 08:57:13 +1300

> It should already be covered in the main document, but the exact way the matching is supposed to be done has to be inferred from the text (and we have got feedback that the inference may not be obv…

I think it is useful. It seems a little unnatural to have this Appendix *before*
the ABNF Appendix, however.

In response to Jim, as a consumer of CDDL I hope I don't need to study the ABNF
and the matching rules, but when I can't avoid it, I want as much precision as
possible. Having this material in appendices seems right to me.

   Brian

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 4 Nov 2017 22:31:30 +0100

On Nov 4, 2017, at 20:57, Brian E Carpenter <[email-masked]> wrote:
> 
>  It seems a little unnatural to have this Appendix *before*

Right.  Since many documents cite appendices of the draft directly (e.g., Appendix G for the extended diagnostic notation), I have been trying to keep their designations the same.
But I don’t think anyone cites the ABNF, so we can shuffle these around.

So the matching rules are now at
https://cbor-wg.github.io/cddl/matching/draft-ietf-cbor-cddl.html#rfc.appendix.C

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 4 Nov 2017 22:51:03 +0100

> various edge cases and specific matching rules? Or would that worsen the

There is always this tension between illustrating defining text via examples and obscuring the defining text by drowning it in a sea of examples.

I believe the really good examples are the ones based on actual protocols (such as the one restating RFC 7071 or the ones *in* RFCs such as 8007, 8152, or RFC-to-be anima-grasp).

(Recent editorial runs were heavier on the side of deleting existing examples than on adding new ones, as some of the early examples are no longer that useful.)

> opposed to having to extract and infer the information from a text.

Well, I hope there actually aren’t that many edge cases.

So far, we have found that map rules with wildcards in them may violate the POLS, and maybe it is indeed useful to collect examples in that space.

> collected, so others could help writing those samples more easily?

The CDDL github repo also has a wiki.  
We could start collecting examples there and then later decide where to put those.

In case that makes sense, I have created a page 

https://github.com/cbor-wg/cddl/wiki/Examples-for-CDDL-Matching

(Note that the actual matching instances can easily be generated in the CDDL tool.
We could leave that as an exercise for the user…  Maybe not.)

Grüße, Carsten

---

From: Sean Leonard <[email-masked]>
Date: Wed, 15 Nov 2017 14:40:29 +0800

> On Nov 5, 2017, at 3:57 AM, Brian E Carpenter <[email-masked]> wrote:
> 

I reviewed the current Appendix B and Appendix C parts.

Appendix C looks “okay” for what it is (not a complete endorsement, but I think it is a contribution to the document, and therefore is useful, so I echo Brian Carpenter’s opinion).

Appendix B’s ABNF is mostly okay, but has some stuff worth addressing.

Looks like Unicode characters in CDDL are generically considered “okay”. That is fine but draft-seantek-unicode-in-abnf-03 should be referenced for the premise of going up to %x10FFFD for Unicode.

Also, it’s not clear why you don’t want %x10FFFE and %x10FFFF, but you allow %xFFFE, %xFFFF, %x1FFFE, %x1FFFF, etc. in the definition of PCHAR and SCHAR. Either allow everything up through %x10FFFF, or use one of the predefined rules that carve out the problematic non character code points. See “UCHAR” and “UVCHAR” in the document above: they are designed for that purpose.

There are conflicts and redundancies with the way that this document defines some basic rules, and RFC 5234. For example: this document defines CRLF as %x0A or %x0D.0A. But RFC 5234’s core rules define CRLF as…wait for it!…CRLF. I don’t think you want to be mixing different line styles. Just let RFC 5234’s Core Rules define CRLF, so you don’t have to redefine it. We all “know” how to deal with line endings, and the canonical line ending format for Internet interchange is CRLF. Another source of conflict may be the way that WS is defined, since WS = SP / NL, which admits bare newline, but omits HTAB. The cddl processor(s) out there do accept HTAB as whitespace, right?

Similarly, HEXDIG is already in the Core Rules. BINDIG isn’t necessary because the Core Rules already have BIT.

If it would help, I can revise this section and post/PR the revisions.

Thanks,

Sean

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 15 Nov 2017 14:52:59 +0800

Hi Sean,

great input on the abnf (should we have abnf doctors?); will pick this up in next version. 

I don’t agree on the line ends; the name crlf is not right and needs to be fixed, but we need to allow Unix line ends.

Sent from my iPad

> On 15. Nov 2017, at 14:40, Sean Leonard <[email-masked]> wrote:
> 
>

---

From: Sean Leonard <[email-masked]>
Date: Wed, 15 Nov 2017 19:37:37 +0800

> On Nov 15, 2017, at 2:52 PM, Carsten Bormann <[email-masked]> wrote:
> 
> Hi Sean,
> 
> […]  the name crlf is not right and needs to be fixed, […]

Okay great.

> I don’t agree on the line ends; […] but we need to allow Unix line ends.

I understand the premise that you are getting at, and do not have a problem with allowing Unix line ends. I dealt with the same problem in prix-textual (RFC 7468).

I am not sure how to resolve it in the confines of ABNF, but I did a survey this afternoon of all ABNF relating to newlines in RFC 2000-RFC 7999. The majority simply call the end-of-line marker <CRLF>; there is a minority that call it other things, namely <EOL>. Of the minority, a large portion of the minority admits CRLF / LF; a much smaller minority admits CR / LF (aka any of bare CR, bare LF, or CRLF -- which means that LF CR is unambiguously two lines, but CRLF is ambiguous as to whether it is one line or two lines).

Just think about it this way: when you define the end-of-line as CRLF / LF, abnfgen will produce random combinations of CRLF and LF-terminated lines.

I will bring the matter up on abnf-discuss@, since there should be an agreed-upon colloquialism in ABNF to say “end of line”.

Thanks,

Sean

~~~~~~~
2017-11-15 Facts about CRLF and new lines in ABNF in RFC Series
Sean Leonard

Facts about CRLF and new lines:

Surveyed ABNF of all RFCs between RFC 2000 to RFC 7999.

“What is the end-of-line marker called, other than CRLF?” (by search of CRLF)

Answer: Most call it CRLF. Amongst the minority, EOL is the most common, followed by line-break or lineBreak. Most define the end-of-line marker as CR LF; in the minority, most in the minority use CRLF / LF, and a smaller subset uses CR / LF (aka any of bare CR, bare LF, or CRLF).

RFC	Hint	Name	=	Definition
2705	Megaco	EOL	=	CRLF / LF
2967*	TISDAG	nl	=	%d13 %d10
2967*	TISDAG	SEP	=	(CR LF) | LF   <-- NOT actually ABNF 
3108	SDP	EOL	=	(CR / LF / CRLF)
3435	Megaco	EOL	=	CRLF / LF
3780	SMIng	lineBreak =	CRLF / LF
4997	ROHC-FN	CRLF	=	%x0A / %x0D.0A
4997	ROHC-FN	NL	=	COMMENT / CRLF
4997	ROHC-FN	COMMENT	=	"//" *(SP / HTAB / VCHAR) CRLF
5228*	Sieve	(Sieve explicitly limits to CRLF-only in text)
5234*	ABNF	c-nl	=	comment / CRLF
5234*	ABNF	comment =	";" *(WSP / VCHAR) CRLF
6020	YANG	line-break =	CRLF / LF
7208*	SPF	CRLF	=	<standard end-of-line token as per [RFC5322]>
7468	PKIXt	eol	=	CRLF / CR / LF
7468	PKIXt	eolWSP	=	WSP / CR / LF
7468	PKIXt	W	=	WSP / CR / LF / %x0B / %x0C

“How is ‘nl’ defined?” (by search of ‘nl’)

Answer: Always CRLF.

RFC	Hint	Name	=	Definition
2957	whoispp	nl	=	%d13 %d10
2958	whoispp	nl	=	%d13 %d10
2967*	TISDAG	nl	=	%d13 %d10
4997	ROHC-FN	NL	=	COMMENT / CRLF
5234*	ABNF	c-nl	=	comment / CRLF
5234*	ABNF	comment =	";" *(WSP / VCHAR) CRLF
6787*	MRCPv2	phrase-nl =	"Phrase-NL" ":" 1*UTFCHAR CRLF

“How is ’newline’ used?” (by search of ‘newline’) 

Answer: ‘newline’ is never defined as a rule name. When it appears in comments, it is *always* associated with CR LF only. (Same as ‘nl’.)

“How is ‘0A’ used?” (by search of 0A)

Answer: ‘0A’ is almost always defined as LF or used as LF; no additional interesting variations were uncovered. The only interesting variation is RFC 4464, which has %x0a / %x0d / (others) as whitespace generically, but that protocol is not line-oriented.

* items do not address the premise of the inquiry, but is included anyway for completeness.
~~~~~~~

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 15 Nov 2017 20:17:01 +0800

Thanks for the pointer to the usage of EOL.  Since CR only line endings are entirely historic (the last major OS using them became obsolete some 15 years ago), I think defining EOL as [CR] LF seems to be the right thing. 

Sent from my iPad

> On 15. Nov 2017, at 19:37, Sean Leonard <[email-masked]> wrote:
> 
>