Subject: [Cbor] Some questions on CDDL

From: Kevin Braun <[email-masked]>
Date: Thu, 6 Jul 2017 17:33:21 -0400

Hi folks,

I am investigating CDDL and have several questions and comments.

1. When a value (e.g. a text string, byte string, integer, or float) 
appears in a rule, what does that means for the CBOR data?  The document 
doesn't make this explicit.  I experimented using the CDDL tool (0.8.5) 
and that didn't bring clarity.  For example, this rule:
     top = { 1 => "turkey" }
validates both of these instances:
     { 1 : "turkey" }
     { 1 : 'turkey' }

I would have thought the second instance ought to be invalid, since a 
byte string was found where a text string is  expected, but the CDDL 
spec isn't decisive on this.

Meanwhile, using the following rule, neither of the above instances will 
validate:
     top = { 1 => 'turkey' }
In both cases, the error is:
     CDDL validation failure (nil for {1=>"turkey"}):
     ["turkey", [:bytes, "turkey"], nil]

2. Related to the above question, numeric handling is confusing.  On the 
one hand,
     top = { ? "test" : [*mix-int-float] }
     mix-int-float = 1 / 2 / 10.0
will validate
     { "test" : [ 1, 2, 10.0, 1.0, 2.0, 10 ] }
which suggests that a literal float can validate an integer value and 
vice-versa.  But then, if I try defining this:
     mixed-range = 1..10.0
CDDL tool gives me an error: Incompatible range [Integer, Float]. For a 
range, integers and floats seem to be treated distinctly.  I don't know 
whether this reflects the CDDL specification's intention or is just a 
limit of the tool.
Also, this rule:
     top = { 1 => "turkey" }
will not validate
     { 1.0 : "turkey" }
which suggests that a literal integer will NOT validate a float value, 
contrary to how mix-int-float is handled.  However, the following DOES 
validate the same instance:
     top = { one => "turkey" }
     one = 1 / 10.0
which just further muddies the waters.  Again, this might all be due to 
limitations or bugs in the CDDL tool, but the CDDL spec doesn't make 
clear what should happen in these cases.

3. CDDL tool reports an error if you try to use an mapentry without a 
key inside a map, such as in:
     top= {mygroup, secondgroup}
     mygroup = ("thing-one" : uint, "thing-two" : tstr)
     secondgroup = (uint, tstr)
This is sensible, but I don't think the specification actually forbids 
this; perhaps it should.

4. When a map has duplicate keys, CDDL tool will generate data using the 
last definition of the key.  I didn't find an instance it would 
validate, though.  Perhaps the specification should require an error in 
such cases.  Example:
     top = { "thing" : uint, "thing" : tstr }
Generated value:
     {h'7468696E67': "tic"}

5. CDDL tool reports an error (`block in grpent') when trying to 
generate instances for the following schema.  I think it is valid CDDL 
and this is probably just a bug in the tool:
     top = { topgroup }

     topgroup = ( first: tstr, second: tstr,
              (option1: uint //
              option2: uint) )

6. Are there restrictions on the use of '&' to create a choice from a 
group?  I think some sensible restrictions would be:
- every mapentry in the group must have a key type which must be a 
single-value
- the group cannot use group choice (//)
- the mapentry values must be unique, single-value types?

At least, I think the above rules would be consistent with the intent of 
the "&" operator as described.

7. What happens when you have an empty group?  Section 3.9 Socket/Plug 
would lead one to believe that an empty group cannot be matched, since 
it says that an undefined socket symbol is "taken to be an empty type 
choice (group choice)" and "It is not an error if there is no definition 
for a socket at all; this then means there is no way to satisfy the rule 
(i.e., the choice is empty)."  But, the CDDL tool does not behave the 
same when a socket symbol is undefined and when it is defined as empty.  
The following instance:
     { "one" : 7, "two" : "seven" }
is validated by:
     top = { one: uint, two: tstr, extra-junk }
     extra-junk = ()
and by:
     top = { one: uint, two: tstr, $$extra-junk }
     $$extra-junk //= ()
but not by:
     top = { one: uint, two: tstr, $$extra-junk }
     ; $$extra-junk undefined

The description of an undefined socket as equivalent to an empty choice 
seems inconsistent with the observed behavior.  I don't see an inherent 
problem with this behavior; I just think that explaining the undefined 
symbol behavior in terms of an empty choice group is inaccurate and 
probably misleading (assuming the CDDL tool is work as desired).

Regards,
Kevin

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 7 Jul 2017 00:15:27 +0200

> On Jul 6, 2017, at 23:33, Kevin Braun <[email-masked]> wrote:
> 
> Hi folks,
> 
> 1. When a value (e.g. a text string, byte string, integer, or float) appears in a rule, what does that means for the CBOR data?  The document doesn't make this explicit.  I experimented using the CDDL tool (0.8.5) and that didn't bring clarity.  For example, this rule:
>    top = { 1 => "turkey" }
>    { 1 : 'turkey’ }

Bug.  Only the first one is valid with this description.

> I would have thought the second instance ought to be invalid, since a byte string was found where a text string is  expected, but the CDDL spec isn't decisive on this.

Do you have a suggestion what we should add to the spec?
The CBOR spec is quite clear that byte strings and text strings are not the same; we could add an example that demonstrates this with CDDL as well (e.g., your example).

> Meanwhile, using the following rule, neither of the above instances will validate:
>    top = { 1 => 'turkey' }

This is another expression of the same bug.

> 2. Related to the above question, numeric handling is confusing.  On the one hand,
> which suggests that a literal float can validate an integer value and vice-versa.  

This is indeed an interesting question.
In a CBOR data model, ints and floats are rather different.
In a JSON data model, they nominally aren’t (although they are very much in many practical applications of JSON).

We will need to make a few decisions here.
I would tend to go for a complete separation of integer and float, but we do need to make up our mind how JSON fits in there.  (We also, in all likelihood, need conversion operators or some such.  Ouch.)

> But then, if I try defining this:
>    mixed-range = 1..10.0
> CDDL tool gives me an error: Incompatible range [Integer, Float]. For a range, integers and floats seem to be treated distinctly.  I don't know whether this reflects the CDDL specification's intention or is just a limit of the tool.

Right, as with strings, the tool is a bit “simple-minded” (read: broken) with its idea of equality.

> Also, this rule:
>    top = { 1 => "turkey" }
> will not validate
>    { 1.0 : "turkey" }
> which suggests that a literal integer will NOT validate a float value, contrary to how mix-int-float is handled.  However, the following DOES validate the same instance:
> which just further muddies the waters.  Again, this might all be due to limitations or bugs in the CDDL tool, but the CDDL spec doesn't make clear what should happen in these cases.

Nice demos for why I’m in the process of rewriting parts of the tool at this very moment…
The current tool essentially exposes idiosyncrasies of the data model of the programming language it is written in; instead, it needs to be based on the more explicit choices we make.

> 3. CDDL tool reports an error if you try to use an mapentry without a key inside a map, such as in:
> This is sensible, but I don't think the specification actually forbids this; perhaps it should.

Yes, the spec should say that a group needs member keys in order to be used in a map.

> 4. When a map has duplicate keys, CDDL tool will generate data using the last definition of the key.  I didn't find an instance it would validate, though.  Perhaps the specification should require an error in such cases.  Example:
>    {h'7468696E67': "tic”}

Another tool bug.  First, the member key should not transmogrify into a byte string.

The CDDL spec is bad in that it can never be satisfied.  It is not really “wrong” in any other way, and it might be hard to detect this particular error in a more general case.

The tool has a bug here in that it enters a key into the map without checking whether it was present already.

> 5. CDDL tool reports an error (`block in grpent') when trying to generate instances for the following schema.  I think it is valid CDDL and this is probably just a bug in the tool:
>    top = { topgroup }
> 

There is a nice “XXX” in that source line 1038 :-)
It just doesn’t know how to handle a group choice here.
(Group choices came in when the logic already was based on knowing there weren’t any and not all cases are actually implemented at this point.)

> 6. Are there restrictions on the use of '&' to create a choice from a group?  I think some sensible restrictions would be:
> - every mapentry in the group must have a key type which must be a single-value

Why do you want this restriction?
(It doesn’t hurt very much, but I also think it doesn’t help.)

> - the mapentry values must be unique, single-value types?

Yes.

> At least, I think the above rules would be consistent with the intent of the "&" operator as described.
> 
> 7. What happens when you have an empty group?  Section 3.9 Socket/Plug would lead one to believe that an empty group cannot be matched, since it says that an undefined socket symbol is "taken to be an empty type choice (group choice)" and "It is not an error if there is no definition for a socket at all; this then means there is no way to satisfy the rule (i.e., the choice is empty)."  But, the CDDL tool does not behave the same when a socket symbol is undefined and when it is defined as empty.  The following instance:

An empty group is very different from an empty group choice.

CDDL does not currently provide a notation for an empty group choice (which would be the bottom or “fail” symbol in a grammar), so the last test case is the only way to get one.
An empty group, however, is approximately a no-op.  It can be very much a part of a group choice, besides other alternatives...
(There can’t be an empty type, as a type is always matched by exactly one data item.)

> The description of an undefined socket as equivalent to an empty choice seems inconsistent with the observed behavior.  I don't see an inherent problem with this behavior; I just think that explaining the undefined symbol behavior in terms of an empty choice group is inaccurate and probably misleading (assuming the CDDL tool is work as desired).

See above.

Thank you Kevin; I’ll turn most of your message into a number of failing tests for the CDDL tool (which will then hopefully be fixed by the partial reimplementation).

The question of equality of ints and floats (and, maybe even more difficult, bignums and decimals and bigfloats) needs some thinking and appropriate spec updates.  I’d love to hear more about the use case where you ran into the current implementation idiosyncrasies around these equality issues; those might help us make the right choices there.

Grüße, Carsten

---

From: Kevin Braun <[email-masked]>
Date: Fri, 7 Jul 2017 11:01:03 -0400

> The CBOR spec is quite clear that byte strings and text strings are not the same; we could add an example that demonstrates this with CDDL as well (e.g., your example).

I wasn't sure about it due to the fact that a byte string can be written 
using characters and due to the behavior of the tool, plus there isn't 
an explicit description of how validation works.  I thought if integer 
values can validate CBOR floats, perhaps text values can validate CBOR 
byte strings and vice versa.  I would suggest spelling out the relation 
between CDDL values and what they will validate/match in CBOR.  That 
would cover the int/float and other numerics issues too.  For example, 
"CDDL values have corresponding CBOR representations against which they 
alone will match.  The following table indicates the CBOR major type and 
(for floats) the extra information that alone can be matched for each 
kind of value:
     integer             0 or 1
     byte string        2
     text string         3
     float                 7.25 or 7.26 or 7.27
"
> (It doesn’t hurt very much, but I also think it doesn’t help.)

Well, someone could write something like:

	colors = &color_group
	color_group = ( int => 0, green: 1, blue:2)

It's pretty clear that isn't how & is intended to be used, but I believe 
it is perfectly legal as things stand.  If a tool is going to give 
special support for "choice from group", such as displaying the name 
associated with a value, it would simplify things if the tool could rely 
on there being a nice 1-to-1 name-value mapping.

> (There can’t be an empty type, as a type is always matched by exactly one data item.)

Okay.  My two cent's worth then - I wouldn't talk about empty types or 
empty choices in that section as you're introducing a concept that the 
rest of the specification doesn't give a meaning to.  I would just say 
"Tools honor that convention by not raising an error if such a type or 
group is not defined at all; the symbol is then taken to represent a 
rule that cannot be satisfied."  and then later on, "It is not an error 
if there is no definition for a socket at all; this then means there is 
no way to satisfy the rule."

> The question of equality of ints and floats (and, maybe even more difficult, bignums and decimals and bigfloats) needs some thinking and appropriate spec updates.  I’d love to hear more about the u…

Unfortunately, I don't have a concrete use case to share.  We're 
interested in commercial CBOR and CDDL tool development, so I'm just 
examining the specification and thinking about possibilities.

Thanks,
Kevin