Subject: [core] CBOR YANG encoding of union & bits [draft-ietf-core-yang-cbor-12]

From: "Rob Wilton (rwilton)" <[email-masked]>
Date: Wed, 6 May 2020 10:43:49 +0000

Hi,

[Cross posting to Netmod for wider YANG expertise.]

I had a couple of questions on the CBOR YANG encoding of bits and unions (based on draft-ietf-core-yang-cbor-12).  Apologies if these have already been discussed/closed in the WGLC.

1) Regarding the encoding of the bits datatype:
[draft-ietf-core-yang-cbor-12, section 6.7]

The CBOR YANG encoding of the bits datatype is defined as a byte string encoding of a bitfield.  However, my concern here is that YANG bit positions are allowed to be up 2^32-1.

E.g. the following, pathologically bad, bits definition would seem to have a very inefficient encoding in CBOR:

   typedef alarm-state {
     type bits {
       bit unknown {
         position 4000000000;
       }
       bit under-repair;
       bit critical;
       bit major;
       bit minor;
       bit warning;
       bit indeterminate;
     }
   }

How much should we be concerned about this pathological case?  Would it be reasonable for implementations to reject bitfields larger than a small set size (e.g. perhaps 256 bits)?

Or, if it is supported by the language then is it reasonable that implementation SHOULD support it?  In which case I think that we might need a second encoding of bits that supports this pathological case.  Perhaps an array of 'set' bit positions, or alternatively the union string encoding of bits could be used.

2) Regarding the encoding of unions:

I was questioning whether the special encoding of bits type within a union was required in CBOR [draft-ietf-core-yang-cbor-12, section 6.7].  Am I right to presume that this is to ensure that the CBOR encoding of unions is always at least as specific as XML?  If so, this seems like a reasonable design choice.  But that leads on to these general YANG questions:

Should the value encoding of a YANG union type behave the same way regardless of whether the encoding is XML/JSON/CBOR?  Or is it reasonable for there to be differences in the case of conflicting values?  Perhaps this is already answered by RFC 7951 that can behave differently from the XML encoding of unions.

Longer term, should YANG be looking for a discriminated-union type?  Or perhaps it would be sufficient for tooling to flag up potentially ambiguous union definitions, particularly those that may be encoding dependent.

Regards,
Rob

[As an individual contributor]

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 6 May 2020 17:52:43 +0200

Hi Rob,

> E.g. the following, pathologically bad, bits definition would seem to have a very inefficient encoding in CBOR:

My knee jerk reaction would be “don’t do that then” [1].

[1]: http://catb.org/jargon/html/D/Don-t-do-that-then-.html

> Or, if it is supported by the language then is it reasonable that implementation SHOULD support it?  In which case I think that we might need a second encoding of bits that supports this pathologic…

This could reduce the attack vector by malicious YANG specification, but I think an arbitrary limit (such as the 256 mentioned above) should work, too.
(Except that arbitrary limits always come back to bite you, but that’s what we have software maintenance contracts for.)

> I was questioning whether the special encoding of bits type within a union was required in CBOR [draft-ietf-core-yang-cbor-12, section 6.7].  Am I right to presume that this is to ensure that the CBOR encoding of unions is always at least as specific as XML?  If so, this seems like a reasonable design choice.  

The semantics of Union types in YANG is closely married to the XML representation of YANG; if two branches of a union have a (non-empty) intersection in their XML encoding, this ambiguity is pretty much considered a feature.
So YANG-CBOR falls back to something that looks more like XML encoding (i.e., is less efficient) in unions.
The one nice thing is that a YANG spec writer can avoid the pathology by not using unions in this way; unfortunately, the fallback cannot rely on recognizing that the intersection is non-empty.

It occurs to me that this fallback could also be used for the above anomalies; we would need to agree on a threshold when that becomes active.

> But that leads on to these general YANG questions:
> 
> Should the value encoding of a YANG union type behave the same way regardless of whether the encoding is XML/JSON/CBOR?  

Of course it “should”, while retaining reasonableness for each of the encodings, but that ship seems to have sailed (at least for 1.0/1.1).

> Or is it reasonable for there to be differences in the case of conflicting values?  Perhaps this is already answered by RFC 7951 that can behave differently from the XML encoding of unions.
> 

These are all good questions.
I don’t have an answer, except that the marriage of YANG to XML semantics probably should get a divorce at some point.

Grüße, Carsten

---

From: Andy Bierman <[email-masked]>
Date: Wed, 6 May 2020 09:14:44 -0700

On Wed, May 6, 2020 at 8:52 AM Carsten Bormann <[email-masked]> wrote:

> > The CBOR YANG encoding of the bits datatype is defined as a byte string
> encoding of a bitfield.  However, my concern here is that YANG bit
> positions are allowed to be up 2^32-1.
> >
> > E.g. the following, pathologically bad, bits definition would seem to
> have a very inefficient encoding in CBOR:
> >
> >   typedef alarm-state {
> >     type bits {
> >       bit unknown {
> >         position 4000000000;
>
I remember this issue was discussed by the WG.
The only real reason to assign explicit bit positions is to match some
protocol specification,
which would never waste space in a bitset.

Rob has a valid concern -- what if a large bit position is encountered?

> >       }
> >       bit under-repair;
> >       bit critical;
> >       bit major;
> >       bit minor;
> >       bit warning;
> >       bit indeterminate;
> >     }
> >   }
> >
> >
> > How much should we be concerned about this pathological case?  Would it
> be reasonable for implementations to reject bitfields larger than a small
> set size (e.g. perhaps 256 bits)?
> >
> > Or, if it is supported by the language then is it reasonable that
> implementation SHOULD support it?  In which case I think that we might need
> a second encoding of bits that supports this pathological case.  Perhaps an
> array of 'set' bit positions, or alternatively the union string encoding of
> bits could be used.
>
> This could reduce the attack vector by malicious YANG specification, but I
> think an arbitrary limit (such as the 256 mentioned above) should work, too.
> (Except that arbitrary limits always come back to bite you, but that’s
> what we have software maintenance contracts for.)
>
>

This is the approach we started with for the union type but ended up
changing it later.
It is better to fully support YANG semantics with a less efficient
encoding, rather than
tag some YANG modules as "unsupportable".

> > I was questioning whether the special encoding of bits type within a
> union was required in CBOR [draft-ietf-core-yang-cbor-12, section 6.7].  Am
> I right to presume that this is to ensure that the CBOR encoding of unions
> is always at least as specific as XML?  If so, this seems like a reasonable
> design choice.
>
> The semantics of Union types in YANG is closely married to the XML
> representation of YANG; if two branches of a union have a (non-empty)
> intersection in their XML encoding, this ambiguity is pretty much
> considered a feature.
> The one nice thing is that a YANG spec writer can avoid the pathology by
> not using unions in this way; unfortunately, the fallback cannot rely on
> recognizing that the intersection is non-empty.
>
> It occurs to me that this fallback could also be used for the above
> anomalies; we would need to agree on a threshold when that becomes active.
>
>

Would it be possible to have 2 encoding formats for bits?
   1) binary encoding using bit positions (only bit positions 0 -- 255
allowed)
   2) YANG XML encoding using bit names (any bit positions over position
255)

The client and server already have to deal with the YANG XML encoding for
union types
so this would not be that different.

> > Should the value encoding of a YANG union type behave the same way
> regardless of whether the encoding is XML/JSON/CBOR?
>
> Of course it “should”, while retaining reasonableness for each of the
> encodings, but that ship seems to have sailed (at least for 1.0/1.1).
>
> > Or is it reasonable for there to be differences in the case of
> conflicting values?  Perhaps this is already answered by RFC 7951 that can
> behave differently from the XML encoding of unions.
> >
> > Longer term, should YANG be looking for a discriminated-union type?  Or
> perhaps it would be sufficient for tooling to flag up potentially ambiguous
> union definitions, particularly those that may be encoding dependent.
>
> I don’t have an answer, except that the marriage of YANG to XML semantics
> probably should get a divorce at some point.
>
>

This is already the case but the RFCs have not caught up yet.
If YANG is ever updated the XML-specific text will probably get moved to a
separate document.

> Grüße, Carsten

Andy

> _______________________________________________
> netmod mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/netmod
>

---

From: Michael Richardson <[email-masked]>
Date: Wed, 06 May 2020 13:21:24 -0400

Carsten Bormann <[email-masked]> wrote:
    >> implementation SHOULD support it?  In which case I think that we might
    >> need a second encoding of bits that supports this pathological case.
    >> Perhaps an array of 'set' bit positions, or alternatively the union
    >> string encoding of bits could be used.

    > This could reduce the attack vector by malicious YANG specification,
    > but I think an arbitrary limit (such as the 256 mentioned above) should
    > work, too.  (Except that arbitrary limits always come back to bite you,
    > but that’s what we have software maintenance contracts for.)

I think that the YANG knows what the maximum value is.
So I think that anyone generating code from specific YANG definitions could
size their array based upon that value.
If it's a generic decoder for YANG encoded content, then it might need
a hint from the application, or it could have a compile-time arbitrary limit.

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: Andy Bierman <[email-masked]>
Date: Thu, 7 May 2020 20:27:35 -0700

> a hint from the application, or it could have a compile-time arbitrary
There are 2 competing goals: fidelity and efficiency

For network management and telemetry, fidelity is critical.
Missing data and altered data are dealbreakers.

I think 1 point Rob is making is that the bitset encoding can end up being
much less efficient
than an XSD-style list of tokens.  Encoding bit position 4 million requires
an octet string 500,000 bytes long.
Obviously not a workable solution, even for an unconstrained device/network.
Also not OK to drop bit values that seem too big for the implementation.
That is not reliable enough.

Why is the bit position allowed to be a uint32 in YANG? Who knows, but it
has to be supported.

--
> _______________________

Andy

> ________________________
> core mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/core
>

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 8 May 2020 08:22:50 +0200

On 2020-05-08, at 05:27, Andy Bierman <[email-masked]> wrote:
> 
> Why is the bit position allowed to be a uint32 in YANG? Who knows, but it has to be supported.

If we think that is the way to go, I like Kio’s proposal over in the CBOR list:
<https://mailarchive.ietf.org/arch/msg/cbor/3sZ4YfWLzmVVnNnQbttkovgjTP8>

We probably should arm that with some text that says (in nicer words) that this is an emergency representation and implementations should not invoke it for sane YANG models (with an operable definition of sane).

Grüße, Carsten

---

From: Andy Bierman <[email-masked]>
Date: Fri, 8 May 2020 08:51:40 -0700

> <https://mailarchive.ietf.org/arch/msg/cbor/3sZ4YfWLzmVVnNnQbttkovgjTP8>

Yes -- this encoding should work well

> We probably should arm that with some text that says (in nicer words) that
> this is an emergency representation and implementations should not invoke
> it for sane YANG models (with an operable definition of sane).
>
>

Not sure what this means.
Wouldn't this map approach be used all the time for a bits type?
The "normal" case would be a small number of octets representing all the
bits present
 (e..g 0x7 for 3 bits : bit 0 to bit 2)
This would simply be 1 map entry with {offset=0,length=1,value=0x7}

The actual bitmap is conceptually constructed by starting with an array of
zero bytes.
Overlapping offsets are allowed. Duplicate offsets are allowed.
There is no requirement to list offsets in ascending order.
There is no canonical representation.

Each map entry is added to the array using a "bit or" operator.
After all map entries are processed the full bits value is known.
This allows an implementation to encode bits serially (often done this way
with YANG bit names)

So the same bitset 0x7 could be sent different ways including:

  { 0, 1, 0x1 }
  { 0, 1, 0x2 }
  { 0, 1, 0x4 }

If the position is really bit 4 million (bit 1 in byte 500,000)

  { 0, 500000, 0x1 }

Andy

---

From: Andy Bierman <[email-masked]>
Date: Fri, 8 May 2020 08:58:14 -0700

On Fri, May 8, 2020 at 8:51 AM Andy Bierman <[email-masked]> wrote:

>> We probably should arm that with some text that says (in nicer words)
>> that this is an emergency representation and implementations should not
>> invoke it for sane YANG models (with an operable definition of sane).
>>
>>
>
oops: correction:  (bit 0 in byte 500000)

  { 500000, 1, 0x1 }

---

From: Jim Schaad <[email-masked]>
Date: Fri, 8 May 2020 10:48:06 -0700

Does yang consider that there is a difference between a bit being present and zero and a bit being absent?

From: core <[email-masked]> On Behalf Of Andy Bierman
Sent: Friday, May 8, 2020 8:58 AM
To: Carsten Bormann <[email-masked]>
Cc: [email-masked]; [email-masked]
Subject: Re: [core] [netmod] CBOR YANG encoding of union & bits [draft-ietf-core-yang-cbor-12]

On Fri, May 8, 2020 at 8:51 AM Andy Bierman <[email-masked] <mailto:[email-masked]> > wrote:

On Thu, May 7, 2020 at 11:22 PM Carsten Bormann <[email-masked] <mailto:[email-masked]> > wrote:

 

 

The "normal" case would be a small number of octets representing all the bits present

The actual bitmap is conceptually constructed by starting with an array of zero bytes.

This allows an implementation to encode bits serially (often done this way with YANG bit names)

  { 0, 1, 0x1 }

  { 0, 1, 0x2 }

  { 0, 1, 0x4 }

  { 0, 500000, 0x1 }

  { 500000, 1, 0x1 }

Grüße, Carsten

Andy

---

From: Andy Bierman <[email-masked]>
Date: Fri, 8 May 2020 10:56:22 -0700

> and zero and a bit being absent?

  leaf bits-leaf {
      type bits {
         bit zero { position 0; }
         bit one { position 1; }
         bit two { position 2; }
         bit 4million { position 4000000; }
     }
  }

I think I see your point.
The value "" is valid for this leaf indicating no bits are set.
In order to send this value:
   { 0, 1, 0x0 }

Any YANG bit that is present is represented by a bit with the value 1 in
the corresponding position and octet.

Andy

*From:* core <[email-masked]> *On Behalf Of *Andy Bierman
> *Sent:* Friday, May 8, 2020 8:58 AM
> *To:* Carsten Bormann <[email-masked]>
> *Cc:* [email-masked]; [email-masked]
> *Subject:* Re: [core] [netmod] CBOR YANG encoding of union & bits
> [draft-ietf-core-yang-cbor-12]
>
>
>
>
>
>
>

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 8 May 2020 20:13:07 +0200

> present and zero and a bit being absent?

In YANG every bit in the bit set is either 0 or 1. The xml / json
encodings send the position of the 1 bits (actually the names bound to
the position). All other bits default to 0.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
Fax:   +49 421 200 3103         <https://www.jacobs-university.de/>