Subject: [netconf] YANG encoding in CBOR

From: "Rob Wilton (rwilton)" <[email-masked]>
Date: Fri, 22 Mar 2019 12:42:47 +0000

Hi,

As part of YANG evolution discussion, that was some talk about using a binary encoding of YANG in NETCONF or RESTCONF.

CBOR looks like a good fit for this, and obviously CORE WG are working on draft-ietf-core-yang-cbor-07, but one comment came up from Andy that the CBOR encoding of YANG cannot handle all YANG data models.  In particular, because of the way that the encoding works there are limitations on how unions of enums work.  Is that still the case?

Hence I was wondering whether it would be better to encode enum values in CBOR using module-qualified names, or also assign them SIDs and use the SIDs.  Has this approach been considered at all?

Or, is there an alternative approach to how we could/should consider using CBOR as a binary encoding for YANG data in NETCONF or RESTCONF?

Do you think it would be possible to get interested parties together to discuss this at some point in Prague?

Thanks,
Rob

---

From: Michel Veillette <[email-masked]>
Date: Fri, 22 Mar 2019 15:45:27 +0000

Hi Rob

==================================================================
About "I was wondering whether it would be better to encode enum values ..."

YANG assigns either explicitly or implicitly to each enumeration, a unique integer value, see https://tools.ietf.org/html/rfc7950#section-9.6.4.2.
In CBOR, these values are used, see https://tools.ietf.org/html/draft-ietf-core-yang-cbor-07#section-6.6
When used outside of a union, I don't see any issues with those values.
Andy, do you have a specific example for which, the current encoding is ambiguous?

==================================================================
The encoding of union is defined in:
https://tools.ietf.org/html/draft-ietf-core-yang-cbor-07#section-6.12

Currently, each YANG datatype in a union is encoded differently to avoid any ambiguities between  them.
For example, integer 4 is encoded as:

04 # unsigned(4)

enumerator value 4 is encoded as (assuming the allocated tag is 99):

D8 63 # tag(99)
   04 # unsigned(4)

The list of these encoding is shown below:
- unsigned integer --> CBOR unsigned integer
- integer --> CBOR unsigned integer, CBOR negative integer
- enumeration --> CBOR tag <TBD>
- identityref as SID --> CBOR tag <TBD>
- string --> CBOR text string
- identityref as name --> CBOR tag <TBD>
- bits --> CBOR tag <TBD>
- binary --> CBOR byte string
- decimal64 --> CBOR tag 4
- boolean --> CBOR simple value

The only potential problem I aware is when multiple enumerations are part of the same union.
Value 4 from enumeration A will be encoded the same way as Value 4 from enumeration B.

Is it a real problem is proactive?
If so, how this can be resolved?

I won't be present at the Prague meeting, but I'm certainly available to discuss this topic by email.
I will be present at the Montreal meeting for any remaining discussions on this draft.

Note: I'm currently updating the draft to remove any dependencies with RFC 7951, to resolve a comment sent by Andy.

Regards
Michel

From: core <[email-masked]> On Behalf Of Rob Wilton (rwilton)
Sent: Friday, March 22, 2019 8:43 AM
To: [email-masked]; [email-masked]
Subject: [core] YANG encoding in CBOR

Hi,

Thanks,
Rob

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 22 Mar 2019 17:08:20 +0100

> Value 4 from enumeration A will be encoded the same way as Value 4 from enumeration B.

… and that is not a problem for the XML version, because the string is being used instead of the value.  (But then if two enumerations share a string, you have the equivalent problem in the XML serialization.)

Anyway, I haven’t seen a piece of real-world YANG that actually has this problem, so I would be a bit reluctant to make CBOR-based implementations more complex (and less efficient) so solve this (non-?)problem.

Grüße, Carsten

---

From: Andy Bierman <[email-masked]>
Date: Fri, 22 Mar 2019 09:37:49 -0700

On Fri, Mar 22, 2019 at 9:08 AM Carsten Bormann <[email-masked]> wrote:

> … and that is not a problem for the XML version, because the string is
> being used instead of the value.  (But then if two enumerations share a
> string, you have the equivalent problem in the XML serialization.)
>
>

The union data type has similar issues with JSON encoding, where the JSON
encoding can
influence the YANG decoding. This is not possible with XML since all values
are strings.

Anyway, I haven’t seen a piece of real-world YANG that actually has this
> problem, so I would be a bit reluctant to make CBOR-based implementations
> more complex (and less efficient) so solve this (non-?)problem.
>
>
I agree, but tools still need to identify the problem if it exists and flag
the object for different processing
or no processing at all (draft says YANG with these conflicts are not
allowed). IMO a string encoding
should be used.

All YANG union types are position-dependent.
The parser goes through the member types in order, and first one that
accepts the value given wins.
(e.g, ip-address, or any union with 2 of the same base type).

We discussed this issue during the NETMOD WG virtual interim this week, and
wanted clarification
on the scope of the problem. It seems to be limited to just multiple
enumerations in a union.
The rest of the corner-cases are just position-dependent, same as before.

Grüße, Carsten
Andy

> _______________________________________________
> core mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/core
>

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 22 Mar 2019 17:41:02 +0100

> Anyway, I haven’t seen a piece of real-world YANG that actually has this problem, so I would be a bit reluctant to make CBOR-based implementations more complex (and less efficient) so solve this (n…

I think we need to make sure that the interpretation of yang-defined
data is _always_ the same and does not depend on the encoding
negotiated between a client and a server. Efficiency is nice,
avoiding ambiguity is a must.

Avoiding ambiguity can be achieved in different ways, such as making
sure all encodings are designed to not allow different interpretations
(even if that means a loss of efficiency in some cases) or by changing
YANG (or YANG usage guidelines) such that constructs that may lead to
ambiguities are impossible or at least flagged and then avoided. I
guess this is why this topic pops up in the context of YANG next and
we need to figure out which road to take.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
Fax:   +49 421 200 3103         <https://www.jacobs-university.de/>

---

From: "Rob Wilton (rwilton)" <[email-masked]>
Date: Fri, 22 Mar 2019 17:30:09 +0000

I guess that the concern is that this introduces more variation in how data is interpreted between the different XML/JSON/CBOR encodings.

E.g. if someone switched from XML to CBOR, suddenly the configuration or state data may have a different meaning.

Thanks,
Rob

> -----Original Message-----
> From: Carsten Bormann <[email-masked]>
> Sent: 22 March 2019 16:08
> To: Michel Veillette <[email-masked]>
> Cc: Rob Wilton (rwilton) <[email-masked]>; [email-masked];
> [email-masked]
> Subject: Re: [netconf] YANG encoding in CBOR
> 
> … and that is not a problem for the XML version, because the string is being used
> instead of the value.  (But then if two enumerations share a string, you have the
> equivalent problem in the XML serialization.)
>

---

From: "Rob Wilton (rwilton)" <[email-masked]>
Date: Fri, 22 Mar 2019 17:32:36 +0000

Hi Michael,

Indeed, it was the encoding of unions containing more than one enum that was the concern raised in the meeting.

Thanks,
Rob

From: Michel Veillette <[email-masked]>
Sent: 22 March 2019 15:45
To: Rob Wilton (rwilton) <[email-masked]>; [email-masked]; [email-masked]
Subject: RE: YANG encoding in CBOR

Hi Rob

04 # unsigned(4)

D8 63 # tag(99)
   04 # unsigned(4)

Regards
Michel

From: core <[email-masked]<mailto:[email-masked]>> On Behalf Of Rob Wilton (rwilton)
Sent: Friday, March 22, 2019 8:43 AM
To: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: [core] YANG encoding in CBOR

Hi,

Thanks,
Rob

---

From: Michel Veillette <[email-masked]>
Date: Fri, 22 Mar 2019 18:34:00 +0000

Hi Rob

Any solutions have been discussed or proposed?

Encoding enumerators part of a union as CBOR text string might be the simples solution.
Would you please discuss this proposed solution with any and any other interested parties in Prague.

Regards
Michel

From: Rob Wilton (rwilton) <[email-masked]>
Sent: Friday, March 22, 2019 1:33 PM
To: Michel Veillette <[email-masked]>; [email-masked]; [email-masked]
Subject: RE: YANG encoding in CBOR

Hi Michael,

Thanks,
Rob

From: Michel Veillette <[email-masked]<mailto:[email-masked]>>
Sent: 22 March 2019 15:45
To: Rob Wilton (rwilton) <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: RE: YANG encoding in CBOR

Hi Rob

YANG assigns either explicitly or implicitly to each enumeration, a unique integer value, see https://tools.ietf.org/html/rfc7950#section-9.6.4.2<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc7950%23section-9.6.4.2&data=02%7C01%7C%7C240258b409ab4199aa2608d6aeec637b%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C0%7C636888727665422412&sdata=Q%2FY2hWDGxB9YnVJRr10eUl1F0PdJAzGC1%2FLZRO0B%2FaI%3D&reserved=0>.
In CBOR, these values are used, see https://tools.ietf.org/html/draft-ietf-core-yang-cbor-07#section-6.6<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-ietf-core-yang-cbor-07%23section-6.6&data=02%7C01%7C%7C240258b409ab4199aa2608d6aeec637b%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C0%7C636888727665432421&sdata=sej1JrJZGbECXHACilc3fe%2BgMHCIGy4CggCra8LPI5Y%3D&reserved=0>
When used outside of a union, I don't see any issues with those values.
Andy, do you have a specific example for which, the current encoding is ambiguous?

==================================================================
The encoding of union is defined in:
https://tools.ietf.org/html/draft-ietf-core-yang-cbor-07#section-6.12<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-ietf-core-yang-cbor-07%23section-6.12&data=02%7C01%7C%7C240258b409ab4199aa2608d6aeec637b%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C0%7C636888727665432421&sdata=AjA49ErkN1IbbIr29dKYPN6oJr%2FbGk12QUP7Nk90W0E%3D&reserved=0>

04 # unsigned(4)

D8 63 # tag(99)
   04 # unsigned(4)

Is it a real problem is real life?
If so, how this can be resolved?

Regards
Michel

Hi,

Thanks,
Rob

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 23 Mar 2019 10:03:32 +0100

Well, if that is a problem, we can go for a longer representation within unions (section 6.12).  Theoretically, we could do that only of there is more than one enum in the union type (so things stay efficient if there is only one), but that might pose difficulties with model evolution.

Going for a string representation repeats the feature of XML YANG (which was ported over to JSON YANG):

typedef foo {
  type union {
    type enumeration {
      enum red { value 1; }
      enum breen { value 2; }
      enum glue { value 3; }
    }
    type enumeration {
      enum tacks { value 1; }
      enum nails { value 2; }
      enum glue { value 3; }
    }
  }
}

If you use “glue”, you don’t know which of the enumerations are being used.

Using SIDs, we can do better.

So what do we have to do to get the SID tool to allocate SIDs for enum values?

We could then define the CBOR tag for enums in unions to take the usual SID difference (delta relative to the environment, I’d think), not an integer value.

Several of us are at the hackathon and could make something happen today and tomorrow.

Grüße, Carsten

> On Mar 22, 2019, at 18:30, Rob Wilton (rwilton) <[email-masked]> wrote:
>

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Sat, 23 Mar 2019 11:10:03 +0100

I think we need to look at the whole picture and in which direction we
want to go. In the longer term, I would prefer a solution where the
values of a union are discriminated. The current XML encoding
behaviour of 'first match wins' is fragile (for example, if someone
adds an enum to a type, the interpretation of data can change).

Look at this:

typedef bar {
  type union {
    type enumeration { enum "1"; value 2; enum "2"; value 1; }
    type uint8;
  }
}

We have some encodings that send the string representations of the
values and some encodings that prefer to send numeric representations
where possible. In order to have a robust solution, encodings should
likely indicate to which type the value belongs.

/js

On Sat, Mar 23, 2019 at 10:03:32AM +0100, Carsten Bormann wrote:
> netconf mailing list
> https://www.ietf.org/mailman/listinfo/netconf

On Sat, Mar 23, 2019 at 10:03:32AM +0100, Carsten Bormann wrote:
> netconf mailing list
> https://www.ietf.org/mailman/listinfo/netconf

---

From: Carsten Bormann <[email-masked]>
Date: Sat, 23 Mar 2019 11:27:08 +0100

On Mar 23, 2019, at 11:10, Juergen Schoenwaelder <[email-masked]> wrote:
> behaviour of ‘first match wins'

Can you point to chapter and verse?

> is fragile (for example, if someone

In a serialization that has more type information (such as yang-cbor), you can reliably distinguish any usage of the uint8 from the usage of the enumerations.  We put in the CBOR tags to enable that even though enumeration values are normally represented by their “value” field.

The part that I don’t understand is what is the plan with handling conflicts between the enumerations.  I’m not sure what the equivalence model is, here — do YANG enums employ structural equivalence?  There is no name up there, so it can’t really be name equivalence, unless there is an implicit name being inferred from the schema tree.

Grüße, Carsten

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Sat, 23 Mar 2019 14:35:19 +0100

> Can you point to chapter and verse?

RFC 7950 section 9.12:

   When generating an XML encoding, a value is encoded according to the
   rules of the member type to which the value belongs.  When
   interpreting an XML encoding, a value is validated consecutively
   against each member type, in the order they are specified in the
   "type" statement, until a match is found.  The type that matched will
   be the type of the value for the node that was validated, and the
   encoding is interpreted according to the rules for that type.
 
> In a serialization that has more type information (such as yang-cbor), you can reliably distinguish any usage of the uint8 from the usage of the enumerations.  We put in the CBOR tags to enable tha…

The JSON encoding (RFC 7951 section 6.10) also can distinguish between
the enum and the uint8 but it can't distinguish between other types
that all map to a JSON string. Can the CBOR encoding distinguish
between two uint8 values (or two string values) in a union?

typedef temperature {
    type union {
    	type fahrenheit; // an int16
	type celsius;    // an int16
    }
}

> The part that I don’t understand is what is the plan with handling conflicts between the enumerations.

Depends on what you consider a conflict. My problem is that I see
three encodings that all appear to do something slightly different and
the problem seems to be that YANG kind of defers the interpretation of
unions to encodings. Things would be robust if even the above case
would be unambiguous in all encodings, i.e., the union would be a
discriminated or tagged union.

/js

> I’m not sure what the equivalence model is, here — do YANG enums employ structural equivalence?  There is no name up there, so it can’t really be name equivalence, unless there is an implicit name being inferred from the schema tree.
> 
> Grüße, Carsten
> 
> 
> 
> 
> > 
> > /js
> >

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 24 Mar 2019 10:47:36 +0100

> RFC 7950 section 9.12:

Thank you, this is very useful.

Let me try to ask my other question in a very concrete way:

If I have

>      enum glue { value 3; }

in one place, and

> typedef bar {
>  type union {
>    type enumeration {
>      enum red { value 1; }
>      enum breen { value 2; }
>      enum glue { value 3; }
>    }
>    type enumeration {
>      enum sparkling { value 1; }
>      enum blinking { value 2; }
>      enum steady { value 3; }
>    }
>  }
> }

in another place, is the value “red” between the two types referring to the “same thing”?

If there were a typedef for the two anonymous enumerations that both look the same, as in

  typedef color {
  }

and that typedef being referenced from both unions instead of copy-pasting it, would that change the answer?

Grüße, Carsten

---

From: Andy Bierman <[email-masked]>
Date: Sun, 24 Mar 2019 04:29:45 -0700

On Sun, Mar 24, 2019 at 2:47 AM Carsten Bormann <[email-masked]> wrote:

> On Mar 23, 2019, at 14:35, Juergen Schoenwaelder <
> [email-masked]> wrote:
> >
> > RFC 7950 section 9.12:
>

no -- and this is not an error, although it is position-dependent:

  typedef baz {
     type union {
       type foo;
       type bar;
    }
  }

The issue for CBOR applies to both bits and enumerations.
The probability of 2 enum names clashing is almost zero.
The probability of 2 value-stmts clashing is almost 100%

The answer seems to be simple but inefficient:
Always encode bits or enumeration as a string, if this is within a union
type.
There are not many of these -- more likely to find strings with different
patterns
where first-match-wins is no different than before (for XML)

> it, would that change the answer?

I think first-match-wins still dictates the result.

The JSON issue (and maybe CBOR) is what Juergen is talking about:

    typedef maybe-first {
       type union {
           type string;
           type int32;
        }
     }

Use 42 as an example...
The XML encoding <foo>42</foo> will always match string.
The JSON (or CBOR) encoding { "foo": 42 } will always match int32
We want YANG data to maintain fidelity and still be encoding-independent,
but this is broken for union types.

Grüße, Carsten
Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Sun, 24 Mar 2019 15:40:48 +0100

> The probability of 2 value-stmts clashing is almost 100%

Enum names like 'unknown' do have a probability to clash that is
larger than 0%.
 
> where first-match-wins is no different than before (for XML)

The first match wins rule is debatable. Someone adds an enum to an
enumeration that you import and suddenly the interpretation of config
changes. Not very robust.

> but this is broken for union types.

If we plan to improve things we have we should improve the union
construct by turning it into a tagged union (and do away with the
'first match wins' rule but leave it for backwards compatibility for
untagged unions as long as they are used).

/js

---

From: Carsten Bormann <[email-masked]>
Date: Sun, 24 Mar 2019 23:24:12 +0100

On Mar 24, 2019, at 12:29, Andy Bierman <[email-masked]> wrote:
> Always encode bits or enumeration as a string, if this is within a union type.

For enum values within unions (which are already handled specially by giving them a CBOR tag) we could simply use a SID (that we then would have to generate for each enum value) instead of a string.  We could also extend this to enums outside unions; but here the recurrence to the (outside unions unambiguous) value sounds better to me.

Bits.  Oh.  Same procedure, I’d say.

Grüße, Carsten

---

From: Andy Bierman <[email-masked]>
Date: Sun, 24 Mar 2019 15:52:21 -0700

On Sun, Mar 24, 2019 at 3:24 PM Carsten Bormann <[email-masked]> wrote:

> For enum values within unions (which are already handled specially by
> giving them a CBOR tag) we could simply use a SID (that we then would have
> to generate for each enum value) instead of a string.  We could also extend
> this to enums outside unions; but here the recurrence to the (outside
> unions unambiguous) value sounds better to me.
>
This seems like it would make the SID file very complicated.
Not sure it would work since leaf/leaf-list can define their own types.
Plus, all the enum typedefs would have to be numbered this way.
Maybe the SID authors can comment on how this would be done.

Andy

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 25 Mar 2019 07:23:44 +0100

Hi Andy,

> On Mar 24, 2019, at 23:52, Andy Bierman <[email-masked]> wrote:
> 

> Plus, all the enum typedefs would have to be numbered this way.

Right!
It would amount to defining an identifier scheme for all the so far anonymous enums (and bits defs, I gather).

> Maybe the SID authors can comment on how this would be done.

Well, such an identifier scheme would benefit the whole YANG ecosystem, so you can have a transition from “first match wins” to proper name equivalence.  So we would need to do it in a way that over time it can become part of the core YANG definition.

We can maybe cheat:  Use strings for enums inside unions for now, and values only outside.
But it would be good if the SID files generated (and registered) for a YANG module already had those enum/bits type identifiers so we only have to do the SID generation once, so I’m not sure if I like cheating; doing the SIDs for those identifiers right from the outset would win so much more.

So does anyone have an idea how such an identifier scheme might look like?

Grüße, Carsten

---

From: ivaylo petrov <[email-masked]>
Date: Mon, 25 Mar 2019 07:52:47 +0100

Hello,

On Mon, Mar 25, 2019 at 7:23 AM Carsten Bormann <[email-masked]> wrote:

> It would amount to defining an identifier scheme for all the so far
> anonymous enums (and bits defs, I gather).
>
> Well, such an identifier scheme would benefit the whole YANG ecosystem, so
> you can have a transition from “first match wins” to proper name
> equivalence.  So we would need to do it in a way that over time it can
> become part of the core YANG definition.
>
> But it would be good if the SID files generated (and registered) for a
> YANG module already had those enum/bits type identifiers so we only have to
> do the SID generation once, so I’m not sure if I like cheating; doing the
> SIDs for those identifiers right from the outset would win so much more.
>
>
[IP]: Carsten, your idea for the good solution would be to keep the way
enum values are sent now outside of unions and only send the SIDs inside
unions, is that correct? If so, this sounds as a very good solution to me.
It might add a bit of complexity in the SID generation, but I would expect
it to be rather small.

> So does anyone have an idea how such an identifier scheme might look like?

Best regards,
Ivaylo Petrov

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 25 Mar 2019 08:53:35 +0100

> We can maybe cheat:  Use strings for enums inside unions for now, and values only outside.

Or, if it takes too long to do the identifier scheme:
We could define SIDs for each module for the enum and bits strings used in that module.
That would mean we’d have multiple SIDs for the same strings all over the YANG universe, but it is something we don’t have to wait for and can do locally.

So a decoder knowing the module at all would also know what string to imagine (it never has to be reified!) for that SID.  An encoder has a slightly harder time, as it has to use a SID from a module the decoder will know; I’d imagine that won’t be too hard to glean from the context though.

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 25 Mar 2019 08:57:55 +0100

On Mar 25, 2019, at 07:52, ivaylo petrov <[email-masked]> wrote:
> 
> [IP]: Carsten, your idea for the good solution would be to keep the way enum values are sent now outside of unions and only send the SIDs inside unions, is that correct?

Yes, as most enums occur outside unions, so good design principles say the cost of the feature of unions of enums should only be paid by those who actually use them.

> If so, this sounds as a very good solution to me. It might add a bit of complexity in the SID generation, but I would expect it to be rather small.

I think we need to make a few examples to check out how that would look like.

Grüße, Carsten

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 25 Mar 2019 09:06:33 +0100

> likely indicate to which type the value belongs.

Perhaps the easiest way would be to use (optional) annotation that
specifies, using an ordinal number, which of the member types is used
for the particular instance. But since there can be unions inside
unions, a list of numbers would be needed in general.

Lada

-- 
Ladislav Lhotka
Head, CZ.NIC Labs
PGP Key ID: 0xB8F92B08A9F76C67

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 25 Mar 2019 09:12:53 +0100

> unions, a list of numbers would be needed in general.

Is the sequential number of the alternative supposed to be “stable”?
(I.e., would there be confusion if an alternative is inserted in the middle?
Which you may want to do while we still do “first match wins”.)

If yes, a list of those numbers may indeed be sufficient to always stay with values for CBOR.

Grüße, Carsten

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 25 Mar 2019 09:30:39 +0100

> Which you may want to do while we still do “first match wins”.)

Inserting a new member type in the middle also affects the current algorithm for
union type resolution, so it is not permitted by the module update rules (sec.
11 in RFC 7950).

> If yes, a list of those numbers may indeed be sufficient to always stay with

And it is also a solution that works the same for any data representation, as
long as it can encode annotations.

Lada

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 25 Mar 2019 09:42:27 +0100

On Mar 25, 2019, at 09:06, Ladislav Lhotka <[email-masked]> wrote:
> 
> use (optional) annotation

What do I have to read to learn more about YANG annotations?

Grüße, Carsten

---

From: Ladislav Lhotka <[email-masked]>
Date: Mon, 25 Mar 2019 09:45:55 +0100

> What do I have to read to learn more about YANG annotations?

RFC 7952.

Lada

---

From: Michel Veillette <[email-masked]>
Date: Wed, 27 Mar 2019 01:12:52 +0000

Hi Ladislav

If I summarize this issue of multiple enumerations or bits in a union, this problem can be solve by the following approaches:

- To not allows these duplicate values or positions to happen in YANG
- To encode enumerations and bits as string (in all unions or only when multiple enumerations or bits are defined)
- To encode enumerations and bits as SID (in all unions or only when multiple enumerations or bits are defined)
- To encode enumerations and bits as delta between the value SID and the leaf SID (in all unions or only when multiple enumerations or bits are defined)

In this email, I will like to focus on the first option, fixing the problem directly in YANG instead of fixing the consequences.

Without any changes in YANG, a union with multiple enumeration or bits can be constructed without value or position overlaps.
For example:

  leaf multiple-enumerations-test-1 {
    type union {
      type enumeration {
        enum "Monday" { value 0; }
        enum "Tuesday" { value 1; }
        enum "Wednesday" { value 2; }
        enum "Thursday" { value 3; }
        enum "Friday" { value 4; }

      }
      type enumeration {
        enum "Saturday" { value 5; }
        enum "Sunday" { value 6; }
      }
    }
  }

  leaf multiple-bits-test-1 {
    type union {
      type bits {
        bit  "Monday" { position  0; }
        bit "Tuesday" { position  1; }
        bit "Wednesday" { position  2; }
        bit "Thursday" { position  3; }
        bit "Friday" { position  4; }

      }
      type bits {
        bit "Saturday" { position 5; }
        bit "Sunday" { position 6; }
      }
    }
  }

When using already defined typedef, avoiding overlap is less obvious without some help.
To help building unions with already defined typedefs, I propose to introduce two extensions. 

  extension value-offset {
    argument offset {
      yin-element true;
    }
    description
      "Offset added to each enum value of the associated enumeration.";
  }
  
  extension position-offset {
    argument offset {
      yin-element true;
    }
    description
      "Offset value added to each bit position of the associated bits.";
  }

The value-offset extension can be used as follow:

    type enumeration {
      enum "Monday";
      enum "Tuesday";
      enum "Wednesday";
      enum "Thursday";
      enum "Friday";
    }
  }

  typedef weekend {
    type enumeration {
      enum "Saturday";
      enum "Sunday";
    }
  }
  
  leaf multiple-enumerations-test-3 {
    type union {
      type weekdays;
      type weekend {
        ext:value-offset 5;
      }
    }
  }

The position-offset extension can be used as follow:

  typedef weekdays-flags {
    type bits {
      bit "Monday";
      bit "Tuesday";
      bit "Wednesday";
      bit "Thursday";
      bit "Friday";
    }
  }

  typedef weekend-flags {
    type bits {
      bit "Saturday";
      bit "Sunday";
    }
  }
  
  leaf multiple-bits-test-3 {
    type union {
      type weekdays-flags;
      type weekend-flags {
        ext:position-offset 5;
      }
    }
  }

The yang file in attachment show different examples based on this approach.
This module have been validated using http://www.yangvalidator.com/validator 
If this approach is accepted, tools like pyang should be updated to produce an error if multiple enumerations or bits are defined with value or position overleaps.

Please comment,
Michel

-----Original Message-----
From: Ladislav Lhotka <[email-masked]> 
Sent: Monday, March 25, 2019 4:07 AM
To: Juergen Schoenwaelder <[email-masked]>; Carsten Bormann <[email-masked]>
Cc: Michel Veillette <[email-masked]>; [email-masked]; [email-masked]
Subject: Re: [netconf] YANG encoding in CBOR

> likely indicate to which type the value belongs.

Perhaps the easiest way would be to use (optional) annotation that specifies, using an ordinal number, which of the member types is used for the particular instance. But since there can be unions inside unions, a list of numbers would be needed in general.

Lada

>> >> On Mar 22, 2019, at 16:45, Michel Veillette 
>> >> <[email-masked]>
>> >> wrote:
>> >>> 
>> >>> The only potential problem I aware is when multiple enumerations 
>> >>> are part of
>> >> the same union.
>> >>> Value 4 from enumeration A will be encoded the same way as Value 
>> >>> 4 from
>> >> enumeration B.
>> >> 
>> >> … and that is not a problem for the XML version, because the 
>> >> string is being used instead of the value.  (But then if two 
>> >> enumerations share a string, you have the equivalent problem in 
>> >> the XML serialization.)
>> >> 
>> >> Anyway, I haven’t seen a piece of real-world YANG that actually 
>> >> has this problem, so I would be a bit reluctant to make CBOR-based 
>> >> implementations more complex (and less efficient) so solve this (non-?)problem.
>> >> 
>> >> Grüße, Carsten
>> > 
>> > 
>> > 
>> 
>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww
>> .ietf.org%2Fmailman%2Flistinfo%2Fnetconf&amp;data=02%7C01%7C%7C343ea8
>> d1cf8f4e39afc708d6b0f8d874%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C1
>> %7C636890980182553400&amp;sdata=u1KFAYAus16B8a7sgsBfPfIquOptMlaOb%2B0
>> kvPZgr4o%3D&amp;reserved=0
>
> -- 
> Fax:   +49 421 200 3103         <https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jacobs-university.de%2F&amp;data=02%7C01%7C%7C343ea8d1cf8f4e39afc708d6b0f8d874%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C1%7C636890980182553400&amp;sdata=TrW2iL3nUDlZ%2BVvhPxWeqdU1X%2BqvFCnXyodX6Bu1e94%3D&amp;reserved=0>
>
>
>> >> On Mar 22, 2019, at 16:45, Michel Veillette 
>> >> <[email-masked]>
>> >> wrote:
>> >>> 
>> >>> The only potential problem I aware is when multiple enumerations 
>> >>> are part of
>> >> the same union.
>> >>> Value 4 from enumeration A will be encoded the same way as Value 
>> >>> 4 from
>> >> enumeration B.
>> >> 
>> >> … and that is not a problem for the XML version, because the 
>> >> string is being used instead of the value.  (But then if two 
>> >> enumerations share a string, you have the equivalent problem in 
>> >> the XML serialization.)
>> >> 
>> >> Anyway, I haven’t seen a piece of real-world YANG that actually 
>> >> has this problem, so I would be a bit reluctant to make CBOR-based 
>> >> implementations more complex (and less efficient) so solve this (non-?)problem.
>> >> 
>> >> Grüße, Carsten
>> > 
>> > 
>> > 
>> 
>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww
>> .ietf.org%2Fmailman%2Flistinfo%2Fnetconf&amp;data=02%7C01%7C%7C343ea8
>> d1cf8f4e39afc708d6b0f8d874%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C1
>> %7C636890980182553400&amp;sdata=u1KFAYAus16B8a7sgsBfPfIquOptMlaOb%2B0
>> kvPZgr4o%3D&amp;reserved=0
>
> -- 
> Fax:   +49 421 200 3103         <https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jacobs-university.de%2F&amp;data=02%7C01%7C%7C343ea8d1cf8f4e39afc708d6b0f8d874%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C1%7C636890980182553400&amp;sdata=TrW2iL3nUDlZ%2BVvhPxWeqdU1X%2BqvFCnXyodX6Bu1e94%3D&amp;reserved=0>
>
> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.
> ietf.org%2Fmailman%2Flistinfo%2Fnetconf&amp;data=02%7C01%7C%7C343ea8d1
> cf8f4e39afc708d6b0f8d874%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C1%7C
> 636890980182553400&amp;sdata=u1KFAYAus16B8a7sgsBfPfIquOptMlaOb%2B0kvPZ
> gr4o%3D&amp;reserved=0

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Wed, 27 Mar 2019 07:16:37 +0100

Hi,

a union can be formed by using member types that are imported and not
under change control of a single author/organization and ideally this
should work without complex coordination of name and number spaces.
Duplicate enum/bits values are legal in YANG today so an encoding has
to deal with this aspect of life.

A robust fix to all these problems will be to tag the type members in
order to discriminate the values in the encodings. This, however, will
take some time to specify and we will need to preserve backwards
compatibility with unions without a tag (but compilers can encourage
people to add tags whenever modules are updated).

/js

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 27 Mar 2019 09:25:42 +0100

> people to add tags whenever modules are updated).

And that may be the way forward:

As XML and JSON already differ in handling “first-match wins” (as Andy has demonstrated), there is probably not a lot of damage in CBOR being different again in handling this arcane case, so we can mostly ship as is (but see below).

We then develop the YANG-wide fix together.  We’ll make sure that the CBOR union representation has the right placeholders to put that in.  For that, it would be nice to have a direction about where exactly that information would be in the instance tree.  And, more importantly, if something needs to be included in the SID files, it would be good to nail that down now.

Grüße, Carsten

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 27 Mar 2019 09:40:08 +0100

> people to add tags whenever modules are updated).

I already opened a new issue for this in yang-next:

https://github.com/netmod-wg/yang-next/issues/72

Lada

> > If I summarize this issue of multiple enumerations or bits in a union, this
> > problem can be solve by the following approaches:
> > 
> > - To encode enumerations and bits as string (in all unions or only when
> > multiple enumerations or bits are defined)
> > - To encode enumerations and bits as SID (in all unions or only when
> > multiple enumerations or bits are defined)
> > - To encode enumerations and bits as delta between the value SID and the
> > leaf SID (in all unions or only when multiple enumerations or bits are
> > defined)
> > 
> > In this email, I will like to focus on the first option, fixing the problem
> > directly in YANG instead of fixing the consequences.
> > 
> > Without any changes in YANG, a union with multiple enumeration or bits can
> > be constructed without value or position overlaps.
> > For example:
> > 
> > If this approach is accepted, tools like pyang should be updated to produce
> > an error if multiple enumerations or bits are defined with value or position
> > overleaps.
> > 
> > Please comment,
> > Michel
> > 
> > specifies, using an ordinal number, which of the member types is used for
> > the particular instance. But since there can be unions inside unions, a list
> > of numbers would be needed in general.
> > 
> > Lada
> > 
> > > /js
> > > 
> > > > Well, if that is a problem, we can go for a longer representation within
> > > > unions (section 6.12).  Theoretically, we could do that only of there is
> > > > more than one enum in the union type (so things stay efficient if there
> > > > is only one), but that might pose difficulties with model evolution.
> > > > 
> > > > We could then define the CBOR tag for enums in unions to take the usual
> > > > SID difference (delta relative to the environment, I’d think), not an
> > > > integer value.
> > > > 
> > > > > I guess that the concern is that this introduces more variation in how
> > > > > data is interpreted between the different XML/JSON/CBOR encodings.
> > > > > 
> > > > > E.g. if someone switched from XML to CBOR, suddenly the configuration
> > > > > or state data may have a different meaning.
> > > > > 
> > > > > Thanks,
> > > > > Rob
> > > > > 
> > > > > 
> > > Fax:   +49 421 200 3103         <
> > > https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jacobs-university.de%2F&amp;data=02%7C01%7C%7C343ea8d1cf8f4e39afc708d6b0f8d874%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C1%7C636890980182553400&amp;sdata=TrW2iL3nUDlZ%2BVvhPxWeqdU1X%2BqvFCnXyodX6Bu1e94%3D&amp;reserved=0
> > > >
> > > 
> > > 
> > > > Well, if that is a problem, we can go for a longer representation within
> > > > unions (section 6.12).  Theoretically, we could do that only of there is
> > > > more than one enum in the union type (so things stay efficient if there
> > > > is only one), but that might pose difficulties with model evolution.
> > > > 
> > > > We could then define the CBOR tag for enums in unions to take the usual
> > > > SID difference (delta relative to the environment, I’d think), not an
> > > > integer value.
> > > > 
> > > > > I guess that the concern is that this introduces more variation in how
> > > > > data is interpreted between the different XML/JSON/CBOR encodings.
> > > > > 
> > > > > E.g. if someone switched from XML to CBOR, suddenly the configuration
> > > > > or state data may have a different meaning.
> > > > > 
> > > > > Thanks,
> > > > > Rob
> > > > > 
> > > > > 
> > > Fax:   +49 421 200 3103         <
> > > https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jacobs-university.de%2F&amp;data=02%7C01%7C%7C343ea8d1cf8f4e39afc708d6b0f8d874%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C1%7C636890980182553400&amp;sdata=TrW2iL3nUDlZ%2BVvhPxWeqdU1X%2BqvFCnXyodX6Bu1e94%3D&amp;reserved=0
> > > >
> > >

---

From: Michel Veillette <[email-masked]>
Date: Wed, 27 Mar 2019 16:00:42 +0000

Hi Juergen

You will find in attachment the updated version of [email-masked]' based on your proposed solution, type tagging.
This module defines an extension called union-tag as follow:

  extension union-tag {
    argument tag {
      yin-element true;
    }
    description
      "In some cases, the interpretation of a value of a union type can differ
      when encoded using the different data format. This difference comes from
      the level of type information supported by the different data formats.
      In the case of XML, all elements and attributes are encoded as string
      and no type information are included. JSON supports string, number and
      boolean. Finally, CBOR supports all YANG basic types.

      This extension can be used to add a tag to a type within a union to
      guarantee its proper interpretation when decoded.";
  }

This extension can be used as follow:

  leaf multiple-enumerations-test-2 {
    type union {
      type enumeration {
        ext:union-tag weekdays;
        enum "Monday";
        enum "Tuesday";
        enum "Wednesday";
        enum "Thursday";
        enum "Friday";
      }
      type enumeration {
        ext:union-tag weekend;
        enum "Saturday";
        enum "Sunday";
      }
    }
  }

If I assume this solution or similar solution is implemented, the next question is how this tag is encoded in CBOR.
Let assume that the data node SID is 1820 and the SID assigned to "weekdays" tag is  1821.

When set to "Tuesday", this data node can be encoded using a CBOR map as follows:

{
   1820 : { "weekdays", 1}
}

Do we need to add a CBOR tag to this encoding to specify the use of a union tag? (99 in this example)
I personally don't think so since the use of a CBOR map for a union data node is not currently possible.

{
   1820 : (99) { "weekdays", 1}
}

Should we use a SID instead of the tag? 

{
   1820 : { 1821, 1}
}

Should we use a delta between the data node SID and the union tag SID?
The drawback of this approach is that the same value assigned to different data nodes is encoded differently.
I won't personally recommend this approach.

{
   1820 : { 1, 1}
}

I hope this email will help moving the discussion closer to the final solution.

Please comments
Michel

-----Original Message-----
From: Juergen Schoenwaelder <[email-masked]> 
Sent: Wednesday, March 27, 2019 2:17 AM
To: Michel Veillette <[email-masked]>
Cc: Ladislav Lhotka <[email-masked]>; Carsten Bormann <[email-masked]>; [email-masked]; [email-masked]
Subject: Re: [netconf] YANG encoding in CBOR

Hi,

a union can be formed by using member types that are imported and not under change control of a single author/organization and ideally this should work without complex coordination of name and number spaces.
Duplicate enum/bits values are legal in YANG today so an encoding has to deal with this aspect of life.

A robust fix to all these problems will be to tag the type members in order to discriminate the values in the encodings. This, however, will take some time to specify and we will need to preserve backwards compatibility with unions without a tag (but compilers can encourage people to add tags whenever modules are updated).

/js

On Wed, Mar 27, 2019 at 01:12:52AM +0000, Michel Veillette wrote:
> - To encode enumerations and bits as string (in all unions or only 
> when multiple enumerations or bits are defined)
> - To encode enumerations and bits as delta between the value SID and 
> the leaf SID (in all unions or only when multiple enumerations or bits 
> are defined)
> 
> This module have been validated using 
> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.y
> angvalidator.com%2Fvalidator&amp;data=02%7C01%7C%7Cc69426dcaaf8448e1d4
> 708d6b27bc735%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C0%7C63689264207
> 3257660&amp;sdata=Xqo40lzBjINVFZhIZZI1LEktAdq5mee2AdxeYqefnsA%3D&amp;r
> eserved=0 If this approach is accepted, tools like pyang should be 
> updated to produce an error if multiple enumerations or bits are defined with value or position overleaps.
> 
> Please comment,
> Michel
> 
> > I think we need to look at the whole picture and in which direction 
> > we want to go. In the longer term, I would prefer a solution where 
> > the values of a union are discriminated. The current XML encoding 
> > values and some encodings that prefer to send numeric 
> > representations where possible. In order to have a robust solution, 
> > encodings should likely indicate to which type the value belongs.
> 
> >> >>> The only potential problem I aware is when multiple 
> >> >>> enumerations are part of
> >> >> the same union.
> >> >> has this problem, so I would be a bit reluctant to make 
> >> >> CBOR-based implementations more complex (and less efficient) so solve this (non-?)problem.
> >> >> 
> >> >> Grüße, Carsten
> >> > 
> >> > 
> >> > 
> >> 
> > Fax:   +49 421 200 3103         <https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jacobs-university.de%2F&amp;data=02%7C01%7C%7Cc69426dcaaf8448e1d4708d6b27bc735%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C0%7C636892642073257660&amp;sdata=RqSYuNplS4kOLU0R3etRzrR%2FU2ccwljkEZkjHHDU1HA%3D&amp;reserved=0>
> >
> >
> >> >>> The only potential problem I aware is when multiple 
> >> >>> enumerations are part of
> >> >> the same union.
> >> >> has this problem, so I would be a bit reluctant to make 
> >> >> CBOR-based implementations more complex (and less efficient) so solve this (non-?)problem.
> >> >> 
> >> >> Grüße, Carsten
> >> > 
> >> > 
> >> > 
> >> 
> > Fax:   +49 421 200 3103         <https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jacobs-university.de%2F&amp;data=02%7C01%7C%7Cc69426dcaaf8448e1d4708d6b27bc735%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C0%7C636892642073257660&amp;sdata=RqSYuNplS4kOLU0R3etRzrR%2FU2ccwljkEZkjHHDU1HA%3D&amp;reserved=0>
> >

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
Fax:   +49 421 200 3103         <https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.jacobs-university.de%2F&amp;data=02%7C01%7C%7Cc69426dcaaf8448e1d4708d6b27bc735%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C0%7C636892642073267660&amp;sdata=zcQv4hR8ykJFdhvsd2qnTPQ3EPdEZijDc4eT%2BDmsqu0%3D&amp;reserved=0>

---

From: Michel Veillette <[email-masked]>
Date: Wed, 27 Mar 2019 16:01:56 +0000

Sorry for this duplicate email, I forgot the attachment.

Regards,
Michel

-----Original Message-----
From: Michel Veillette 
Sent: Wednesday, March 27, 2019 12:01 PM
To: 'Juergen Schoenwaelder' <[email-masked]>
Cc: Ladislav Lhotka <[email-masked]>; Carsten Bormann <[email-masked]>; [email-masked]; [email-masked]
Subject: RE: [netconf] YANG encoding in CBOR

Hi Juergen

  extension union-tag {
    argument tag {
      yin-element true;
    }
    description

{
   1820 : { "weekdays", 1}
}

Do we need to add a CBOR tag to this encoding to specify the use of a union tag? (99 in this example) I personally don't think so since the use of a CBOR map for a union data node is not currently possible.

{

{
   1820 : { 1821, 1}
}

{
   1820 : { 1, 1}
}

Please comments
Michel

-----Original Message-----
From: Juergen Schoenwaelder <[email-masked]>
Sent: Wednesday, March 27, 2019 2:17 AM
To: Michel Veillette <[email-masked]>
Cc: Ladislav Lhotka <[email-masked]>; Carsten Bormann <[email-masked]>; [email-masked]; [email-masked]
Subject: Re: [netconf] YANG encoding in CBOR

Hi,

/js

---

From: Andy Bierman <[email-masked]>
Date: Wed, 27 Mar 2019 09:13:44 -0700

> https://github.com/netmod-wg/yang-next/issues/72
We already explored the solution of giving the member types numbers
so they could be used in CBOR but this was rejected because it is so
complex to implement.

Consider when union is within union, and the types are named types from
other modules. Union types can be legally updated in new versions of the
module,
but the position assignments for SID can never change.

Even without this complexity this solution would cause the encoder/decoder
to
be very schema-aware.

BTW, creating SIDs for enums and bits will break if the server uses
'deviate replace type'.
There is no way to number the deviated type since this is server-specific
not part of the original module
and the deviation is not actually a data-def-stmt.

Lada
Andy

> the
> > > When using already defined typedef, avoiding overlap is less obvious
> without
> > > some help.
> http://www.yangvalidator.com/validator
> > >
> > > To: Juergen Schoenwaelder <[email-masked]>;
> Carsten
> > > Bormann <[email-masked]>
> > > Cc: Michel Veillette <[email-masked]>;
> [email-masked];
> > > [email-masked]
> we
> representations
> for
> > > > > Going for a string representation repeats the feature of XML YANG
> (which
> > > > > was ported over to JSON YANG):
> > > > >
> > > > > typedef foo {
> > > > >   type union {
> > > > >     type enumeration {
> > > > >       enum red { value 1; }
> > > > >       enum breen { value 2; }
> > > > >       enum glue { value 3; }
> > > > >     }
> > > > >     type enumeration {
> > > > >       enum tacks { value 1; }
> > > > >       enum nails { value 2; }
> > > > >       enum glue { value 3; }
> > > > >     }
> > > > >   }
> > > > > }
> > > > >
> > > > > If you use “glue”, you don’t know which of the enumerations are
> being
> > > > > used.
> > > > >
> > > > > So what do we have to do to get the SID tool to allocate SIDs for
> enum
> > > > > values?
> > > > >
> > > > > Several of us are at the hackathon and could make something happen
> today
> > > > > and tomorrow.
> > > > >
> > > > > Grüße, Carsten
> > > > >
> > > > >
> > > > > > On Mar 22, 2019, at 18:30, Rob Wilton (rwilton) <
> [email-masked]>
> > > > > > wrote:
> > > > > >
> enumerations
> CBOR-based
> > > > > .ietf.org
> %2Fmailman%2Flistinfo%2Fnetconf&amp;data=02%7C01%7C%7C343ea8
> > > > >
> > > > > Going for a string representation repeats the feature of XML YANG
> (which
> > > > > was ported over to JSON YANG):
> > > > >
> > > > > typedef foo {
> > > > >   type union {
> > > > >     type enumeration {
> > > > >       enum red { value 1; }
> > > > >       enum breen { value 2; }
> > > > >       enum glue { value 3; }
> > > > >     }
> > > > >     type enumeration {
> > > > >       enum tacks { value 1; }
> > > > >       enum nails { value 2; }
> > > > >       enum glue { value 3; }
> > > > >     }
> > > > >   }
> > > > > }
> > > > >
> > > > > If you use “glue”, you don’t know which of the enumerations are
> being
> > > > > used.
> > > > >
> > > > > So what do we have to do to get the SID tool to allocate SIDs for
> enum
> > > > > values?
> > > > >
> > > > > Several of us are at the hackathon and could make something happen
> today
> > > > > and tomorrow.
> > > > >
> > > > > Grüße, Carsten
> > > > >
> > > > >
> > > > > > On Mar 22, 2019, at 18:30, Rob Wilton (rwilton) <
> [email-masked]>
> > > > > > wrote:
> > > > > >
> enumerations
> CBOR-based
> > > > > .ietf.org
> %2Fmailman%2Flistinfo%2Fnetconf&amp;data=02%7C01%7C%7C343ea8
> > > > >
> > > > ietf.org
> %2Fmailman%2Flistinfo%2Fnetconf&amp;data=02%7C01%7C%7C343ea8d1
> > > >

---

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 27 Mar 2019 20:54:07 +0100

Andy Bierman píše v St 27. 03. 2019 v 09:13 -0700:
> so they could be used in CBOR but this was rejected because it is so complex
> to implement.

I don't think it is too complex. Get the type in the schema, get the member,
that's all. It is much easier and more robust than the current algorithm.

> Consider when union is within union, and the types are named types from

Union within union is no problem: it jut requires a sequence of numbers.

> other modules. Union types can be legally updated in new versions of the

Again, I don't agree. Sec. 11 in RFC 7950 says:

   o  A "type" statement may be replaced with another "type" statement
      that does not change the syntax or semantics of the type.
      ...

Adding a new member clearly changes the semantics of the type, if not syntax.

> but the position assignments for SID can never change.

The annotation would also help resolve the differences between JSON and XML. SID
is only available in CBOR.

> 
> Even without this complexity this solution would cause the encoder/decoder to
> be very schema-aware.

The current method of resolving unions is totally schema-aware (somewhat less so
in JSON).

Lada

>  
> 
> BTW, creating SIDs for enums and bits will break if the server uses 'deviate
> replace type'.
> There is no way to number the deviated type since this is server-specific not
> part of the original module
> > > > Cc: Michel Veillette <[email-masked]>; [email-masked];
> >  
> > > > [email-masked]
> > > > > > > On Mar 22, 2019, at 18:30, Rob Wilton (rwilton) <[email-masked]
> > >
> > > > > > > wrote:
> > > > > > > 
> > > > > > > On Mar 22, 2019, at 18:30, Rob Wilton (rwilton) <[email-masked]
> > >
> > > > > > > wrote:
> > > > > > >

---

From: Andy Bierman <[email-masked]>
Date: Wed, 27 Mar 2019 13:11:33 -0700

On Wed, Mar 27, 2019 at 12:54 PM Ladislav Lhotka <[email-masked]> wrote:

> complex
no it isn't easier because the algorithm is error-prone.
Renumbering un-named types nested within unions and referenced named types
over many revisions of the module is not easy.
The encoding MUST NOT change EVER -- even if the union type is expanded.

> Union within union is no problem: it jut requires a sequence of numbers.
do not agree since the union can have a member type added

>       that does not change the syntax or semantics of the type.

   type uint32 { range "1..8"; }

I can refactor that into 2 types (1..4 and 5..8)
This does not change the semantics of the union type but it adds a member
type
Other refactoring is possible that will also change the number of member
types.

Also, deviations do not follow this rule

> Adding a new member clearly changes the semantics of the type, if not
no it doesn't always change the semantics

> The current method of resolving unions is totally schema-aware (somewhat
No -- it just needs to know string vs. number.
Not schema aware at all.

Andy

> 'deviate
> > > > > If I summarize this issue of multiple enumerations or bits in a
> union,
> > > this
> when
> > > > > Without any changes in YANG, a union with multiple enumeration or
> bits
> > > can
> > > > > the particular instance. But since there can be unions inside
> unions, a
> > > list
> > > > > > > Well, if that is a problem, we can go for a longer
> representation
> > > within
> > > > > > > more than one enum in the union type (so things stay efficient
> if
> > > there
> > > > > > > We could then define the CBOR tag for enums in unions to take
> the
> > > usual
> > > > > > > SID difference (delta relative to the environment, I’d think),
> not
> > > an
> > > > > > > integer value.
> > > > > > >
> [email-masked]
> > > > > > > > I guess that the concern is that this introduces more
> variation in
> > > how
> > > > > > > > > > Value 4 from enumeration A will be encoded the same way
> as
> > > Value
> > > > > > > > > > 4 from
> > > > > > > > > enumeration B.
> > > > > > > > >
> CBOR-
> > > based
> > > > > > > > > implementations more complex (and less efficient) so solve
> this
> > > > > > > > > (non-?)problem.
> > > > > > > > >
> > > > > > > > > Grüße, Carsten
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > > Well, if that is a problem, we can go for a longer
> representation
> > > within
> > > > > > > more than one enum in the union type (so things stay efficient
> if
> > > there
> > > > > > > We could then define the CBOR tag for enums in unions to take
> the
> > > usual
> > > > > > > SID difference (delta relative to the environment, I’d think),
> not
> > > an
> > > > > > > integer value.
> > > > > > >
> [email-masked]
> > > > > > > > I guess that the concern is that this introduces more
> variation in
> > > how
> > > > > > > > > > Value 4 from enumeration A will be encoded the same way
> as
> > > Value
> > > > > > > > > > 4 from
> > > > > > > > > enumeration B.
> > > > > > > > >
> CBOR-
> > > based
> > > > > > > > > implementations more complex (and less efficient) so solve
> this
> > > > > > > > > (non-?)problem.
> > > > > > > > >
> > > > > > > > > Grüße, Carsten
> > > > > > > >
> > > > > > > >
> > > > > > >

---

From: Andy Bierman <[email-masked]>
Date: Wed, 27 Mar 2019 18:37:15 -0700

Hi Lada,

> that's all. It is much easier and more robust than the current algorithm.

I think a discriminated union is too complex and overkill, considering the
low probability
of the problem.

It's not as if the client or server developers know which union member type
each value is using.
This is not going to be hard-wired either.  Senders do not usually validate
data, especially
NETCONF servers.  It will be very expensive (and very slow) to validate
union types
in order to determine the member type number.

We cannot rely on YANG authors to follow the update rules, especially when
they change type definitions to fix bugs.  The encoding for a union cannot
ever
change over time or it will break clients. We want to support clients
without requiring
any code changes, even when the server is upgraded.

I prefer a much easier and faster solution:

The client or server must figure out which data nodes have this problem at
boot-time.
These data nodes will always be sent with a new tag (for DUP-UNION) and the
format will always be a string. A receiver must be able to handle DUP-UNION
strings
and parse them against the union type correctly.

All other union type data nodes are sent as normal.
Annotations are not needed.

The JSON vs. XML ordering problem can be handled with best practices
https://tools.ietf.org/html/rfc8407#section-4.11.4

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 28 Mar 2019 10:33:06 +0100

> I can refactor that into 2 types (1..4 and 5..8)

This looks like a pretty contrived example, why would somebody want to
do this?

Anyway, regarding updates: one can have this union

type union {
  type uint32 {
    range "1..3";
  }
  type string;
}

So, in XML representation, the value of 4 will be classified as
a string. Now, according to sec. 11, it is legal to expand the range to
1..10, say. Suddenly, the same value becomes a number.

> No -- it just needs to know string vs. number.

Of course it is. With both strings and numbers, you also have to check
restrictions if they are specified in the schema (pattern, range etc.),
deref leafrefs etc.

I think we all agree that unions in YANG are fragile and may cause
subtle problems. If a better solution is found, then fine, but IMO it
needs to be addressed, and not only in CBOR.

Lada

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 28 Mar 2019 11:03:17 +0100

Hi Michel,

string tags look like a more robust solution than numeric member
positions. It would be better though to have them as a built-in
statement, or at least a "critical" extension.

Lada

Michel Veillette <[email-masked]> writes:

>> Carsten Bormann <[email-masked]>
>> [email-masked]; [email-masked]

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 28 Mar 2019 12:30:18 +0100

Michel,

I think the YANG part of the problem needs to be resolved as part of
the yang next effort. We now have an issue for it so this will not get
lost. YANG extensions have their limits; you can not expect that
client and server both understand an extension and hence it is a
stretch to make the encoding of data depending on tags provided by
extensions. So what we need is:

a) A CBOR encoding that works with the current YANG rules (even if it
   may be less efficient CBOR).
b) Work on a general solution for tagged unions as part of the YANG
   next effort.
c) Eventually update the XML,JSON,CBOR encodings such that tagged
   unions are handled properly.

Assuming you won't be willing to wait for b), you have to define
something that works reasonably with YANG 1.1 (task a) and get
prepared to update the CBOR encoding when the YANG next effort has
settled on a solution (task c).

/js

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 28 Mar 2019 14:46:52 +0100

So for enum/bits we are back to value outside, string inside union?
Can we use SIDs instead of the strings for the latter?
This can’t be much more complicated than assigning a SID to each string that occurs in an enum/bits in a union.

Grüße, Carsten

---

From: Michel Veillette <[email-masked]>
Date: Thu, 28 Mar 2019 14:06:17 +0000

+1 for value outside, string inside union

My rational:

- SID can have up to 8 bytes and will provide a marginal compression.
- Generating SID for all 'enumeration' and 'bits' in case they are used in 'union' is lot of overhead to revolve a corner case.
- YANG also support the 'choice' statement (https://tools.ietf.org/html/rfc7950#section-4.2.7) which is similar to
   the concept of type tagging within union. This construct can be used by YANG modules specifically created for
  embedded applications to take advantage of 'enumeration' encoded as value and 'bits' encoded as bit.

Michel

-----Original Message-----
From: Carsten Bormann <[email-masked]> 
Sent: Thursday, March 28, 2019 9:47 AM
To: Juergen Schoenwaelder <[email-masked]>
Cc: Michel Veillette <[email-masked]>; Ladislav Lhotka <[email-masked]>; [email-masked]; [email-masked]
Subject: Re: [netconf] YANG encoding in CBOR

Grüße, Carsten

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 28 Mar 2019 15:30:31 +0100

I agree with your points. It seems this is the best we can do right
now and hopefully we can do better in the future.

/js

---

From: ivaylo petrov <[email-masked]>
Date: Thu, 28 Mar 2019 15:39:50 +0100

Hello,

I also support value outside and string inside unions. Michel gave a rather
good rationale why this is a good option and for me this seems as the best
option available right now.

Best regards,
Ivaylo

On Thu, Mar 28, 2019 at 3:32 PM Juergen Schoenwaelder <
[email-masked]> wrote:

> > - Generating SID for all 'enumeration' and 'bits' in case they are used
> in 'union' is lot of overhead to revolve a corner case.
> > - YANG also support the 'choice' statement (
> https://tools.ietf.org/html/rfc7950#section-4.2.7) which is similar to
> > Cc: Michel Veillette <[email-masked]>; Ladislav Lhotka <
> [email-masked]>; [email-masked]; [email-masked]

---

From: Andy Bierman <[email-masked]>
Date: Thu, 28 Mar 2019 08:50:34 -0700

> +1 for value outside, string inside union
This is simple, but I was trying to support some very common types (like
ip-address)
that are unions of strings, and are desirable to send as binary instead of
string representation.

Andy

My rational:
> - Generating SID for all 'enumeration' and 'bits' in case they are used in
> 'union' is lot of overhead to revolve a corner case.

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 28 Mar 2019 17:07:14 +0100

> that are unions of strings, and are desirable to send as binary instead of

How does CBOR deal with say inet:ipv4-address? Does inet:ipv4-address
magically get converted into a binary format? If not, the surprise that
it is a string in a union is not a bit surprise.

It seems to take full advantage of binary formats efficiently we need
some additional magic. The YANG language requires the definition of
canonical (textual) format. Do we need a way to define corresponding
canonical binary formats for some of the types (where the default
textual format is inefficient to send as a string)?

/js

---

From: Andy Bierman <[email-masked]>
Date: Thu, 28 Mar 2019 09:30:28 -0700

> textual format is inefficient to send as a string)?
I thought there was a magic tag for an IPv4 address.
I don't see it now.  Maybe it got removed.

Andy

---

From: Michel Veillette <[email-masked]>
Date: Thu, 28 Mar 2019 18:01:36 +0000

Hi Andy

The current discussion on 'union' don't help reducing the size and complexity of some of the 'ietf-inet-types' types such as:
typedef ip-address
typedef ipv4-address
typedef ipv6-address
typedef ip-address-no-zone
typedef ipv4-address-no-zone
typedef ipv6-address-no-zone
typedef ip-prefix
typedef ipv4-prefix

Or some of the 'ietf-yang-types' types such as:
typedef date-and-time
typedef timestamp
typedef phys-address
typedef mac-address
typedef hex-string
typedef uuid
typedef dotted-quad

The simples way to address those is to create a new version of these YANG modules by adding a binary option using a union.
I hope this can eventually happen when the CBOR encoding get more popular.

Regards,
Michel

From: Andy Bierman <[email-masked]>
Sent: Thursday, March 28, 2019 11:51 AM
To: Michel Veillette <[email-masked]>
Cc: Carsten Bormann <[email-masked]>; Juergen Schoenwaelder <[email-masked]>; [email-masked]; [email-masked]
Subject: Re: [netconf] YANG encoding in CBOR

On Thu, Mar 28, 2019 at 7:06 AM Michel Veillette <[email-masked]<mailto:[email-masked]>> wrote:
+1 for value outside, string inside union

This is simple, but I was trying to support some very common types (like ip-address)
that are unions of strings, and are desirable to send as binary instead of string representation.

Andy

My rational:

- YANG also support the 'choice' statement (https://tools.ietf.org/html/rfc7950#section-4.2.7<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Frfc7950%23section-4.2.7&data=02%7C01%7C%7C916ff06ce9384b8a104a08d6b3952561%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C0%7C636893850490334294&sdata=O9f0GNcRvg9s5aE%2BB9R43DjI2quiKzAfMfQcHhdnrMg%3D&reserved=0>) which is similar to

Michel

-----Original Message-----
From: Carsten Bormann <[email-masked]<mailto:[email-masked]>>
Sent: Thursday, March 28, 2019 9:47 AM
To: Juergen Schoenwaelder <[email-masked]<mailto:[email-masked]>>
Cc: Michel Veillette <[email-masked]<mailto:[email-masked]>>; Ladislav Lhotka <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [netconf] YANG encoding in CBOR

Grüße, Carsten

_______________________________________________
netconf mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/netconf<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Fnetconf&data=02%7C01%7C%7C916ff06ce9384b8a104a08d6b3952561%7C4f6fbd130dfb415085c3d43260c04309%7C0%7C0%7C636893850490344308&sdata=8%2BGjrFucM%2Bfs6L2TY%2FvHUYl83Cq9YkuqcxQANkR4hkQ%3D&reserved=0>