Subject: Re: [Netconf] Issue #2 for binary encoding: Should the new encoding include the Message layer?

From: Mahesh Jethanandani <[email-masked]>
Date: Mon, 25 Jun 2018 17:14:38 -0700

Picking up on this thread ...

> On Mar 23, 2018, at 4:30 PM, Andy Bierman <[email-masked]> wrote:
> 
> 
> 
> On Wed, Mar 21, 2018 at 1:25 AM, Juergen Schoenwaelder <[email-masked] <mailto:[email-masked]>> wrote:
> There is encoding of YANG defined data and there is the encoding of
> protocol messages. Lets not confuse the two.
> 
> 
> 
> good point.

Agree.

> We have usually left out YANG for protocol messages, but now we have YANG
> Data Extensions so the complete message can be defined in YANG (sort of)

The preference, at least on part of the authors, is to NOT change protocol message formats, at least for now. In addition, as Juergen states, changing the message layer would involve opening up RFC 6241 and splitting the message layer from other parts of NETCONF, work I am not sure the WG is ready to adopt.

> 
> I have concerns about encoding of metadata.
> This is critical for real-world NM tools.
> IMO XML designers really got this right.
> JSON and everything else - not so much.
> 
> The practice of stuffing JSON objects into XML string nodes seems like
> the worst of both worlds.  NETCONF relies on metadata in the content
> layer, so encoding <rpc> in XML does not even help a little.
> Switching parsers from XML to something else in mid-message is a non-starter.

Agreed. Our proposal is to keep XML parsing, and encode only YANG data. BTW, the idea was to encode YANG data into CBOR+SID, not the XML version of the YANG data, in case there was any confusion. 

> 
> 
> - RESTCONF: Protocol encoding is handled by HTTP. HTTP/1.1 uses a
>   textual format, HTTP/2 uses a binary format. HTTP was designed to
>   handle arbitrary content formats. Hence, using RESTCONF with XML,
>   JSON, CBOR, ... just works (but note that the HTTP protocol layer
>   does not use XML, nor JSON nor CBOR encoding).
> 
> 
> 
> We should use IANA media types and the Accept header, as designed,
> to add support for new message formats.
> 
>  
> - NETCONF: NETCONF does not have the capability to support different
>   content formats. We can (a) retrofit this onto the existing protocol
>   format (allowing other content encoding on top of the XML encoded
>   protocol format) or (b) we create a new protocol encoding that can
>   handle different content formats more efficiently than (a) would do.
>   But this largely boils down to a new protocol version, stripped down
>   to the messaging layer.
> 
>   I do not think it is desirable to have N different message layer
>   encodings for N different content layer encodings since you need at
>   least something common to negotiate the encodings. That said, one
>   could define the messaging layer using the yang-data extension and
>   then the rpc message formats translate to XML, JSON, CBOR, ... but
>   we would still have to have something common that initially
>   negatiates the encodings.
> 
> 
> I prefer the switch-over after the <hello> exchange.
> This would be NETCONF's version of the Accept header.

The draft will be updated to reflect this.

> 
>    
> 
> Anyway, to do all this properly, we would have to open up RFC 6241 and
> to split it into pieces, separating the messaging layer cleanly from
> the well-known NETCONF operations and their semantics.
> 
> 
> Properly for documentation and standards track purposes,
> but not really for interoperability.
> 
>  
> This of course raises the question whether this work is useful to
> entertain given that we already have RESTCONF that happily supports
> different content encoding formats and which can be used with both a
> textual encoding and a binary encoding of the messaging layer. (And
> outside the IETF we have protocols that use entirely different RPC
> layers.) Perhaps the simpler answer is that if you really need
> something more efficient than NETCONF, use RESTCONF. Available today.
> 
> 
> 
> It is true that NMDA RESTCONF exposes many details that were intentionally left
> of RFC 8040.  IMO we should support both protocols for the foreseeable future.
> 
> My wishlist for NETCONF binary data:
>   - keep NETCONF 1.1 message framing

Agree.

>   - define protocol mapping to use CBOR + SID message encoding

Agree.

>   - define extension to encode metadata in CBOR + SID using the annotation-stmt in RFC 7952

Ok.

>   - define some binary data types for IP addresses etc. so they are not encoded as CBOR strings

Would this be for this draft or something like CBOR+SID to define this?

Thanks.

> 
> 
> /js
> 
> Andy
>  
> 
> On Wed, Mar 21, 2018 at 08:02:11AM +0000, Mahesh Jethanandani wrote:
> > Added another issue to GitHub here - https://github.com/netconf-wg/binary-encoding/issues/2 <https://github.com/netconf-wg/binary-encoding/issues/2><https://github.com/netconf-wg/binary-encoding/issues/2 <https://github.com/netconf-wg/binary-encoding/issues/2>> that reads:
> >
> > The -00 version of the draft is silent about which layer the alternate form of encoding impacts. After some discussion, the authors clarified on the mailing list, that the draft was going to propose that encoding of alternate forms of encoding will impact layers above the Message layer per Figure 1 of RFC 6241.
> >
> > An alternate proposal is for the Message layer to be binary encoded also. That means that whatever form of encoding is agreed upon has to provide a way to encode Message layer RPCs, such as <rpc>, <rpc-reply>, <rpc-error> and <ok>.
> >
> > The pro of not encoding the Message layer is that alternate form of encoding does not have to define encoding for that layer. For example, CBOR defines a way to encode YANG. It does not define how to encode RPCs, at least currently. BTW, that draft is in LC, if not past it.
> >
> > The con is that one would lose 25% efficiency in doing a base64 encoding, not to include the fact that one would have to deal with two forms of encoding.
> >
> > If we do agree that the Messaging layer should be binary encoded, (a separate) draft would have to define a way to encode the Message layer. And, this would have to be done for each form of encoding.
> >
> > Question for the WG, should the new encoding include the Message layer?
> >
> > Mahesh Jethanandani
> > [email-masked] <mailto:[email-masked]>
> >
> 
> > _______________________________________________
> > Netconf mailing list
> > [email-masked] <mailto:[email-masked]>
> > https://www.ietf.org/mailman/listinfo/netconf <https://www.ietf.org/mailman/listinfo/netconf>
> 
> 
> --
> Juergen Schoenwaelder           Jacobs University Bremen gGmbH
> Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
> Fax:   +49 421 200 3103         <https://www.jacobs-university.de/ <https://www.jacobs-university.de/>>
> 
> _______________________________________________
> Netconf mailing list
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/netconf <https://www.ietf.org/mailman/listinfo/netconf>
Mahesh Jethanandani
[email-masked]

---

From: Andy Bierman <[email-masked]>
Date: Mon, 25 Jun 2018 18:46:41 -0700

On Mon, Jun 25, 2018 at 5:14 PM, Mahesh Jethanandani <
[email-masked]> wrote:

> On Wed, Mar 21, 2018 at 1:25 AM, Juergen Schoenwaelder <j.
> [email-masked]> wrote:
>
> The preference, at least on part of the authors, is to NOT change protocol
> message formats, at least for now. In addition, as Juergen states, changing
> the message layer would involve opening up RFC 6241 and splitting the
> message layer from other parts of NETCONF, work I am not sure the WG is
> ready to adopt.
>
>

I am not sure how this would be implemented.
You seem to be proposing a hybrid parser:

   <rpc> ... binary data ... </rpc>

   <rpc-reply> ... binary data ... </rpc-reply>

   <notification> ... binary data ... </notification>

I do not know of any XML tools that let you switch to a different (binary)
parser
for the start-tag of a complex element, then switch back for the end-tag.

I suggest assigning SID values for these nodes and encoding the entire
message
in CBOR.

> Agreed. Our proposal is to keep XML parsing, and encode only YANG data.
> BTW, the idea was to encode YANG data into CBOR+SID, not the XML version of
> the YANG data, in case there was any confusion.
>
>
I do not see what benefits exist from keeping XML parsing
(besides the <hello> that has to stay the same: XML + base:1.0 framing).
I am not sure how that even works since attributes are in the YANG data.

CBOR does not support metadata at all.
NETCONF <edit-config> does not really work without it.
There are also retrieved attributes (with-defaults, with-origin) that
cannot be supported.
There is also the "message-id" attribute in NETCONF that cannot be
supported.

Andy

>> > Added another issue to GitHub here - https://github.com/netconf-
>> wg/binary-encoding/issues/2<https://github.com/netconf-wg/
>> binary-encoding/issues/2> that reads:
>> >
>> > The -00 version of the draft is silent about which layer the alternate
>> form of encoding impacts. After some discussion, the authors clarified on
>> the mailing list, that the draft was going to propose that encoding of
>> alternate forms of encoding will impact layers above the Message layer per
>> Figure 1 of RFC 6241.
>> >
>> > An alternate proposal is for the Message layer to be binary encoded
>> also. That means that whatever form of encoding is agreed upon has to
>> provide a way to encode Message layer RPCs, such as <rpc>, <rpc-reply>,
>> <rpc-error> and <ok>.
>> >
>> > The pro of not encoding the Message layer is that alternate form of
>> encoding does not have to define encoding for that layer. For example, CBOR
>> defines a way to encode YANG. It does not define how to encode RPCs, at
>> least currently. BTW, that draft is in LC, if not past it.
>> >
>> > The con is that one would lose 25% efficiency in doing a base64
>> encoding, not to include the fact that one would have to deal with two
>> forms of encoding.
>> >
>> > If we do agree that the Messaging layer should be binary encoded, (a
>> separate) draft would have to define a way to encode the Message layer.
>> And, this would have to be done for each form of encoding.
>> >
>> > [email-masked]
>> > https://www.ietf.org/mailman/listinfo/netconf
>>
>>
>> --
>> <https://maps.google.com/?q=Campus+Ring+1+%7C+28759+Bremen+%7C+Germany&entry=gmail&source=g>
>> Fax:   +49 421 200 3103         <https://www.jacobs-university.de/>
>>
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/netconf
>
>
> Mahesh Jethanandani
> [email-masked]
>
>

---

From: Mahesh Jethanandani <[email-masked]>
Date: Thu, 28 Jun 2018 20:48:50 -0700

> On Jun 25, 2018, at 6:46 PM, Andy Bierman <[email-masked]> wrote:
> 
> 
> 
> On Mon, Jun 25, 2018 at 5:14 PM, Mahesh Jethanandani <[email-masked] <mailto:[email-masked]>> wrote:
>> On Mar 23, 2018, at 4:30 PM, Andy Bierman <[email-masked] <mailto:[email-masked]>> wrote:
>> 
>> 
>> 
> I do not know of any XML tools that let you switch to a different (binary) parser
> I suggest assigning SID values for these nodes and encoding the entire message
> in CBOR.

In talking to Andy and getting clarification from him, here is my understanding of what he is proposing.

First of all he is suggesting that the entire message, message header and yang-data be encoded using the agreed upon form of encoding. While the clear favorite is cbor/sid, the exchange does allow for other forms of encoding to be negotiated. For the message header, and in case of cbor/sid encoding, the SID values will have to come from a new range TBA.

The initial exchange, session setup would look as follows:

- <hello> exchange between client and server happens as before
- client sends an ordered list of encodings it supports.
- server sends an unordered list of encodings it supports
- both client and server check capabilities and decide to switch to base:1.1 framing
- the first match is picked. If no match is found the session is dropped
- if match is found, both client and server switch to the matched encoding even before the first rpc is sent.

If the match found is cbor/sid, the data exchange consists of message header encoded using cbor/sid and yang-data attributes encoded in cbor/sid as child nodes.

This approach has the advantage that all RPC operations and notifications are supported with the new encoding format.

Cheers.

> There are also retrieved attributes (with-defaults, with-origin) that cannot be supported.
> There is also the "message-id" attribute in NETCONF that cannot be supported.
> 
>> 
>> 
>> Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany <https://maps.google.com/?q=Campus+Ring+1+%7C+28759+Bremen+%7C+Germany&entry=gmail&source=g>

Mahesh Jethanandani
[email-masked]

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 29 Jun 2018 09:46:27 +0200

> CBOR does not support metadata at all.

I’m trying to follow this discussion, but I can’t process this sentence.

What is “metadata” here?  Or, maybe, what is “support”?
(You certainly can encode any kind of metadata in CBOR.)

Grüße, Carsten

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 29 Jun 2018 10:16:17 +0200

> (You certainly can encode any kind of metadata in CBOR.)

Carsten,

YANG definitions translate to values of XML elements. We in addition
use XML attributes to carry metadata, for example following RFC 7952.
See section 5 of RFC 7952 for details how this works with the XML and
JSON encodings.

Note that what we call meanwhile metadata did exist before 7952 came
along: the NETCONF protocol operations use XML attributes to control
how the edit is carried out, see section 7.2 of RFC 6241. Hence, in
order to run NETCONF over a different encoding, we need a convention
to carry metadata along with the data.

/js

---

From: Andy Bierman <[email-masked]>
Date: Fri, 29 Jun 2018 02:10:46 -0700

> to carry metadata along with the data.

I think CBOR+SID can support attributes without any changes to either of
them.
If each attribute is numbered (e.g., SID numbers annotation-stmts), then
the encoding looks the same as an augmenting leaf. It is just another leaf
that happens
to have a SID value from a different module range.

/js

Andy

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 29 Jun 2018 14:47:46 +0200

Hi Juergen,

thanks for the translation; I hadn’t read RFC 7952 yet.
The various hacks in Section 5.2 can probably be replaced by a single CBOR tag.
Let’s call that ANNO (or maybe ATTR?):

So what’s 

   "foo:flag": true,
   "@foo:flag": {
     "example-last-modified:last-modified":
       "2015-09-16T10:27:35+02:00"
   }

in Section 5.2 could be

   ANNO([
      { SID-for-foo-flag: true },
      { SID-for-elm-lm: …date… },
   ])

or maybe

   SID-for-foo-flag: ANNO([
      true,
      { SID-for-elm-lm: …date… },
   ])

or maybe even shorter:

   SID-for-foo-flag: ANNO([
      true,
      SID-for-elm-lm, …date…,
   ])

in CBOR (where SID-for-.. is the SID for the identifier).

I haven’t checked how to do RFC 6241 attributes, but the principle should be the same.

I also haven’t checked how the netconf protocol elements themselves could be encoded in CBOR, but that should be straightforward.
Let me just say that another SDO that shall not be named here has come up with a rather mechanical way to translate XML into CBOR — a solution for netconf could be just that or could be tweaked to be more efficient for that specific XML schema.

Grüße, Carsten

> On Jun 29, 2018, at 10:16, Juergen Schoenwaelder <[email-masked]> wrote:
>