Subject: [core] YANG Packages for CoMI

From: Andy Bierman <[email-masked]>
Date: Sun, 26 Jul 2015 10:44:31 -0700

Hi,

Michel raised some issues with the YANG library draft
http://datatracker.ietf.org/doc/draft-ietf-netconf-yang-library/

Maybe CoMI needs its own optimized module library.
I am concerned that data sizes of the YANG library will be
too big in some really constrained environments.

I wrote a draft that defines a way to specify the contents of the
YANG library in a single static document, called a YANG package.
http://datatracker.ietf.org/doc/draft-bierman-netmod-yang-package/

If there was a "shorthand" resource that listed YANG packages, instead
of YANG modules, then the CoMI client that supported YANG packages
could read that instead of the module library.

The YANG package message response can identify a single package
(e.g. match the firmware) so the size of the response will remain very
small, and not depend on the entire list of modules, features,
and deviations. The data savings could 1000X for 100 modules.

The client needs to retrieve the library first-time and anytime it changes.
The module-set-id values are per-server, not global. This could be a
significant bottleneck when a CoMI client starts or restarts, and
manages lots of servers.

Andy

---

From: peter van der Stok <[email-masked]>
Date: Mon, 27 Jul 2015 08:26:02 +0200

HI Andy,

Is the proposal to make YANG packages compulsory over yang-library in 
Comi servers and clients?

Peter

Andy Bierman schreef op 2015-07-26 19:44:
> _______________________________________________
> core mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/core

---

From: Andy Bierman <[email-masked]>
Date: Sun, 26 Jul 2015 23:29:44 -0700

On Sun, Jul 26, 2015 at 11:26 PM, peter van der Stok <[email-masked]>
wrote:

> HI Andy,
>
> Is the proposal to make YANG packages compulsory over yang-library in Comi
> servers and clients?
>
>
No, it would be optional.
Another solution approach would be to make the module-set-id values
global instead of per-server. That might be less work perhaps.

Peter

Andy

---

From: Alexander Pelov <[email-masked]>
Date: Mon, 27 Jul 2015 12:25:14 +0200

Hi Andy!

Thanks for the draft! Seems a good start to get the things moving in the 
direction of having a good, structured introspection mechanism for 
module discovery!

I'm not completely sure that it will be a static document. You still 
need to think of the possibility of a firmware upgrade, in which case 
you have to consider the clients which have obtained your previous 
package descriptions.

Why don't you want it to be a YANG module? I would suppose that for a 
structured numbering scheme (such as CoOL) we could have a nice fixed 
entry point for such introspection resource, e.g. moduleID = 1. 
Alternatively, I would think that it should be discoverable through 
.well-known/core

A point to make is that we can have alias mappings through this 
introspection mechanism. Defining such an introspection mechanism for 
the aliases was actually one of the work I was considering for CoOL.

Best,
Alexander

Le 27/07/2015 08:29, Andy Bierman a écrit :
>
>
> On Sun, Jul 26, 2015 at 11:26 PM, peter van der Stok 
> <[email-masked] <mailto:[email-masked]>> wrote:
>
>     HI Andy,
>
>     Is the proposal to make YANG packages compulsory over yang-library
>     in Comi servers and clients?
>
>
>         [email-masked] <mailto:[email-masked]>

---

From: Andy Bierman <[email-masked]>
Date: Mon, 27 Jul 2015 11:01:08 -0700

On Mon, Jul 27, 2015 at 3:25 AM, Alexander Pelov <[email-masked]> wrote:

> direction of having a good, structured introspection mechanism for module
> discovery!
>
> I'm not completely sure that it will be a static document. You still need
> to think of the possibility of a firmware upgrade, in which case you have
> to consider the clients which have obtained your previous package
> descriptions.
>
>
Each package has a name and a revision-date.
The expected use for CoMI would be a new
package revision for each firmware release.

> structured numbering scheme (such as CoOL) we could have a nice fixed entry
> point for such introspection resource, e.g. moduleID = 1. Alternatively, I
> would think that it should be discoverable through .well-known/core
>
>
Not sure what you mean.
This is static data used to identify a set of YANG modules.
Do you mean why not have the contents of a package be YANG objects
to read from the server?  This would be even more data than the
current YANG module library.

The point of a package list is to avoid listing all the modules, features,
and revision dates.  There is a YANG Package Identifier in sec. 4

  urn:ietf:params:xml:ns:yang:pkg?name=example-sensor314&rev=2015-07-01

This would be the entire response needed to discover the entire YANG
implementation by the server. The size remains the same no matter how many
modules and features are added over time.

> introspection mechanism. Defining such an introspection mechanism for the
> aliases was actually one of the work I was considering for CoOL.
>
>
RESTCONF and NETCONF provide the ability for the client to retrieve
the schema, and get all the meta-data, for every module.
IMO this should be done by a centralized server in CoMI.

But I agree the package data could include alias mappings.
This might be a way to convert YANG Hash to COOL for example,
to optimize frequently used identifiers..

Andy

> On Sun, Jul 26, 2015 at 11:26 PM, peter van der Stok <
> <[email-masked]>[email-masked]> wrote:
>
>> HI Andy,
>>
> core mailing [email-masked]://www.ietf.org/mailman/listinfo/core
>
>
>

---

From: Alexander Pelov <[email-masked]>
Date: Tue, 28 Jul 2015 09:04:09 +0200

Hi Andy,

Le 27/07/2015 20:01, Andy Bierman a écrit :
>
>
> On Mon, Jul 27, 2015 at 3:25 AM, Alexander Pelov <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     Hi Andy!
>
>     Thanks for the draft! Seems a good start to get the things moving
>     in the direction of having a good, structured introspection
>     mechanism for module discovery!
>
>     I'm not completely sure that it will be a static document. You
>     still need to think of the possibility of a firmware upgrade, in
>     which case you have to consider the clients which have obtained
>     your previous package descriptions.
>
>
I meant that there should be a mechanism for the client to be notified 
when the YANG package definition changes. This may be a rare event, but 
imagine a new client arrives and discovers the server 1 minute before 
the server gets a new firmware release (or some other event, making it 
change its package definition).

I would think that if a client is expected to keep long-term association 
to the server, it should use Observe or NOTIFY. The latter will work if 
the package is defined as module.

>
>     Why don't you want it to be a YANG module? I would suppose that
>     for a structured numbering scheme (such as CoOL) we could have a
>     nice fixed entry point for such introspection resource, e.g.
>     moduleID = 1. Alternatively, I would think that it should be
>     discoverable through .well-known/core
>
>
> Not sure what you mean.

Ah, OK! Now I understand! It was not obvious to me that you were aiming 
at having the package definition be detached from the server. I like 
that! This, however makes me think about something else - you should 
have a way to obtain the package from the sensor. However, this goes a 
little bit in the sense of having YANG packages being mandatory for 
CoMI. I like the "uses-package" statement, as this could be used to have 
full definition of the modules on a server.

I would suppose that you could represent the information of a YANG 
package as a YANG module. Then, a client could query the server with a 
select/fields statement to obtain only the package identifier. IF the 
client wants, it could obtain the whole package content from the server:

E.g. for a YANG module definition such as:
module yang-package {
   ...
   leaf yang-package-uri {
     type string;
   }

   container package {
     leaf organization {type string;}
     leaf contact {type string;}
     leaf description {type string;}
     leaf revision {type string;}
     container imports-module {
        key module-name;
        leaf module-name {type string;}
        leaf module-cool-id {type int32;}
     }
     container uses-module {
        ...
     }
     ...
   }
}

module ID for this module would be fixed in CoOL to 1, so 
yang-package-uri will have CoOL ID 2049.

In CoOL this would be as getting :
GET /cdat?fields(2049)

Response:
{
2049:"urn:ietf:params:xml:ns:yang:pkg?name=example-sensor314&rev=2015-07-01"
}

Maybe this could be even more optimized.

Now, if the client has no idea how to find this URN, it would simply  
GET /cdat?fields(2048) and retrieve the YANG package information from 
the server.

>     introspection mechanism. Defining such an introspection mechanism
>     for the aliases was actually one of the work I was considering for
>     CoOL.
>
>

Maybe this should also be the way to handle alias attribution, e.g. add 
a "alias" statement, which indicates the mapping alias-id -> CoOL id / 
CoMI hash / full identifier.

module yang-package {
....
   container alias {
     key cool-id;
     leaf cool-id {type int32};
     leaf alias-id {type int32};
   }
}

So you can GET /cdat?fields(2058) and have
{
   2058: [
     {11: 29834394, 12:1},
     {11: 99999999, 12:2}
     {11: 88888384, 12:3}
   ]
}

assuming the following CoOL id automatic attribution:
alias -> node ID=10
cool-id -> node Id=11
alias-id -> node ID=12
and some random IDs to be mapped (29834394, 99999999, 88888384).

Ideally, of course, you would be fetching this information from a server 
somewhere on the Internet, so that you don't fetch from your constrained 
server.

Best,
Alexander

>>         Is the proposal to make YANG packages compulsory over
>>         yang-library in Comi servers and clients?
>>
>>
>>             The client needs to retrieve the library first-time and
>>             anytime it
>>             changes.

---

From: Andy Bierman <[email-masked]>
Date: Tue, 28 Jul 2015 19:19:01 -0700

On Tue, Jul 28, 2015 at 12:04 AM, Alexander Pelov <[email-masked]> wrote:

> On Mon, Jul 27, 2015 at 3:25 AM, Alexander Pelov < <[email-masked]>
> [email-masked]> wrote:
>
>>  Hi Andy!
>>
> imagine a new client arrives and discovers the server 1 minute before the
> server gets a new firmware release (or some other event, making it change
> its package definition).
>

I agree.
There can be a YANG Package Library URI

  /mg/pkg.uri

URI of the YANG Package Library resource to GET or Observe

The package data tree could be much smaller than the module library tree.

  container packages {
     leaf pkg-set-id { type uint32; }
     list package {
        key name;
        leaf name { type string; }
        leaf rev { type yang:revision-date; }
     }
   }

  { "packages": {
     "pkg-set-id":42,
     "package": [
       {
          "name":"example-sensor314",
          "rev":"2015-06-14"
       }
     ]
   }
 }

> to the server, it should use Observe or NOTIFY. The latter will work if the
> package is defined as module.
>
>
>
>
> Ah, OK! Now I understand! It was not obvious to me that you were aiming at
> having the package definition be detached from the server. I like that!
> This, however makes me think about something else - you should have a way
> to obtain the package from the sensor. However, this goes a little bit in
> the sense of having YANG packages being mandatory for CoMI. I like the
> "uses-package" statement, as this could be used to have full definition of
> the modules on a server.
>
> I would suppose that you could represent the information of a YANG package
> as a YANG module. Then, a client could query the server with a

The point of the YANG package is that it serves as an offline handle
for lots of YANG details related to a set of modules.

NETCONF and RESTCONF just advertise the conformance details
for each module (name, revision, features, deviations). There is
a <get-schema> operation in both protocols to allow the client to retrieve
the actual YANG files.  There is no YANG module that modules
YANG module contents.

The same approach <get-pkg-schema> should be used in CoMI,
except I would expect a centralized server to have that data,
not any constrained servers.

module ID for this module would be fixed in CoOL to 1, so yang-package-uri
> will have CoOL ID 2049.

I don't know if package numbers are really needed.
The entire package library above would fit in 128 bytes.

Andy

>
> Now, if the client has no idea how to find this URN, it would simply  GET
> /cdat?fields(2048) and retrieve the YANG package information from the
> server.
>
>
>
>
>
>
> Maybe this should also be the way to handle alias attribution, e.g. add a
> "alias" statement, which indicates the mapping alias-id -> CoOL id / CoMI
> hash / full identifier.
>
> module yang-package {
> ....
>   container alias {
>     key cool-id;

---

From: Alexander Pelov <[email-masked]>
Date: Wed, 29 Jul 2015 10:33:13 +0200

Hi Andy,

Le 29/07/2015 04:19, Andy Bierman a écrit :
>
>
> On Tue, Jul 28, 2015 at 12:04 AM, Alexander Pelov <[email-masked] 
> <mailto:[email-masked]>> wrote:
>
>     Hi Andy,
>
>>     On Mon, Jul 27, 2015 at 3:25 AM, Alexander Pelov
>>     <[email-masked] <mailto:[email-masked]>> wrote:
>>
>>         Hi Andy!
>>
>>         Thanks for the draft! Seems a good start to get the things
>>         moving in the direction of having a good, structured
>>         introspection mechanism for module discovery!
>>
>>         I'm not completely sure that it will be a static document.
>>         You still need to think of the possibility of a firmware
>>         upgrade, in which case you have to consider the clients which
>>         have obtained your previous package descriptions.
>>
>>
>     I meant that there should be a mechanism for the client to be
>     notified when the YANG package definition changes. This may be a
>     rare event, but imagine a new client arrives and discovers the
>     server 1 minute before the server gets a new firmware release (or
>     some other event, making it change its package definition).
>
>
>
> I agree.

I suppose in a YANG hash world you need to multiply the file locations 
so that you can have this information in a convenient place. In a 
numbering scheme with managed IDs you can allocate predefined module IDs 
for the important modules. This one seems to be of great importance!

Do you think that we could re-use the RFC 6022 - YANG module for NETCONF 
monitoring? It seems to me that it does exactly what the YANG package 
you are proposing.

In both cases (packages or RFC 6022) if we have the introspection 
information exposed as YANG module, we don't need to invent additional 
mechanisms. By performing a select/fields filtering, a client can obtain 
in a lightweight fashion the information that concerns it. If the client 
wants, it could obtain the full schema information, but that is not 
necessary.

>
>
>     I would think that if a client is expected to keep long-term
>     association to the server, it should use Observe or NOTIFY. The
>     latter will work if the package is defined as module.
>
>>
>>         Why don't you want it to be a YANG module? I would suppose
>>         that for a structured numbering scheme (such as CoOL) we
>>         could have a nice fixed entry point for such introspection
>>         resource, e.g. moduleID = 1. Alternatively, I would think
>>         that it should be discoverable through .well-known/core
>>
>>
>>     Not sure what you mean.
>     Ah, OK! Now I understand! It was not obvious to me that you were
>     aiming at having the package definition be detached from the
>     server. I like that! This, however makes me think about something
>     else - you should have a way to obtain the package from the
>     sensor. However, this goes a little bit in the sense of having
>     YANG packages being mandatory for CoMI. I like the "uses-package"
>     statement, as this could be used to have full definition of the
>     modules on a server.
>
>     package as a YANG module. Then, a client could query the server
>     with a select/fields statement to obtain only the package
>     identifier. IF the client wants, it could obtain the whole package
>     content from the server:
>

I don't understand the need of modeling the YANG module contents. Once 
you have the conformance details (name, revision, features, deviations), 
you can try and fetch them from an remote store (or have them locally). 
If you don't have access to this information, you need to fetch their 
schema. Do you see an intermediate level of details?

For me, the introspection mechanism should be implemented as a YANG 
module. There is no need to invent new mechanisms, when we have an 
existing one. I would imagine that you could declare the YANG hash of 
this module in the .well-known/core if you want to be flexible.

Best,
Alexander

>     Now, if the client has no idea how to find this URN, it would
>     simply  GET /cdat?fields(2048) and retrieve the YANG package
>     information from the server.
>
>>
>>
>>
>>
>>         A point to make is that we can have alias mappings through
>>         this introspection mechanism. Defining such an introspection
>>         mechanism for the aliases was actually one of the work I was
>>         considering for CoOL.
>>
>>
>     Maybe this should also be the way to handle alias attribution,
>     e.g. add a "alias" statement, which indicates the mapping alias-id
>     -> CoOL id / CoMI hash / full identifier.
>
>     module yang-package {
>     ....
>       container alias {
>         key cool-id;
>     Ideally, of course, you would be fetching this information from a
>     server somewhere on the Internet, so that you don't fetch from
>     your constrained server.
>
>     Best,
>     Alexander
>
>>
>>
>>
>>
>>         Best,
>>         Alexander
>>
>>
>>     Andy
>>
>>
>>>                 The client needs to retrieve the library first-time
>>>                 and anytime it
>>>                 changes.