Subject: Re: [netmod] instance file parsing - what is the target YANG module set

From: Balázs Lengyel <[email-masked]>
Date: Sat, 1 Dec 2018 09:28:46 +0000

Hello Andy,

Give me a few days and I will come out with -01 version of the
draft that addresses this issue. We propose 3 options:

Include the needed information as part of instance data as
defined by ietf-yang-library, that documents module name,
revision, supported features, deviations

Include a URI that points to the same information (if you
don't want to repeat the info again and again)

Include nothing as the target YANG module set may already be
known, or the information is available through external
documents.

I also see a major reason for this draft: once it is agreed we
can use instance data to document other standard things, like
server capabilities.

regards Balazs

On 2018. 11. 30. 18:48, Andy Bierman
wrote:

Hi,

I don't see much standards value in this draft so far.

It seems the parser of the file needs to know the YANG
library information

for the instance file data in some out-of-scope
non-standard manner.

This is what we already have today just by naming the file
in a specific way.

Is the intent that the instance-data-set leafs will be used
to determine

the module revisions, features, and deviations used in the
children

or the 'data' node?

Andy

_______________________________________________
netmod mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/netmod

--
Balazs Lengyel Ericsson Hungary Ltd.
Senior Specialist
Mobile: +36-70-330-7909 email: [email-masked]

---

From: Robert Wilton <[email-masked]>
Date: Mon, 3 Dec 2018 10:21:20 +0000

On 30/11/2018 19:28, Juergen Schoenwaelder wrote:
> I doubt it makes sense to carry an entire yang library schema with the
> instance data. The modules are actually known via the namespaces.

Knowing the module name or namespace isn't sufficient to be able to 
parse the data:
  - E.g. if the contents are in the format of RFC7895bis, but the client 
tries to parse it using the schema from RFC7895, then it will fail.
  - Once NBC changes are allowed via YANG versioning it will become more 
necessary to know the revision or version of the modules to be able to 
parse the data.  E.g. not even just the latest module revision will 
necessarily work.   It is entirely plausible that different server 
revisions may be generating instance data using slightly different schema.

If the server has deviations then the client may also need to know those 
deviations to correctly parse the file.

So, I think that it is pretty clear that knowing the right schema is 
required to be able to correctly parse and interpret the instance data.

I think that there are 4 ways that this can be achieved:
  1) embedded the necessary schema information into the YANG instance data.
  2) put the necessary schema information online somewhere and have a 
URI reference to it.
  3) some combination of (1) and (2), e.g. packages defined centrally, 
with deviations listed in the file.
  4) the schema is determined using some out of bound mechanism, or 
possibly it is hard coded.

I don't think that there is a one size fits all answer here.  I think 
that that it depends on the use case.  Certainly, I think that 
facilitating 1 -3 is useful, but they should be optional rather than 
mandatory.  I.e. defining nodes for these doesn't seem to cost much if a 
server isn't obliged to populate them.

I do think that YANG packages (themselves defined in instance data 
documents) could be very helpful here.  I.e. rather than listing all the 
modules, instead list the packages + any deviations from that.  I'm 
presuming here that the definitions of the packages are available via a URI.

>   If
> you want to capture the schema, dump the relevant yang library into
> another instance file.

That just means another file to carry around and manage.

Thanks,
Rob

>
> /js
>
> On Fri, Nov 30, 2018 at 07:07:03PM +0000, Robert Wilton wrote:
>> I also think that the instance data header needs to have a way of indicating
>> which modules (and versions/revisions) the instance data applies to.
>>
>> It can be optional, so producers that know it is not required can leave it
>> out.  But I think that it is required, at least for situations where being
>> able to programmatically consume the instance data in a robust way is
>> important.
>>
>> Thanks,
>> Rob
>>
>>
>> On 30/11/2018 17:48, Andy Bierman wrote:
>>> Hi,
>>>
>>> It seems the parser of the file needs to know the YANG library information
>>> for the instance file data in some out-of-scope non-standard manner.
>>> This is what we already have today just by naming the file in a specific
>>> way.
>>>
>>> Is the intent that the instance-data-set leafs will be used to determine
>>> the module revisions, features, and deviations used in the children
>>> or the 'data' node?
>>>
>>> Andy
>>>
>>>
>>>

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 3 Dec 2018 12:36:33 +0100

On Mon, Dec 03, 2018 at 10:21:20AM +0000, Robert Wilton wrote:
> Knowing the module name or namespace isn't sufficient to be able to parse
> the data:
> necessary to know the revision or version of the modules to be able to parse
> the data.  E.g. not even just the latest module revision will necessarily
> work.   It is entirely plausible that different server revisions may be
> generating instance data using slightly different schema.

Perhaps this is a problem of the versioning solution then. ;-)
 
>  2) put the necessary schema information online somewhere and have a URI
> reference to it.
>  3) some combination of (1) and (2), e.g. packages defined centrally, with
> deviations listed in the file.
>  4) the schema is determined using some out of bound mechanism, or possibly
> it is hard coded.

Perhaps we need to define first what "parse" means. I am not sure
parsing requires to know all schema details. Anyway, if all schema
details are needed, then the simplest solution seems to be to read the
schema from another instance file. This then only requires to
bootstrap the schema model version.

> I don't think that there is a one size fits all answer here.  I think that
> that it depends on the use case.  Certainly, I think that facilitating 1 -3
> is useful, but they should be optional rather than mandatory.  I.e. defining
> nodes for these doesn't seem to cost much if a server isn't obliged to
> populate them.
> 
 
> That just means another file to carry around and manage.

Yes, but compared to solutions that require new and/or much more
elaborate data formats, this is very cheap and efficient, in
particular for systems where the schema itself is relatively
static. Also in terms of engineering time this is a rather cheap
solution since you do not need to invent a new way to document and
communicate a schema.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1 | 28759 Bremen | Germany
Fax:   +49 421 200 3103         <https://www.jacobs-university.de/>

---

From: Andy Bierman <[email-masked]>
Date: Mon, 3 Dec 2018 09:56:48 -0800

On Mon, Dec 3, 2018 at 3:36 AM Juergen Schoenwaelder <
[email-masked]> wrote:

> parse

I agree with Rob.
The solution has to support accurate parsing of the instance data.
This means that the parser has the correct schema tree to use for
validating an instance document against the schema.

Since the new yang-library can have a different schema tree for each
datastore,
clearly the option of specifying the datastore is needed.

> that

Yes -- this has been my goal for YANG Packages since the start.
By using nested packages and offline caching, the entire YANG library for a
device
could be recognized in a few URIs.

> solution since you do not need to invent a new way to document and
I agree the cost of an extra instance document is not a problem, especially
since
it would be optional and only used if the defaults are not sufficient:
   - latest revision of module will be OK
   - assume all features present will be OK
   - assume no deviations will be OK

If the defaults are not OK, then the parser will incorrectly accept or
reject certain instance data,
unless the correct schema tree is obtained somehow.

/js

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Mon, 3 Dec 2018 19:13:23 +0100

> clearly the option of specifying the datastore is needed.

We need to get the terminology straight. I can parse XML and JSON
files without the need to have schema information available. So I
think the word 'parsing' is quite misleading here. And I can extract
data out of XML and JSON files that I find interesting without schema
information as well. So its a certain type of tools that may take
advantage of being schema aware but not all tools need to be schema
aware.

> could be recognized in a few URIs.

An instance file storing the schema tree is an offline cache. It falls
out of the instance file format for free. Yes, packages are yet
something entirely different but so far no work in this direction is
chartered so we should get instance file formats defined without
solving another bigger problem first.

> unless the correct schema tree is obtained somehow.

Or the tool processing instance file content just does not care about
the schema and it simply assumes that a certain (module,path)
combination has fixed semantics. I am big fan of being able to use
generic tools to filter and process XML or JSON encoded data.

/js

---

From: Robert Wilton <[email-masked]>
Date: Tue, 4 Dec 2018 17:14:54 +0000

> solving another bigger problem first.

In the IETF 103 discussion on YANG versioning, it seemed to be the 
consensus in the room that the versioning design team need to consider 
also versioning of sets of modules, rather than just individual 
modules.  I.e. something along the lines of YANG packaging.

Thanks,
Rob

---

From: Andy Bierman <[email-masked]>
Date: Tue, 4 Dec 2018 09:23:30 -0800

On Tue, Dec 4, 2018 at 9:14 AM Robert Wilton <[email-masked]> wrote:

> URI
> >>>> that it depends on the use case.  Certainly, I think that
> facilitating 1
> >>> -3
> >>>> presuming here that the definitions of the packages are available via
> a
> >>> URI.
> >>>
> >>>

I am not suggesting YANG Packages has to be solved in order to
work on this instance document format. The IETF thinks YANG modules
are the appropriate level of abstraction.

Issues like "the extra metadata takes up too much space" are entirely
subjective.
Obviously a parser does not need any schema to parse anydata.
This is only needed if there is a real schema expected within the anydata.

It is optional for the writer to add the extra metadata.
It is optional for the reader to use the extra metadata.
Not sure why the solution should forbid this functionality.

Andy