Subject: [netmod] overlays

From: Martin Bjorklund <[email-masked]>
Date: Wed, 25 Jun 2008 22:42:50 +0200 (CEST)

Hi,

Starting a new thread for the overlay discussion.

I want to solve the same problem as Phil, i.e. being able to attach
vendor-specific implementation directives to standard modules, w/o
actually having to modify the standard module text.

My original idea was to use annotation files for this, where these
files would be applied as overlays to the original module.  For
example:

Given this standard module:

  module ietf-interface {
    namespace "urn:ietf:params:xml:ns:interface";
    prefix if;

    ...

    container interfaces {

      list interface {
        key name;

        leaf name {
          type string;
        }
 
        leaf mtu {
          type uint32;
        }

        ...
       }
     }
  }

I could write an annotation file (my-interface.ann) like this:

  annotation ietf-interface {
 
    // add an import of the vendor-specific extensions
    import tailf-extensions {
      prefix tailf;
    }

    container interface {
 
      list interface {
        // add an important implementation detail
        tailf:callpoint my-cp;
      }
    }
  }
         
Then I would give this information to my YANG compiler:

  # pyang --emit-code ietf-interface.yang --annotation my-interface.ann

Annotations file can cascade, so I can have one annotation file with
CLI-hints, one with code generation hints and so on.  Or put
everything in one file.

This was supposed to be completely proprietary.

But I think the same idea can be useful to specify
implementation-specific defaults.  Something like this:

  annotation ietf-interface {

    container interface {
     
      list interface {

        leaf mtu {
          // add a product-specific default
          default 1500;
        }
      }
    }
  }

This would require some changes to the current YANG semantics (can I
add a default value to a mandatory leaf?) and for it to be useful, a
manager MUST be able to get this information, e.g. by using the schema
discovery mechanism.

/martin

_______________________________________________
netmod mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/netmod

---

From: "Randy Presuhn" <[email-masked]>
Date: Wed, 25 Jun 2008 15:29:01 -0700

Hi -

> From: "Martin Bjorklund" <[email-masked]>
> To: <[email-masked]>
> Sent: Wednesday, June 25, 2008 1:42 PM
> Subject: [netmod] overlays
...
...

I like the general concept a lot.  It was one of the things I found most
interesting about the the work Alex Clemm and TJ Tjong had done
leading up to the canmod BOF.  At a previous company, we got a lot
of experience using different annotation strategies with our GDMO
and SNMP SMI MIB compilers.  A few conclusions I've reached:

    (1) For software (release) configuration management sanity, it is essential
          to be able to keep annotations separate from the annotated
          material.

    (2) If the annotation in any way changes the "interface contract"
          represented, it is essential to be able to treat this as a distinct
          contract (= class definition or whatever) even if it is allomorphic
          to the original definition.  (this means, for example, being able
          to retrieve the modified interface definition)

   (3) Having "chains" of refinement, while perhaps theoretically elegant,
         can be a maintenance pain.  It's ultimately easier to derive different
         versions of a definition from the same base definition, than working
         back through a chain of versions to reach that base.  This is particularly
         true if an organization does development work on multiple releases
         concurrently.

   (4) It's important to separate annotations that elaborate on the "interface
        contract" from the ones that elaborate on the implementation.

   (5) It's important to think through annotations for IMPORTed bits very
        carefully.  In our particular case, we encountered situations where
        instrumentation from different vendors meant that "the same"
        bit of management information in different contexts on a single
        system required different annotations.

Randy

---

From: Phil Shafer <[email-masked]>
Date: Thu, 26 Jun 2008 02:33:27 -0400

>        // add an important implementation detail

This gives two distinct "container" statements, one in
real YANG and one in annotations.  What would you think
of using an augmentation-like approach:

    import tailf-extensions { ... }

    annotate "interface/interface" {
        // add an important implementation detail
        tailf:callpoint my-cp;
    }
}

Thanks,
 Phil

---

From: "Randy Presuhn" <[email-masked]>
Date: Wed, 25 Jun 2008 23:46:31 -0700

Hi -

> From: "Phil Shafer" <[email-masked]>
> To: "Martin Bjorklund" <[email-masked]>
> Cc: <[email-masked]>
> Sent: Wednesday, June 25, 2008 11:33 PM
> Subject: Re: [netmod] overlays
...
...

Yet another way of looking at the problem is to consider the
annotation as a transform applied to one model to produce
another.  The kinds of transform that are possible set the
limits of what can be considered in some sense to be
"compatible" models.

Randy

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 26 Jun 2008 09:21:31 +0200 (CEST)

Phil Shafer <[email-masked]> wrote:
> real YANG and one in annotations.

Yes, just like it's currently being done in 'uses' refinements.  (but
then we might want to change that).

> What would you think

This is the strategy we currently use in our proprietary DML.  The
disadvantage is that it becomes clumsy and error-prone if you have
many annotations:

     annotate "interface/interface/mtu" {
         tailf:display-hint "...";
     }

     annotate "interface/interface/unit" {
         tailf:callpoint my-cp2;
     }

     annotate "interface/interface/unit/vlan-id" {
         tailf:display-hit "...";
     }

     ...

Maybe a hybrid scheme would work:

         annotate mtu {
             tailf:display-hint "...";
         }

         annotate unit {
             annotate vlan-id { ... }
         }
     }

Hmmm I like this for the same reasons that I like an explicit 'refine'
statement in 'uses'.  With the "overlay" method (for annotations and
refinements), both the syntax and semantics for a particular statement
is different depending on context.  For example:

   uses my-grouping {
     container foo {
       default 17;
     }
   }

The 'container' statement in 'uses' does not mean the same thing
as it does outside 'uses' - outside it creates a new container, inside
it modifies an existing one.  And syntax-wise, the substatements that
are allowed in 'container' is different inside and outside the 'uses'.

With such an 'annotate' statement, everything can be put into a
normal module:

  module my-interface-annotations {
    
    import ietf-interface { ... }
    import tailf-extensions { ... }
   
    annotate /if:interfaces/if:interface {
       ...
    }
  }

/martin

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 26 Jun 2008 09:41:45 +0200

>         // add an important implementation detail

We had this at some point in SMIng but it got dropped in the published
version. Note that you can use the normal module framework with this
approach - no need for "annotation" to start the module. While more
verbose for large modules, I find the "annotate" statement approach
nicer because it allows me to cut&paste an annotation without loosing
the context what precisely is being annotated.

/js

-- 
Juergen Schoenwaelder           Jacobs University Bremen gGmbH
Phone: +49 421 200 3587         Campus Ring 1, 28759 Bremen, Germany
Fax:   +49 421 200 3103         <http://www.jacobs-university.de/>
_______________________________________________
netmod mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/netmod

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 26 Jun 2008 13:53:49 +0200

Hi Martin,

the simplest possible approach - and one that works pretty well for me
in the RELAX NG context - is to specify that statements with foreign
namespaces MAY be ignored and their effect is unspecified on the general
YANG level.

If this could be accepted, I would also vote for removing the extension
statement from YANG - it is actually quite weak, essentially telling the
parser that such a symbol exists and how to transform it to YIN. It
doesn't provide any syntactic constraints as to where the new statement
may appear, what are its substatements etc. Extensions could be handled
in the same way as annotations, and ignored by default.

I believe it is much easier (and, at the end of the day, necessary
anyway) to modify the generic parser to understand the annotations and
proprietary extensions, rather that try to invent smart extension
mechanisms for the generic parser.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Thu, 26 Jun 2008 05:11:01 -0700

>         // add an important implementation detail

What about nested groupings and typedefs?

There is no way to identify the name of a nested
typedef or grouping with Xpath because they are not in the same
naming scope.

>  Phil

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 26 Jun 2008 14:46:14 +0200 (CEST)

Hi,

> typedef or grouping with Xpath because they are not in the same

You are right.  Maybe they should be in the same naming scope.  Are
there any compelling reasons for having separate naming scopes?

/martin

---

From: Phil Shafer <[email-masked]>
Date: Thu, 26 Jun 2008 10:08:57 -0400

Ladislav Lhotka writes:
>parser that such a symbol exists and how to transform it to YIN.

It allows the parser to know when a statement is an extension
and when it's a typo.  If I have an extension "junos:foo" and
misspell the prefix, a compiler that understands "junos" won't
see that "junss:foo" is a mistake, but will ignore it.  We want
to be able to catch errors at compile time.

>mechanisms for the generic parser.

For proprietary YANG modules (and eventually for well-known
extensions), putting them inline with the YANG they are part of is
a big win.  Asking the reader to look in two places is something
we try to avoid (but is unavoidable when extending standards).

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Thu, 26 Jun 2008 08:01:58 -0700

> there any compelling reasons for having separate naming scopes?

I keep bringing this up:

typedef foo ...

grouping foo ...

leaf foo ...

extension foo ...

It doesn't really matter what nest level the typedef or grouping
is at -- /foo always refers to the accessible object (leaf foo).
Phil claims that it would be too restrictive on the vendor.
I keep claiming that it will useful to be able to have a
unique identifier, even for non-data like typedefs and groupings.

It is OK with me to reproduce the hierarchy rather than
use Xpath, and not change all the naming scope rules now.

Andy

---

From: Andy Bierman <[email-masked]>
Date: Thu, 26 Jun 2008 08:19:26 -0700

> use Xpath, and not change all the naming scope rules now.

I think it has to be done with an overlay tree.
There is absolutely no way to use Xpath to reference the sub-tree
under a 'uses' or 'augment' node.  (I brought up the issue of names
for uses already.)

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 26 Jun 2008 19:24:01 +0200

> to be able to catch errors at compile time.

Well, a generic parser would just ignore it - with the typo or without.
And any application that is able to interpret the extension should check
not only for typos but also for proper placement of the statement, its
substatements etc.

> we try to avoid (but is unavoidable when extending standards).

I am not sure that we talk about the same thing. I am advocating exactly
what you are saying - combine the annotations/extensions in-line with
the module text. My point just is that saying "ignore all statements in
other namespaces" provides a reasonable path for extensions but does not
introduce any new statements. Before coming up with YANG language
constructs, I'd like to see the cases where this simple approach fails.

Cheers, Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Thu, 26 Jun 2008 10:45:53 -0700

> not only for typos but also for proper placement of the statement, its

What exactly is a 'generic' parser for a language that is brand new?
I hope we don't follow the path of the 'generic MIB walker' in SNMP,
where bad decisions are made to maintain dumb tools.

The extension-stmt allows a parser to generate YIN correctly,
but mostly, it makes vendor hooks visible in the modules
and the language.  This allows extensions to be tracked
like other definitions, with versioning and schema advertisement.

> constructs, I'd like to see the cases where this simple approach fails.

I agree with Randy.
It seems better to keep the meta-cruft out of the data model.
This does not belong inside standard data models.
What if two different products or versions have different cruft
for a given DM definition?  I guess we would need to add CPP-style
#if, #else, #elif constructs to deal with that.  No thanks.
I prefer the simplest possible DML that we can use to support
standards-based content.

I have some concerns that since we have no architecture whatsoever
for standards-based multi-protocol network management, that we
have no common vision of the goals for YANG.  Clearly there is
some advantage in having a common language for all proprietary
NETCONF data models, but we sort-of already had that with XSD.

IMO, YANG and NETCONF are mostly worthless unless actual
standard content is defined and published in RFCs.  Therefore,
YANG's usefulness for standard data models and multi-vendor
interoperability should be the highest priorities.

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Thu, 26 Jun 2008 21:04:28 +0200

> not only for typos but also for proper placement of the statement, its

An application that is able to interpret an extension foo will also
have to skip over things it does not understand, including any typos
of loo^hfoo.

/js

---

From: Phil Shafer <[email-masked]>
Date: Thu, 26 Jun 2008 15:18:46 -0400

>Phil claims that it would be too restrictive on the vendor.

I don't think I claimed that.  My claim is that a rule that
we don't need is a CLR.  There's nothing in the langauge
that needs these namespaces to be common.  If we need
the namespaces to be common to allow some new feature
like annotations, it stops being a CLR.

Thanks,
 Phil

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 26 Jun 2008 21:28:43 +0200 (CEST)

Andy Bierman <[email-masked]> wrote:
> under a 'uses' or 'augment' node.

Right.  But do we really need to annotate the 'uses' and 'augment'
statements themselves?  Hmm... pretty weak argument, and it smells
like a CLR ("you can annotate all statements but two, b/c we didn't
think it was necessary").

/martin

---

From: Andy Bierman <[email-masked]>
Date: Thu, 26 Jun 2008 12:54:27 -0700

> think it was necessary").

Are you trying to convince yourself?

One use-case mentioned is attaching a bunch
of i18n translation strings (or constants)
to a description clause, and those can show up everywhere.
Nobody wants to read a data model draft full
of that sort of 'tool stuff' inline, and a 1:1 overlay
seems the most simple to manage.

IMO, a tree overlay makes the most sense because it is
complete, already defined, and doesn't throw out the existing
design in order to work.

Andy

---

From: Martin Bjorklund <[email-masked]>
Date: Thu, 26 Jun 2008 22:07:03 +0200 (CEST)

> Are you trying to convince yourself?

Yes.  I haven't succeeded yet though.

> of that sort of 'tool stuff' inline

With both proposals the DM writer can choose to do everything inline,
or in an extra annotation file.

> and a 1:1 overlay

The drawback is that with the overlay we'll have to say that some
statements can be used to add to the module (like the import in my
first example), but some others are only valid if the corresponding
statement exist in the original module (e.g. 'container'; you cannot
add new the data definition statements in an overlay).

/martin

---

From: Phil Shafer <[email-masked]>
Date: Thu, 26 Jun 2008 16:51:27 -0400

>add new the data definition statements in an overlay).

My hangup is this is really overloading the YANG statements, making
them mean one thing in annotations in one one thing in normal
modules.  While sort of conditional logic and overloading is confusing
to readers.  You can't say "a list statement needs to specify a
key", since a list statement in an annotation doesn't need to do
this.

This was a mistake I made in the JUNOS DDL.  Several statements
have meanings that are dependent on which parent statement they
appear under.  This has been a continuing source of irritation for
developers and I regret it.

A statement should have one meaning and that meaning should be
clear, consistent, and unchanging.

Thanks,
 Phil

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 27 Jun 2008 10:25:52 +0200

> where bad decisions are made to maintain dumb tools.

A generic parser is one whose behaviour will be defined in the YANG
standard. It is an exact analogy of RELAX NG.

> The extension-stmt allows a parser to generate YIN correctly,

As we have already discussed with Martin off the list, these YANG<->YIN
translation rules can be easily specified in a text file, so it is no
problem to extend such a file. And there could be a default translation
rule that converts the statement in an element with the same name and
attribute "arg" or something like that.

> like other definitions, with versioning and schema advertisement.

Applications can track the extensions if they want to, they just cannot
check that the name is correct. Or, they can extend YANG grammar and
deal handle the extension as any other YANG statement - the extension
statement falls short in this respect.

> I prefer the simplest possible DML that we can use to support

An application that wants to keep this cruft out of the DM can do so,
but I am not sure that the mechanism for doing this has to be specified
together with the core YANG.
 
> interoperability should be the highest priorities.

I agree with this, and it is IMO better to start with the simplest
possible language and add features only after there is enough evidence
that they are really needed, and perhaps how to do it right.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 27 Jun 2008 10:31:38 +0200

> have to skip over things it does not understand, including any typos

No, it will know that it is able to completely interpret vocabulary in
namespace X, and all used namespace will have to be defined in the
module, so it will recognize all typos and syntax errors within that
namespace.

In fact, the typo in Phil's example was in a namespace prefix, so this
should be caught even by a generic parser.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Phil Shafer <[email-masked]>
Date: Fri, 27 Jun 2008 10:05:13 -0400

Ladislav Lhotka writes:
>problem to extend such a file.

I'm having trouble seeing any of this as an improvement on what's
currently in the spec.  How is an external text file a win?  What's
the issue with the current extension mechanism?  What problem are
you trying to solve by removing it?

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Fri, 27 Jun 2008 07:42:29 -0700

> you trying to solve by removing it?

I agree with Phil on this one.
At first, I was really opposed to the extension-stmt,
mostly because I didn't want to encourage non-standard mechanisms.

But now I think they are kind of useful, and easy to parse.
For example, the smidump program adds extensions that are specialized for
NETCONF applications that want access to SMIv2 defined data.
This is not something that should be built into the YANG language,
and the extension-stmt handles it very well.

It is possible to design tools which allow the user to
attach functionality to the extensions.  At a minimum,
tools can gather the extensions used within each construct
and make them available within the manager or agent as meta-data.

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 27 Jun 2008 17:06:19 +0200

> currently in the spec.  How is an external text file a win?  What's

The spec could remain untouched wrt YANG<->YIN translation. The text
file would be used by pyang and other tools.

> you trying to solve by removing it?

Simplify YANG.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 27 Jun 2008 17:23:33 +0200

> mostly because I didn't want to encourage non-standard mechanisms.

The extension statement is lame. This is because YANG syntax is defined
by the ABNF grammar and this is where the extension should be formally
specified. The extension statement does it at a wrong level. It is as if
a statement in a programming language could change the grammar of the
language. This could lead to inconsistencies. For example: can I use the
newly defined statement as a substatement of the extension statement
that defines it, or before the extension statement?

> and the extension-stmt handles it very well.

What happens if smidump just adds the extensions without declaring them
via extension-stmt?
 
> and make them available within the manager or agent as meta-data.

Right, and every such program has to check that the use of the
extensions is syntactically correct, which the extension statement
doesn't.

Lada 

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Andy Bierman <[email-masked]>
Date: Fri, 27 Jun 2008 08:45:27 -0700

> that defines it, or before the extension statement?

IMO the extension-stmt is a good compromise between standards needs
and vendor needs.  It allows hooks to be added, new experimental
constructs to be tested, but it does not allow everyone to
create their own version of YANG.  That kind of flexibility
does not help interoperability.

> What happens if smidump just adds the extensions without declaring them

That's what happened at first, and yangdump generated errors
for every leaf :-(   Then Juergen added yang-smi.yang
with the extensions.  Yangdump checks the following:
   - extension is defined in the specified module
   - argument should match the 2nd field (present or not)

It is true that the compiler cannot validate that the proper
sub-fields (if any) are present in an extension usage,
based on the extension-stmt.  That's OK, because the
low-level parsing is the same as what standard YANG uses,
and if my tool doesn't understand the extension, it can still
save it as 'appinfo'.

> extensions is syntactically correct, which the extension statement

If my tool does understand the extension, then it can do
all the validation needed within the code.  If it doesn't
understand the extension, then there isn't much value
in validating the nested sub-clauses.  Standard YANG is
different.  I can't just ignore a range-stmt without
breaking something and being non-compliant.  Extensions
are optional.  Only an agent implementing/advertising
the module containing the extension-stmt is expected to support it.

Andy

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 27 Jun 2008 19:07:29 +0200

> that defines it, or before the extension statement?

The extension statement does not "define" an extension, it just
"declares" the existence of an extension.

In SMIng, we did embed ABNF in the extension statement but if was
clear that nobody would like to write an ABNF runtime interpreter - so
the ABNF was more meant as documentation. In addition, I note that
while ABNF is great for capturing some properties, it does not catch
everything you expect from to be catched by a compiler which
implements all the semantics of a statement.

In other words, I assume we agree that the _definition_ of a YANG
extension is out of scope for YANG itself. So the question left is
whether the _declaration_ of an extensions is useful to have or not.

I personally find it nice if generic tools (that do not necessarily
implement all possible extensions) can identify all undeclared
extensions used in a module; some of them might be simply typos.

/js

---

From: "Randy Presuhn" <[email-masked]>
Date: Fri, 27 Jun 2008 09:56:58 -0700

Hi -

> To: "Ladislav Lhotka" <[email-masked]>
> Cc: <[email-masked]>
> Sent: Friday, June 27, 2008 7:05 AM
...
> How is an external text file a win?
...

If the annotations go beyond the externally visible "adjustments"
to the management interface, the ability to put stuff in an external
text file is a huge win from a source-code configuration management
perspective.  Been there, done that with MIB compilers.

I think there are two categories of stuff here.  Applying the "adjustments"
to the management interface defined by the file should effectively result
in a new interface definition, which would be darn convenient to have
available as a single file, possibly an output of the processing.
Applying the "elaborations" (annotations having to do with the guts
of the implementation, such as data structure or callback bindings)
would obviously be very specific to a particular implementation environment,
and out of our scope here.

Randy

---

From: Phil Shafer <[email-masked]>
Date: Fri, 27 Jun 2008 14:51:26 -0400

Ladislav Lhotka writes:
>language. This could lead to inconsistencies.

The ABNF defines the grammar, which allows for extensions.  The
extension statement declares the extension.  Using undeclared
extensions is an error.

This is similar to defining structs in C.  The C language defines
how they are used, but you need to declare them before using them.

Thanks,
 Phil

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 27 Jun 2008 22:05:40 +0200

> whether the _declaration_ of an extensions is useful to have or not.

Perhaps a declaration of a namespace and prefix used for the extensions
could be sufficient. And of course, the extension statements must follow
the general structure of YANG statements.

> extensions used in a module; some of them might be simply typos.

If I can't do anything with the statement, then it is pretty much
irrelevant whether it contains typos or not.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Juergen Schoenwaelder <[email-masked]>
Date: Fri, 27 Jun 2008 22:13:38 +0200

> irrelevant whether it contains typos or not.

We don't have to agree here. You seem to assume that every module
writer uses the same tools as a model implementor, which is not
generally true, at least not in the SMI space.

/js

---

From: Ladislav Lhotka <[email-masked]>
Date: Fri, 27 Jun 2008 22:20:35 +0200

> how they are used, but you need to declare them before using them.

Actually C structures always have to be defined, too. C structures are
comparable to YANG groupings. On the other hand, an extension statement
declares a new statement and there is no analogy for it in C or any
other language I am aware of.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

---

From: Phil Shafer <[email-masked]>
Date: Fri, 27 Jun 2008 17:30:27 -0400

>other language I am aware of.

XML ;^)

But seriously, the ability to define values for language tokens is
universal in languages.  In YANG, we just allow the definition of
the token that appears as a peer to the built-in YANG statements.
Allowing random stuff that isn't defined anywhere would make for a
bad language.  (XML? ;^)

(Sorry, couldn't resist)

Thanks,
 Phil