Subject: [netmod] architecture draft - DSDL

From: Ladislav Lhotka <[email-masked]>
Date: Wed, 15 Oct 2008 19:45:32 +0200

Hi,

as a follow-up to our discussion at the interim, I propose the following
text for inclusion in the YANG architecture draft, perhaps as a separate
section. Comments and suggestions are certainly welcome.

Lada

========================================================================
   The hierarchical structure, datatypes of nodes and semantics
   specified in a YANG module have their counterparts in similar (but
   not identical) structural, datatype and semantical constraints on
   NETCONF protocol data units (PDU) that are exchanged within a
   NETCONF session.

   In [YANG], the implications of YANG constructs and constraints for
   NETCONF PDUs are explained in plain English and by giving numerous
   examples. However, as the PDUs are supposed to be the only means of
   communication between the agent and manager, the ability to
   formally validate the entire contents of NETCONF PDUs is crucial
   for verifying that the contract expressed by the YANG module is
   fulfilled.

   Since NETCONF PDUs are encoded in XML, it is natural to use XML
   Schema Languages for their validation. To facilitate this, YANG
   offers a standardized mapping of YANG modules Document Schema
   Description Languages (DSDL) [DSDL] that are being developed as the
   set of international standards ISO/IEC 19757.

   DSDL are considered to be the best choice for the given purpose
   because they address not only grammar and datatypes of XML
   documents but also semantic constraints and rules for modifying
   information set of the document, such as application of default
   values.

   In addition, DSDL offers formal means for coordinating multiple
   independent schemas and specifying how to apply the schemas to the
   various parts of the document. This is useful since NETCONF PDUs
   are typically composed of multiple vocabularies. For example, an
   edit-config request consists of several nested elements specified
   in [RFC4741] (<rpc>, <edit-config> and <config>) that encapsulate
   configuration data and, moreover, some configuration data elements
   may carry "operation" attributes that are again defined in
   [RFC4741]. Rather than merging the schema of [RFC4741] with that of
   a particular data model into a new monolithic schema,
   Namespace-based Validation Dispatching Language (NVDL, part 4 of
   DSDL) allows for setting up a validation procedure that applies
   specific schemas to various levels and sections of the XML
   document.

   While it is true that some parts of the DSDL set need further
   development before they can be used in applications, the standard
   mapping from YANG to DSDL uses only those DSDL parts that already
   have the status of an ISO/IEC International Standard or are at the
   stage of a FCD (Final Committee Draft):
   - Part 2: Regular-grammar-based validation - RELAX NG
     (ISO/IEC 19757-2:2003)
   - Part 3: Rule-based validation - Schematron (ISO/IEC 19757-3:2006)
   - Part 4: Namespace-based Validation Dispatching Language - NVDL
     (ISO/IEC 19757-4:2006)
   - Part 8: Document schema renaming language - DSRL
     (ISO/IEC 19757-8 FCD)

   For several details of YANG semantics that cannot be adequately
   represented in the four DSDL schema languages mentioned above, the
   mapping introduces a small set of new XML attributes that annotate
   appropriate elements in the RELAX NG schema. In addition, metadata
   contained in YANG modules such as contact person, organization and
   revision date are mapped to analogical Dublin Core metadata terms.

   The mapping of YANG modules to DSDL schemas is further complicated
   by the inherent variability of NETCONF PDUs. A number of NETCONF
   RPC methods are defined in [RFC4741] and other RPC methods may be
   defined by data models. Yet another type of PDUs are asynchronous
   notifications. Contents of each of these PDU types is constrained
   in a specific way and in some cases other constraints may follow
   from the context - for example, contents of a get-config reply
   depends on filters that may be present in the corresponding
   get-config request, on available features etc.

   Last but not least, in order to get a complete schema for a
   particular NETCONF session, one has to take into account all YANG
   modules that have been negotiated as capabilities in the hello
   message. These modules may also be interrelated via the augment
   statement.

   In order to cope with this complexity, the mapping procedure is
   divided into two phases illustrated in the following figure:

   1. In the first phase (transformation T in Fig. 1), one or more YANG
      modules are mapped into a schema for the conceptual tree that
      encompasses the entire contents as specified by the YANG
      modules: tree of configuration and non-configuration data, input
      and output parts of RPC messages and notifications.

   2. In the second phase, the schema for the conceptual tree is
      transformed to "production" schemas for different NETCONF PDU
      types that may also take into account other contextual
      information such as get-config filters or available features.
      This is accomplished by an array of specialized transformations
      (T_g, T_e, T_r and T_n in the figure) that will be typically
      implemented as XSLT stylesheets.

              +----------------+
              | YANG module(s) |
              +----------------+
                      |
                      | T
                      |
             +-----------------+
             | DSDL schema for |
             | conceptual tree |
             +-----------------+
              /    |   |     \
          T_g/  T_e|   |T_r   \T_n
            /      |   |       \
      +----------+ | +---+  +------------+
      |get-config| | |rpc|  |notification|
      +----------+ | +---+  +------------+
                   |
             +-----------+
             |edit-config|
             +-----------+

   The conceptual tree can be schematically represented as follows:

   <nmt:netmod-tree yang-module="foo"
       xmlns:nmt="urn:ietf:params:xml:ns:netmod:tree:1">
     <nmt:data>
       ... configuration and status data ...
     </nmt:data>
     <nmt:rpcs>
       <nmt:rpc>
         <nmt:method>...</nmt:method>
         <nmt:input>
           ...
         </nmt:input>
         <nmt:output>
           ...
         </nmt:output>
       </nmt:rpc>
       ...
     </nmt:rpcs>
     <nmt:notifications>
       <nmt:notification>
         <nmt:name>...</nmt:name>
         ...
         <nmt:name>...</nmt:name>
       </nmt:notification>
       ...
     </nmt:notifications>
   </nmt:netmod>

   The first transformation phase (T), described in [ref], maps the
   logic expressed in YANG modules to the slightly different logic of
   DSDL schema languages. In contrast, the second phase
   transformations implement mainly NETCONF rules specified in
   [RFC4741] and other documents. They will be described in separate
   documents.

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C

_______________________________________________
netmod mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/netmod

---

From: Phil Shafer <[email-masked]>
Date: Wed, 15 Oct 2008 17:06:53 -0400

>section. Comments and suggestions are certainly welcome.

Three points of contention:

(a) The implication that DSDL is needed for the YANG spec is
incorrect.  The YANG spec needs to stand on its own, even if the
constraints are in "plain English".  The spec must articulate these
contraints in a "stand alone" manner.

(b) The claim that DSDL is needed to validate YANG PDUs is incorrect.
When YANG-based validation software is available, the need to
generate DSDL will disappear.  You only need DSDL files if you use
DSDL for validation;  if you can validate directly based on the
YANG modules, you don't need them.

(c) The concentration on PDUs is unfortunate.  As we discussed at
the interim, there are three places where validation is needed.

   (1) rpc input (did you send me the right stuff?)
   (2) rpc output (did I send you the right stuff?)
   (3) database contents (do I have the right stuff?)

PDUs cover (1) and (2), but not (3).  In selling us on DSDL, Rohan
implied that the same schema file could support all three scenarios.

>   information set of the document, such as application of default

Does XSD not do default values?

>   - Part 8: Document schema renaming language - DSRL

I thought we only needed 2 and 3.  What are 4 and 8 and why
do we need them?  Are implementations available?

Looks like part 8 went out for review last December.  dsdl.org
lists it as a draft, but their copy of the draft is older than
the one sent out in December.  Do you know the current status?

Are we comfortable with our depth of knowledge in these technologies?

The rest of this material is not appropriate for an architecture
document.  We need only the high level view.  I'd skip the earlier
"best choice" part, since we need only say what we are doing.

Thanks,
 Phil

---

From: Andy Bierman <[email-masked]>
Date: Wed, 15 Oct 2008 14:52:51 -0700

> implied that the same schema file could support all three scenarios.

4 scenarios:
    (4) notification content

I agree 100% that DSDL based validation is just one implementation
choice, and validation is just one task among many that
can be performed on a YANG file.

A NETCONF manager or agent implementation is not required
to use any particular DML, or any at all.  Maybe the
DM content support is hard-wired, and there are no YANG or DSDL
files anywhere near the implementation.

If the agent supports the NETCONF operations on the
particular data model content associated with module 'foo',
then the manager should not care how the agent
figured out how to send the correct <rpc-reply> responses.

Isn't the DSDL mapping supposed to be an Informational RFC?

Andy

---

From: Ladislav Lhotka <[email-masked]>
Date: Thu, 16 Oct 2008 09:57:55 +0200

> contraints in a "stand alone" manner.

I agree, but still I claim it's important to be able to do formal
validation. The spec also explains YANG syntax in plain English but any
implementor will peruse the ABNF quite often, too. The problem with
YANG->PDU mapping is that there is no formal language for writing down
the mapping rules as ABNF does for context-free grammars, so having it
in the same spec isn't really an option.

> (b) The claim that DSDL is needed to validate YANG PDUs is incorrect.

Where did I say that? I only said this: "Since NETCONF PDUs are encoded
in XML, it is natural to use XML Schema Languages for their
validation." (Actually, "schema languages" shouldn't be capitalized
here).

> When YANG-based validation software is available, the need to

AFAIK, there is no such thing in sight, and I think it would be waste of
time. The existing schema languages and validating software are good
enough for this task.

> implied that the same schema file could support all three scenarios.

The schemas that result from the second phase of the mapping may have a
different structure - the figure is just an example. One option is to
have all client PDUs in one schema (except hello) and then all server
rpc-reply messages in another and notifications in yet another. Any
ideas?

However, I do have problems with "database contents validation" since it
is not clear what a configuration database is - what its information
model is etc.

I am also not saying that PDU validation covers everything, just that it
is an important step in the validation process.

> Does XSD not do default values?

It sure does.

> do we need them?  Are implementations available?

Part 4 will only appear in phase 2 schemas. I am aware of two
implementations:
- http://www.oxygenxml.com/onvdl.html
- http://jnvdl.sourceforge.net/

Part 8 is used only for specifying default values. There is an
XSLT-based implementation, but in fact implementing insertion of
defaults when they are missing is trivial.

> the one sent out in December.  Do you know the current status?

The second draft that you mention has been approved five weeks ago:
http://broadcast.oreilly.com/2008/09/dsrl-a-new-standard-that-can-r.html

> Are we comfortable with our depth of knowledge in these technologies?

As I said, just dealing with defaults is no rocket science.

> "best choice" part, since we need only say what we are doing.

Sure, select what you like, the rest will appear in the DSDL draft.

Lada

-- 
Ladislav Lhotka, CESNET
PGP Key ID: E74E8C0C