Subject: [core] YANG list to CBOR mapping

From: peter van der Stok <[email-masked]>
Date: Thu, 19 Nov 2015 11:05:51 +0100

Hi CoOL authors,

This e_mail complements my earlier e-mail on YANG to CBOR mapping, and 
explains the proposed list encoding.

Both the draft yang-json and the CoOL draft encode a list as an array. 
In the yang-json case as a JSON array, and in the CoOL case as major 
type 4 of major type 5 objects.
This encoding prevents the selection of list instances on the basis of 
the key values from the payload alone. In CoMI the list is encoded as a 
major type 5 of major type 5 objects. This deviation from the yang-json 
mapping is motivated by the wish to transport individual list instances 
and identify the instances in the payload from the key values. This is 
especially important in case of PATCH operation on list instances.
Two cases are to be looked at: (1) the list definition specifies one or 
more key elements, and (2) the list definition does not specify a key 
element. We will start with a list containing key elements.
It is the wish to transport only those list elements with a unique key 
value, either to support a PATCH or a FETCH command. Therefore it is 
required that the payload allows the selection and identification of 
these list element on the basis of their key values. Using the 
diagnostic JSON notation used for CBOR diagnostics, the payload 
containing a number of elements with key values should look like a JSON 
object with the structure list-name: {key: value}. Both key and value 
are composed of JSON objects separated by commas. For example consider 
the following YANG list specification:
module foo {
     namespace http://example.com/book;
     prefix “fo”;
     revision  2015-06-07;

    list B {
       key “key1 key2”;
       leaf key1 {type string;}
       leaf key2 {type string;}
       leaf coll {type int32;}
       leaf counter {type int32;}
    }
}

Consider the transport of two instances specified by <“key1” = “book1”, 
“key2” = “author2”> and by <“key1” = “book5”, “key2” = “author3”> out of 
a larger set of instances. The representation is different from the 
yang-json one for the reasons explained above.

The payload will look like:
{"B" :{
     {"key1" : "book1", "key2" : "author2"} :
     {"coll" : 500, "counter": 66} ,
     {"key1" : "book5", "key2" : "author3"} :
     {"coll" : 444, "counter": 11}
     }
}

Which translated to CBOR yields:

a1                         # map(1)
    61                      # text(1)
       42                   # "B"
    a2                      # map(2)
       a2                   # map(2)
          64                # text(4)
             6b657931       # "key1"
          65                # text(5)
             626f6f6b31     # "book1"
          64                # text(4)
             6b657932       # "key2"
          67                # text(7)
             617574686f7232 # "author2"
       a2                   # map(2)
          64                # text(4)
             636f6c6c       # "coll"
          19 01f4           # unsigned(500)
          67                # text(7)
             636f756e746572 # "counter"
          18 42             # unsigned(66)
       a2                   # map(2)
          64                # text(4)
             6b657931       # "key1"
          65                # text(5)
             626f6f6b35     # "book5"
          64                # text(4)
             6b657932       # "key2"
          67                # text(7)
             617574686f7233 # "author3"
       a2                   # map(2)
          64                # text(4)
             636f6c6c       # "coll"
          19 01bc           # unsigned(444)
          67                # text(7)
             636f756e746572 # "counter"
          0b                # unsigned(11)

The corresponding yang-json draft encoding - that allows transport of a 
selection of instances- does not specify the key fields and would have 
looked like:

{"B" : [
     {"key1" : "book1",
       "key2" : "author2”,
        "coll" : 500,
        "counter": 66
      } ,
     {"key1" : "book5",
       "key2" : "author3",
       "coll" : 444,
       "counter": 11,
     }
   ]
}

Suppose there are no key elements specified. In that case it is 
impossible to specify the wanted list elements, and always the whole 
list must be transported. In the case of a list without key 
specification is is best to use major type 4 composed of major type 5 to 
transport the list, as is specified in yang-json draft.

Consider the following YANG list

Module foo{
     Namespace http://example.com/book;
     Prefix “bo”
     Revision  2015-06-7
List B {
    Leaf title {type string;}
    Leaf author {type string;}
    Leaf coll {type int32}
    Leaf counter {type int32}
    }
}

Suppose the list contains two elements. The transport payload of the two 
elements looks like:

{"B" :[
     {"title" : "book1", "author" : "author2", "coll" : 500, "counter": 
66} ,
     {"title" : "book5", "author" : "author3",  “coll" : 444, "counter": 
11}
     ]
}

This payload is identical to the yang-json payload.
Which yields the corresponding CBOR code.

a1                         # map(1)
    61                      # text(1)
       42                   # "B"
    82                      # array(2)
       a4                   # map(4)
          65                # text(5)
             7469746c65     # "title"
          65                # text(5)
             626f6f6b31     # "book1"
          66                # text(6)
             617574686f72   # "author"
          67                # text(7)
             617574686f7232 # "author2"
          64                # text(4)
             636f6c6c       # "coll"
          19 01f4           # unsigned(500)
          67                # text(7)
             636f756e746572 # "counter"
          18 42             # unsigned(66)
       a4                   # map(4)
          65                # text(5)
             7469746c65     # "title"
          65                # text(5)
             626f6f6b35     # "book5"
          66                # text(6)
             617574686f72   # "author"
          67                # text(7)
             617574686f7233 # "author3"
          64                # text(4)
             636f6c6c       # "coll"
          19 01bc           # unsigned(444)
          67                # text(7)
             636f756e746572 # "counter"
          0b                # unsigned(11)

-- 
Peter van der Stok
vanderstok consultancy
mailto: [email-masked]

---

From: Michel Veillette <[email-masked]>
Date: Thu, 19 Nov 2015 13:40:35 +0000

Hi Peter

The list example show why CoOL propose a direct mapping (YANG -> CBOR) instead of a two steps mapping (YANG -> JSON -> CBOR) as proposed by CoMI.

Two aspects of your proposal for lists are invalid in JSON.
- The use of an integer as object name
- The use of an object as object value
See http://jsonlint.com/ for testing compliance.

Your proposed solution effectively seem to make sense but this solution is possible only in the context of a direct mapping (YANG -> CBOR)
CoOL didn't "ignore the yang-json draft", it use it as a template.

The two steps mapping (YANG -> JSON -> CBOR) do not allow to take full advantage of the specificities of CBOR.
This is true for the list, true for the bits datatype and probably true for other datatypes like  enumeration , identityref, instance-identifier, leafref
The direct mapping will also allow the use of the currently discussed deterministic map for container.

I hope this provide more background about why CoOL propose a direct mapping and the necessity of this approach.

About your proposal to submit this work the netmod wg, I personally prefer to perform all this work (the 4 or 6 drafts) under the core group.
The CORE group is where the CBOR expertise is which is critical for this draft. 

Regards,

Michel Veillette
System Architecture Director
Trilliant Inc.
Tel: 450-375-0556 ext. 237
[email-masked]
www.trilliantinc.com 

-----Original Message-----
From: core [mailto:[email-masked]] On Behalf Of peter van der Stok
Sent: November-19-15 5:06 AM
To: Core <[email-masked]>
Cc: [email-masked]
Subject: [core] YANG list to CBOR mapping

Hi CoOL authors,

This e_mail complements my earlier e-mail on YANG to CBOR mapping, and explains the proposed list encoding.

Both the draft yang-json and the CoOL draft encode a list as an array. 
In the yang-json case as a JSON array, and in the CoOL case as major type 4 of major type 5 objects.
This encoding prevents the selection of list instances on the basis of the key values from the payload alone. In CoMI the list is encoded as a major type 5 of major type 5 objects. This deviation from the yang-json mapping is motivated by the wish to transport individual list instances and identify the instances in the payload from the key values. This is especially important in case of PATCH operation on list instances.
Two cases are to be looked at: (1) the list definition specifies one or more key elements, and (2) the list definition does not specify a key element. We will start with a list containing key elements.
It is the wish to transport only those list elements with a unique key value, either to support a PATCH or a FETCH command. Therefore it is required that the payload allows the selection and identification of these list element on the basis of their key values. Using the diagnostic JSON notation used for CBOR diagnostics, the payload containing a number of elements with key values should look like a JSON object with the structure list-name: {key: value}. Both key and value are composed of JSON objects separated by commas. For example consider the following YANG list specification:
module foo {
     namespace http://example.com/book;
     prefix “fo”;
     revision  2015-06-07;

    list B {
       key “key1 key2”;
       leaf key1 {type string;}
       leaf key2 {type string;}
       leaf coll {type int32;}

Consider the transport of two instances specified by <“key1” = “book1”, “key2” = “author2”> and by <“key1” = “book5”, “key2” = “author3”> out of a larger set of instances. The representation is different from the yang-json one for the reasons explained above.

a1                         # map(1)
    61                      # text(1)
       42                   # "B"
    a2                      # map(2)
       a2                   # map(2)
          64                # text(4)
             6b657931       # "key1"
          65                # text(5)
             626f6f6b31     # "book1"
          64                # text(4)
             6b657932       # "key2"
          67                # text(7)

The corresponding yang-json draft encoding - that allows transport of a selection of instances- does not specify the key fields and would have looked like:

{"B" : [
     {"key1" : "book1",
       "key2" : "author2”,
        "coll" : 500,
        "counter": 66
      } ,
     {"key1" : "book5",
       "key2" : "author3",
       "coll" : 444,
       "counter": 11,
     }
   ]
}

Suppose there are no key elements specified. In that case it is impossible to specify the wanted list elements, and always the whole list must be transported. In the case of a list without key specification is is best to use major type 4 composed of major type 5 to transport the list, as is specified in yang-json draft.

Module foo{

Suppose the list contains two elements. The transport payload of the two elements looks like:

{"B" :[

a1                         # map(1)
    61                      # text(1)
       42                   # "B"
    82                      # array(2)
       a4                   # map(4)
          65                # text(5)
             7469746c65     # "title"
          65                # text(5)
             626f6f6b31     # "book1"
          66                # text(6)
             617574686f72   # "author"
          67                # text(7)

--
Peter van der Stok
vanderstok consultancy
mailto: mailto:[email-masked]

_______________________________________________
core mailing list
mailto:[email-masked]
https://www.ietf.org/mailman/listinfo/core

---

From: Michel Veillette <[email-masked]>
Date: Fri, 20 Nov 2015 20:41:55 +0000

Reference: YANG list to CBOR mapping https://core-wg.github.io/yang-cbor/#rfc.section.3.1.17

Hi Peter

Your proposed solution for lists have some drawbacks and I'll like to understand better its advantages to evaluate the PROS and CONS. 

List of CONS
- Your proposed solution is not aligned with draft-ietf-netmod-yang-json-06
- Your proposed solution take one extra byte per list entry
- Your proposed solution use a map as key for a map, which is uncommon (Illegal in JSON but not in CBOR)
- The encoding of list defined with keys or without keys are done differently

CoMI-CoOL clients and servers known which data nodes of a list are keys and which ones are associated information.
I don’t the see value of implementing the keys on the left side of the ':' (key) and the rest of the data nodes on the right side of the ':" (value).
This semantic is already known by both ends, why we need to add it to the encoding?

=== Examples

Map of map solution with CoMI IDs (88 Bytes)
  {
    11111111 : {
      {22222222 : "book1", 33333333 : "author2"} : {44444444 : 500, 55555555: 66} ,
      {22222222 : "book5", 33333333 : "author3"} : {44444444 : 444, 55555555: 11}
    }
  }

Array of map solution with CoMI IDs (86 Bytes)
  {
    11111111 : [
      {
        22222222 : "book1",
        33333333 : "author2",
        44444444 : 500,
        55555555: 66
      } ,
      {
        22222222 : "book5",
        33333333 : "author3",
        44444444 : 444,
        55555555: 11
      }
    ]
  }

Map of map solution with CoOL IDs (56 Bytes)

    {
      11111111 : {
        {2 : "book1", 3 : "author2"} : {4 : 500, 5: 66} ,
        {2 : "book5", 3 : "author3"} : {4 : 444, 5: 11}
      }
    }

Array of map solution with CoOL IDs (54 Bytes)

    {
      11111111 : [
        {
          2 : "book1",
          3 : "author2",
          4 : 500,
          5: 66
        } ,
        {
          2 : "book5",
          3 : "author3",
          4 : 444,
          5: 11
        }
      ]
    }

You say "It is the wish to transport only those list elements with a unique key value, either to support a PATCH or a FETCH command".
I don’t see why this is not the case for both representations.

Michel Veillette

-----Original Message-----
From: core [mailto:[email-masked]] On Behalf Of peter van der Stok
Sent: November-19-15 5:06 AM
To: Core <[email-masked]>
Cc: [email-masked]
Subject: [core] YANG list to CBOR mapping

Hi CoOL authors,

    list B {
       key “key1 key2”;
       leaf key1 {type string;}
       leaf key2 {type string;}
       leaf coll {type int32;}

a1                         # map(1)
    61                      # text(1)
       42                   # "B"
    a2                      # map(2)
       a2                   # map(2)
          64                # text(4)
             6b657931       # "key1"
          65                # text(5)
             626f6f6b31     # "book1"
          64                # text(4)
             6b657932       # "key2"
          67                # text(7)

{"B" : [
     {"key1" : "book1",
       "key2" : "author2”,
        "coll" : 500,
        "counter": 66
      } ,
     {"key1" : "book5",
       "key2" : "author3",
       "coll" : 444,
       "counter": 11,
     }
   ]
}

Module foo{

{"B" :[

a1                         # map(1)
    61                      # text(1)
       42                   # "B"
    82                      # array(2)
       a4                   # map(4)
          65                # text(5)
             7469746c65     # "title"
          65                # text(5)
             626f6f6b31     # "book1"
          66                # text(6)
             617574686f72   # "author"
          67                # text(7)

[email-masked]

---

From: peter van der Stok <[email-masked]>
Date: Mon, 23 Nov 2015 09:58:01 +0100

Hi Michel,

the point is to express in the payload the replacement criteria 
independent of application.
It is a CBOR extension to the patch format of RFC 7396.
The latter allows the replacement of complete arrays, not a subset of 
the array.

Peter

Michel Veillette schreef op 2015-11-20 21:41:
> Reference: YANG list to CBOR mapping
> https://core-wg.github.io/yang-cbor/#rfc.section.3.1.17
> 
> Hi Peter
> 
> Your proposed solution for lists have some drawbacks and I'll like to
> understand better its advantages to evaluate the PROS and CONS.
> 
> List of CONS
> CoMI-CoOL clients and servers known which data nodes of a list are
> keys and which ones are associated information.
> I don’t the see value of implementing the keys on the left side of the
> ':' (key) and the rest of the data nodes on the right side of the ':"
> (value).
> You say "It is the wish to transport only those list elements with a
> unique key value, either to support a PATCH or a FETCH command".
> the key values from the payload alone. In CoMI the list is encoded as
> a major type 5 of major type 5 objects. This deviation from the
> yang-json mapping is motivated by the wish to transport individual
> list instances and identify the instances in the payload from the key
> values. This is especially important in case of PATCH operation on
> list instances.
> Two cases are to be looked at: (1) the list definition specifies one
> or more key elements, and (2) the list definition does not specify a
> key element. We will start with a list containing key elements.
> containing a number of elements with key values should look like a
> JSON object with the structure list-name: {key: value}. Both key and
> value are composed of JSON objects separated by commas. For example
> consider the following YANG list specification:
> module foo {
> Consider the transport of two instances specified by <“key1” =
> “book1”, “key2” = “author2”> and by <“key1” = “book5”, “key2” =
> “author3”> out of a larger set of instances. The representation is
> different from the yang-json one for the reasons explained above.
> 
> The corresponding yang-json draft encoding - that allows transport of
> a selection of instances- does not specify the key fields and would
> have looked like:
> 
> {"B" : [
>      {"key1" : "book1",
>        "key2" : "author2”,
>         "coll" : 500,
>         "counter": 66
>       } ,
>      {"key1" : "book5",
>        "key2" : "author3",
>        "coll" : 444,
>        "counter": 11,
>      }
>    ]
> }
> 
> specification is is best to use major type 4 composed of major type 5
> to transport the list, as is specified in yang-json draft.
>

---

From: Michel Veillette <[email-masked]>
Date: Mon, 23 Nov 2015 16:16:24 +0000

Hi Peter

Have you defined an MergePatch() function similar to RFC 7396 section 2 for your proposed extension?
Do we intent to support two PATCH approaches, RFC 7396 style and draft-ietf-netconf-yang-patch style?
IF so, why list need to be supported by the RFC 7396 style since this approach provide a partial coverage (list without keys and user ordered list is not fully supported) ? Should we address list only in the draft-ietf-netconf-yang-patch style?

Michel Veillette

-----Original Message-----
From: peter van der Stok [mailto:[email-masked]] 
Sent: November-23-15 3:58 AM
To: Michel Veillette <[email-masked]>
Subject: RE: [core] YANG list to CBOR mapping

Hi Michel,

the point is to express in the payload the replacement criteria independent of application.
It is a CBOR extension to the patch format of RFC 7396.
The latter allows the replacement of complete arrays, not a subset of the array.

Peter

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 23 Nov 2015 17:50:38 +0100

> Have you defined an MergePatch() function similar to RFC 7396 section 2 for your proposed extension?

AFAIK, the function defined by RFC 7396 should "just work".

> Do we intent to support two PATCH approaches, RFC 7396 style and draft-ietf-netconf-yang-patch style?

Good question.  yang-patch seems a bit more complicated than is needed
for many constrained applications.  It also is not idempotent.

> IF so, why list need to be supported by the RFC 7396 style since this approach provide a partial coverage (list without keys and user ordered list is not fully supported) ? Should we address list o…

I would expect many constrained applications to make use of maps, which
typically are represented in YANG as lists with keys*).  So I would
prefer to be able to address these with a simple, merge-patch style
update mechanism.

Grüße, Carsten

*) leaf-lists are a special case where updating isn't quite addressed
yet by CoMI as far as I know.

---

From: peter van der Stok <[email-masked]>
Date: Tue, 24 Nov 2015 12:23:23 +0100

> AFAIK, the function defined by RFC 7396 should "just work".
<pvds>
"just" is the word, for example CBOR allows more identifier types.
I would really like to encourage Carsten to write a CBOR-merge draft.
It looks useful in more and more applications.
</pvds>
> for many constrained applications.  It also is not idempotent.
<pvds>
Their use is also very different.
The one is part of the transport format, the other is embedded in it.
</pvds>
> 
>> IF so, why list need to be supported by the RFC 7396 style since this 
>> approach provide a partial coverage (list without keys and user 
>> ordered list is not fully supported) ? Should we address list only in 
>> the draft-ietf-netconf-yang-patch style?
<pvds>
IMO user-ordered is orthogonal to keyed modifications.
Why punish keyed lists because non-keyed lists do not provide the 
required handles?
</pvds>
> prefer to be able to address these with a simple, merge-patch style
</pvds> so do I</pvds>
> yet by CoMI as far as I know.
<pvds> They represent the same characteristics as a JSON array.
In json-merge you change all array members or none.
Otherwise take refuge to the verbose RFC 6902 approach.
</pvds>