Subject: [Cbor] Application specific tagging

From: Jim Schaad <[email-masked]>
Date: Tue, 11 Jul 2017 12:04:45 -0700

I have not really thought this through, and I don't know that it can easily
be implemented in such a way as to make it small.  

It might be nice to define the ability for an application specific tagging
mechanism to be defined.  While this to some extent can be thought of as
duplicating the functionality of maps, when one wants to do a more canonical
encoding it will produce smaller data than having a key for every field.

Consider the following

Array_item = [
   version: int,
   field1 : int,
   ? field2 : int,
   field 3 : int,
   ? field4 : int
]

As written, while the syntax is valid one cannot decide for [ 9, 10, 11, 12]
which of the fields are present and which are absent.  ASN.1 solves this
problem by the ability to do tagging so that one can write

Array_item = [
   version: int,
   field1 : int,
   ? [1] field2 : int,
   field 3 : int,
   ? [2] field4 : int
]

And now it is easy to identify which field is present.   

However this leads to problems in trying to determine how to actually encode
in CBOR.  While the nice thing would be to do something like  [9, 10, 11,
2(12)] so that you have a a tag which  has arbitrary cbor as a child.  This
is not currently something that can be written.  The best I can come up with
at the moment would be [9, 10, 11, X([2, 12])] but I would like to not have
the array.

Note that this might be helpful for the SEML people as I seem to remember an
argument where application tagging was needed.

Jim

---

From: Laurence Lundblade <[email-masked]>
Date: Tue, 11 Jul 2017 19:17:08 +0000

Maps with small integer keys are very cheap in CBOR. You can’t express this in JSON easily. Would be hesitant to add the complexity without a compelling example.

LL

> On Jul 11, 2017, at 12:04 PM, Jim Schaad <[email-masked]> wrote:
> 
> _______________________________________________
> CBOR mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/cbor

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 11 Jul 2017 21:31:32 +0200

>   ? [2] field4 : int

I’d solve this as

Array_item = [
  version: int,
  field1 : int,
  field2 : int / nil,
  field3 : int,
  ? field4 : int
]

This is more compact than any tagging scheme I can come up with.

(It would not be hard to reserve a number of CBOR tags for application-specific tagging.
I’ll actually come up with a rather different application for this very soon;
those who were in the W3C WoT meeting today have gotten a glimpse.)

Grüße, Carsten

---

From: Jim Schaad <[email-masked]>
Date: Wed, 12 Jul 2017 10:52:09 +0200

That is true for relatively small maps.  If you get a large enough map then things start going down hill.

Also, part of my worry was the question of getting a canonicalization.  I find this to be difficult with maps but not with arrays.

Jim

-----Original Message-----
From: Laurence Lundblade [mailto:[email-masked]] 
Sent: Tuesday, July 11, 2017 9:17 PM
To: Jim Schaad <[email-masked]>
Cc: [email-masked]
Subject: Re: [Cbor] Application specific tagging

LL

> I have not really thought this through, and I don't know that it can 
> easily be implemented in such a way as to make it small.
> 
> It might be nice to define the ability for an application specific 
> tagging mechanism to be defined.  While this to some extent can be 
> thought of as duplicating the functionality of maps, when one wants to 
> do a more canonical encoding it will produce smaller data than having a key for every field.
> 
> Consider the following
> 
> Array_item = [
>   version: int,
>   field1 : int,
>   ? field2 : int,
>   field 3 : int,
>   ? field4 : int
> ]
> 
> As written, while the syntax is valid one cannot decide for [ 9, 10, 
> 11, 12] which of the fields are present and which are absent.  ASN.1 
> solves this problem by the ability to do tagging so that one can write
> 
> Array_item = [
>   version: int,
>   field1 : int,
>   ? [1] field2 : int,
>   field 3 : int,
>   ? [2] field4 : int
> ]
> 
> However this leads to problems in trying to determine how to actually 
> encode in CBOR.  While the nice thing would be to do something like  
> [9, 10, 11, 2(12)] so that you have a a tag which  has arbitrary cbor 
> as a child.  This is not currently something that can be written.  The 
> best I can come up with at the moment would be [9, 10, 11, X([2, 12])] 
> but I would like to not have the array.
> 
> Note that this might be helpful for the SEML people as I seem to 
> remember an argument where application tagging was needed.
> 
> Jim
> 
>

---

From: Jim Schaad <[email-masked]>
Date: Wed, 12 Jul 2017 10:52:47 +0200

Yes, that is how I have done it today in the COSE spec.  However, I think it might be a useful feature to kick around and see if it can be done.

Jim

-----Original Message-----
From: Carsten Bormann [mailto:[email-masked]] 
Sent: Tuesday, July 11, 2017 9:32 PM
To: Jim Schaad <[email-masked]>
Cc: [email-masked]
Subject: Re: [Cbor] Application specific tagging

I’d solve this as

Array_item = [
  version: int,
  field1 : int,
  field2 : int / nil,
  field3 : int,
  ? field4 : int
]

(It would not be hard to reserve a number of CBOR tags for application-specific tagging.
I’ll actually come up with a rather different application for this very soon; those who were in the W3C WoT meeting today have gotten a glimpse.)

Grüße, Carsten

---

From: "Paul Hoffman" <[email-masked]>
Date: Wed, 12 Jul 2017 07:43:25 -0700

On 12 Jul 2017, at 1:52, Jim Schaad wrote:

> Yes, that is how I have done it today in the COSE spec.  However, I 
> think it might be a useful feature to kick around and see if it can be 
> done.

I respectfully disagree. Its use in ASN.1 has always been problematic 
for data designers, and I think we had some problems with some proposed 
CMS specs that only got caught late in the process.

A better design rule is "if you have optional fields, it's an object".

--Paul Hoffman

---

From: Laurence Lundblade <[email-masked]>
Date: Wed, 12 Jul 2017 17:24:37 +0000

What is the issue with canonicalization of maps?  I can’t think of what it is, but hopefully it is just mildly inconvenient, not actually difficult.

LL

> On Jul 12, 2017, at 1:52 AM, Jim Schaad <[email-masked]> wrote:
>

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 12 Jul 2017 19:57:51 +0200

> What is the issue with canonicalization of maps?  I can’t think of what it is, but hopefully it is just mildly inconvenient, not actually difficult.

Traditionally, the problem is that people get it wrong.  (So we tried to define it well.  And to make the definition simple enough that it is not that easy to get wrong.  See below :-)

It is also slightly inconvenient, as you cannot generate the map on the fly, but need to wait with outputting bytes until you have all fields (well, at least all the keys).

We have aggravated this a bit by defining map ordering in such a way in Section 3.9 that it violates the POLS (principle of least surprise) for at least some implementers.  Discussion ongoing in another thread…

Grüße, Carsten

---

From: Laurence Lundblade <[email-masked]>
Date: Wed, 12 Jul 2017 19:02:57 +0000

On Jul 12, 2017, at 10:57 AM, Carsten Bormann <[email-masked]> wrote:
> ...

I am now surprised. Can you point me to the other discussion thread?

LL

---

From: Sean Leonard <[email-masked]>
Date: Sat, 15 Jul 2017 21:54:19 -0700

> On Jul 12, 2017, at 7:43 AM, Paul Hoffman <[email-masked]> wrote:
> 
> I respectfully disagree. Its use in ASN.1 has always been problematic for data designers, and I think we had some problems with some proposed CMS specs that only got caught late in the process.
> 
> A better design rule is "if you have optional fields, it's an object”.

I agree with Paul on this one.

It appears from RFC 7049 that CBOR tags are “universal”: once a tag is registered, it is supposed to have that same meaning across all CBOR specs for all time. Tag numbers are supposed to be cheap, at least beyond the 1-byte range. There is congestion in the 1-byte range but there is pretty much no congestion in the 2-byte range, and the 4-byte range is pretty much inexhaustible.

I do not see a problem with having duplicate tag registrations, where a tag that was originally very application-specific (and assigned a high tag number, e.g., 99999999) now appears more useful and to save space on the wire, it is doubly-assigned to a low number (99 or some such). But that is a philosophical issue that has not been debated extensively in CBOR as far as I know.

If you want something that is specific to your application, make it a key in a map, or make it implicit in the index of the array. You can always put a one-byte CBOR item in an array as a pad, namely the simple values false, true, null, and undefined.

Sean