Subject: Re: [Json] Update Standard to support ECMA-262 BigInt

From: Pete Cordell <[email-masked]>
Date: Tue, 4 May 2021 11:11:14 +0100

On 30/04/2021 21:53, Carsten Bormann wrote:
> On 30. Apr 2021, at 21:36, Daniel P <[email-masked]> wrote:
>>
>> Do you mean in the context of a schema specification?
> 
> CDDL (RFC 8610) can already do much of what is probably needed here.
> If there is anything else to be added, CDDL has an extension point (control operators) that is rather flexible.

Hi Daniel and Carsten,

I was speaking in very general and abstract terms.  More along the lines 
of "one could imagine that ..." with no specific work item in mind.

But....

Since you raise the issue of JCR vs CDDL, here are my thoughts...

Firstly, Carsten, do you or anyone else knows of any IETF WGs using CDDL 
to define JSON data?

As for what I think JCR offers that I don't think CDDL currently offers:

1.
JCR has a simple way to combine multiple rulesets.  In one ruleset you 
might have:

    #ruleset-id org.ietf.example1

    $foo = int32

Then in another ruleset:

    #import org.ietf.example1 as ex1

    {
       "id" : $ex1.foo
    }

Here, $ex1.foo refers to $foo defined in the ruleset with id 
org.ietf.example1.

2.
Although possibly not formally released, we were considering a way of 
declaring that a string had to adhere to an arbitrary format.  This was 
a way of adding additional string based types (as there is no other 
option in JSON for extended types!) without adding new base types (like 
int32, date-http etc).  So, in a ruleset you could specify something like:

    #ruleset-id org.ietf.extended-types

    $rfc2822-date-time =
          @{format org.ietf.extended-types.rfc2822.date-time} string

Then use it as:

    #import org.ietf.extended-types as iet

    {
       "start-time" : $iet.rfc2822-date-time,
       "end-time" : $iet.rfc2822-date-time
    }

The spec for the "org.ietf.extended-types.rfc2822.date-time" format 
might read something like:

    Formatted according to the RFC2822 date-time ABNF but omitting
    the trailing [CFWS] term.

So, "org.ietf.extended-types.rfc2822.date-time" is not machine 
resolvable to a specific format on its own.  It is just an opaque 
character string used as an identifier for the format.  A validator 
would either know about strings formatted according to 
"org.ietf.extended-types.rfc2822.date-time" (possibly via inbuilt hard 
coding or using some plugin arrangement) or just issue a warning saying 
I don't know what the format "org.ietf.extended-types.rfc2822.date-time" 
is so I'll check it is a string but leave it to other layers to verify 
that it is the right format.

Such a feature may not be something so useful for CBOR but I think it is 
very useful for JSON strings.

In conclusion, I need to revisit things in more detail, but currently my 
gut feeling is that if those sorts of features are supported in CDDL, it 
would be worth adopting CDDL for JSON format specification.  Primarily 
because it has a formally published spec!

Many thanks,

Pete.
-- 
---------------------------------------------------------------------
Pete Cordell
Codalogic Ltd
C++ tools for C++ programmers, http://codalogic.com
Read & write XML in C++, http://www.xml2cpp.com
---------------------------------------------------------------------

---

From: Pete Cordell <[email-masked]>
Date: Tue, 4 May 2021 11:38:00 +0100

>    the trailing [CFWS] term.

Actually, @{format} doesn't have to be limited to strings.  You could 
also format other base types.  For example, @{format} on an int might 
look something like:

   #ruleset-id org.example.my-types

   $int-div-3 =
         @{format org.example.my-types.int-div-3} int

Where the definition for "org.example.my-types.int-div-3" is "positive 
integers that are multiples of 3".

Cheers,

Pete.

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 4 May 2021 13:49:41 +0200

Hi Pete,

> Firstly, Carsten, do you or anyone else knows of any IETF WGs using CDDL to define JSON data?

Sure.

I’m not keeping a list, and the info is a bit scattered, but you can start from 

https://datatracker.ietf.org/doc/rfc8610/referencedby/

E.g.,

https://datatracker.ietf.org/doc/draft-ietf-asdf-sdf/
https://datatracker.ietf.org/doc/draft-ietf-ace-aif/
https://datatracker.ietf.org/doc/draft-ietf-ace-mqtt-tls-profile/
https://datatracker.ietf.org/doc/draft-ietf-anima-bootstrapping-keyinfra/

Sorry, I stopped at WGs starting with “a”, but you can continue if you want…

(The data is skewed a bit because WGs that are aware about CDDL often also are aware about CBOR and are using that for encoding what would maybe have been encoded in JSON otherwise.)

If you want to get started with using CDDL for JSON, RFC 8610 has a little appendix with a worked example for the JSON in RFC 7071:
https://www.rfc-editor.org/rfc/rfc8610.html#appendix-H

> JCR has a simple way to combine multiple rulesets.  In one ruleset you might have:
> 
> Here, $ex1.foo refers to $foo defined in the ruleset with id org.ietf.example1.

Yep, the top item on the list for CDDL 2.0, probably this year.
Right now, you have to concatenate by hand.

(As we already did before with CDDL “groups” (*), we are certainly going to learn from JCR when defining the CDDL import/export/module feature.  We are certainly also going to learn from YANG, the other IETF data definition language.)

> 2.
> Although possibly not formally released, we were considering a way of declaring that a string had to adhere to an arbitrary format.  This was a way of adding additional string based types (as there is no other option in JSON for extended types!) without adding new base types (like int32, date-http etc).  So, in a ruleset you could specify something like:
> 
> The spec for the "org.ietf.extended-types.rfc2822.date-time" format might read something like:
> 

I don’t think one ever, ever wants to use RFC 2822 (5322) date-time, but here is a snippet from https://datatracker.ietf.org/doc/draft-ietf-cbor-cddl-control/, which is in pre-WGLC review (with X and Y for things that are not relevant here):

   X = text .abnf full-date
   Y = text .abnf date-time

   full-date = "full-date" .det rfc3339
   date-time = "date-time" .det rfc3339

   ; Note the trick of idiomatically starting with a newline, separating
   ;   off the element in the concatenations above from the rule-list
   rfc3339 = '
      date-fullyear   = 4DIGIT
      date-month      = 2DIGIT  ; 01-12
      date-mday       = 2DIGIT  ; 01-28, 01-29, 01-30, 01-31 based on
                                ; month/year
      time-hour       = 2DIGIT  ; 00-23
      time-minute     = 2DIGIT  ; 00-59
      time-second     = 2DIGIT  ; 00-58, 00-59, 00-60 based on leap sec
                                ; rules
      time-secfrac    = "." 1*DIGIT
      time-numoffset  = ("+" / "-") time-hour ":" time-minute
      time-offset     = "Z" / time-numoffset

      partial-time    = time-hour ":" time-minute ":" time-second
                        [time-secfrac]
      full-date       = date-fullyear "-" date-month "-" date-mday
      full-time       = partial-time time-offset

      date-time       = full-date "T" full-time
   ' .cat rfc5234-core

   rfc5234-core = '
      DIGIT          =  %x30-39 ; 0-9
      ; abbreviated here
   '

These are some examples the cddl tool generates for X:

"6962-26-77"
"7578-39-91"
"5073-91-77"
"1258-31-15"
"9316-20-59"
"8308-23-92"
"2131-42-71"
"6573-08-96"
"3120-88-30"
"4322-41-98”

Well, the ABNF is a bit loose :-)

RFC 8610 also has a .regexp control operator (using W3C XSD regexps, but other flavors can be added as needed).

[…]

> currently my gut feeling is that if those sorts of features are supported in CDDL, it would be worth adopting CDDL for JSON format specification.  

Indeed.

> Primarily because it has a formally published spec!

It also is pretty alive and responding to user requirements.

Grüße, Carsten

(*) From RFC 8610:

Acknowledgements

   Inspiration was taken from the C and Pascal languages, MPEG's
   conventions for describing structures in the ISO base media file
   format, RELAX NG and its compact syntax [RELAXNG], and, in
   particular, Andrew Lee Newton's early proposals on JSON Content Rules
   (JCR) as found in draft version four (-04) of [JCR].

---

From: Pete Cordell <[email-masked]>
Date: Tue, 4 May 2021 14:05:56 +0100

Thanks Carsten,

Comments below...

On 04/05/2021 12:49, Carsten Bormann wrote:
>>...

Indeed.  I picked an intentionally grizzly example that you wouldn't 
want to capture in a regular expression or similar.

>... >

That's where the concept of @{format 
org.ietf.extended-types.rfc2822.date-time} has extra mileage.  (Or 
.format org.ietf.extended-types.rfc2822.date-time if translated to CDDL.)

The example wasn't intended to be about ABNF, but saying specifying 
formats in purely machine readable ways usually ends up being 
insufficient.  Having provision for human understood formats done in 
such a way that a machine can ask 'does this value conform to this 
format that you, smart human, have coded?' allows a simple way to 
side-step that problem.

So in addition to the ABNF (or any other equivalent you might choose) 
you can also include to narratively define the constraints on say, 
date-mday etc.

Humans can then write code just for that format, such as:

   class Rfc3339FullDateValiadtor
     def IsValidFormat str
       # Code goes here
       return true or false
     end
   end

Then inject it into a validator:

   cddl_validator = CDDLValidator.new
   cddl_validator.add_type_validator 'org.ietf.rfc3339fulldate', 
Rfc3339FullDateValiadtor.new

which puts the data into a hash of string => object.

When cddl_validator finds a value that should be .format 
org.ietf.rfc3339fulldate it does:

   is_valid_external_type 'org.ietf.rfc3339fulldate', value

where:

   def is_valid_external_type format_key, value
      if @external_validators.include? format_key
         return @external_validators[format_key].IsValidFormat value
      end
      return ???
   end

I hope that all makes sense!

>...

A nice touch :)

Pete.

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 4 May 2021 15:38:30 +0200

On 2021-05-04, at 15:05, Pete Cordell <[email-masked]> wrote:
> 
> The example wasn't intended to be about ABNF, but saying specifying formats in purely machine readable ways usually ends up being insufficient.  Having provision for human understood formats done in such a way that a machine can ask 'does this value conform to this format that you, smart human, have coded?' allows a simple way to side-step that problem.
> 
> So in addition to the ABNF (or any other equivalent you might choose) you can also include to narratively define the constraints on say, date-mday etc.

Sure, we can add something like that, that is an interesting thought.

I’m mostly in favor of also providing a machine-executable form, which may not cover all the bases (if I say there is a “signature” of type “bstr”, I won’t be able to specify in CDDL what exactly I want from that :-).

Grüße, Carsten

---

From: Pete Cordell <[email-masked]>
Date: Tue, 4 May 2021 17:56:25 +0100

> I’m mostly in favor of also providing a machine-executable form, which may not cover all the bases (if I say there is a “signature” of type “bstr”, I won’t be able to specify in CDDL what exactly I…

I agree that an ABNF option is good.  In fact, if I had to choose 
between ABNF and regex, I'd go with ABNF.

But your example did make me think, did you consider HEREDOC format in 
CDDL? e.g. something like:

    rfc3339 = <<ABNFEND
       date-fullyear   = 4DIGIT

       date-time       = full-date "T" full-time
    ABNFEND .cat rfc5234-core

    rfc5234-core = <<ABNFEND
       DIGIT          =  %x30-39 ; 0-9
       ; abbreviated here
    ABNFEND

(I think HEREDOCs would normally end when only the end token appears on 
its own line, but CDDL could say it has to be the first token on the 
line ad other tokens could follow so you can get your .cat rfc5234-core in.)

And I forgot earlier to say thanks for the links to WGs using CDDL for 
JSON.  So thanks for that.  I'll have a look at them.

Looking forward to the the ability to combine multiple CDDL files 
together being in place.

Cheers,

Pete.

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 4 May 2021 21:40:42 +0200

On 2021-05-04, at 18:56, Pete Cordell <[email-masked]> wrote:
> 
> But your example did make me think, did you consider HEREDOC format in CDDL? e.g. something like:

For the .abnf/.abnfb control operators, we tried to make do with what we had in RFC 8610 (no changes in the syntax, just two new control operators — and a third, .det, which is needed to handle ABNF’s indentation intolerance).

A CDDL 2.0 might have a HEREDOC-style mechanism, if people find that useful.

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 5 May 2021 06:56:19 +0200

Ah, I missed this point:

> On 4. May 2021, at 18:56, Pete Cordell <[email-masked]> wrote:
> 
>   rfc3339 = <<ABNFEND
>      date-fullyear   = 4DIGIT
> (I think HEREDOCs would normally end when only the end token appears on its own line, but CDDL could say it has to be the first token on the line ad other tokens could follow so you can get your .cat rfc5234-core in.)

Using classical HEREDOC syntax, this would actually be:

>   rfc3339 = <<ABNFEND .cat rfc5234-core
>      date-fullyear   = 4DIGIT

…which is maybe the best reason why one might want to prefer HEREDOCs over strings (or not!).

Grüße, Carsten