Subject: Re: [core] [T2TRG] Review of CoRAL

From: Klaus Hartke <[email-masked]>
Date: Thu, 1 Nov 2018 11:02:23 +0100

Hej Christian,

thanks a lot for this very detailed review! You're making a lot of
good points. Since several points seem to me worth a discussion on
their own, I will go through the list in the upcoming days one-by-one
and comment on them individually.

A question to the chairs of T2TRG and CoRE WG as this thread was
started on both mailing lists: Where would you like to see the
extended discussion to happen?

Best regards,
Klaus

Christian Amsüss wrote:
> Hello CoRAL observers,
>
> I've read and partially implemented draft-hartke-t2trg-coral-05, and
> would like to offer a review of it.
>
> My two largest individual issues with this are that I'm not convinced of
> the necessity or suitability of the textual format, and that I don't see
> yet how forms would be used in practice (that's out of scope for the
> document, but limits the depth of my review).
>
> In general, the described format would be useful to me, both for new
> applications and to fill in for link-format where RFC6690 is too
> limited.
>
> The following is a rather unsorted list of things that struck me as
> unclear or odd; none of them should stand in the way of a research or
> working group adoption or be otherwise insurmountable; many are also
> just editorial comments.
>
> * "The CoRAL data and interaction model is a superset of the Web Linking
>   model" / "can describe simple resource metadata in a way similar to
>   the Resource Description Framework (RDF)":
>
>   From the examples in 2.1 (in particular, because the rel becomes a
>   full URI), it appears the othr way round: That the data model is a
>   superset of the RDF model, and that the document describes an
>   equivalence of a subset of RDF statements to typed links (as used in
>   link headers, link format and HTML).
>
> * Browsing context: This growing history appears hard to do in a
>   constrained context. Can an agent still work with any given maximum
>   history length, in particular 1 or even 0?
>
> * Forms: I haven't followed all the W3C & surroundings discussions on
>   forms, so I can't say much there as I don't yet understand how they
>   were to actually work. Examples that go beyond "POST to create" and
>   "DELETE to delete" would be helpful when they'd go beyond the
>   intuitive. -- But I assume that as this matures, any formative github
>   discussions will evolve into referrable documents.
>
> * If the first use of the http://www.iana.org/assingments/relation/
>   prefix for rel values pointed to RFC4287, it would be clearer to the
>   reader that this is already established practice.
>
> * Representations: I think it would be helpful to state something in the
>   area of the second paragraph along the lines of "The producer of the
>   CoRAL document claims that the embedded representation is fresh at
>   least for as long as the CoRAL document is fresh", if so intended.
>
>   It's obvious that the "full, partial or inconsistent" part is
>   intentionally giving much leeway to the CoRAL producer. Could this
>   possibly be narrowed down for resources under the same authority? A
>   statement like "(under some condition), the presence of a
>   representation states that there exists a request to the target that
>   would result in a response with the given payload" could go into a
>   direction where an agent could populate its own cache with it and work
>   on from there.
>
> * Binary data format: When relations expressed as uint are first
>   mentioned (and possibly also with numeric form-field-name), a forward
>   reference to the topic of profiles would be helpful.
>
> * Textual format: I find this more confusing than helpful. The format is
>   too far from turtle to allow intuition to be taken from there
>   (especially as there are simple and qualified names, where turtle has
>   only qualified names but the qualifier may be empty), too far from the
>   binary representation to help understanding the binary format (eg.
>   someone only editing in text format would not see that naming a
>   resource linked as rel="index" <index> would be beneficial, but could
>   be misled to believe that application/octet-stream can be a compact
>   default or that representations can have more attributes -- plus
>   people were led to think that CoRAL is large on the wire).
>
>   My impression is that CoRAL would not be written by humans, let alone
>   stored or transported as that. I think the document would be better
>   off if no textual format were defined, but full turtle (or a slim
>   superset thereof that's still within N3) were used to express the
>   semantics of a document (which would need an RDF serialization of
>   forms and representations, but that's more of a benefit than a
>   downside IMO).
>
>   Concrete example from 2.1:
>
>     @prefix : <http://www.iana.org/assignments/relation/> .
>
>     <> :next <./chapter4>;
>        :icon </favicon.png>;
>        :license <http://creativecommons.org/licenses/by/4.0/>.
>
>   or from 2.2:
>
>     @prefix : <http://example.org/vocabulary#> .
>     @prefix coral: <urn:ietf:rfc:XXXX#> .
>
>     <> :task [ = </tasks/1>;
>          :description "Pick up the kids"
>       ];
>       :task [ = </tasks/2>;
>          :description "Return the books to the library";
>          coral:delete [ form:method coap:DELETE;
>                         form:iri </tasks/2>
>                       ]
>       ]
>
> * Why is <http://TBD/> used for attrs, but <urn:ietf:rfc:XXXX#...> used
>   for the core form relations? They could at least share structural
>   similarity.
>
> * attrs: title and title* are described as target attributes in RFC8288
>   and can thus easily occur in CoRAL documents.
>
> * ibd: Then, it can say something like "MUST NOT occur in a CoRAL
>   document as attributes, but are expressed as link relations, nested
>   links and link relations, respectively" to indicate that their
>   information is not lost in the transition.
>
> * Any test vectors or examples to get started on the binary format would
>   be helpful, and could be augmented by (CDDL automatically?) annotated
>   extended diagnostic notation, like (as in the above, assuming a
>   navigation profile where someone forgot that favicon is a typical rel)
>
>   [[/link/ 2, /rel:next/ 4,
>     /<./chapter4>/ [6, "chapter4"]],
>    [/link/ 2, "http://www.iana.org/assignments/relation/icon",
>     /</favicon.png>/ [5, 0, 6, "favicon.png"]],
>    [/link/ 2, /rel:license/ 10,
>     /<http://creativecommons.org/licenses/by/4.0/>/ [1, "http", 2:
>     "creativecommons.org", 6, "licenses", 6, "by", 6, "4.0", 6, ""]]
>   ]
>
> * FoaF example: Just for my understanding, would anything be wrong with
>   some application using <http://www.w3.org/1999/02/22-rdf-syntax-ns#type>
>   as the rel to foaf:Person here? (I figure that few RDF applications
>   treat rdf:type and iana:type as equivalent properties).
>
>   Later that's clarified a bit in A.1 where it says "same purpose", but
>   is rel:type in actual use anywhere? Otherwise, I suggest to go for
>   rdf:type right away, as that is in use. (It's more verbose when
>   expressed in non-CoRAL link formats, though.)
>
> * Only occurred to me when writing the above
>   extended-diagnostic-notation example: Might it make sense to allow
>   profile defined URIs as well, like common licenses or (when looking a
>   the FoaF example) types? They'd need to be told apart from numeric
>   literals that could just as well show up in a link target, but the iri
>   rule could get an additional `?(profiled: 9, uint)` entry that could
>   only exist on its own and expands like a numeric rel.
>
> * What to do with language-tagged attributes (like title*)? The takeaway
>   from links-json is that they can be decoded easily but need to keep
>   their language information. That's easily expressed in RDF (turtle:
>   `<> :title "Überschrift"@de`). links-json went for
>   `{"title":{"de":"Überschrift"}}`, which seems a bit crude to me but
>   could work just as well for CoRAL, as could a tagged CBOR array or a
>   plain array if we stared using arrays with discriminators that are not
>   part of the IRI discriminators as something different than IRIs (a la
>   `[/link/ 2, /attr:title/ 42, [/language-tagged/ -1, "de",
>   "Überschrift"]]`).
>
> * Appendix C: Do I understand correctly that there are some CBOR encoded
>   relative references that can not (even when knowing the rel in
>   append-relation) be expressed in a relative-ref? In particular, those
>   would be the append-* types. If so, it would be prudent to point out
>   right next to where it says that "CBOR-encoded IRI References are not
>   capable of expressing all IRI references".
>
> * Most of the URI resolution gets away without string comparisons. I'd
>   like to suggest making "../" an explicit URI option (say, a `*(parent:
>   5.5)` that'd go between path.type and and path. References with
>   dot-segments inside them are probably safe to put in the "not capable
>   of expressing all" category.
>
>   As I understand URI resolution, we don't need a "./" because that only
>   needs to be explicit when starting with something that has a colon in
>   it that's not the scheme's colon, otherwise "./x/y" is always
>   equivalent to "x/y".
>
> * Speaking of expressiveness of the encoded URIs: (Especially as urn: is
>   used in the document,) would it hurt to grow the encoded references
>   such that they can encode urns? It might be sufficient to allow scheme
>   without a host/port but a path.
>
>   Apropos port: Why must this be present after a host name? This
>   complicates round-tripping to URIs (because the algorithm has to know
>   to fill in default ports of all protocols), limits applicability to
>   portless protocols and adds some bytes -- and in the embedded
>   implementation, adding a default port can be as easy as just setting
>   it at initialization time before parsing.
>
> * Appendix C: A state transition diagram like the ones on
>   http://json.org/ could be as expressive as the list in C.3 while being
>   easier to grasp.
>
> * How are compressed URIs expressed in CBOR? Flat [6, "foo", 6, "bar"]
>   or nested [[6, "foo"], [6, "bar"]]? My first reading (and current
>   implementation) resulted in "nested", supported by the `(option,
>   value) = href[0]` line in C4, but reading C.1 with the CDDL spec next
>   to it (and running the cddl tool) rather indicates "flat" to me.
>
>   Either way, some examples would be great.
>
> * Nit: calling a sequence of options _absolute_ brings up the
>   association of the "absolute URI", which is not the same as "URI"
>   (which is by definition not a reference, but sometimes it helps to say
>   "full URI"). An Absolute URI would not have a fragment identifier (and
>   is defined to ease definign protocols where the fragment is never
>   sent), whereas the non-relative or full option sequence described here
>   can easily (and should be able to) contain a fragment identifier.
>
> * Is there a difference between a CoRAL registry and a CoRAL profile?
>   Both terms are used.
>
> Best regards
> Christian

---

From: Klaus Hartke <[email-masked]>
Date: Sun, 4 Nov 2018 15:56:28 +0100

Hej Christian,

again, thanks a lot for your feedback. I'll start with some
low-hanging fruits and create separate threads for larger topics.

As part of the IETF103 Hackathon this weekend, I've started doing some
basic interop testing of our two CoRAL implementations. There was one
blocking issue and a few minor issues, the blocking issue being this

>   to it (and running the cddl tool) rather indicates "flat" to me.

The representation of the list of options is indeed different in the
draft for CBOR and Python. In Python, it feels natural to me to
iterate over the list as a list of pairs, while in CBOR I don't see a
need to wrap each pair in an extra array. So, in CBOR, the array
should be flat: [6, "foo", 6, "bar"].

If it improves clarity, the Python code could be updated to operate
on flat lists instead, e.g., using the grouper function from
<https://docs.python.org/3.6/library/itertools.html#itertools-recipes>:

-        for option, _ in href:
+       for option, _ in grouper(href, 2):

Minor interop issues:

body.py:
>                if ':' in i.relation:
>                    rel = i.relation
>                else:
>                    rel = 'http://www.iana.org/assignments/relation/' + i.relation

When a relation is provided as a string, the intention is that the
string always contains the IRI. I don't think the current set of
IANA-registered link relation types is relevant enough in the context
of IoT that it should get special treatment.

body.py:
>            elif isinstance(i, BaseDirective):
>                base = base / i.iri

The draft currently specifies that a base IRI is resolved against the
current context IRI, not the previous base IRI.

> * Is there a difference between a CoRAL registry and a CoRAL profile?

The draft uses the "profile" media type parameter defined in
<https://tools.ietf.org/html/rfc6906> to indicate the CoAL registry,
which I think makes more sense to define a new "registry" parameter
with essentially the same semantics. So "profile" should appear only
as the parameter name. I've replaced all instances where "profile"
appeared without this meaning with "registry".

>   for the core form relations? They could at least share structural

In general, I think URNs would make more sense for identifiers than
HTTP URIs, as the latter always look like they could be successfully
dereferenced. But it seems that HTTP URIs are more popular in the
Semantic Web...

Right now, both <http://TBD/...> and <urn:ietf:rfc:XXXX#...> (which
I've replaced with <http://TBD2/...> and <urn:TBD1#...> in my working
copy, respectively) are just visually distinct placeholders until
IETF-controlled IRIs are assigned.

>   reader that this is already established practice.

The whole Examples section probably needs an overhaul, but I've moved
the text about expressing registered link relations into its own
section and added a note that the prefix is already established
practice.

>   navigation profile where someone forgot that favicon is a typical rel)

My idea was to maintain a collection of test vectors in a GitHub
repository, so that the test vectors can be more numerous and more
easily fixed than in an RFC. Now that we creating a bunch of examples
for use cases like RD, it would actually be a good time to start that.

Best regards,
Klaus

---

From: Christian M. Amsüss <[email-masked]>
Date: Mon, 5 Nov 2018 13:01:10 +0100

Hello Klaus,

(recipients reduced as CoRE indicated interest in adoption)

On Sun, Nov 04, 2018 at 03:56:28PM +0100, Klaus Hartke wrote:
> >   or nested [[6, "foo"], [6, "bar"]]?
> 

Very well (saves ~10% message size in the 6690 example), now fixed in
micrurus.

> on flat lists instead.

The grouper version is probably more concise than the note that "the
Python code expects base and href not to be input in the trivial
deserialization of a CBOR array, but as a list of (option number, option
value) pairs" that should otherwise be there.

> of IoT that it should get special treatment.

Indeed. I don't know where the mistake came from -- my mental model
after my current reading of the document was yours, just my the code and
example from March did not match it.

> current context IRI, not the previous base IRI.

Thanks; likewise, I missed updating the current relation number. Both
being fixed.

> appeared without this meaning with "registry".

Having read up on them, a "profile" has a no-changes-to-interpretation
condition:

  A profile MUST NOT change the semantics of the resource representation
  when processed without profile knowledge, so that clients both with
  and without knowledge of a profiled resource can safely use the same
  representation

Given that a client without knowledge of the profile can't process a
representation, the parameter should probably be called something else.

> dereferenced. But it seems that HTTP URIs are more popular in the

It's a matter of style; my take: Devices that dereference URIs on a whim
will cause trouble in some form anyway, better let that show on expected
input and not only when under attack. And as a developer, I'd rather
have a link to follow than know where to look up values in
subregistries.

Thanks
Christian

-- 
To use raw power is to make yourself infinitely vulnerable to greater powers.
  -- Bene Gesserit axiom

---

From: Klaus Hartke <[email-masked]>
Date: Tue, 6 Nov 2018 08:03:04 +0100

> value) pairs" that should otherwise be there.

Right, but it also makes the code slightly less self explanatory. I'll
think about it. Thanks!

> representation, the parameter should probably be called something else.

Good catch! I've changed the parameter name to "registry" in my working copy.

Klaus

---

From: Christian Amsüss <[email-masked]>
Date: Thu, 8 Nov 2018 15:45:48 +0100

Hi Klaus,

some more points came up pushing forward my implementation:

* On "Why are there bare-words as relation strings": During
  refactoring[1], I found why I had that -- the append-rel path type
  seemed to indicate that the rel should be something usable here.

  Now the code in the draft is a bit vague here (`format(relation,
  "x")`), and there's no text on it yet. Any ideas yet on how I'd come
  from a URI to an appendable string (like uri_to_rel(x) = rightmost
  substring of x where x in [a-zA-Z.0-9-])? Will it be sufficent on a
  constrained device that has hardcoded knowledge of what the URIs mean
  to only store uri_to_rel(rel_number_to_uri(n)) for all n in the
  supported profiles?

* current base IRI: Current base IRI keeps being updated to
  current context IRIs. That is a good idea in terms of compact
  representations, but means that local references can not be expressed
  "deep in there".

  Say I want to say that "</temp> is described by
  <http://example.com/TemperatureSensor>, where
  <http://example.com/TemperatureSensor> has author <mailto:..>, change
  log at <...> *and is also available at </descr>*" (like a local cached
  version).  By the time I talk about the thing at example.com, my
  context and current base URI is http://example.com/..., and I can't
  say </descr> any more meaning a local resource.

  (Note that this is a bit of a similar situation as with RFC6690, but
  sans all the confusion.)

  In this very example, one could work around by saying that "is also
  available at" is symmetric, so one could instead say "Furthermore,
  </descr> is also available at <http://example.com/TemperatureSensor>",
  which would be another toplevel link, but it means that a *generic*
  link-format document can not necessarily be expressed easily.

  I have no concrete proposal of how this could be done better. One idea
  is to allow expressing "rev" and not only "rel", but have not thought
  through whether this solves the issue completely at all.

Best regards, and sorry for the somewhat chaotic way in which my
observations and questions arrive
Christian

[1] which has led to micrurus to be what I believe to be correct again,
not only w/rt flat CIRIs, but also with base URI changes and
delta-compressed relation numbers

---

From: Christian Amsüss <[email-masked]>
Date: Fri, 9 Nov 2018 13:17:58 +0100

Running larger tests of CoRAL conversion from RDF (because that's what I
have large quantities of) I ran into issues converting IRIs, which I
think makes sense to discuss (and maybe write down in a section in what
I only just noticed now to be the standalone CIRI document) -- "Which
IRIs can we express and which not".

Topics there are:

* userinfo (ftp://user:[email-masked]/): Can not be expressed; that
  way of transporting credentials is deprecated anyway.

* Some syntax-based denormalization in IRIs is implicitly normalized
  (following RFC 3986 section 6.2.2) when converted to CoRAL. In
  particular, this removes

  * The difference of having a trailing slash after the netloc when the
    path is empty (http://example.com vs. http://example.com/), and

  * any denormalization in percent encoding.

  * Denormalization in the capitalization of scheme and host name may
    be persisted.

* Any differences in IP literal representation: Can not be expressed.
  (I'm not sure whether that falls in the syntax-based normalization
  category; at any rate, there's no guarantees that a CoRAL system
  actually outputs the normalized version.)

* "Using the scheme's default port" can not be expressed.

  As this precludes to-CoRAL conversion (and normalized from-CoRAL) on
  schemes the processor has no explicit knowledge of, I already
  suggested that this be allowed. (It's also a form of normalization,
  but not syntax-based.)

* URIs that do not use a netloc are not supported.

  Anticipating interaction with the dev-urn draft, I think that the
  netloc parts in the CIRI should just be optional; micrurus can run the
  recompose code with just one more bit of state (has_netloc) and
  round-trip URNs with that as well [1]. I did not check whether relative
  resolution works correctly there, but don't see why it shouldn't if
  it's defined at all.

  (I ran into this converting the various mailto: and xmpp: addresses in
  my RDF files, but urn:dev: is probably a more suitable example.)

I'm currently expanding my test section that decomposes RFC3986 example
URIs and tests recomposition, but so far, I'm confident that CIRIs can
be used for all I'd ever intend them :-)

Christian

[1]: https://gitlab.com/chrysn/micrurus/commit/b734de8fdf6cfb701ca0f662c70562574048f621

---

From: Klaus Hartke <[email-masked]>
Date: Sat, 10 Nov 2018 10:08:55 +0100

> Thanks; likewise, I missed updating the current relation number. Both

I just checked how Turtle does it, and it seems that in Turtle a base
IRI is resolved against the previous base IRI. The difference is as
follows:

Document without base IRIs:

    item <foo/a>
    item <foo/b>
    item <bar/c>
    item <bar/d>

Document with base IRIs relative to context IRI:

    #base <foo/>
    item <a>
    item <b>
    #base <bar/>
    item <c>
    item <d>

Advantage: Since link targets are more likely to be relative to the
context than to the previous link targets, this should generate
slightly more compact CBOR representations.

Document with base IRIs relative to previous base IRI:

    #base <foo/>
    item <a>
    item <b>
    #base <../bar/>
    item <c>
    item <d>

Advantages: Since the context IRI is not used after the first base IRI
directive, a processor can reuse the memory for the context IRI to
store the base IRI instead of having to allocate memory for both. It
would match the intuition of people who are familiar with Turtle.

Would it make sense to switch?

Klaus

---

From: Christian M. Amsüss <[email-masked]>
Date: Sat, 10 Nov 2018 10:54:35 +0100

> > Thanks; likewise, I missed updating the current relation number. Both

Oh: I did not fix all of it. What my original code did was change the
context IRI as well (which I had fixed), but I kept working off the same
base IRI.

> IRI is resolved against the previous base IRI.

I checked HTML, but they only allow a single base href. XML's (and thus
Atom's) xml:base URIs can't be relative references anyway. In JSON-LD,
the @base must be in the @context if I understand the text right, so
there can only be one like in HTML.

> slightly more compact CBOR representations.

I'd think that situations of deep links will be rather unlikely anyway
(if / says something about /foo/bar/, it probably says something about
/foo/ too, and /foo/bar/ will be linked to from /foo/ and not / in the
likely cases.)

> would match the intuition of people who are familiar with Turtle.

I'd rather think of it as "updated in-place" than "freed and reused",
but yes, it might be more RAM efficient (esp. interesting with non-zero
nesting limits). Transfering intuition from related formats is a bonus
too, so...

> Would it make sense to switch?

yes.

Christian