Subject: Re: [core] Benjamin Kaduk's Discuss on draft-ietf-core-senml-14: (with DISCUSS and COMMENT)

From: Ari Keränen <[email-masked]>
Date: Mon, 7 May 2018 17:55:59 +0000

Thank you for the review Benjamin!

We have now submitted a new revision of the SenML draft that addresses all the IESG review comments:
https://tools.ietf.org/html/draft-ietf-core-senml-15

For answers to your review comments, please see below.

Thanks,
Ari

> On 19 Apr 2018, at 5.33, Benjamin Kaduk <[email-masked]> wrote:
> 
> Benjamin Kaduk has entered the following ballot position for
> draft-ietf-core-senml-14: Discuss
> 
> When responding, please keep the subject line intact and reply to all
> email addresses included in the To and CC lines. (Feel free to cut this
> introductory paragraph, however.)
> 
> 
> Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
> for more information about IESG DISCUSS and COMMENT positions.
> 
> 
> The document, along with other ballot positions, can be found here:
> https://datatracker.ietf.org/doc/draft-ietf-core-senml/
> 
> 
> 
> ----------------------------------------------------------------------
> DISCUSS:
> ----------------------------------------------------------------------
> 
> I agree with Ben's DISCUSS.

This was addressed in the updated security considerations text:
https://github.com/core-wg/senml-spec/pull/106/commits/b911090ce2d12d394d243658f636522bb3dcc335

https://tools.ietf.org/html/draft-ietf-core-senml-15#section-13

> Additionally, I have serious reservations about introducing the
> concept of "base fields" that apply to subsequent array elemnets
> unless overridden.  It seems to violate an abstraction barrier for
> at least some of the serialization formats, and prevents snippets
> from being composable and commutable absent the
> resolution/normalization process.  It does not seem like the markup
> language and the document contain suffient safeguards against misuse
> to prevent security holes (both sensor data and commands could be
> misinterpreted).  It seems that some substantially expanded text
> should be added on the hazards of the non-resolved format and giving
> guidance on when resolution/normalization must be performed in order
> to avoid correctness and security issues.

The base values feature is something that the WG decided is important to have. We added now text about the security implications and remedies, also taking into account Alexey's follow-ups in this thread: https://github.com/core-wg/senml-spec/pull/109/files

> There also seem to be sizeable risks associated with the semantics
> for time values.  In particular, both with the use of an
> implicit-"now-ish", and with positive and negative values being
> interpreted with respect to a different absolute time base.  (The
> involvement of base time is a further complication -- I do not
> remember any discussion of the interaction of a (positive) base time
> and a negative regular time, for example.  I also do not remember
> any discussion of how the "now-ish" semantics make it actively
> harmful to do things like store-and-forward or archive SenML data
> (again, absent normalization), or what sort of granularity the
> "now-ish" semantics are expected to adhere to.  (Is "yesterday"
> still considered "roughly now"?)  I understand the desire for this
> sort of semantics, but the current specification seems to leave many
> potential problems exposed.

We clarified the "now" issue (also to address other comments on this topic). The new text in section 4.5.3 says:

  Obviously, "now"-referenced SenML records are only useful within a
  specific communication context (e.g., based on information on when
  the SenML pack, or a specific record in a SensML stream, was sent) or
  together with some other context information that can be used for
  deriving a meaning of "now"; the expectation for any archival use is
  that they will be processed into UTC-referenced records before that
  context would cease to be available.  This specification deliberately
  leaves the accuracy of "now" very vague as it is determined by the
  overall systems that use SenML.  In a system where a sensor without
  wall-clock time sends a SenML record with a "now"-referenced time
  over a high speed RS 485 link to an embedded system with accurate
  time that resolves "now" based on the time of reception, the
  resulting time uncertainty could be within 1 ms.  At the other
  extreme, a deployment that sends SenML wind speed readings over a LEO
  satellite link from a mountain valley might have resulting reception
  time values that are easily a dozen minutes off the actual time of
  the sensor reading, with the time uncertainty depending on satellite
  locations and conditions.

> ----------------------------------------------------------------------
> COMMENT:
> ----------------------------------------------------------------------
> 
> Section 4.4
> 
> Just "Considerations" is an unusual subject title.

We re-structruted this part a bit and added sub-titles to this section and split it into "Extensibility" (4.4) and "Records and Their Fields (4.5). Should be more clear & usual now.

> Having no Unit and no Base Unit is allowed, but you don't say what
> the semantics are in that case (presumably just a dimensionless
> counter for integers, with units not really being applicable to
> non-numeric types).  Interestingly, Section 5.1.7 deems it fit to
> use "/" for the unit for a boolean value, even though "/" is
> supposed to be a (continuous/floating-point) ratio.

The application using SenML may have additional ways to convey the unit information (where needed), as is the case with e.g., LwM2M use of SenML. We clarified this in the text now:

  If the Record has no Unit, the Base Unit is used as the Unit. Having
  no Unit and no Base Unit is allowed; any information that may be
  required about units applicable to the value then needs to be provided 
  by the application context.

> Section 4.5
> 
> Just to double-check: you really do want to privilege this
> specification's version for eternity, for the purpose of being
> omitted from resolved records?

Yes.

> Section 12.1 is there not some other units registry we can use?  I
> fear begetting https://xkcd.com/927/  .  Also, how is/should the
> table be sorted?

We are not trying to replace all other such registries. In WISHI work, we haven’t encountered such a registry either. (The problem this one solves is to give ASCII names to units that are useful in SenML.)

The table is in no particular order (and doesn’t have to be); the current order seemed useful to the authors.

> Also in Section 12.1, number 9, is the need for case sensitivity in
> Units (or otherwise?) normatively covered anywhere?  If not, should
> it?

We added to end of section 3:

  All comparisons of text strings 
  are performed byte-by-byte (and therefore necessarily case-sensitive).

> Section 12
> 
> Are Base fields supposed to get negative CBOR labels
> (and not other fields)?  Is this mentioned explicitly somewhere?
> (Yes, I know that the intent is for no more CBOR lablels to be
> allocated, but that is only a SHOULD-level requirement.)

Added to section 6: "The base values are given negative CBOR labels and others non-negative labels."

> In
>  Extensions that are mandatory to understand to correctly process the
>  Pack MUST have a label name that ends with the '_' character.
> should that say something about "mandatory to understand but not
> defined in this document"?  (Also in 12.3.1 et seq?)

This specification doesn't actually define yet any extensions and all the labels in this specification are by definition mandatory to understand, so we don't think this is necessary.

> Section 13
> 
> Why are we talking about "executable content" at all?  It seems
> quite unrelated to the rest of the document.

This was requested by Alexey due to fact that media type reviews often ask about this.

---

From: Benjamin Kaduk <[email-masked]>
Date: Mon, 7 May 2018 19:01:57 -0500

> For answers to your review comments, please see below.

Thanks for the updates, they are definitely improvements!  A few
comments inline.

> https://tools.ietf.org/html/draft-ietf-core-senml-15#section-13

I would of course like to hear from Ben as well, but it would
probably be good to say a bit more about what "proper security
mechanisms" are, perhaps that they are use "to provide data
integrity, confidentiality, and source authentication as appropriate
for the usage".

> The base values feature is something that the WG decided is important to have. We added now text about the security implications and remedies, also taking into account Alexey's follow-ups in this t…

The text about use outside of a Pack is probably the minimal change
needed on this issue (though of course I would not object if more
text was added, perhaps noting that "the smallest functional quantum
for non-resolved SenML Records is the Pack, that provides the needed
context for the Base Values").  But please use your judgment on what
volume of discussion is appropriate given the intended usage and
audience.

>   locations and conditions.

This is definitely helpful to have, but I think I still have some
questions about how things work.  (Perhaps they can be answered in
this thread and no change to the document would be needed.)

The existing text seems to be pretty clear that the base time and
(non-base) time are first added, and then the special semantics for
positive and non-positive values are applied.  Things could
potentially get exciting in some scenarios, though, as a collection
of Records (whether Pack or streaming) could cross over the
discontinuity.  Consider, for example, a small negative base time,
and a series of records with increasing time values, that cross the
zero threshold and suddenly appear decades in the past.  Do we need
to say more to warn about this possibility, or is it perhaps better
to only give the special semantics to one of the (base time, time)
values and not their sum?  (I guess it would have to be something a
little complicated like "if base time is present, it gets the
special semantics and time is added to it, but if base time is not
present then time gets special semantics".)

Thanks for all these updates, they look good.

-Benjamin

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 8 May 2018 10:07:41 +0200

One interesting consequence of the "now" interpretation is that we can't send an actuator message that says "switch on the light five seconds from now". I think this limitation (which became relevant late as we started thinking about actuator messages late) is indeed worth mention -- more so than saying the more inconsequential "you can't express a measurement you will take five seconds later than sending it". 

Sent from mobile

>> On 8. May 2018, at 02:01, Benjamin Kaduk <[email-masked]> wrote:
>> 
> _______________________________________________
> core mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/core
> 
>

---

From: Ari Keränen <[email-masked]>
Date: Tue, 8 May 2018 18:57:12 +0000

> On 8 May 2018, at 3.01, Benjamin Kaduk <[email-masked]> wrote:
> 

Thank you for your fast reply Ben! 

I'll answer now below (mostly) just on my behalf (it takes always a bit of extra time to get consensus across multiple authors in multiple time zones and with busy schedules).

> integrity, confidentiality, and source authentication as appropriate

This sounds good to me. I made a simple PR for this:
https://github.com/core-wg/senml-spec/pull/126/files

I added "e.g." to point out these are not the only considerations and Cullen also pointed out that this could be better without "source" in "source authentication" since also other authentication could be important. I agree with Cullen here; is that OK change to you too?

> volume of discussion is appropriate given the intended usage and

I think the last part of the updated (in -15) security section now essentially says that:

   SenML Records are intended to be interpreted in the context of any
   applicable base values.  If records become separated from the record
   that establishes the base values, the data will be useless or, worse,
   wrong.  Care needs to be taken in keeping the integrity of a Pack
   that contains unresolved SenML Records (see Section 4.6).

(https://tools.ietf.org/html/draft-ietf-core-senml-15#section-13)

> present then time gets special semantics".)

Good point with the discontinuity. It seems like worth pointing out the risk or making best use of it.

I'll have a look how to best address this.

> Thanks for all these updates, they look good.

Thank you once more!

Cheers,
Ari

---

From: Benjamin Kaduk <[email-masked]>
Date: Tue, 8 May 2018 14:01:52 -0500

> I'll answer now below (mostly) just on my behalf (it takes always a bit of extra time to get consensus across multiple authors in multiple time zones and with busy schedules).

Understood.

> I added "e.g." to point out these are not the only considerations and Cullen also pointed out that this could be better without "source" in "source authentication" since also other authentication c…

Yes, thanks.

> (https://tools.ietf.org/html/draft-ietf-core-senml-15#section-13)

Okay.  (Now I have forgotten if I actually said or only intended to
say, do you want to have a note about how when the streaming media
type is used, the underlying transport must be reliable?)

> I'll have a look how to best address this.

Sure, I expect it to require some thought to figure out what's best.

-Benjamin

---

From: Ari Keränen <[email-masked]>
Date: Fri, 11 May 2018 17:05:40 +0000

Hi all,

While addressing the IESG review comments regarding the use of time in SenML we came up with a simple way to enable SenML Records to express also relative times in the future that would have minimal impact to any existing use of SenML. We could use the following definition:

  Values greater than or equal to 2**28 represent an absolute time relative to the Unix epoch. Values less than 2**28 represent time relative to the current time.

That is, instead of only values less than zero, also values less than 2**28 (268,435,456) would be used to express relative time. Negative values and zero are still times in past and "now" respectively, as before. Time values from zero to 2**28 would be relative times in the future.

The only change this causes in the current use of SenML is that the smallest absolute time expressible in SenML becomes 1978-07-04 21:24:16 UTC instead of 1970-01-01 00:00 UTC. The absolute times after 1978 are still exactly the same ("Seconds after Unix epoch"). We are not aware of any deployments with SenML data between 1970 and 1978 that would be impacted negatively by this. 

For details, see:
https://github.com/core-wg/senml-spec/pull/129/files

Would anyone have concerns about including this change in SenML before publication?

Since we are already very late in the process and we need to get SenML published as RFC very soon, we would also need to agree on this very soon.

Thanks,
Ari, Carsten, Cullen

---

From: Christian Amsüss <[email-masked]>
Date: Sat, 12 May 2018 14:39:41 +0200

Hello Ari, hello ADs,

On Fri, May 11, 2018 at 05:05:40PM +0000, Ari Keränen wrote:
>   Values greater than or equal to 2**28 represent an absolute time
>   relative to the Unix epoch. Values less than 2**28 represent time
>   relative to the current time.

I appreciate that change. It does not impact my deployments of SenML but
allows using it with CoRE interfaces more consistently.

Best regards
Christian

-- 
To use raw power is to make yourself infinitely vulnerable to greater powers.
  -- Bene Gesserit axiom

---

From: Alexey Melnikov <[email-masked]>
Date: Sun, 13 May 2018 00:02:29 +0300

Hi Ari,

> On 11 May 2018, at 20:05, Ari Keränen <[email-masked]> wrote:
> 
> Hi all,
> 

No objections from me personally, but please double check with the WG.

Best Regards,
Alexey

---

From: Michael Koster <[email-masked]>
Date: Sat, 12 May 2018 14:15:22 -0700

+1

Michael Koster

> On May 12, 2018, at 2:02 PM, Alexey Melnikov <[email-masked]> wrote:
> 
> Hi Ari,
> 
> On 11 May 2018, at 20:05, Ari Keränen <[email-masked] <mailto:[email-masked]>> wrote:
> 
>> Hi all,
>> 
>> https://github.com/core-wg/senml-spec/pull/129/files <https://github.com/core-wg/senml-spec/pull/129/files>
>>

---

From: Klaus Hartke <[email-masked]>
Date: Sun, 13 May 2018 19:20:40 +0200

+1

On Sat 12. May 2018 at 23:15, Michael Koster <[email-masked]>
wrote:

> While addressing the IESG review comments regarding the use of time in
> SenML we came up with a simple way to enable SenML Records to express also
> relative times in the future that would have minimal impact to any existing
> use of SenML. We could use the following definition:
>
> relative to the Unix epoch. Values less than 2**28 represent time relative
> to the current time.
>
> That is, instead of only values less than zero, also values less than
> 2**28 (268,435,456) would be used to express relative time. Negative values
> and zero are still times in past and "now" respectively, as before. Time
> values from zero to 2**28 would be relative times in the future.
>
> The only change this causes in the current use of SenML is that the
> smallest absolute time expressible in SenML becomes 1978-07-04 21:24:16 UTC
> instead of 1970-01-01 00:00 UTC. The absolute times after 1978 are still
> exactly the same ("Seconds after Unix epoch"). We are not aware of any
> deployments with SenML data between 1970 and 1978 that would be impacted
> negatively by this.
>
> For details, see:
> Since we are already very late in the process and we need to get SenML
> published as RFC very soon, we would also need to agree on this very soon.
>
>

---

From: Ari Keränen <[email-masked]>
Date: Sun, 13 May 2018 18:06:35 +0000

> On 13 May 2018, at 0.02, Alexey Melnikov <[email-masked]> wrote:
> 
> Hi Ari,
> 

Great!

I saw that Christian, Michael, and Klaus from the WG already +1'd (thanks guys!). If there are no concerns raised, we could submit a new version with this change included, e.g., at the end of the work week.

Cheers,
Ari

---

From: Jaime Jiménez <[email-masked]>
Date: Tue, 15 May 2018 10:09:13 +0000

Hi all,

there seems to be consensus on the changes to the Times values among those who participated in the discussion. 
Just to be sure lets have a last consensus call on this, allowing for any other concerns to be voiced, ending by EOB Thursday 17th. 

Ciao!
- - Jaime Jiménez

> On 13 May 2018, at 21.06, Ari Keränen <[email-masked]> wrote:
> 
> 
> 
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/core <https://www.ietf.org/mailman/listinfo/core>

---

From: Mohit Sethi <[email-masked]>
Date: Wed, 16 May 2018 11:11:13 +0300

Hi,

I have previously implemented senml. I am fine with the amendments and 
would like to see the draft move forward.

--Mohit

On 05/15/2018 01:09 PM, Jaime Jiménez wrote:
> Hi all,
>
> there seems to be consensus on the changes to the Times values among 
> those who participated in the discussion.
> Just to be sure lets have a last consensus call on this, allowing for 
> any other concerns to be voiced, ending by *EOB Thursday 17th*.
>
> Ciao!
> - - Jaime Jiménez
>
>> On 13 May 2018, at 21.06, Ari Keränen <[email-masked] 
>> <mailto:[email-masked]>> wrote:
>>
>>
>>
>>> On 13 May 2018, at 0.02, Alexey Melnikov <[email-masked] 
>>> <mailto:[email-masked]>> wrote:
>>>
>>> Hi Ari,
>>>
>>> On 11 May 2018, at 20:05, Ari Keränen <[email-masked] 
>>> <mailto:[email-masked]>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> While addressing the IESG review comments regarding the use of time 
>>>> in SenML we came up with a simple way to enable SenML Records to 
>>>> express also relative times in the future that would have minimal 
>>>> impact to any existing use of SenML. We could use the following 
>>>> definition:
>>>>
>>>> That is, instead of only values less than zero, also values less 
>>>> than 2**28 (268,435,456) would be used to express relative time. 
>>>> Negative values and zero are still times in past and "now" 
>>>> respectively, as before. Time values from zero to 2**28 would be 
>>>> relative times in the future.
>>>>
>>>> smallest absolute time expressible in SenML becomes 1978-07-04 
>>>> 21:24:16 UTC instead of 1970-01-01 00:00 UTC. The absolute times 
>>>> after 1978 are still exactly the same ("Seconds after Unix epoch"). 
>>>> We are not aware of any deployments with SenML data between 1970 
>>>> and 1978 that would be impacted negatively by this.
>>>>
>>>> For details, see:
>>>> Since we are already very late in the process and we need to get 
>>>> SenML published as RFC very soon, we would also need to agree on 
>>>> this very soon.
>>>
>> I saw that Christian, Michael, and Klaus from the WG already +1'd 
>> (thanks guys!). If there are no concerns raised, we could submit a 
>> new version with this change included, e.g., at the end of the work week.
>>
>>
>> Cheers,
>> Ari
>>

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 18 May 2018 13:39:52 +0200

Hi Benjamin,

after WG input about this late change (all positive) has petered out, https://tools.ietf.org/html/draft-ietf-core-senml-16 is now out with the boundary between “now”-relative and absolute moved from 0 (1970) to 2**28 (1978).  Thank you for making us think this through once more; the result is so much better.

The authors believe that -16 should be addressing all DISCUSSes and COMMENTs.  As we are looking at an OMA deadline (an SDO depending on this) that could be picking up the new version, we could make good use of swift approval.

Grüße, Carsten

> On May 13, 2018, at 20:06, Ari Keränen <[email-masked]> wrote:
> 
> 
>

---

From: Benjamin Kaduk <[email-masked]>
Date: Fri, 18 May 2018 07:22:08 -0500

Hi Carsten,

Thanks for the updates; the 2**28 threshold does seem like a big
improvement (and the natural semantics of "add base time to relative
time and then compare against the threshold" seems to make perfect
sense in that context).

I have changed my position to Abstain, since the non-hierachical and
especially (potentially) interleaving of base values is still not
something I support, but I recognize that there is consensus to move
forward with it.

If you do have chance to make additional updates, it might still be
worth adding an explicit note that the streaming format only makes
sense in the presence of a reliable, in-order transport.  Since this
requirement really ought to be obvious to anyone who understands
what is going on, I will not attempt to make this a blocking
comment, though.

Adding Ben to the To: field since he also has an outstanding DISCUSS
position in the datatracker.

-Benjamin