Subject: [core] Observing core-interfaces batches live and in history

From: Christian Amsüss <[email-masked]>
Date: Mon, 31 Mar 2014 22:37:43 +0200

Hello working group,

I've been using CoAP for some time in data monitoring applications, and
observed patterns emerge that work well, but not necessarily fully
conform to the specifications; I'd like to work out with you which parts
of it can be generally useful, among other things allowing continuous
logging of batches.

* Having history available: core-interfaces mentions returning the
  latest measurements as a SenML time series. We are using the same
  concept for batches too, and internally compress lengthier histories
  (ie. more that you'd just send along in a single-block piggy-backed
  response) in our nodes' ram.

  I think it would be wise to advertise the capability of providing
  history, and to provide a common interface to querying it. Something
  like if="core.s core.h" could indicate that some form of history is
  provided by the sensor, and a query with parameters similar to those
  in core-interfaces 5.9 would be ?tmin=1396134000, returning values no
  earlier than 00:00 today. A tmax query would set an upper limit.

  (So far, this is a minor suggestion for addition in coap-interfaces,
  but it is relevant for setting the context for the next items.)

* A proxy might understand the semantics of a core.h (as it might
  understand a core.b and update included resources after querying a
  newer batch), and provide time slices by itself, provided it has the
  complete slice freshly cached. (The original server will probably
  assign a long max-age to an reply that has a tmax in its clock's
  past.)

* If a proxy observes such a resource, that observation alone would not
  be sufficient to keep the cache accurate, as observation only provides
  eventual consistency, and intermediate events could be missed based on
  the guarantees of core-observe.

  The only reason I found in observe-12 that would prevent more than just
  eventual consistency from being provided is the presence of proxies,
  which might eat an intermediate state.

  (Of course, the original server would need to send every single state
  as CON, or include the last NON packages' states until a CON is sent).

  Have mechanisms to support such behavior been discussed before?

* A comparatively different mode of observations we are currently using
  are partial updates of batches. As having many observations running
  for individual resources can be costly, and some resources are often
  queried together (eg. individual multi-value sensors in a multi-sensor
  device, or all sensor values of a device), observable batches are a
  practical thing.

  If only parts of that batch change, our test implementations send only
  the modified "e" items. This would confuse non-aware proxies, and
  generally has the same issues as above.

  On the practical side, this not only saves bandwidth, but helps with
  latency, as the typical update now fits in a single block again. (I
  don't know how it could at all fit with multiblock updates.)

If no mechanism to provide fast updates of batches and event streams
exists, I would consider proposing an option (let's call it
Observe-Aggregate, an unsafe option that may be present in observe
replies multiple times, and has an integer value similar to Accept; a
proxy that wants to utilize the information needs to aggregate it
according to a well-known method indicated by the option value) -- but
that is something that'd need some input first, especially on other use
cases.

There are some still unresolved issues with SenML batches related to
this, which I brought up a while ago (back then, not with my business
but with my free-software developer hat on and under the nickname of
chrysn <[email-masked]>); if you'd find time to review the mails
concerning 'SenML unit in link format response' and '"n" in SenML
examples vs SenML spec' of 2013-08-02, they fit very well with the
issues discussed above.

The latest SenML proposal is generally a little outdated, but used in
recent core-interfaces drafts. If desired, I'd happily provide patches
where I can; I think it is a good thing to have in the context of CoRE.

Best regards
Christian

-- 
Christian Amsüss
Energy Harvesting Solutions GmbH
http://www.energyharvesting.at/