Subject: Re: [core] Second Resource Directory plug test

From: Christian Amsüss <[email-masked]>
Date: Wed, 3 Oct 2018 11:37:16 +0200

Hello RD plug test participants and watching mailing lists,

these are the promised details about the upcoming second RD
interoperability event.

A -15 version of Resource Directory was just published[1] and will be
the reference draft; it contains no changes to actual mechanisms
compared to -14. The tests we will run through are described at [2].

The plug test will take place online, using the Jitsi at [3], which
provides screen sharing and a common note taking area.

As no time was found for all participants to meet, we'll hold the event
in two time slots:

    2018-10-10 at 10:00 CEST (08:00 UTC)

and

    2018-10-12 at 16:00 CEST (14:00 UTC, 07:00 MST).

Participants who did not register in the schedule sent with the original
mail are asked to contact me directly for further planning.

See you there
Christian

[1]: https://tools.ietf.org/html/draft-ietf-core-resource-directory-15
[2]: http://htmlpreview.github.io/?https://github.com/core-wg/resource-directory/draft-ietf-core-resource-directory-15/interop-spec.html
[3]: https://jitsi.tools.ietf.org/resource-directory-interop-2

-- 
There's always a bigger fish.
  -- Qui-Gon Jinn

---

From: Christian Amsüss <[email-masked]>
Date: Wed, 10 Oct 2018 13:20:30 +0200

Hello CoRE list,

we've just completed part I of the previously announced RD plug test
(between the RIOT and aiocoap implementations), and I'm sending the
results to the list right away to have them available in today's interim
meeting.

Summary:

  The tests that can be expected to be performed by a device implements
  the registrant-EP side of RD passed, with some changes due to limits
  of the underlying CoAP library. One test was missed because tests were
  executed out-of-order.

  So far, nothing group related has been tested as nothing has been
  implemented by either participant.

Details below.

Best regards
Christian

---

Implementations under test were:

* RIOT's built-in endpoint, where RIOT is an operating system for class
  C0 to C2 devices. The endpoint implementation is designed to do either
  a simple or a as a full registration, to keep both alive, and to
  explicitly deregister again. Due to limitations in the underlying CoAP
  library, it can not perform multicast discovery or send target
  attributes or an anchor in links yet. (Those features are being worked
  on.)

  The registrant EPs were run in iot-lab with full IPv6 connectivity and
  operated by Hauke Petersen; later tests also include an endpoint run
  locally at his site.

* aiocoap's resource directory, where aiocoap is a Python implementation
  of CoAP that ships several tools for use on unconstrained systems. It
  implements all components of the RD server except groups, as well as
  regular registrant endpoint. No commissioning tool (CT) is implemented
  per se, but it provides a program called rd-relay. The relay provides
  links to an RD proper in multicast lookups, and performs third party
  registration for local registrants that attempt simple registration.

  The RD server was run on a public IPv6 address and operated by
  yours truly.

Test execution:

* Test 3 passed. The execution deviated from the protocol by providing
  no additional link or target attributes (see above), a varied endpoint
  name and an unexpected Content-Format header sent by the registrant
  (fixed).

* Test 1 passed with modifications, test 4 passed. The target address in
  1 was changed from multicast to unicast to accomodate library
  limitations, and because the implementations were not colocated. In
  test 4, the same links as in test 3 were used (an oversight), and
  again no link attributes were present.

  The registrant provided an endpoint name varying from the test
  specification. The RD used a different registration path than the
  example path given in the test specification.

* Test 12 passed (the device was manually prompted to de-register).

* Test 13 was missed when going through the list, but a version of it
  (minus the new base address) was executed regularly by the active
  registrants.

* Tests 2, 5, 6, 7, 8, 9, 10, 11, 14 and 15 were not applicable to an
  endpoint-only client implmentation. Some of those steps were filled in
  by an aiocoap implementation for verification purposes:

  * Tests 2 and 5 were executed by running an rd-relay at Hauke's site,
    where he tasked an endpoint to simply register at the local
    rd-relay. The relay looked up the details of the RD server and then
    performed a thrid party registration.

  * Tests 10, 12 and 15 were executed continuously by means of an active
    observation on the server, and gave the expected results. As no link
    attributes were set by the registrants, the filter parameters were
    left out.

  As additional tests, the links produced in lookup were handed to an
  client to be dereferenced. Links from test 3 and 4 (simple and
  regular registration) could be read. Links from test 5 could not be
  read because the endpoint had no route to the internet; the CT thus
  entered them as
  <coap://[fe80::2c7c:b7ff:feac:bda8%tapbr0]/sensors/temp>. That might
  be an issue to consider during discovery, and was taken to the draft's
  issue tracker.

Various obervations:

* Sending more link attributes would make the link-format serialization
  on small devices hard in terms of flash size, link-format+cbor might
  be of help here and will require support by RDs.

* Issues discovered in the server were failure to deregister timed out
  simple registrations, and needless sending of observe notifications
  when nothing really changed.

* The GETs performed during simple registration were issued from a
  different IP address than the one targeted in the registration. That
  caused no trouble in the test, but might confuse readers of the logs.

-- 
To use raw power is to make yourself infinitely vulnerable to greater powers.
  -- Bene Gesserit axiom

---

From: Christian Amsüss <[email-masked]>
Date: Wed, 17 Oct 2018 09:47:19 +0200

Hello observers of the Resource Directory plug tests,

we have concluded part II of the second installment of the RD plug tests.

(DNS-SD: Report part I only went to CoRE as it was time critical
to an interim. The summary covers both, details on [1].)

Summary:

  Three implementations were tested, and all eventually passed the
  implemented subsets of RD. (Jim's covered everything, Christian's
  everything but groups, Hauke's embedded one acted as an endpoint).
  Four details that need claraification in the specification were
  discovered (handling of link-local addresses, and the below), but
  nothing major. Two implementations could not interoperate due to IPv6
  not having been universally deployed.

  No implementations of extensions to RD (like RD-DNS-SD) were known so
  none were tested. We hope for participation of implementing parties in
  future events.

Questions raised:

  * When matching against endpoint registrations, can an href filter
    query be matched as a relative reference (eg. /lkp/ep?href=/reg/1)?
  * Can a registration be "taken over" by a Commissioning Tool?
  * Is displaying the base of a registration mandated in EP lookup?

They're being taken up in the document's issue tracker for discussion
among the authors, and expected to be clarified in the next version of
the draft.

Details below.

Best regards
Christian

[1]: https://mailarchive.ietf.org/arch/msg/core/enMOFPTiK9qcS-lxpQ-VVyyKbuc

---

* Jim Schaad's resource directory, implemented in C# and running on an
  unconstrained system.

* aiocoap's resource directory (see [1]; Christian)

* RIOT's registrant endpoint (see [1]; Hauke)

Test execution:

* Christian's server with Jim's registrant and lookup client

  Tests were executed in sequence and passed unless noted otherwise,
  with the following alterations and notes:

  * Test 1, 2: Unicast address used as devices were not in multicast
    range.
  * Tests 6-7, 9 and 14 (anything group related) were not supported by
    the server
  * Test 11 was not implemented on the server.
  * Bugs in the implementations were found in tests 1 (when modified by
    the client to include ct lookup; actually a bug in the server's
    implementation of RFC6690), 13 (client sent content-format
    indication payload while it shouldn't, due to a limitation of the
    underlying library).

* Jim's server with Hauke's client

  * Test could not be executed because the test sites did not share a
    common IP version.
  * Registration through a relay (see rd-relay description in [1]) was
    attempted but failed for reasons discovered later.

  The paring will be tried again in the next plug test, either with
  IPv6 everywhere or some sort of tunneling in place, possibly
  facilitated by F-Interop.

* Jim's server with Christian's client

  Tests were executed out of sequence because several tests are grouped
  together in a single client application run. All tests except where
  noted did pass, with several alterations and notes:

  * Test 1, 2: Unicast address used as devices were not in multicast
    range.
  * Tests 6-7, 9 and 14 (anything group related) were not supported by
    the client.
  * The .well-known/core resource was advertised in all registrations in
    addition to the prescribed resources, and consequently shown in the
    lookups.
  * In test 13, the wrong registration was updated. This was noticed in
    test 15; the lookup results were consequently in conflict with the
    test description, but consistent with the course of the experiment.

  * Bugs were found in tests 2 (the * in lookup was matched to "one or
    more characters", actually a bug in the server's implementation of
    RFC6690) and 4 (a slash was expected to trail the base value), and
    fixed before continuing.
  * The participants spent some time debugging the spurious absence of
    GET requests triggered by simple registration, which was actually an
    artifact of all network scanners having been configured to port 5683
    exclusively, and the registrant using a different port to avoid
    collision with an already active non-simple endpoint.