Subject: [core] YANG notification within CoMI

From: Michel Veillette <[email-masked]>
Date: Fri, 8 Jun 2018 18:17:19 +0000

Support the YANG notifications in the current CoMI draft (i.e https://www.ietf.org/id/draft-ietf-core-comi-03.txt) is based on the observe option as defined by RFC7641.

This solution has the following issues which make it not practical for the targeted use case.

1) This approach in incompatible with load balancers, notifications are directly returned to the specific server within the cluster which have initiated the observe request.

2) Typical CoAP implementation (e.g. Californium) doesn't support the persistence of the observe context. These contexts can't be recovered after a server restart and can't be shared between servers.

3) Registrations to event streams are not resilient, they can terminate unexpectedly upon transmission error or reception of a Reset message.

I recommend to delete draft-ietf-core-comi section 5.5 and start a specific effort on this topic.

Alignment with the [I-D.ietf-netconf-yang-push] also need to be considered.

Those interested to this topic, please send me a email so I can schedule conference calls or a face to face meeting during the next IETF.

The following diagram shows a typical deployment scenario.

  +---------+  +---------+       +---------+

  | CoMI    |  | CoMI    |  ...  | CoMI    |

  | server  |  | server  |       | server  |

  +---------+  +---------+       +---------+

       |            |                 |

       |            |                 |

       +----------+-+-----------------+

                  |

                  |

   +---------------------------------+

   | CoMI         |                  |

   | client       |                  |

   |         +----------+            |

   |         | Load     |            |

   |         | Balancer |            |

   |         +----------+            |

   |              |                  |

   |        +-----+---------+        |

   |        |               |        |

   |   +---------+     +---------+   |

   |   | Server  | ... | Server  |   |

   |   +---------+     +---------+   |

   |        |               |        |

   |        +-------+-------+        |

   |                |                |

   |           +----------+          |

   |           | Database |          |

   |           +----------+          |

        |                                 |

   +---------------------------------+

Regards,
Michel

---

From: Michael Richardson <[email-masked]>
Date: Sat, 09 Jun 2018 09:18:03 -0400

Why would a COMI client be behind a load balancer?

(I can't understand why some ports on such a machine might be subject to
load balancing, but why the CoMI communications?)

-- 
]               Never tell me the odds!                 | ipv6 mesh networks [ 
]   Michael Richardson, Sandelman Software Works        | network architect  [ 
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: Peter van der Stok <[email-masked]>
Date: Mon, 11 Jun 2018 09:19:46 +0200

Hi Michel,

Thanks for the message.
I think that load balancers, stable storage for messages, californium
implementation restrictions are all separate subjects, and merit a
solution independent of CoMI.

I recommend to maintain section 5.5, as is, in the comi draft and start
looking at the other subjects separately.

Peter

---

From: Michel Veillette <[email-masked]>
Date: Mon, 11 Jun 2018 19:41:52 +0000

Hi Michael

About "Why would a COMI client be behind a load balancer?"

The main target of the CoMI protocol is network management. We typically deploy this protocol between a Network Management System (i.e. CoMI client) and hundreds of thousand or million(s) of constrained network elements (i.e CoMI server). At that scale, the NMS is implemented using a server cluster. The load balancer is used to distribute the load of notifications between the different servers. The load balancer also presents a stable IP address independent of the currently available servers behind it.

This is just a typical NMS configuration, noting special.

Regards,
Michel

-----Original Message-----
From: Michael Richardson [mailto:[email-masked]] 
Sent: Saturday, June 9, 2018 9:18 AM
To: Michel Veillette <[email-masked]>
Cc: Peter van der Stok <[email-masked]>; Andy Bierman <[email-masked]>; Alexander Pelov <[email-masked]>; Eric Voit (evoit) <[email-masked]>; Henk Birkholz <[email-masked]>; [email-masked]; Core <[email-masked]>
Subject: Re: [Yot] YANG notification within CoMI

(I can't understand why some ports on such a machine might be subject to load balancing, but why the CoMI communications?)

---

From: Michel Veillette <[email-masked]>
Date: Mon, 11 Jun 2018 21:12:51 +0000

Hi Peter

About " I recommend to maintain section 5.5, as is, in the CoMI draft and start looking at the other subjects separately."

The current 'observe' method might still be useful for some use cases which requires a temporary subscription. For example, this method can be used to temporarily subscribe to the syslog for debug purposes. This method can also be used to wait for the completion of a transaction initiated using the normal CoAP methods.

To address the more common event stream use cases (i.e. a something equivalent to SNMP traps), it seem we need to introduce a new mechanism.

About how to solution should be packaged (i.e. one of two drafts), we should probably wait for a better understanding of the direction we want to take.

Regards,
Michel

From: Peter van der Stok [mailto:[email-masked]]
Sent: Monday, June 11, 2018 3:20 AM
To: Michel Veillette <[email-masked]>
Cc: Andy Bierman <[email-masked]>; Alexander Pelov <[email-masked]>; Eric Voit (evoit) <[email-masked]>; Henk Birkholz <[email-masked]>; [email-masked]; Core <[email-masked]>
Subject: Re: YANG notification within CoMI

Hi Michel,

Thanks for the message.
I think that load balancers, stable storage for messages, californium implementation restrictions are all separate subjects, and merit a solution independent of CoMI.

I recommend to maintain section 5.5, as is, in the comi draft and start looking at the other subjects separately.

Peter

       |            |                 |

       |            |                 |

                  |

                  |

   | CoMI         |                  |

   | client       |                  |

   |         +----------+            |

   |         | Load     |            |

   |         | Balancer |            |

   |         +----------+            |

   |              |                  |

   |        +-----+---------+        |

   |        |               |        |

   |        |               |        |

   |        +-------+-------+        |

   |                |                |

   |           +----------+          |

   |           | Database |          |

   |           +----------+          |

        |                                 |

Regards,
Michel

---

From: Michael Richardson <[email-masked]>
Date: Mon, 11 Jun 2018 21:18:40 -0400

Michel Veillette <[email-masked]> wrote:
    > The main target of the CoMI protocol is network management. We
    > typically deploy this protocol between a Network Management System
    > (i.e. CoMI client) and hundreds of thousand or million(s) of
    > constrained network elements (i.e CoMI server). At that scale, the NMS
    > is implemented using a server cluster. The load balancer is used to
    > distribute the load of notifications between the different servers. The
    > load balancer also presents a stable IP address independent of the
    > currently available servers behind it.

yes, in IPv4 land, this is what you'd be forced to do.
In IPv6, it doesn't that much sense to me.

It seems that there many possible stateless ways of making what you describe
work without resorting to storing a bunch of state in a
single-point-of-failure Load balancer, and then having problems with keeping
context.

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 12 Jun 2018 11:48:29 +0200

Hi Michel,

sorry for being slow to answer; I finally had a nice discussion with Henk today about your concerns that I want to summarize.

Let me propose some terminology first:  “servers” are a CoAP term, so when we talk about “heavy boxes with noisy fans installed into a rack”, I’ll use “machines” (as opposed to “devices” that are the light bulbs being managed).  Not great terminology, but better than using the term “server” for both.

> 1) This approach in incompatible with load balancers, notifications are directly returned to the specific server within the cluster which have initiated the observe request.

Whether that is true depends a lot on what kind of load balancer you are thinking about.
For UDP CoAP, an anycast mechanism is the obvious choice for a load balancer.

Clearly, for any load balancing mechanism to be useful, the machines sharing the load need to share state (symbolized by “database” in your diagram).
If they are clients, they need to share communication state as well as application state (e.g., addresses of servers they talk to, the tokens they use for outgoing requests).  They don’t need to do this for every single transaction (they don’t if there is no problem in an occasional loss), but they do have to share observe state.

When people talk about load balancers and resilience, they sometimes mean that the peer with initiative (here: the CoAP server sending another notification) needs to perform a full rendezvous with the other side (e.g., in the Big Web, a fresh DNS lookup that might lead to a completely different machine, potentially a new TLS session).  See (3) below for how we see this, for now let’s just say that we are trying to achieve generally rendezvous-free notifications.

> 2) Typical CoAP implementation (e.g. Californium) doesn't support the persistence of the observe context. These contexts can’t be recovered after a server restart and can't be shared between servers.

That is indeed a problem.  But the real problem is that they are not ready for sharing their client state at all.  Once they are, adding persistence to that sharing becomes a rather straightforward exercise.

So what I’m reading out of your message is that, to employ load balancing for COMI notifications, CoAP implementations need to grow support for sharing communication state (preferably with a persistent mode).  That is an important message to implementers, and thank you for highlighting this.

> 3) Registrations to event streams are not resilient, they can terminate unexpectedly upon transmission error or reception of a Reset message.

Now how does the resilience come in?

In the Big Web situation mentioned above, it comes from redoing the rendezvous each time a notification is needed (potentially with some caching, both of DNS state [leading to defined periods of blackholing] and of any connections still open [which will hopefully time out if there is a problem].  

In a rendezvous-free world, we have to do this explicitly.  For a CoAP server that cares about delivering the notifications, it will send (at least some of) the notifications as confirmable messages [it actually has to, once every 24 hours, but can do that more often if resilience calls for it].  So it will notice when the recipient of the notifications is no longer there.  [It will also notice if the recipient is confused enough to send a Reset, but persistence of communication state is supposed to make this a non-event.]  RFC 7641 tells us that the CoAP server is to cease delivering notifications when the client machine goes away.  It doesn’t tell us what else the implementation might want to do in that event.  In a COMI call home scenario, I would expect the device to notice that its relationship to home broke and redo the rendezvous (call home) — once, when needed, not for every transaction (mitigated only by caching).  

So, in effect, we can have all the advantages of the “do the rendezvous always, with caching” world with much less black-holing and unnecessary message overhead.

For the above to be actionable, we do have to have implementations that:
— on the big machines, can share enough communication state so they can take part in anycast-based load balancing,
— on the small devices, can react to loss of an observation interest by redoing a call-home transaction.

The second one clearly is an implementation quality requirement.  Let’s work on that with the implementers.
The first could be thought to call for a protocol for coordinating the machines.  The IETF has not been very successful in establishing “reliable server pooling”.  I would actually expect implementations that want to provide that coordination to come with their own high-performance mechanisms, involving the usual state sharing databases such as Redis — they already have to do this with the management (application) state shared between the machines.

Grüße, Carsten

---

From: Peter van der Stok <[email-masked]>
Date: Tue, 12 Jun 2018 14:09:42 +0200

I think Carsten and also Michael have started to discuss an approach to
the current implementation short comings and resilience additions.

Peter
Michel Veillette schreef op 2018-06-11 23:12:

> FROM: Peter van der Stok [mailto:[email-masked]] 
> SENT: Monday, June 11, 2018 3:20 AM
> TO: Michel Veillette <[email-masked]>
> CC: Andy Bierman <[email-masked]>; Alexander Pelov <[email-masked]>; Eric Voit (evoit) <[email-masked]>; Henk Birkholz <[email-masked]>; [email-masked]; Core <[email-masked]>
> SUBJECT: Re: YANG notification within CoMI 
> 
> Hi Michel,
> 
> Thanks for the message.
>> Support the YANG notifications in the current CoMI draft (i.e https://www.ietf.org/id/draft-ietf-core-comi-03.txt [1]) is based on the observe option as defined by RFC7641. 
>> 

Links:
------
[1] https://www.ietf.org/id/draft-ietf-core-comi-03.txt

---

From: Michel Veillette <[email-masked]>
Date: Tue, 12 Jun 2018 14:54:57 +0000

Hi Carsten

Thanks for sharing your points of view on this subject.

I'll first validate if the anycast address can effectively address my first issue.
I need to verify if I can configure the infrastructure to use an anycast source address for my CoAP traffic.

The CoAP library already support sharing of DTLS states.
This feature at least enable a solution based on a POST form the CoMI server.
I assume that the same technique is possible for the observe states, something to be validated.

I hope to get these answers prior to the next IETF in Montreal.

Regards,
Michel
 
-----Original Message-----
From: Carsten Bormann [mailto:[email-masked]] 
Sent: Tuesday, June 12, 2018 5:48 AM
To: Michel Veillette <[email-masked]>
Cc: Peter van der Stok <[email-masked]>; Andy Bierman <[email-masked]>; Alexander Pelov <[email-masked]>; Eric Voit (evoit) <[email-masked]>; Henk Birkholz <[email-masked]>; [email-masked]; Core <[email-masked]>
Subject: Re: [core] YANG notification within CoMI

Hi Michel,

For the above to be actionable, we do have to have implementations that:
— on the big machines, can share enough communication state so they can take part in anycast-based load balancing, — on the small devices, can react to loss of an observation interest by redoing a call-home transaction.

Grüße, Carsten