Subject: [netconf] Magnus Westerlund's Discuss on draft-ietf-netconf-subscribed-notifications-25: (with DISCUSS and COMMENT)

From: Magnus Westerlund via Datatracker <[email-masked]>
Date: Thu, 02 May 2019 05:24:46 -0700

Magnus Westerlund has entered the following ballot position for
draft-ietf-netconf-subscribed-notifications-25: Discuss

When responding, please keep the subject line intact and reply to all
email addresses included in the To and CC lines. (Feel free to cut this
introductory paragraph, however.)

Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
for more information about IESG DISCUSS and COMMENT positions.

The document, along with other ballot positions, can be found here:
https://datatracker.ietf.org/doc/draft-ietf-netconf-subscribed-notifications/

----------------------------------------------------------------------
DISCUSS:
----------------------------------------------------------------------

My focus when reviewing this document was from a perspective of how to handle
overload. I have a hard time understanding how this will actually work,
especially in a fashion that preservers goodput and ensure what is considered
the most important subscriptions are delivered. Not having good undertanding
into netconf and restconf don't hesitate to call out likely missunderstanding
by me and provide clarification and pointers.

A) The QoS and priority sending mechanism discussed in 2.3 and furhter defined
by the YANG model.

I do want to raise the usage of the DSCP code point to a discuss. As the DSCP
defines different PHB and relative priorities in the router queues a DSCP value
does not provide the publisher any information about priority. I get the
feeling from the text that this may be intended. If the only intention is to
have the transport perform differential treatment in the network between the
publisher and the receiver there are still more considerations are needed.
First of all I think these sentence needs a total rewrite:

   If the publisher supports the "dscp" feature, then a subscription
   with a "dscp" leaf MUST result in a corresponding [RFC2474] DSCP
   marking being placed within the IP header of any resulting
   notification messages and subscription state change notifications.
   Where TCP is used, a publisher which supports the "dscp" feature
   SHOULD ensure that a subscription's notification messages are
   returned within a single TCP transport session where all traffic
   shares the subscription's "dscp" leaf value.

I think one need to put a requriement on the transport to use a transport that
utilize the DSCP marking on its traffic. Which for the current set of
connection oriented transport protocols, TCP, SCTP, and QUIC all currently only
support using a single DSCP per connection. Implying multiple transport
protocol connections using a particular transport to enable this mapping.

A.2 Queuing model of a publisher.
With the DSCP and the Weight and dependency model I think it is important to
clarify the model of the queueing in the publisher. So is the intention that
several subscriptions with different weights and possibly dependencies have
their individual queues that goes into a scheduler? To avoid complex queue
interactions on this level I think there need to be seperate scheduler
instances per DSCP. I would also note that Dependency mechanism can't ensure
that a dependent stream arrive at receiver after the identified dependency if
they are on different DSCP. In fact if one would have HTTP/3 (over QUIC) we may
not even guarantee it within a single connection and same DSCP due to packet
loss impact. To me this model and what relationship one need to consider here
need to be clarified. I think RFC 7540 Section 5.3.1 is an excellent indication
of just the importance of considering what is in the same dependency tree and
what it means to have different weighting. To conclude I think this needs a
model description and clearer definition and possibly requirements towards the
transport. Espceially if you actually need an in-order delivery requirement?

B. The unpredictability of the circuit breaker overload mechanism.

My description of the overload handling in this document is that it is a
circuit breaker based mechanism that can blow a fuse on subscriptions that it
fails to honor in overload situations. What worries me deply is the total
unpredictability of this mechanism.

First, is it the intention to derive what subscriptions are least important
from the DSCP, weighting and Dependency parameters? If it is, I think that may
be a misstake as priority on what subscriptions are most important to retain
are not necessarily reflected in their QoS parameters.

Secondly, what are the values when a subscription are considered to be to heavy
or not be handled accordingly. Are there any parameter sets that actually
describe what SLA the subscriber expect that can be converted into timeout
timers or buffer depth thresholds to indicate that publisher side isn't
delivering these in time?

Third, I what I understand there are no any additional back pressure mechanism
between the receiver and the publisher than the transport protocols flow
control? So a receiver that is not keeping up processing the data it process
will not read out the data out of the flow controlled buffers in the receiver
and thus prevent the publisher to write to the transport conncetion, thus
causing the publisher to eventually trigger a suspension due to its queue build
up?

To my understanding the current mechanism places all subscriptions on the same
importance and with the same SLA. Thus likely causing short term overload
situations to cause subscription suspensions in random subscriptions. Is the WG
fine with this type of randomness occuring and expecting that normally there
will be such amount of overprovisioning that being able to indicate priority
and SLA are overkill?

At a minimal a change of this sentence in Section 2.5.1 is needed:

  This could
   be for reasons of an unexpected but sustained increase in an event
   stream's event records, degraded CPU capacity, a more complex
   referenced filter, or other higher priority subscriptions which have
   usurped resources.

As it states that subscriptions has priorities without reference to a mechanism
that provides that priority.

C. 2.4.5.  Killing a Dynamic Subscription

   The "kill-subscription" operation permits an operator to end a
   dynamic subscription which is not associated with the transport
   session used for the RPC.  A publisher MUST terminate any dynamic
   subscription identified by the "id" parameter in the RPC request, if
   such a subscription exists.

Can someone please clarify the use case for this functionality. To my
understanding it allows another receiver given authority to kill the
subscription being delivered to another receiver. Based on the otherwise rather
strict that all manipulations of dynamic subscriptions happens from the
transport session context that established it I want to better understand the
use case it appears out place.

D. The Requirements on a transport supporting Configured Subscriptions
Reading through Section 2.5 I arrived at a number of questions. I tried to
understand what the requirements are for the transport that supports Configured
Subscriptions. I did note that neither draft-ietf-netconf-restconf-notif-13 nor
draft-ietf-netconf-netconf-event-notifications-17 supports configured
subscriptions. Thus, there appear no template for a solution either, or does
there exist another document that is being worked on defining such a transport?

Questions that arose for me related to Configured Susbription Transport where
the following: 1. Can Transport Session be initiated in both direction. RFC
8071 provides a solution for Publisher to Receiver initiation. It is unclear if
all parts are in place to have a receiver to publisher initiated transport to
be bound to the subscription. 2. What is "name" really? It appears to be
defined as an empty container. Despite that it appears to have requirements on
being both a security identity as well as network address. 3. In Figure 9,
which is stated to be for the receiver. What information does the receiver use
to determine the transition (d)? And what does it do in this step. Related to
Discuss part B). 4. RFC 8071 appears to lack any considerations for how often
and how many times it attempts to connect to the receiver. So applying that
mechanism appears to require some usage guidance to avoid causing overload
situations or DoS potential by misconfiguring network devices with this
soltution to dial out frequently to a target.

As the transport solution requirements are not detail it is actually hard to
determine if there are short comings in the description in Section 2.5 and the
corresponding YANG model. Is it an reasonable request to document these
requirements?

----------------------------------------------------------------------
COMMENT:
----------------------------------------------------------------------

1. Section 2.3: DSCP domains
I think the text could benefit from pointing out that the subscriber actually
needs to know what the DSCP values represents in the diffserv domain of the
publisher. As these could be different, they also create an interesting
problems for transports of how they establish a transport connection that uses
a particular DSCP, as the receiver if initiating need to know the local DSCP
value that corresponds to the behavior in the publisher's domain.

2. In general I think there are more than one description that are fuzzy about
if it describes a publisher or receiver action/requirement.

---

From: "Eric Voit (evoit)" <[email-masked]>
Date: Mon, 6 May 2019 21:19:10 +0000

Hi Magnus,

Thanks very much for your comments.   Thoughts in-line...

> From: Magnus Westerlund, May 2, 2019 8:25 AM
> 
> When responding, please keep the subject line intact and reply to all email
> addresses included in the To and CC lines. (Feel free to cut this introductory
> paragraph, however.)
> 
> 
> My focus when reviewing this document was from a perspective of how to
> handle overload. 

You are absolutely correct to focus on overload.  Mitigating different dimensions of the overload risk has been a design goal since this effort’s inception.  And this is the reason a variety of QoS mechanisms were included in the document.

> I have a hard time understanding how this will actually work,
> into netconf and restconf don't hesitate to call out likely missunderstanding by
> me and provide clarification and pointers.

Few of the mechanisms are specific to either RESTCONF or NETCONF.  For completeness reasons, let me summarize the overload mechanisms available...

1. The publisher is allowed to decline a dynamic subscription.  One of these reasons is that the incremental traffic generated by a particular incremental subscription will be too high.  There are errors back to the subscriber indicating this condition exist.
2. A publisher can suspend any subscription for capacity reasons, and a receiver must be able to gracefully accept this suspension. 
3. Much like with HTTP2 streams, higher priority subscriptions intended for a particular receiver can be dequeued first from a publisher. 
4. Much like with HTTP2 streams, proportional subscription dequeuing intended for a particular receiver can be performed a publisher.
5. DSCP markings can be placed on subscribed data.
6. Mechanisms for detecting and reacting to different types of subscribed data loss have been embedded.
7. Methods have been included to ensure a configured receiver “ok’s” information push before subscribed data is sent. (This should reduce one DDoS vector.) 
8. Keep alive mechanisms are established for different transport types, so that subscribed data isn’t being sent when the is no receiver listening.

Mechanisms (3) & (4) will likely be seen only with HTTP2 based transports.*   This is because (as documented within draft-ietf-netconf-restconf-notif section 4), these capabilities are to integrated directly HTTP2 RFC-7450 sections 5.3.1 & 5.3.2.   

* One background/review note...  Earlier versions of draft-ietf-netconf-subscribed-notifications included specific parallels to RFC-7450 when describing (3) & (4).  However WG members wanted to abstract away what they felt were HTTP2 specific references. This is despite the fact that what was being referenced was the desired functional behavior rather than anything HTTP2 specific.   I can understand the WG reviewers' concern.  This is already a long document, and a reader who only cares about NETCONF hopefully won't need to wade through complex issues which they are unlikely to worry about in deployment.

> does not provide the publisher any information about priority. I get the feeling
> from the text that this may be intended. If the only intention is to have the
> transport perform differential treatment in the network between the publisher
> and the receiver 

Yes, this is the case.

> there are still more considerations are needed.
> utilize the DSCP marking on its traffic. 

I believe you are asking for a  the publisher respect the DSCP markings for traffic egressing an interface on a publisher.  Yes this requirement was assumed, and can be explicitly added.

> Which for the current set of connection
> oriented transport protocols, TCP, SCTP, and QUIC all currently only support
> using a single DSCP per connection. Implying multiple transport protocol
> connections using a particular transport to enable this mapping.

Yes fully agree, a connection would be needed per DSCP.     Is your objection with the text above the words "SHOULD ensure" rather than "MUST ensure"?    If yes, I can ask the WG objects to whether this requirement should be a MUST.

> clarify the model of the queueing in the publisher. So is the intention that several
> subscriptions with different weights and possibly dependencies have their
> individual queues that goes into a scheduler?

As you know, queuing models are non-trivial.   For that reason we wanted to 100% adopt the functional behavior RFC-7540 Section 5.3.1 and 5.3.2 without having to re-document/mirror that content at a higher level of the network stack.   We are a hoping that a reader of network subscriptions can look to well documented, implemented HTTP2 behaviors as applicable.  Providing an intermediate layer of functional description could easily result in some mis-alignments from what is intended.

> To avoid complex queue
> interactions on this level I think there need to be seperate scheduler instances
> per DSCP. I would also note that Dependency mechanism can't ensure that a
> dependent stream arrive at receiver after the identified dependency if they are
> on different DSCP. In fact if one would have HTTP/3 (over QUIC) we may not
> even guarantee it within a single connection and same DSCP due to packet loss
> impact. To me this model and what relationship one need to consider here need
> to be clarified. I think RFC 7540 Section 5.3.1 is an excellent indication of just the
> importance of considering what is in the same dependency tree and what it
> means to have different weighting. To conclude I think this needs a model
> description and clearer definition and possibly requirements towards the

I think we have the same technical objective in mind.   And it is absolutely the desire to have identical behavior to RFC-7540 Section 5.3.1 and 5.3.2.    For the current set of documents before the IESG, we include within draft-ietf-netconf-restconf-notif Section 4, a 1:1 mapping between draft-ietf-netconf-subscribed-notifications and RFC-7540.    

We are hoping that the transport documents like draft-ietf-netconf-restconf-notif can be the place where such supporting documentation and mappings occur.  

> My description of the overload handling in this document is that it is a circuit
> breaker based mechanism that can blow a fuse on subscriptions that it fails to
> honor in overload situations. What worries me deply is the total unpredictability
> of this mechanism.

At the beginning of this email, there are eight methods of overflow handling listed.  We optimized these eight mechanisms for different failure scenarios and congestion conditions.   Our goal was to depend on existing mechanisms/technologies wherever possible.  Reuse of existing mechanisms should reduce at least some of the unpredictability.

> First, is it the intention to derive what subscriptions are least important from the
> DSCP, weighting and Dependency parameters? If it is, I think that may be a
> misstake as priority on what subscriptions are most important to retain are not
> necessarily reflected in their QoS parameters.

At this point the document does not attempt to define "important".  All that is done is to provide guidance relating to some elements of network transport.   If there is a dimension of "importance" which an implementer would like to layer onto this solution, it could be done.  For example, "importance" could applied in areas such as what subscriptions should be suspended  in case of CPU exhaust.   However guidance on such enhancements are not included in this document.

> timers or buffer depth thresholds to indicate that publisher side isn't delivering
> these in time?

There is no guidance on this provided in the document.   As equipment types, subscription volumes, available memory, will vary between solutions, this will take a while to dimension properly.  I can imagine someday that we might have something like "Erlang for subscriptions" much like we used Erlangs in the old telephony network to dimension call handling capabilities of phone switches.  But we are a long way from that. 
 
> control? So a receiver that is not keeping up processing the data it process will
> not read out the data out of the flow controlled buffers in the receiver and thus
> prevent the publisher to write to the transport conncetion, thus causing the
> publisher to eventually trigger a suspension due to its queue build up?

There is nothing beyond transport flow control.  We thought about it initially, but we were not ready to pick up even more complexity than we already had.

> To my understanding the current mechanism places all subscriptions on the
> same importance and with the same SLA. Thus likely causing short term overload
> will be such amount of overprovisioning that being able to indicate priority and
> SLA are overkill?

Yes.   We needed a starting point.  And there are technical solutions which can be layered on top.   But what we have now took many years to finalize, and should be a big enough technical jump considering our current knowledge.

>    referenced filter, or other higher priority subscriptions which have

I have removed "higher priority".

> understanding it allows another receiver given authority to kill the subscription
> being delivered to another receiver. Based on the otherwise rather strict that all
> manipulations of dynamic subscriptions happens from the transport session
> context that established it I want to better understand the use case it appears
> out place.

A network operator needs a very secure mechanism to end a dynamic subscription in progress which it sees as harmful.   The operator cannot do this via configuration operations, as the dynamic subscription is not configured (and therefore cannot be deleted in the configuration datastore).  

> understand what the requirements are for the transport that supports
> Configured Subscriptions. I did note that neither draft-ietf-netconf-restconf-
> notif-13 nor

This is the case.   Originally both of those documents *did* include configured subscriptions.  However within the WG there was a decision not to progress configured subscriptions yet.  One reason is that other YANG model drafts moving in the NETCONF WG were seen as pre-requisites for securely creating transport sessions via call home for the configured subscriptions.  As a result, support for configured subscriptions was extracted from those transport documents.   It is expected that that updated versions of just those transport documents will be driven when the YANG models complete.

> 8071 provides a solution for Publisher to Receiver initiation. It is unclear if all
> parts are in place to have a receiver to publisher initiated transport to be bound
> to the subscription. 

This will be up to a specific transport draft to make this determination.  

To see what might be viable for NETCONF, check out the earlier version of the document at:
https://datatracker.ietf.org/doc/draft-ietf-netconf-netconf-event-notifications/10/ 

This was seen as a complete version of such a solution.  However the WG wanted a YANG model for session parameters discussed in Section 5.2.

>2. What is "name" really? It appears to be defined as an
> empty container. Despite that it appears to have requirements on being both a
> security identity as well as network address. 

You are correct.  In previous versions of this draft, a receiver was identified by the combination of address + port.  However due to the YANG doctor reviews, and call home discussions referenced above, the WG wasn't ready to finalize this YANG structure.  The compromise was the current structure, plus the example YANG model of Appendix A to show how this might be built out.

> 3. In Figure 9, which is stated to be
> for the receiver. What information does the receiver use to determine the
> transition (d)? And what does it do in this step. Related to Discuss part B). 

This determination is implementation specific.

> 4. RFC
> 8071 appears to lack any considerations for how often and how many times it
> attempts to connect to the receiver. So applying that mechanism appears to
> require some usage guidance to avoid causing overload situations or DoS
> potential by misconfiguring network devices with this soltution to dial out
> frequently to a target.
> 

The requirements were documented for both NETCONF and RESTCONF.   However these requirements were removed when the WG decided to wait until there was a YANG model for RFC-8071 ready to go to the IESG.   For a preview on what these requirements might look like, I refer you  to Section 5.2 of 
https://datatracker.ietf.org/doc/draft-ietf-netconf-netconf-event-notifications/10/

> I think the text could benefit from pointing out that the subscriber actually needs
> to know what the DSCP values represents in the diffserv domain of the publisher.
> As these could be different, they also create an interesting problems for
> transports of how they establish a transport connection that uses a particular
> DSCP, as the receiver if initiating need to know the local DSCP value that
> corresponds to the behavior in the publisher's domain.

This makes sense.  

I have added in Section 5.3 Transport Requirements a new entry which states:

"A subscriber which includes a "dscp" leaf within an "establish-subscription" request will need to understand and consider what the corresponding DSCP value represents within the domain of the publisher."

Let me know if this is not sufficient.

> 2. In general I think there are more than one description that are fuzzy about if it
> describes a publisher or receiver action/requirement.

It would be great if you have some specifics.   The authors and previous reviewers likely have looked at this often enough where things look obvious which perhaps are not.

Thanks again for the review,
Eric

---

From: Magnus Westerlund <[email-masked]>
Date: Tue, 7 May 2019 12:50:06 +0000

Hi Eric,

Thanks for the answers. I think we are making progress here, see below.

What I see is that there need to be some text changes in the QoS parts.

Regarding the priority I get the impression that it is not the WG's intention to clarify this. However, I would wish that the document is a bit more explicit about the lack of any interface to affect the publisher's action when it comes to dropping subscriptions. Yes there is this paragraph in Section 1.3:

   A publisher MAY terminate a dynamic subscription at any time.
   Similarly, it MAY decide to temporarily suspend the sending of
   notification messages for any dynamic subscription, or for one or
   more receivers of a configured subscription.  Such termination or
   suspension is driven by internal considerations of the publisher.

So I guess I have to drop this aspect now when the explicit reference to priority based actions are removed.

On 2019-05-06 23:19, Eric Voit (evoit) wrote:

Hi Magnus,

To me the big weakness of this mechanism are actually in the API between the receiver and the publisher. The current API defined by this document does not include any information that allow the receiver to express its actual view on priority for a subscription. It has some other tools that has completely different meanings namely:

- Transport level DSCP expressing PHB and routing priority

- Weight: Proportional bandwidth distribution

- Dependency: Deliver this only after this other subscription

None of this expresses anything related to the priority or importance to maintain a particular subscription, nor does the interface carry any information regarding what delivery requirements the receiver have on the subscription. Thus, the behavior in overload situation is going to be totally random.

So if the WG is okay with this being the case I will hold my nose. However, I think the fact that random implementation dependent things will happen in overload should be explicitly stated in the document.

I did a brief look at the transports. They did not answer my questions when I wrote this discuss. Also, to my understanding the HTTP/2 priority mechanism is that is far from easy to use and also intended to distribute resources in an ongoing transmission. That doesn't actually match the higher level need of determining which subscription are more important than another. Doesn't a 5 level priority value cover a lot of the missing ground here?

Yes, this is the case.

Please do.

If you think you should use RFC 2119 language, yes then I reuqest a MUST. However, I think reformulating this so state it as a fact that different DSCP code points requires different transport connections, unless a transport supporting multiple DSCP simultanous are used (No standardized option that has reliable object or stream semantics exist to my knowledge).

Hmm, that is fine, accept that you added DSCP to the mix, where RFC 7540 is all within a single TCP connection. Thus, to avoid complexities I think you at least need to be explicit that one can only perform Dependency and Weight operations between subscriptions that share DSCP, and that they become independent queues.

To avoid complex queue

I still think that this document needs to have that sentence in their stating that Dependency and Weight applies only per subscription sharing DSCP value. Because these values are define on this document level, not on the mapping level that the individual transport exists on. And this is the interface the WG have defined for this. And as it currently stands you appear to have something that lacks a clear meaning.

Still it is both implied and explicitly stated in one place.

To be clear what I mean with SLA here is to express some boundaries on when the subscribed to information will be received by the receiver compared to when it come to existence on the publisher side. However, I do hear you that there is nothing defined and nothing in the pipeline.

Ok, which just contribute to what I call random things happen at overload.

  This could

Thanks.

Thanks, that clarification resolves my question mark. So lets close this issue without any actions.

Look at the responses below I see that we are in one of these situation where things are deliberately left open. I will simply assume that this is done in a way that supports those future definitions and clarifications. So lets close this issue without any actions.

4. RFC

Thanks, so this is clearly something to have in mind when the WG do get to specify a solution for this. So I don't see a need for any actions here.

This makes sense.

Sorry, didn't take note about those confusion.

Cheers

Magnus Westerlund

----------------------------------------------------------------------
Network Architecture & Protocols, Ericsson Research
----------------------------------------------------------------------
Ericsson AB                 | Phone  +46 10 7148287
Torshamnsgatan 23           | Mobile +46 73 0949079
SE-164 80 Stockholm, Sweden | mailto: [email-masked]<mailto:[email-masked]>
----------------------------------------------------------------------

---

From: "Eric Voit (evoit)" <[email-masked]>
Date: Tue, 7 May 2019 19:22:25 +0000

Hi Magnus,

Thanks again for your thorough review on the overload aspect.   It is appreciated.  Some thoughts/proposals back to you in-line..

From: Magnus Westerlund, May 7, 2019 8:50 AM

Hi Eric,

<eric>  I agree it would be great to have a structure of subscription handling precedence on the publisher.  However there are not any solutions that I know of being discussed today.   Building a model here would be hard, and would likely take quite a bit of time/effort.  It is primarily the time cost which is the biggest consideration at this point.  Since NETCONF got started on these subscription specifications, Google's GNMI alternative has been introduced.    GNMI has fewer overload controls, yet has gained industry traction.    Based on this, it doesn't seem like standardized handling of subscription precedence on a publisher is a market necessity yet.

Hi Magnus,

paragraph, however.)

DISCUSS:

handle overload.

You are absolutely correct to focus on overload.  Mitigating different dimensions of the overload risk has been a design goal since this effort's inception.  And this is the reason a variety of QoS mechanisms were included in the document.

7. Methods have been included to ensure a configured receiver "ok's" information push before subscribed data is sent. (This should reduce one DDoS vector.)

8. Keep alive mechanisms are established for different transport types, so that subscribed data isn't being sent when the is no receiver listening.

<eric> I think the documents reflect the state-of-the-art across industry players.  Attempting to layer-in something new which hasn't been considered would be non-trivial.  But your intent should not be lost.  So to reflect your request above, how about I add the following statement at the very end of Section 2.3 QoS:

There are additional types over publisher capacity overload which this specification does not address within its scope. For example, the prioritization of which subscriptions have precedence when the publisher CPU is overloaded is not discussed.  As a result, implementation choices will need to be made to address such considerations.

Does this work for you?

<eric> Agree that the HTTP2 QoS mechanisms described do not address anything other than handling dequeuing congestion between a publisher and a receiver as part of an ongoing transmission.  This actually is very relevant problem.  Early implementations of telemetry for SDN have many network element publishers pushing data to a few central data collectors.

Beyond this, I don't think the working group has ever considered a multi-level subscription priority mechanism.  I believe the problem to be a very hard one, even if there are just 5 levels (or two levels).

by the YANG model.

and the receiver

Yes, this is the case.

Please do.

<eric> Section 2.3 QoS now says..

A publisher MUST respect the DSCP markings for subscription traffic egressing that publisher.

If you think you should use RFC 2119 language, yes then I request a MUST.  However, I think reformulating this so state it as a fact that different DSCP code points requires different transport connections, unless a transport supporting multiple DSCP simultanous are used (No standardized option that has reliable object or stream semantics exist to my knowledge).

<eric>  Adding a statement of fact here is not an issue.  And thinking about your comment regarding RFC2119 language, I don't think RFC2119 language is needed here.  So I have turned the "SHOULD" into "must".   This results in the following proposed text:

Different DSCP code points require different transport connections.  As a result where TCP is used, a publisher which supports the "dscp" feature must ensure that a subscription's notification messages are returned within a single TCP transport session where all traffic shares the subscription's "dscp" leaf value.

Does this work for you?

Where TCP is used, a publisher which supports the "dscp" feature must ensure that a subscription's notification messages are returned within a single TCP transport session where all traffic shares the subscription's "dscp" leaf value.

<eric> To cover this, are you ok with adding as a last sentence of Section 2.3:

"Dependency" and "weight" parameters will only be respected and enforced between subscriptions that share the same "dscp" leaf value.

To avoid complex queue

<eric> Hopefully the last suggested text covers this.

of this mechanism.

<eric> Hopefully the changes suggested above cover this concern.

these in time?

<eric> Understood.  We simply don't have anything.

<eric> agree

SLA are overkill?

  This could

   usurped resources.

Thanks.

out place.

notif-13 nor

to the subscription.

4. RFC

frequently to a target.

requirements?

COMMENT:

This makes sense.

<eric> Magnus, once again your comments are thoughtful and appreciated.  Thanks for taking such a good look at this document.

Eric

Cheers

Magnus Westerlund

---

From: Magnus Westerlund <[email-masked]>
Date: Wed, 8 May 2019 09:12:06 +0000

On 2019-05-07 21:22, Eric Voit (evoit) wrote:
Hi Magnus,

Hi Eric,

> <eric> I agree it would be great to have a structure of subscription > handling precedence on the publisher. However there are not any > solutions that I know of being discussed today. Building a model > here would be hard, and would likely take quite a bit of time/effort. > It is primarily the time cost which is the biggest consideration at > this point. Since NETCONF got started on these subscription > specifications, Google’s GNMI alternative has been introduced. > GNMI has fewer overload controls, yet has gained industry traction. > Based on this, it doesn’t seem like standardized handling of > subscription precedence on a publisher is a market necessity yet.

Understood. And I will most definitely not insist on a solution at this time. I was considering if there should be more word on that this is something suitable for future extension, but I guess it does not provide any good benefit.

Hi Magnus,

paragraph, however.)

DISCUSS:

handle overload.

<eric> I think the documents reflect the state-of-the-art across industry players.  Attempting to layer-in something new which hasn’t been considered would be non-trivial.  But your intent should not be lost.  So to reflect your request above, how about I add the following statement at the very end of Section 2.3 QoS:

Does this work for you?

Yes.

Ok

Beyond this, I don’t think the working group has ever considered a multi-level subscription priority mechanism.  I believe the problem to be a very hard one, even if there are just 5 levels (or two levels).

Ok. Just a suggestion and I agree that solution to this problem will have to be in a future extension, if ever.

by the YANG model.

and the receiver

Yes, this is the case.

Please do.

I think that work, but I like to see it in its full context.

<eric>  Adding a statement of fact here is not an issue.  And thinking about your comment regarding RFC2119 language, I don’t think RFC2119 language is needed here.  So I have turned the “SHOULD” into “must”.   This results in the following proposed text:

Does this work for you?

Yes.

“Dependency” and “weight” parameters will only be respected and enforced between subscriptions that share the same “dscp” leaf value.

Yes, I think that clarifies the issue.

To avoid complex queue

Yes, it does.

of this mechanism.

Yes, I think it is sufficiently well handled.

these in time?

<eric> Understood.  We simply don’t have anything.

<eric> agree

SLA are overkill?

  This could

   usurped resources.

Thanks.

out place.

notif-13 nor

to the subscription.

4. RFC

frequently to a target.

requirements?

COMMENT:

This makes sense.

Good, I will await the updated document, but I expect to be able to clear with the changes discussed here.

Cheers

Magnus Westerlund

---

From: "Eric Voit (evoit)" <[email-masked]>
Date: Wed, 8 May 2019 15:29:13 +0000

Hi Magnus,

All items discussed below have now been uploaded as part of:
draft-ietf-netconf-subscribed-notifications-26

Thanks again,
Eric

> From: Magnus Westerlund, May 8, 2019 5:12 AM
> 
> Thanks again for your thorough review on the overload aspect.   It is
> appreciated.  Some thoughts/proposals back to you in-line..
> 
> From: Magnus Westerlund, May 7, 2019 8:50 AM Hi Eric,
> 
> Regarding the priority I get the impression that it is not the WG's intention to
> clarify this. However, I would wish that the document is a bit more explicit about
> the lack of any interface to affect the publisher's action when it comes to
> dropping subscriptions. Yes there is this paragraph in Section 1.3:
> > <eric> I agree it would be great to have a structure of subscription > handling
> precedence on the publisher. However there are not any > solutions that I know
> of being discussed today. Building a model > here would be hard, and would
> likely take quite a bit of time/effort. > It is primarily the time cost which is the
> biggest consideration at > this point. Since NETCONF got started on these
> subscription > specifications, Google's GNMI alternative has been introduced. >
> GNMI has fewer overload controls, yet has gained industry traction. > Based on
> this, it doesn't seem like standardized handling of > subscription precedence on
> a publisher is a market necessity yet.
> Understood. And I will most definitely not insist on a solution at this time. I was
> considering if there should be more word on that this is something suitable for
> future extension, but I guess it does not provide any good benefit.
> 
> 
> 
> You are absolutely correct to focus on overload.  Mitigating different
> dimensions of the overload risk has been a design goal since this effort's
> inception.  And this is the reason a variety of QoS mechanisms were included in
> the document.
> 
> I have a hard time understanding how this will actually work, especially in a
> fashion that preservers goodput and ensure what is considered the most
> important subscriptions are delivered. Not having good undertanding into
> netconf and restconf don't hesitate to call out likely missunderstanding by me
> and provide clarification and pointers.
> 
> Few of the mechanisms are specific to either RESTCONF or NETCONF.  For
> completeness reasons, let me summarize the overload mechanisms available...
> 
> 1. The publisher is allowed to decline a dynamic subscription.  One of these
> reasons is that the incremental traffic generated by a particular incremental
> subscription will be too high.  There are errors back to the subscriber indicating
> this condition exist.
> 2. A publisher can suspend any subscription for capacity reasons, and a receiver
> must be able to gracefully accept this suspension.
> 3. Much like with HTTP2 streams, higher priority subscriptions intended for a
> particular receiver can be dequeued first from a publisher.
> 4. Much like with HTTP2 streams, proportional subscription dequeuing intended
> for a particular receiver can be performed a publisher.
> 7. Methods have been included to ensure a configured receiver "ok's"
> information push before subscribed data is sent. (This should reduce one DDoS
> vector.) 8. Keep alive mechanisms are established for different transport types,
> so that subscribed data isn't being sent when the is no receiver listening.
> 
> Mechanisms (3) & (4) will likely be seen only with HTTP2 based transports.*   This
> is because (as documented within draft-ietf-netconf-restconf-notif section 4),
> these capabilities are to integrated directly HTTP2 RFC-7450 sections 5.3.1 &
> 5.3.2. To me the big weakness of this mechanism are actually in the API between
> the receiver and the publisher. The current API defined by this document does
> not include any information that allow the receiver to express its actual view on
> priority for a subscription. It has some other tools that has completely different
> meanings namely:
> - Dependency: Deliver this only after this other subscription None of this
> expresses anything related to the priority or importance to maintain a particular
> subscription, nor does the interface carry any information regarding what
> delivery requirements the receiver have on the subscription. Thus, the behavior
> in overload situation is going to be totally random.
> So if the WG is okay with this being the case I will hold my nose. However, I think
> the fact that random implementation dependent things will happen in overload
> should be explicitly stated in the document.
> <eric> I think the documents reflect the state-of-the-art across industry
> players.  Attempting to layer-in something new which hasn't been considered
> would be non-trivial.  But your intent should not be lost.  So to reflect your
> request above, how about I add the following statement at the very end of
> Section 2.3 QoS:
> There are additional types over publisher capacity overload which this
> specification does not address within its scope. For example, the prioritization of
> which subscriptions have precedence when the publisher CPU is overloaded is
> not discussed.  As a result, implementation choices will need to be made to
> address such considerations.
> Does this work for you?
> Yes.
> 
> 
> * One background/review note...  Earlier versions of draft-ietf-netconf-
> subscribed-notifications included specific parallels to RFC-7450 when describing
> (3) & (4).  However WG members wanted to abstract away what they felt were
> HTTP2 specific references. This is despite the fact that what was being
> referenced was the desired functional behavior rather than anything HTTP2
> specific.   I can understand the WG reviewers' concern.  This is already a long
> document, and a reader who only cares about NETCONF hopefully won't need
> to wade through complex issues which they are unlikely to worry about in
> deployment.
> I did a brief look at the transports. They did not answer my questions when I
> wrote this discuss. Also, to my understanding the HTTP/2 priority mechanism is
> that is far from easy to use and also intended to distribute resources in an
> ongoing transmission. That doesn't actually match the higher level need of
> determining which subscription are more important than another. Doesn't a 5
> level priority value cover a lot of the missing ground here?
> <eric> Agree that the HTTP2 QoS mechanisms described do not address anything
> other than handling dequeuing congestion between a publisher and a receiver as
> part of an ongoing transmission.  This actually is very relevant problem.  Early
> implementations of telemetry for SDN have many network element publishers
> pushing data to a few central data collectors. Ok Beyond this, I don't think the
> working group has ever considered a multi-level subscription priority
> mechanism.  I believe the problem to be a very hard one, even if there are just 5
> levels (or two levels).
> I believe you are asking for a  the publisher respect the DSCP markings for traffic
> egressing an interface on a publisher.  Yes this requirement was assumed, and
> can be explicitly added.
> Please do.
> Yes fully agree, a connection would be needed per DSCP.     Is your objection
> with the text above the words "SHOULD ensure" rather than "MUST ensure"?    If
> yes, I can ask the WG objects to whether this requirement should be a MUST.
> 
> If you think you should use RFC 2119 language, yes then I request a
> MUST.  However, I think reformulating this so state it as a fact that different
> DSCP code points requires different transport connections, unless a transport
> supporting multiple DSCP simultanous are used (No standardized option that has
> reliable object or stream semantics exist to my knowledge).
> <eric>  Adding a statement of fact here is not an issue.  And thinking about your
> comment regarding RFC2119 language, I don't think RFC2119 language is
> needed here.  So I have turned the "SHOULD" into "must".   This results in the
> following proposed text:
> Different DSCP code points require different transport connections.  As a result
> where TCP is used, a publisher which supports the "dscp" feature must ensure
> that a subscription's notification messages are returned within a single TCP
> transport session where all traffic shares the subscription's "dscp" leaf value.
> Does this work for you?
> Yes.
> Where TCP is used, a publisher which supports the "dscp" feature must ensure
> that a subscription's notification messages are returned within a single TCP
> transport session where all traffic shares the subscription's "dscp" leaf value.
> 
> As you know, queuing models are non-trivial.   For that reason we wanted to
> 100% adopt the functional behavior RFC-7540 Section 5.3.1 and 5.3.2 without
> having to re-document/mirror that content at a higher level of the network
> stack.   We are a hoping that a reader of network subscriptions can look to well
> documented, implemented HTTP2 behaviors as applicable.  Providing an
> intermediate layer of functional description could easily result in some mis-
> alignments from what is intended.
> Hmm, that is fine, accept that you added DSCP to the mix, where RFC 7540 is all
> within a single TCP connection. Thus, to avoid complexities I think you at least
> need to be explicit that one can only perform Dependency and Weight
> operations between subscriptions that share DSCP, and that they become
> independent queues.
> "Dependency" and "weight" parameters will only be respected and enforced
> between subscriptions that share the same "dscp" leaf value.
> I think we have the same technical objective in mind.   And it is absolutely the
> desire to have identical behavior to RFC-7540 Section 5.3.1 and 5.3.2.    For the
> current set of documents before the IESG, we include within draft-ietf-netconf-
> restconf-notif Section 4, a 1:1 mapping between draft-ietf-netconf-subscribed-
> notifications and RFC-7540.
> 
> We are hoping that the transport documents like draft-ietf-netconf-restconf-
> notif can be the place where such supporting documentation and mappings
> occur. I still think that this document needs to have that sentence in their stating
> that Dependency and Weight applies only per subscription sharing DSCP value.
> Because these values are define on this document level, not on the mapping
> level that the individual transport exists on. And this is the interface the WG have
> defined for this. And as it currently stands you appear to have something that
> lacks a clear meaning.
> At the beginning of this email, there are eight methods of overflow handling
> listed.  We optimized these eight mechanisms for different failure scenarios and
> congestion conditions.   Our goal was to depend on existing
> mechanisms/technologies wherever possible.  Reuse of existing mechanisms
> should reduce at least some of the unpredictability.
> 
> At this point the document does not attempt to define "important".  All that is
> done is to provide guidance relating to some elements of network transport.   If
> there is a dimension of "importance" which an implementer would like to layer
> onto this solution, it could be done.  For example, "importance" could applied in
> areas such as what subscriptions should be suspended  in case of CPU
> exhaust.   However guidance on such enhancements are not included in this
> document.
> There is no guidance on this provided in the document.   As equipment types,
> subscription volumes, available memory, will vary between solutions, this will
> take a while to dimension properly.  I can imagine someday that we might have
> something like "Erlang for subscriptions" much like we used Erlangs in the old
> telephony network to dimension call handling capabilities of phone
> switches.  But we are a long way from that.
> To be clear what I mean with SLA here is to express some boundaries on when
> the subscribed to information will be received by the receiver compared to when
> it come to existence on the publisher side. However, I do hear you that there is
> nothing defined and nothing in the pipeline.
> There is nothing beyond transport flow control.  We thought about it initially,
> but we were not ready to pick up even more complexity than we already had.
> Yes.   We needed a starting point.  And there are technical solutions which can
> be layered on top.   But what we have now took many years to finalize, and
> should be a big enough technical jump considering our current knowledge.
> 
> A network operator needs a very secure mechanism to end a dynamic
> subscription in progress which it sees as harmful.   The operator cannot do this
> via configuration operations, as the dynamic subscription is not configured (and
> therefore cannot be deleted in the configuration datastore).
> This is the case.   Originally both of those documents *did* include configured
> subscriptions.  However within the WG there was a decision not to progress
> configured subscriptions yet.  One reason is that other YANG model drafts
> moving in the NETCONF WG were seen as pre-requisites for securely creating
> transport sessions via call home for the configured subscriptions.  As a result,
> support for configured subscriptions was extracted from those transport
> documents.   It is expected that that updated versions of just those transport
> documents will be driven when the YANG models complete.
> Look at the responses below I see that we are in one of these situation where
> things are deliberately left open. I will simply assume that this is done in a way
> that supports those future definitions and clarifications. So lets close this issue
> without any actions.
> 
> This was seen as a complete version of such a solution.  However the WG
> wanted a YANG model for session parameters discussed in Section 5.2.
> 
> You are correct.  In previous versions of this draft, a receiver was identified by
> the combination of address + port.  However due to the YANG doctor reviews,
> and call home discussions referenced above, the WG wasn't ready to finalize this
> YANG structure.  The compromise was the current structure, plus the example
> YANG model of Appendix A to show how this might be built out.
> 
> The requirements were documented for both NETCONF and
> RESTCONF.   However these requirements were removed when the WG decided
> to wait until there was a YANG model for RFC-8071 ready to go to the IESG.   For
> a preview on what these requirements might look like, I refer you  to Section 5.2
> of
> Thanks, so this is clearly something to have in mind when the WG do get to
> specify a solution for this. So I don't see a need for any actions here.
> 
> 
> "A subscriber which includes a "dscp" leaf within an "establish-subscription"
> request will need to understand and consider what the corresponding DSCP
> value represents within the domain of the publisher."
> 
> It would be great if you have some specifics.   The authors and previous
> reviewers likely have looked at this often enough where things look obvious
> which perhaps are not.
> <eric> Magnus, once again your comments are thoughtful and
> appreciated.  Thanks for taking such a good look at this document.
> SE-164 80 Stockholm, Sweden | mailto:
> mailto:[email-masked]