Subject: Re: [Netconf] LC on subscribed-notifications-10

From: Kent Watsen <[email-masked]>
Date: Tue, 8 May 2018 20:04:26 +0000

Hi Eric,

I'm just noticing that the github repo for this draft also (same as restconf-notif) does not contain source files and scripts to compile the drafts.  Is this the case with all the yang-push++ drafts?

See below for more <KENT5>

Kent // contributor

On 4/26/18, 12:49 PM, "Eric Voit (evoit)" <[email-masked]<mailto:[email-masked]>> wrote:

Hi Kent,

All changes below are also in the latest in the pending v13 draft within:
https://github.com/netconf-wg/rfc5277bis/blob/master/draft-ietf-netconf-subscribed-notifications-13.txt<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_netconf-2Dwg_rfc5277bis_blob_master_draft-2Dietf-2Dnetconf-2Dsubscribed-2Dnotifications-2D13.txt&d=DwMGaQ&c=HAkYuh63rsuhr6Scbfh0UjBXeMK-ndb3voDTXcWzoCI&r=9zkP0xnJUvZGJ9EPoOH7Yhqn2gsBYaGTvjISlaJdcZo&m=pyf5B1nLn0f-7e_2ibrHycJ2En-BzzKhdNIWD42jpgs&s=zuc13Lg3JIr9bsMfPGYNGeV9Kedfi-uCEfHYVm00pZI&e=>

From: Kent Watsen, April 25, 2018 6:07 PM

[further trimming]

 <Eric4>  Ahhh.  I got it now.  The two reasons are:

·         Insufficient resources (e.g., CPU)

·         Unsupportable volume (i.e., a bandwidth constraint)

I adjusted the diagram to:
                      .........
                      : start :
                      :.......:
                          |
                 establish-subscription
                          |
                          |   .------modify-subscription-------.
                          v   v                                |
                    .-----------.                        .-----------.
         .--------. | receiver  |-insufficient CPU, b/w->| receiver  |
     modify-       '|  ACTIVE   |                        | SUSPENDED |
     subscription   |           |<---CPU, b/w sufficient-|           |
         ---------->'-----------'                        '-----------'
                          |                                    |
               delete/kill-subscription                   delete/kill-
                          |                               subscription
                          v                                    |
                      .........                                |
                      :  end  :<-------------------------------'
                      :.......:

With the supporting bullet items under the diagram:

·         A publisher may choose to suspend a subscription when there is insufficient CPU or bandwidth available to service the subscription. This is notified to a subscriber with a "subscription-suspended" state change notification.

·         A suspended subscription may be modified by the subscriber (for example in an attempt to use fewer resources).  Successful modification returns the subscription to an active state.

·         Even without a "modify-subscription" request, a publisher may return a subscription to the active state should the resource constraints clear.  This is announced to the subscriber via the "subscription-resumed" subscription state change notification.

<KENT4> yes, this is it.  Just a couple nits, can you widen the diagram a couple characters to give more dashes (-) around the "insufficient" line?

<Eric5> Done.

Also, in the 3rd bullet, maybe replace "resource constraints clear" to "resource constraints become sufficient again" so that it binds with the words in the diagram?

<Eric5> Done.  I also inserted changes matching your “insufficient CPU, b/w” request into the Configured receivers Section 2.5.1.

<snip/>

<Eric3>  Added the descriptive paragraph requested in the middle of the three paragraphs below...

It is possible to place a start time on a configured subscription.  This enables streaming of logged information immediately after restart.

Replay of events records created since restart can be quite useful.  This allows event records generated before transport connectivity was supportable by a publisher to be passed to a receiver.  In addition, event records logged before restart are not sent.  This avoids the potential for accidental event record duplication.  Such duplication might otherwise be likely as a configured subscription’s identifier before and after the reboot is the same, and there may be not be evidence to a receiver that a restart has occurred.  By establishing restart as the earliest potential time for event records to be included in notification messages, a well-understood timeframe for replay is defined.

Therefore, when configured replay subscription receivers first become ACTIVE, buffered event records (if any) will be sent immediately after the "subscription-started" notification.  And the leading event record sent will be the first event record subsequent to the latest of four different times: the "replay-log-creation-time", "replay-log-aged-time", "replay-start-time", or the most recent publisher boot time.

<Kent3> Hmmm, I'm having a negative reaction to the "event records logged before restart are not sent" bit.  I know what you are trying to do, but I worry that this behavior might drop important logs, perhaps to the advantage of an adversary.  Note that some devices implement an <edit-config> with a restart.  Maybe the solution should require publishers to maintain a per configured-subscription awareness of (roughly) which log was sent last?   - and notify the receiver when a restart has occurred, or when the replaying of events occurs, so that they can be aware that there might be some duplicates?

<Eric4>  The current solution guarantees no duplicates, and also informs the receiver of each new “start-time”.  This allows the receiver to attempt to reconstruct any gaps from the last event previously pushed, should the choose to attempt such reconstruction.   As a dynamic subscription has no such boundary constraints on replay and boot time, all a subsequent dynamic subscription needs to do is to request the events between the last received event previously received from that configured subscription and the new replay-start-time.

<Kent4> So, the receiver is informed of each new "start-time" via the "subscription-started" control message, and then MUST do a short-lived dynamic subscription to scoop-up any possibly-missed logs, for which there may be none?   If we choose to keep this behavior, the draft should say this more clearly, perhaps in the Security Considerations section…

<Eric5> Added the following text as the last paragraph in the Implementation Considerations Section...

For configured replay subscriptions, the receiver is protected from duplicated events being pushed after a publisher is rebooted.  However it is possible that a receiver might want to acquire event records which failed to be delivered just prior to the reboot. Delivering these event records be accomplished by leveraging the “eventTime” from the last event record received prior to the receipt of a “subscription-started” state change notification.  With this “eventTime” and  the “replay-start-time” from the “subscription-started” notification, an independent dynamic subscription can be established which retrieves any event records which may have been generated but not sent to the receiver.

 <KENT5> Is this really limited to reboots, of any restarting of a configured subscription?  Next, I view this as a Security Consideration, since nefarious things can happen when a device reboots and note that a DoS could extend the gap to hours or days.  I think that this draft (the text above) is watering down the issue.  In my view, this is a huge consideration, along the lines of a receiver really MUST always try to use a dynamic subscription to fill in any gaps.  It almost begs the question for why configured subscriptions are supported at all.

<Eric4>  Note that this solution acts identically for loss of events when the platform *doesn’t* reboot, and events are just lost due to some overflow.  See the Section 2.5.2 text:
   “However if events are lost (rather than just delayed) due to replay buffer overflow, a new "subscription-started" must be sent.  This new "subscription-started" indicates an event record discontinuity.”
I.e., this way the receiver doesn’t have to do forensics to determine and attempt to determine the cause of a transient loss of events on a publisher.

<Kent4> okay, but note that this section refers to Section 2.4.2.1 (not 2.5.2).   I understand what you mean, but I think more text is needed to convey it to readers…

<Eric5> Added the sentence:

The most recent publisher boot time ensures that duplicate event records are not replayed from a previous time the publisher was booted.

In any case, tracking the last event sent to each receiver will be a pretty hard requirement to meet during a publisher crash.  Things are simpler to just let the receiver attempt a reconstruction should they need to.

<Kent4> this I agree with, but I really don't like the fact that receiver MUST do a short-lived dynamic subscription to scoop-up any possibly-missed logs, for which there may be none.  Perhaps we could add more values into the "subscription-started" notification message that would enable to receiver to make a local determination if such a dynamic subscription would be  helpful?

<Eric5> I recommend against providing extra objects/reasons in the “subscription-started” at this time.  Publishers might not want to advertise a reboot, and they might not want to advertise why there was loss in event continuity.   All that should matter to a receiver is that such a discontinuity existed, and they have a way to try to fill event the gap should they care.  If the need for more data and the cause of the discontinuity turns out to be required, we can always augment here with future objects.

<KENT5> first, I'm still not 100% sure if this is just a reboot problem, or any time the subscription is restarted/resumed.  Next, can you explain what info could be provided that might allow a client to make a local decision as to whether a dynamic subscription is needed of not, and does the "replay-start-time" node provide it?

<Kent3> Going back to my original comment, the new paragraph helps, it certainly caught my attention regarding reboots wiping out the replay log buffer.

<Eric4>  There is no requirement that the reboot wipe outs out the buffer (the solution is agnostic to that).   The only requirement is that a configured subscription replay start no earlier than the last reboot time.

<Kent4> I'm glad to hear that the logs before restart aren't lost, just rather that there is no attempt to send them by default.   This wasn't all that obvious to me from what you wrote before.

<Eric4>  Tweaked a Section 2.4.2.1 sentence to say:

This document puts no restrictions on the size or form of the log, where it resides within the publisher, or when event record entries in the log are purged.

I suggest adding text that clarifies this, and details the need for a short-lived dynamic-subscription.

<Eric4> The tweak above, with the suggested text in the Implementation Considerations section above hopefully covers this.

<KENT5> the "purged" part helps, but whyis this information buried inside a section titled "Requesting a replay of event records"?

<snip/>

>   Re: the 6th paragraph, I'm surprised that requirements for transport-

>   bindings wasn't discussed before in its own section.  It seems like

>   a new thing here, that a receiver's transport might not be secure.

>   I'm okay with and support this, btw, as its sometimes better to

>   offload devices thru the use of a local collector node, for which

>   encryption may not be needed...

Agree with your comments.

<KENT> but where's the change?  Shouldn't this have been discussed

previously in the draft somewhere?

<Eric2> The vast majority of transport binding discussions are addressed in the transport document.  So I see this as guidance to a documenter of a transport document.  Perhaps that is unnecessary for this document, and the paragraph should be removed.  I would be fine with that.

<Kent2> wait, I don't think you can offload transport-requirements to the transport-binding documents.   I think that this document needs to define the requirements and the transport-binding documents then show how they adhere to them.   Does this make sense?

 <Eric3>With the varied transports of NETCONF, HTTP/RESTCONF, UDP, CoAP already in drafts my belief is that only a high level subset of transport requirements spanning the universe of potential transports can potentially be abstracted in this document.  The secure transport requirement is one such example, and that is a recommendation.  The Security Considerations section is a good place for that one.  Beyond the security recommendation there aren’t too many transport independent possibilities.   I did just added one new transport requirement to the very end of “Event Streams” section though (which perhaps wasn’t explicit enough elsewhere).  This requirement is:

“Event records MUST NOT be delivered to a receiver in a different order than they were placed onto an event stream.”

What other transport-independent transport requirements might there be which are not already documented?

Stepping back, I see the transport draft plus this drafts providing the aggregate set of requirements for a full solution.  And I had thought it would be up to the draft authors plus WGs to validate that the sum of the documents is sufficient.

<Kent3> unsure.  For example, RFC 6241has Section 2 (Transport Protocol Requirements) that the SSH and TLS binding drafts refer to.  It seems that this draft should have a similar section that highlights what MUST or MUST NOT be supported.  It could even include some additional text indicating that bindings MAY introduce additional requirements.

<Eric4> I re-read RFC6241 Section 2 a couple times.  There are a comparisons can be made from that document to a subset of requirements currently in this document’s security section.  But I don’t see anything missing on the MUST and MUST NOT side of things.   FYI: the specific requirements I am thinking of are:

   For both configured and dynamic subscriptions the publisher MUST

   authenticate and authorize a receiver via some transport level

   mechanism before sending any updates.

   A secure transport is highly recommended and the publisher MUST

   ensure that the receiver has sufficient authorization to perform the

   function they are requesting against the specific subset of content

   involved.

   With configured subscriptions, one or more publishers could be used

   to overwhelm a receiver.  Notification messages SHOULD NOT be sent to

   any receiver which does not support this specification.  Receivers

   that do not want notification messages need only terminate or refuse

   any transport sessions from the publisher.

That is about it for common stuff.  Considering the wide variety of potential transports, and ubiquity for the need of stream transports, I am simply not aware of any more common requirements.  If you need me to,  I can extract these three requirements, and put this under a separate transport requirements section.   But this seems excessive, especially as we have transport specific documents with eyes on them from the WG.  But if really do want this, I will place these into a new, separate section; and I will add your text: “bindings MAY introduce additional requirements.”

<Kent4> yes, this is what I'm thinking is needed, even if just for these 3 requirements + a statement that each transport MAY impose additional limitations (not so much a "requirement" as a "fact of life using said transport" I think)

<Eric5>  Move to a new section just before Security Considerations.   Added the last sentence:

Additional transport requirements will be dictated by the choice of transport used with a subscription. For an example of such requirements with NETCONF transport, see [I-D.draft-ietf-netconf-netconf-event-notifications].

<KENT5> This new section "Transport Considerations" seems to be a mashup of requirements, security considerations, and recommendations and, as such, is a bit confusing.  I recommend ensuring the right info is in the right sections.  It only mentions the need for one-way authentication, is that correct?  The first paragraph's use of the word "authorize" confuses me, especially in context of the 2nd paragraph, what would the "transport" authorize?  BTW, maybe this section needs to more clearly specify that it only regards the transport for dedicated notification channels (not the transport for NC or RC).

---

From: "Eric Voit (evoit)" <[email-masked]>
Date: Wed, 9 May 2018 21:39:46 +0000

Hi Kent,

See <Eric6>.  Current version can be seen at:
https://github.com/netconf-wg/rfc5277bis/blob/master/draft-ietf-netconf-subscribed-notifications-13.txt

From: Kent Watsen, May 8, 2018 4:04 PM

Hi Eric,

<Eric6> I will git post source for each draft every time I do an official IETF revision posting.

Kent // contributor

Hi Kent,

[further trimming]

 <KENT5> Is this really limited to reboots, of any restarting of a configured subscription?

<Eric6> The protection from event replication is just for reboots – this is because the configured replay start time might duplicate events already sent from a previous boot.

But the mechanism to retrieve lost events is common between reboots and restarts.  I.e., using this solution, a receiver never needs to know a reboot occurred.  It just knows that a subscription restart occurred, and there might be a gap in events which should be retrieved via a dynamic subscription.

By having the receiver follow the same process for both reboots and restarted subscriptions, it simplifies recovery procedures.

Next, I view this as a Security Consideration, since nefarious things can happen when a device reboots and note that a DoS could extend the gap to hours or days.  I think that this draft (the text above) is watering down the issue.  In my view, this is a huge consideration, along the lines of a receiver really MUST always try to use a dynamic subscription to fill in any gaps.

<Eric6> I agree this is a best practice, but a MUST would require telemetry receivers to have to support dynamic subscriptions.  I can see IoT receiver implementations where this wouldn’t be likely.

To cover your desire to highlight this in security, I have added the following requirement to the Security Considerations Section:

When a configured receiver gets a new “subscription-started” message for a known subscription where it is already consuming events, the receiver SHOULD retrieve any event records generated since the last event record was received.  This can be accomplish by establishing a separate dynamic replay subscription with the same filtering criteria with the publisher.

It almost begs the question for why configured subscriptions are supported at all.

<Eric6> Because there have been lots of industry requests.  And lots of implementations with non-IETF solutions.  Personally I prefer dynamic subscriptions.

<KENT5> first, I'm still not 100% sure if this is just a reboot problem, or any time the subscription is restarted/resumed.

<Eric6> Per above: retrieving missing event records is not a reboot specific problem.  But unintentionally replicating event records is reboot specific.  (Otherwise the configured replay-start-time would drive a repeat of everything on each and every reboot.)

Next, can you explain what info could be provided that might allow a client to make a local decision as to whether a dynamic subscription is needed of not, and does the "replay-start-time" node provide it?

<Eric6> The local decision is easy.   As a configured receiver, I have just received a new subscription-started state change notification for a subscription I am tracking.  If I *need* to know about any events which occurred since the last event record I received for that subscription-id, then create a separate replay to cover that time gap.

<Eric6>   This is the second paragraph of the first section which discussed replay.  It is the sentence after the optional feature of replay is introduced.

<snip/>

   involved.

<KENT5> This new section "Transport Considerations" seems to be a mashup of requirements, security considerations, and recommendations and, as such, is a bit confusing.  I recommend ensuring the right info is in the right sections.

<Eric6> Look at <Kent4>.  My attempt was to do exactly what you requested.

Per the above, there simply are not many common transport requirements.    If you want me to split them up, that is fine.   If you want me to house them in “transport considerations” that is fine too.   I think that collapsing everything back to Security Considerations is the cleanest approach.    But a consistent target would be help us close this questions.   If you have cycles to propose a specific distribution, that would be great.

It only mentions the need for one-way authentication, is that correct?

<Eric6>: One way authentication is needed before pushing events from the perspective of the publisher.   The receiver certainly might want to verify the publisher.  Mechanisms to accomplish that can be transport dependent.  If you want to add a simple statement “A receiver MUST authenticate a publisher”, that can be done as generic guidance.

The first paragraph's use of the word "authorize" confuses me, especially in context of the 2nd paragraph, what would the "transport" authorize?

<Eric6>  Tweaked the wording to:

For both configured and dynamic subscriptions the publisher MUST authenticate a receiver via some transport level mechanism before sending any event records for which they are authorized to see.

BTW, maybe this section needs to more clearly specify that it only regards the transport for dedicated notification channels (not the transport for NC or RC).

<Eric6>  Added the sentence at the beginning of the section:

This section provides requirements which must be supportable via any transport technology supporting this solution.

/Eric6

---

From: Kent Watsen <[email-masked]>
Date: Tue, 15 May 2018 00:00:28 +0000

Hi Eric,

Now <Kent6>

Kent

On 5/9/18, 5:39 PM, "Eric Voit (evoit)" <[email-masked]<mailto:[email-masked]>> wrote:

Hi Kent,

See <Eric6>.  Current version can be seen at:
https://github.com/netconf-wg/rfc5277bis/blob/master/draft-ietf-netconf-subscribed-notifications-13.txt<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_netconf-2Dwg_rfc5277bis_blob_master_draft-2Dietf-2Dnetconf-2Dsubscribed-2Dnotifications-2D13.txt&d=DwMGaQ&c=HAkYuh63rsuhr6Scbfh0UjBXeMK-ndb3voDTXcWzoCI&r=9zkP0xnJUvZGJ9EPoOH7Yhqn2gsBYaGTvjISlaJdcZo&m=RZ_qNzuDRX0D4rDeJJ8fe76NaiVfDiQLNf2edHaFvnI&s=cjTkob7LnjltmMqskx_dCrJN1h-qlRRBc2KUblzNNGg&e=>

[further trimming]

 <KENT5> Is this really limited to reboots, or any restarting of a configured subscription?

<Kent6> So, in the case of a suspend/resume, the device picks up where it left off?   Or is it also just starting with "current"?  (strangely, I'm having trouble finding this answer in -13).  If it picks up where it left off, does this mean that the device has index of some sort and, if so, why can't that state be persisted in the case of a graceful shutdown/restart?

<Kent6> I'm unsure about the "same process" (mentioned in my previous comment).

<Kent6> Good point, but I don't see in the YANG module establish-subscription being optional to implement.  Is that an oversight?

<Kent6> that's helpful.  Perhaps tack onto the end of the last sentence something like ", assuming the publisher supports the ['dynamic' and] 'replay' feature[s]"?

<Kent6> I can see the appeal of configured subscriptions, but if each receiver of a configured subscription SHOULD always do a dynamic subscription on every publisher reboot, it is already being coded to support dynamic subscriptions, at which point it seems that maybe it should've just been implemented to support dynamic subscriptions from the start (on top of the publisher-initiated call-home transport, of course).   Can you say some more about these industry requests and, in particular, if the requests are still strong even when made aware of the need to also do dynamic subscriptions?

<snip/>

<Kent6> okay, I think I got it this time.  Having a *configurable* replay-start-time is so confusing.  Is it really worth having?

<Kent6> it's not "if I need to know" so much as "I *know* that I'm missing some events because X and Y and, therefore I will create a dynamic subscription otherwise, since I know that I didn't miss any events, I will not create a dynamic subscription."   I'm trying to figure out what X and Y might be…

<Kent6> I understand how it fits into this section, but it seems like it should be in a section called something like "replay log", since it's equally applicable for configured subscriptions (and s2.4 is about dynamic subscriptions).  Of course, if we remove replay from configured subscriptions, then leaving it here makes sense…

<snip/>

   involved.

<Kent6> sorry, I wasn't clear before.  Requirements are having MUST and MAY.  Security Considerations are more SHOULD and RECOMMENDED.  I'm seeing a mix in this section.  I think that all the SHOULD and RECOMMENDED statements should be moved to the Security Considerations section (assuming they are indeed Security related).

<Kent5?>It only mentions the need for one-way authentication, is that correct?

<Kent6> Yes, exactly, if it is true.   Perhaps have the two statements and then say something about "mutual authentication" so that it is crystal clear.

<Kent5?> The first paragraph's use of the word "authorize" confuses me, especially in context of the 2nd paragraph, what would the "transport" authorize?

<Kent6> better, thx.

<Kent6> perhaps s/this solution/the solution presented in this document/???

---

From: "Eric Voit (evoit)" <[email-masked]>
Date: Tue, 15 May 2018 15:54:37 +0000

Hi Kent,

Now <Eric7>

Eric

From: Kent Watsen, May 14, 2018 8:00 PM

Hi Eric,

Now <Kent6>

Kent

Hi Kent,

[further trimming]

<Eric7> It would be *very* difficult to persist on the publisher which event actually was received last by a receiver.  However the receiver can make that determination, as it does maintain that state.  Look for my proposal below on how to handle this with a new leaf in “subscription-started” “replay-previous-event-time”

<Eric7> It is the same process.  Note that below, the “replay-previous-event-time” tweak allows a receiver to determine if there are events missed.  And with the tweak, the fact a reboot has happened is still not explicitly exposed.

<Eric7> This is ok.  It is totally fine for a publisher to support dynamic subscriptions.  But a receiver need not.  Perhaps a lightweight IoT client just will just be a configured subscription receiver.

<Eric7>  Added
“, assuming the publisher supports the "replay" feature.”
As dynamic isn’t an optional feature.

<Eric7>  Here is an industry request:  stream event record entries being placed into security logs (e.g., Integrity Measurement Architecture audit events).   These events are only relevant starting at boot time.  Losing events means a system cannot be validated.  And days after a boot, streaming all the events since boot would result in too many events.   So in this environment, doing a dynamic subscription to fill the gap is acceptable.

<snip/>

<Eric7>   Yes it is worth having.

(a) In many environments, reboot is very infrequent.  Without configurable start time, an operator setting up a configured subscription would not have the ability to designate what to send.  It could only send the full log (at whatever size).

(b) on-publisher security or troubleshooting diagnostics might identify a breach or some event where streaming recent historical event records is a MUST.  As a result, it might want to stream a subset of event records off a box going back in time to potential events which might have been evidence or contributing factors.

<Eric7>  Right now the evidence a configured receiver has is:

X = A subscription #3333 exists, the last event record came in at 2:01:55UTC.  I never got a “subscription-terminated” or “subscription-concluded”.

Y = a new “subscription-started” arrives.

Thinking about how to accomplish what you are asking, I do think there might be a possibility.   We could place a new object “replay-previous-event-time” in the subscription-started notification:

       +---n subscription-started {configured}?

          +--ro (target)

             +--:(stream)

            +--ro replay-start-time?       yang:date-and-time        {replay}?

            +--ro replay-previous-event-time?       yang:date-and-time        {replay}?

What this object would do is to provide the timestamp of the event record just prior to the “replay-start-time” in the event buffer (should such a previous event exist).   If this timestamp doesn’t match the last event a receiver got (e.g., 2:01:55UTC), then a dynamic subscription could be established to fill the information gap.

<Eric7> Per above, there is lots of value in configured replay.

<snip/>

<snip7>

<Eric7>
Renamed section to “Transport Requirements”

Moved the SHOULD NOT requirement back to the security considerations section.

Moved the “One subscription identifier can be used for two or more receivers” statement back to the security considerations section.

<Eric7>  Now Says:

For both configured and dynamic subscriptions the publisher MUST authenticate a receiver via some transport level mechanism before sending any event records for which they are authorized to see.  In addition, the receiver MUST authenticate the publisher at the transport level.  The result is mutual authentication between the two.

<Kent6> better, thx.

<Eric7>   Tweaked the section lead-off to:

This section provides requirements for any subscribed notification transport supporting the solution presented in this document.

<Eric7>

Change made.

---

From: Kent Watsen <[email-masked]>
Date: Thu, 24 May 2018 00:44:15 +0000

Hi Eric,

Now <Kent7>.    BTW, I wish we switched to ASCII a while back…
Also, I removed highlighting because it was getting confusing.

Kent

<snip/>

<Kent7> good point about what was *received* as opposed to what was *sent*, as they may differ, but then would the publisher be able to say the last event it sent?  As I recall your proposal from before, we the receiver *always* needs a dynamic subscription to fill in any *potentially* missing gaps.  My goal is to try to change the "always" to a "sometimes" by giving the receiver enough info to know there is a gap, or perhaps just know when there wasn't a gap but, in either case, it becomes less than "always".

<Kent7> Sorry, I didn't mean I didn't know if the process was the same between the two interactions, but what the "same process" is.   Nonetheless, what you write above ("allows a receiver to determine if there are events missed") is music to my ears, and I see now that you've just now added the "replay-previous-event-time” (that wasn't clear to me at first, I had to look at -13 to be sure).  I'll look at it more closely in the section where it is introduced below.  Regarding your last sentence, I take it that it not being explicitly exposed is a good thing, right?

<Kent7> I think you misunderstood me.  Using your IoT example, even though a device (or the entire IoT space) only uses configured subscriptions, the current module doesn't enable a server to not support dynamic subscriptions.   For constrained devices, having to implement something never used could be a problem…

<Eric7>  Added

<Kent7> okay, that's expected, but see my previous Kent7 comment, I think that we might want dynamic to be optional.

<Eric7>  Here is an industry request:  stream event record entries being placed into security logs (e.g., Integrity Measurement Architecture audit events).   These events are only relevant starting at boot time.  Losing events means a system cannot be validated.  And days after a boot, streaming all the events since boot would result in too many events.   So, in this environment, doing a dynamic subscription to fill the gap is acceptable.

<Kent7> meh

<snip/>

<Kent7> Let me come at this another way.  Assume we drop all support for *configurable* replay-start-time.  As such, configured subscriptions always start with the next-generated event (no replay at all).   This covers most use-cases, right?   For those receivers that really wanted the older logs, can't they just do a dynamic subscription to collect them, same as we've been discussing above?

          +--ro (target)

             +--:(stream)

<Kent7> Right, something like that.  Do you think this does it, or are there a bunch of disclaimers around its effectiveness?

<snip/>

<Kent7> Yes, but see again my challenge to that assertion.  I'm leaving this here in case the decision is flipped…

<snip/>

<Eric7>

<Kent7> sounds good

<Eric7>  Now Says:

<Kent7> perfect, very clear now!

<Kent6> better, thx.

<Kent7> okay

<Eric7>

Change made.

<Kent7> good!

---

From: "Eric Voit (evoit)" <[email-masked]>
Date: Thu, 24 May 2018 17:18:23 +0000

And <Eric8>.

Comments below reflected within:
https://github.com/netconf-wg/rfc5277bis/blob/master/draft-ietf-netconf-subscribed-notifications-13.txt

From: Kent Watsen, May 23, 2018 8:44 PM

Hi Eric,

Kent

<snip/>

<Eric8> Yes, with the proposal now embodied in the latest draft version linked above, it is now a “sometimes”.

In the “subscription-started” notification, the controller is passing the timestamp of the immediately prior event stored in its replay buffer.  Whether the publisher tried and failed to send this immediately prior event a receiver or not doesn’t actually matter.  All that matters is that that receiver can sees whether this time matches the its last received event record.  If it does not, then a dynamic replay can be initiated.

<Eric8>  Yes, a publisher not being absolutely forced to expose a reboot is good.

<Eric8>  The requirement is that a publisher must support a dynamic subscription.  There is no requirement for that on a configured receiver.   To clarify this, I have tweaked early “Configured Subscriptions” section text to say:

On the publisher, supporting configured subscriptions is optional and advertised using the "configured" feature. On a configured receiver, support for dynamic subscriptions is optional except where replaying missed event records is required.

<Eric7>  Added

<Eric8> Per above, receiver support for dynamic is what is optional.

<Kent7> meh

<snip/>

<Eric8> Some reasons this might not always be practical:

(a) IoT devices just might want to passively listen to event streams of Telemetry.  (I.e., this would force configured receivers to support dynamic subscriptions.)

(b) This forces complexity onto applications which only ever need to track what has happened since boot.  (E.g., per above, continuous Integrity Measurement Architecture (IMA) boot log streaming and evaluation.)

(c) Publisher access permissions for who can use the establish-subscription RPC might have to be expanded to include lots of configured receivers.  This might open up a vector to control plane DDoS.  Right now the access permissions would just have to allow the receiver read access to the event records.

(d) A publisher may choose to firewall classes of receivers (or locations of receivers) into a listen-only mode without the ability to establish subscriptions.

          +--ro (target)

             +--:(stream)

<eric8>  I know of no disclaimers.

/Eric8

<snip/>

<snip/>

---

From: Kent Watsen <[email-masked]>
Date: Thu, 31 May 2018 18:11:22 +0000

Please look for <Kent8> below.

 <snip/>

<Kent7> good point about what was *received* as opposed to what was *sent*, as they may differ, but then would the publisher be able to say the last event it sent?  As I recall your proposal from before, the receiver *always* needs a dynamic subscription to fill in any *potentially* missing gaps.  My goal is to try to change the "always" to a "sometimes" by giving the receiver enough info to know there is a gap, or perhaps just know when there wasn't a gap but, in either case, it becomes less than "always".

 <Kent8> okay, I accept this resolution.

<Kent8> thanks.

<Kent8> I understand that supporting dynamic subscriptions is currently a requirement.  I am challenging that requirement.  Why is it a requirement?  Does it have to be a requirement?  What is an IoT device only wants to support configured subscriptions and having code to support dynamic is wasting space?    FWIW, I realize that not supporting dynamic subscriptions also means that it would be impossible to filling in gaps introduced by a reboot, but maybe that's a decision that the vendor can/should make for themselves?

<Kent8> Separately, "configured receiver" is not a defined term, though I see it appearing eight times in the -13 draft.  FWIW, there are no instances of "configured publisher".   Firstly, I'm not entirely sure what a "configured receiver" is, I think it's suppose to mean "a receiver of a configured subscription", yes?  Next, I'm unsure if the fact that the subscription was configured vs dynamic is important/relevant for these eight instances; if not, then s/configured receiver/receiver/, else if it is relevant/important, then maybe s/configured receiver/receiver of a configured subscription/?

<Eric7>  Added

<Kent8> I understand that this is what's written, but I'm challenging that premise (see my previous comment on this)

<Kent7> meh

<snip/>

<Kent8> This response seems to address the "can't they just do a dynamic subscription" aspect of my comment, but doesn't really address the "why is it important" (I paraphrase) part.  My contention is that the concept of a *configurable* replay-start-time seems confusing and of low value.   I acknowledge that there is some value, but it seems like the value is limited to a one-time start-up optimization that can be alternatively addressed by a dynamic subscription to fetch earlier events (assuming it's allowed, per your points b-d).   Additionally, FWIW, I've never seen such a feature implemented before, and logging mechanisms have been around for decades, so this makes me think that this is something that probably isn't worth having.

          +--ro (target)

             +--:(stream)

<Kent8> excellent, I think this sub-thread is now resolved, thanks.

/Eric8

<snip/>

<snip/>

---

From: "Eric Voit (evoit)" <[email-masked]>
Date: Fri, 1 Jun 2018 00:24:45 +0000

Hi Kent

Look for <Eric9>

From: Kent Watsen, May 31, 2018 2:11 PM

 <snip/>

<snip>

<Kent8> I understand that supporting dynamic subscriptions is currently a requirement.  I am challenging that requirement.  Why is it a requirement?  Does it have to be a requirement?

What is an IoT device only wants to support configured subscriptions and having code to support dynamic is wasting space?    FWIW, I realize that not supporting dynamic subscriptions also means that it would be impossible to filling in gaps introduced by a reboot, but maybe that's a decision that the vendor can/should make for themselves?

<Eric9> In RFC-5277, all you have is dynamic subscriptions.  So support for that older spec by definition makes dynamic subscriptions mandatory.  Beyond that, newer specifications like RFC-7923 as well as sections of other documents like RFC-7921, section 7.6 identify dynamic subscriptions as mandatory for a subscription service.  So at least some use cases exist where such dynamic support is mandatory.

With your IoT publisher use case above you are asserting that dynamic subscriptions are not needed for configured subscription only publishers – i.e., there are a class of publishers which have been driven by use cases not considered by the documents referenced above.  So who has documented the need configured subscription only publishers?   I can’t point to such documentation (beyond IoT case above).  Is such a possibility worth slowing down this spec?     In the end making the fix for this specification which you seem to want is itself really quite trivial: we can make both dynamic and configured subscriptions optional.  The reason I have been resisting it is that this solution (a) leads to more complexity for implementers as yet another feature would have to be advertised as optional, (b) this waters down the mandatory capabilities support of the YANG module, and (c) we would need to include some a constraint that at least one of the two optional features needs to be supported.  Also for (c) AFAIK, features don’t support the application of such constraints, so it would have to be done in the feature descriptions themselves.

I guess the text above is a long way of saying that if you assert the optional dynamic subscription is mandatory to progress the document, I will make the change.  But the change will impose complexity costs which to me are hard to justify.

<eric9> Made the change to “receiver of a configured subscription”

<Eric7>  Added

<Eric9> This comment will then fate-share the resolution from the previous comment.

<Kent7> meh

<snip/>

<Eric9> As you point out, the why "can't they just do a dynamic subscription" is covered, and we shouldn’t always assume away (b)-(d) as they can matter in some scenarios.  So if we want to support the use case of streaming log entries made after boot, but before the transport session is available, the only alternative I see is to have a configured replay-flag rather than a configuring a start-time.  Are you ok with a flag instead?  Or do you have an alternative suggestion?

In terms of using this configured replay capability, Cisco’s Integrity Verification application

https://www.cisco.com/c/dam/en/us/td/docs/cloud-systems-management/application-policy-infrastructure-controller-enterprise-module/1-5-x/integrity_verification/user-guide/Cisco_Integrity_Verification_Application_APIC-EM_User_Guide_1_5_0_x.pdf

does do a shell access event log fetch of the full log after boot, and then just does incremental fetch the deltas of the log (based on log line numbers).  This application is interested in configured subscriptions subsequent to boot for this purpose.  So such incremental streaming of portions of syslog after boot seems like a typical/common need to me.

/Eric9

<snip>

/Eric8

<snip/>

<snip/>