Subject: [sfc] Proposal for congestion control in draft-farrel-sfc-convent

From: Adrian Farrel <[email-masked]>
Date: Mon, 12 Feb 2018 22:33:22 +0000

Hi Mirja and Martin,

Here is a proposal to address Mirja's Discuss. It is a new top-level section.

It is for discussion, so please tell me what it is missing and how we should work on it.

It makes sense (to me) for the WG to comment about this as well.

Thanks,
Adrian

====
6. Management and Congestion Control Considerations

The mechanisms described in this document allow SFC-aware nodes in an SFC network to generate additional packets.  While these packets are in the nature of in-band control packets and not intended to be sent frequently for any flow, there is still a risk that they might flood the network. For example, if an attempt is made to use this mechanism for "per-packet metadata" (Section 5.6) then this might double the number of packets in the network. Similarly, if this mechanism is used for a form of aliveness detection OAM that requires very frequent test messages, then the number of additional messages may be very high. Such additional messages risk causing congestion in the network.

The underlay network (that is the tunnels across the underlay between SFC nodes) will not distinguish between data-carrying packets and those packets with next protocol set to none. All packets will be treated the same and will need to fall within the capabilities of the underlay network to process and forward packets.

Nodes in the SFC overlay network will need to perform special processing on the additional packets according to their roles and according to the application for the metadata. For example, an SFF will likely only have to forward per-SFC metadata, while an SF will need to extract it and process it as it would if the metadata was carried in a packet with user data. On the other hand, metadata might also be used to cause actions at all nodes (see Sections 5.3, 5.4 and 5.5) and could increase the processing load.

In view of these potential issues, all implementations SHOULD implement rate limits on the generation of SFC packets with next protocol set to none. Furthermore, these rate limits SHOULD be configurable and applied per SFC and per application so that one application on one SFC does not encumber a different application on a different SFC. When an implementation finds that it is unable to generate or send a packet it SHOULD increment a counter that is accessible by the operator, and MAY raise an alert (although such alerts SHOULD, themselves, be rate limited).

Additionally, an SFC node needs to protect itself against another node in the network not applying suitable rate limits. Therefore, implementations SHOULD apply incoming rate limits for SFC packets with next protocol set to none. Such rate limits SHOULD be application-aware, per SFC or interface, and SHOULD be configurable, but implementations MAY be more subtle if they are aware of internal processing loads and have access to queues/buffers. In any case, when an implementation drops a received packed because of these rate limits it SHOULD increment a counter that is accessible by the operator, and MAY raise an alert (although such alerts SHOULD, themselves, be rate limited).
====

---

From: "Mirja Kuehlewind (IETF)" <[email-masked]>
Date: Fri, 16 Feb 2018 12:40:39 +0100

Hi Adrian,

this text looks really good to me. Thanks!

I only comment I have is that is would be even better if you could give some more concrete guidance what the rate should be limited to. I don’t know if that makes sense for the SFC use case but in RFC8085 the guidance would be: „... not sending on average more than one UDP datagram per RTT to a destination“.

Please update the draft, depending on the wg feedback of course, and I’ll clear my discuss.

Mirja

> Am 12.02.2018 um 23:33 schrieb Adrian Farrel <[email-masked]>:
> 
> Hi Mirja and Martin,
>  
> Here is a proposal to address Mirja’s Discuss. It is a new top-level section.
>  
> The mechanisms described in this document allow SFC-aware nodes in an SFC network to generate additional packets.  While these packets are in the nature of in-band control packets and not intended to be sent frequently for any flow, there is still a risk that they might flood the network. For example, if an attempt is made to use this mechanism for “per-packet metadata” (Section 5.6) then this might double the number of packets in the network. Similarly, if this mechanism is used for a form of aliveness detection OAM that requires very frequent test messages, then the number of additional messages may be very high. Such additional messages risk causing congestion in the network.
>

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Fri, 16 Feb 2018 13:48:40 +0000

Hi, Adrian,

Thank you for producing this text and actively addressing all comments and review input on this document! Many thanks also because “soon” was very soon indeed!

Some of this text, however, seems pretty heavy-handed. Please find some comments inline.

—
Carlos Pignataro, [email-masked]

> On Feb 12, 2018, at 5:33 PM, Adrian Farrel <[email-masked]> wrote:
> 
> Hi Mirja and Martin,
>  
> The mechanisms described in this document allow SFC-aware nodes in an SFC network to generate additional packets.  While these packets are in the nature of in-band control packets

Are these “control” packets? These are packets with a specific characteristic (not with a specific function), and according to the Abstract, “this document illustrates some of the functions that may be achieved”. It does not say “These are control packets …"

> and not intended to be sent frequently for any flow

Might be useful to define flow (by way of a pointer at least), since it’s not used in RFC 8300.

> , there is still a risk that they might flood the network.

Flood the whole network?

> For example, if an attempt is made to use this mechanism for “per-packet metadata” (Section 5.6) then this might double the number of packets in the network. Similarly, if this mechanism is used for a form of aliveness detection OAM that requires very frequent test messages, then the number of additional messages may be very high. Such additional messages risk causing congestion in the network. 
>  
> The underlay network (that is the tunnels across the underlay between SFC nodes) will not

:s/will/may/g ?

> distinguish between data-carrying packets and those packets with next protocol set to none. All packets will be treated the same and will need to fall within the capabilities of the underlay network to process and forward packets.

:s/will/may/g ?

>  
> Nodes in the SFC overlay network will need to perform special processing on the additional packets according to their roles and according to the application for the metadata. For example, an SFF will likely only have to forward per-SFC metadata, while an SF

per-SFP instead of per-SFC?

>  will need to extract it and process it as it would if the metadata was carried in a packet with user data. On the other hand, metadata might also be used to cause actions at all nodes (see Sections 5.3, 5.4 and 5.5) and could increase the processing load.
>  
> In view of these potential issues, all implementations SHOULD implement rate limits on the generation of SFC packets with next protocol set to none. Furthermore, these rate limits SHOULD be configurable and applied per SFC and per application so that one application on one SFC does not encumber a different application on a different SFC.

I think “per-SFC” means “per-SFP”  throughout above. However, what is the mapping of applications? Not all nodes may have per-application visibility (or knowledge). 

> When an implementation finds that it is unable to generate or send a packet it SHOULD increment a counter that is accessible by the operator, and MAY raise an alert (although such alerts SHOULD, themselves, be rate limited).
>  

From an operational standpoint what are the defaults for these proposed configuration knobs?

Another important consideration: if the protocol-none packet is for a specific function, what are the implications to that application or function leveraging this mechanism? For example, if, as mentioned above, this is used for OAM, rate-liming without visibility to the OAM rate limiting or function can result in actually more congestion with false-negatives, oscillations, instability, etc.

I strongly recommend some per-use implications, and a SHOULD understand implications before blindly rate-limiting.

> Additionally, an SFC node needs to protect itself against another node in the network not applying suitable rate limits. Therefore, implementations SHOULD apply incoming rate limits for SFC packets with next protocol set to none.

This places an interesting requirement: it does not allow for incremental deployment. Not only protocol-none feature generating nodes need to be upgraded, but every node at once.

> Such rate limits SHOULD be application-aware, per SFC or interface,

This granularity seems beyond what might be possible. Per-SFC should be per-SFP. Not always there’s app-awareness. What’s an interface in this context?

> and SHOULD be configurable, but implementations MAY be more subtle if they are aware of internal processing loads and have access to queues/buffers. In any case, when an implementation drops a received packed because of these rate limits it SHOULD increment a counter that is accessible by the operator, and MAY raise an alert (although such alerts SHOULD, themselves, be rate limited).
> ====
>  

Thanks again, Adrian!

Carlos.

>  
>  
> _______________________________________________
> sfc mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/sfc

---

From: "Adrian Farrel" <[email-masked]>
Date: Fri, 16 Feb 2018 21:31:32 -0000

Hi Carlos,

Thanks for the mail. It is important, IMHO, that this sort of change is reviewed by the WG and not just slipped in through secret agreement between the document editor and a Discussing AD.

> Some of this text, however, seems pretty heavy-handed.

My hand has always been heavy - it is the secret of good childcare.

> > The mechanisms described in this document allow SFC-aware nodes in an SFC
> network to generate additional packets.  While these packets are in the nature of
> in-band control packets
> 
> Are these “control” packets? These are packets with a specific characteristic (not
> with a specific function), and according to the Abstract, “this document illustrates
> some of the functions that may be achieved”. It does not say “These are control
> packets …"

Are we sure this is a question of meaningful semantics and not philology?

The text you quoted does not say that the packets are control packets. But they *do* have that nature.

Of course, we could take this opportunity to open the debate as to whether OAM packets are control plane, data plane, or management plane packets. That might be fun, but if life has taught me anything, it is that we would not actually be advancing the discussion of the Internet.

So, we don't need the commentary and can reduce to "These are not intended to be sent frequently for any flow, but there is  still a risk that they might flood the network."

> Might be useful to define flow (by way of a pointer at least), since it’s not used in

I'm going to refer you back to Section 5.2, but not put a pointer in the text. A reader who gets this far without having first read Section 5.2 is lost.

> > , there is still a risk that they might flood the network.

English usage?
A single packet will not, of course, flood the whole network.
But the whole (SFC) network might be flooded with such packets.

> > For example, if an attempt is made to use this mechanism for “per-packet
> metadata” (Section 5.6) then this might double the number of packets in the
> network. Similarly, if this mechanism is used for a form of aliveness detection
> OAM that requires very frequent test messages, then the number of additional
> messages may be very high. Such additional messages risk causing congestion in
> the network.
> >

Now we are really into English usage!
Did you mean "might" when you said "may", or did you mean to be proscriptive?

Anyway, I intended "will not". That is mainly because it "cannot" without a form of DPI that is, I suspect, beyond the powers of most routers today.

But it is more civilised to say what it will or will not do than it is to assert capability deficiencies :-)

> > distinguish between data-carrying packets and those packets with next
> protocol set to none. All packets will be treated the same and will need to fall
> within the capabilities of the underlay network to process and forward packets.
> 
> :s/will/may/g ?

Text as intended.
When underlays start to make forwarding decisions based on the payload and not the encapsulation, then networking is in a mess!

> > Nodes in the SFC overlay network will need to perform special processing on
> the additional packets according to their roles and according to the application for
> the metadata. For example, an SFF will likely only have to forward per-SFC
> metadata, while an SF
> 

Yes, per 5.1.

> >  will need to extract it and process it as it would if the metadata was carried in a
> packet with user data. On the other hand, metadata might also be used to cause
> actions at all nodes (see Sections 5.3, 5.4 and 5.5) and could increase the
> processing load.
> >
> > In view of these potential issues, all implementations SHOULD implement rate
> limits on the generation of SFC packets with next protocol set to none.
> Furthermore, these rate limits SHOULD be configurable and applied per SFC and
> per application so that one application on one SFC does not encumber a different
> application on a different SFC.
> 
> I think “per-SFC” means “per-SFP”  throughout above. However, what is the
> mapping of applications? Not all nodes may have per-application visibility (or
> knowledge).

Yes, as above.

> > When an implementation finds that it is unable to generate or send a packet it
> SHOULD increment a counter that is accessible by the operator, and MAY raise an
> alert (although such alerts SHOULD, themselves, be rate limited).
> 

I don't believe this text defines any configuration knobs.
I suppose you could make them so, but this text talks about implementation.

> Another important consideration: if the protocol-none packet is for a specific
> function, what are the implications to that application or function leveraging this
> mechanism? For example, if, as mentioned above, this is used for OAM, rate-
> liming without visibility to the OAM rate limiting or function can result in actually
> more congestion with false-negatives, oscillations, instability, etc.
> 
> I strongly recommend some per-use implications, and a SHOULD understand
> implications before blindly rate-limiting.

Erm, the sentence previous to the one you quoted says...

       Furthermore, these rate limits 
       SHOULD be configurable and applied per SFP and per application so that one application on 
       one SFP does not encumber a different application on a different SFP.

Change to ...
     encumber a different application on this or a different SFP.

> > Additionally, an SFC node needs to protect itself against another node in the
> network not applying suitable rate limits. Therefore, implementations SHOULD
> apply incoming rate limits for SFC packets with next protocol set to none.
> 
> This places an interesting requirement: it does not allow for incremental
> deployment. Not only protocol-none feature generating nodes need to be
> upgraded, but every node at once.

No, that would be crazy.
No one would do that, and I don't think anyone would read it like that.
Backward compatibility is what it is (see Section 4).

Besides, "SHOULD" is very useful :-)

> This granularity seems beyond what might be possible. Per-SFC should be per-
> SFP. Not always there’s app-awareness. What’s an interface in this context?

Obviously, this is different from packet generation, but surely the debate could center on where the rate limits are applied.

You specifically worried about "one size fits all" rate limits, and that is wise because you probably want to allow more packets for one use than for another.

OK to drop this to "MAY"?

Hope this helps,
Adrian

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Fri, 16 Feb 2018 21:59:42 +0000

Hi, Adrian,

> On Feb 16, 2018, at 4:31 PM, Adrian Farrel <[email-masked]> wrote:
> 
> Hi Carlos,
> 

Indeed. It is more than important and not a matter of opinion, IMHO. 

Thanks for the Cc to the list!

> My hand has always been heavy - it is the secret of good childcare.

:-(

Focusing again on the new “Congestion paragraph", I’d try to right-size it. Thanks for entertaining my feedback, see below.

> Are we sure this is a question of meaningful semantics and not philology?

Is this question an answer to my question or a new question on its own right? Are we discussing the comment or the discussion?

It really is quite simple, and I did not mean to hide any text in-between lines. The proposal says "are in the nature of in-band control packets” and I think that is too limiting. So...

> So, we don't need the commentary and can reduce to "These are not intended to be sent frequently for any flow, but there is  still a risk that they might flood the network.”

… that sounds great :-) 

> I'm going to refer you back to Section 5.2, but not put a pointer in the text. A reader who gets this far without having first read Section 5.2 is lost.

Fair enough.

> But the whole (SFC) network might be flooded with such packets.

Ah, I understood network to be *the* network (fabric all the way to layer zero), not the “SFC network”.

In any case, you are right, there is a risk that the network might get flooded. No concern.

> Anyway, I intended "will not". That is mainly because it "cannot" without a form of DPI that is, I suspect, beyond the powers of most routers today.

Thinking the the spec might survive more than today, and knowing DPI capabilities that do this easily, as well as other potential mechanisms (such as reflecting the protocol into some underlay field), I think there are cases in which “it will”, and hence “will not” is incorrect as a definitive statement.

I just intended some forward compatibility and wiggle room.

> When underlays start to make forwarding decisions based on the payload and not the encapsulation, then networking is in a mess!

But you are assuming that decisions are based on the payload. It is possible that the encapsulation reflects / copies / lets-bleed-in fields from the payload and the decision is based on the encapsulation. Or other creative ways. I think that being too narrow in specs can backfire.

>> per-SFP instead of per-SFC?

Thanks./

>> mapping of applications? Not all nodes may have per-application visibility (or

Thanks!

> I don't believe this text defines any configuration knobs.

I meant “these rate limits SHOULD be configurable”. I was thinking of RFC 5706, Appendix A.1, item #9, sub-item 1.

>     encumber a different application on this or a different SFP.

Thanks!

> Besides, "SHOULD" is very useful :-)

Ack.

> You specifically worried about "one size fits all" rate limits, and that is wise because you probably want to allow more packets for one use than for another.

You got it.

> 
> OK to drop this to "MAY”?

That would be perfect.

Many thanks again, Adrian!

Carlos.