Subject: [tcpm] Fwd: New Version Notification for draft-kuehlewind-tcpm-ecn-fallback-00.txt

From: Brian Trammell <[email-masked]>
Date: Thu, 4 Jul 2013 14:14:17 +0200

Greetings, all,

We've posted a draft on ECN path probing and fallback, which we'd like to discuss at the TCPM meeting in Berlin. In a recent work [1], we found that though the ECN-readiness of popular webservers has significantly increased even in the last couple of years, there still exist paths on which attempts to use ECN after successful ECN negotiation will cause connection disruption.

This draft proposes an experimental approach to determine at runtime whether a path is usable, by sending segments after connection startup and ECN negotiation with the CE codepoint set, and disabling ECN if all probe segments were lost, on a per-flow or per-destination basis.

Experiments with an implementation of the approach within the Linux kernel are underway; we plan to be able to present initial results in Berlin.

Best regards,

Mirja and Brian

[1] Kuehlewind, M., Neuner, S., and Trammell, B., “On the state of ECN and TCP Options on the Internet”, in Proceedings of the the 2013 Passive and Active Measurement Conference, Hong Kong SAR, China, 19 March 2013.

Begin forwarded message:

> From: [email-masked]
> Subject: New Version Notification for draft-kuehlewind-tcpm-ecn-fallback-00.txt
> Date: 3 July 2013 17:40:32 GMT+02:00
> To: Mirja Kuehlewind <[email-masked]>, Brian Trammell <[email-masked]>
> 
> 
> A new version of I-D, draft-kuehlewind-tcpm-ecn-fallback-00.txt
> has been successfully submitted by Mirja Kuehlewind and posted to the
> IETF repository.
> 
> Filename:	 draft-kuehlewind-tcpm-ecn-fallback
> Revision:	 00
> Title:		 A Mechanism for ECN Path Probing and Fallback
> Creation date:	 2013-07-02
> Group:		 Individual Submission
> Number of pages: 5
> URL:             http://www.ietf.org/internet-drafts/draft-kuehlewind-tcpm-ecn-fallback-00.txt
> Status:          http://datatracker.ietf.org/doc/draft-kuehlewind-tcpm-ecn-fallback
> Htmlized:        http://tools.ietf.org/html/draft-kuehlewind-tcpm-ecn-fallback-00
> 
> 
> Abstract:
>   Explicit Congestion Notification (ECN) is a TCP/IP extension that is
>   widely implemented but hardly used due to the perceived unusablilty
>   of ECN on many paths through the Internet caused by ECN-ignorant
>   routers and middleboxes.  This document specifies an ECN probing and
>   fall-back mechanism in case ECN has be successfully negotiated
>   between two connection endpoints, but might not be usable on the
>   path.
> 
> 
> 
> 
> The IETF Secretariat

---

From: Michael Welzl <[email-masked]>
Date: Thu, 4 Jul 2013 15:29:05 +0200

Hi,

Without having read this draft yet, just from the description, I'd like to express that I applaud this approach. I have always thought that the TCP/IP suite is weak on "fallback" approaches - why do we have dismiss sooo many things once and for all, just because some middle-boxes or routers may play wrong with them? Why can't we simply *try* things, and fall back to a default behavior if something doesn't work?

ECN is one particular case where I was asking myself this question, so I'm happy to see this work appear - but I think that this method could probably be used to improve a lot of things at the transport layer.

Cheers,
Michael

On Jul 4, 2013, at 2:14 PM, Brian Trammell <[email-masked]> wrote:

> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Tue, 16 Jul 2013 12:55:30 +0000

Brian, Mirja,

Having read this draft (I think the rationale for not probing in the IW is a good one, but perhaps a little more text in the draft would do good - i.e. a short flow will be effectively non-reactive anyway, thus any ECN signal would be superfluous), I like it.

I have one (probably academic) point though:

(I would like to see a diagram visualizing the probing process).

In Step 4, the algo sends the next 3 segments after IW with "CE". In addition to the "first hop must not have experienced congestion" scenario, a middlebox might also clear CE if these three segments are pure ACKs, as per 3168, pure ACKs should not be marked ECN-capable... also, you expect an ACK for these. Thus you need to send 3 "data segments" not simply segments, right?

In Step 6, you mention an interaction with ECN Nonce... Perhaps having an explicit example for an ECN-Nonce enabled session would do well (which can detect the removal of the CE-flag in probing data segments by sending one ECT(1) data segment).

In Step 7, CWR should be sent with the segment (not necessarily data segment) ACKing any of the 1st up to the 3rd probing CE data segment. This could be 1, 2 or 3 segments (the text only hints on a single such segment).

Including the optional ECN Nonce probing (4th segment with ECT1; 4th ACK with NS=1) in the algorithm might be good...

Richard Scheffenegger

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Brian Trammell
> Sent: Donnerstag, 04. Juli 2013 14:14
> To: [email-masked]
> Subject: [tcpm] Fwd: New Version Notification for draft-kuehlewind-tcpm-
> ecn-fallback-00.txt
> 
> Greetings, all,
> 
> We've posted a draft on ECN path probing and fallback, which we'd like to
> discuss at the TCPM meeting in Berlin. In a recent work [1], we found that
> though the ECN-readiness of popular webservers has significantly increased
> even in the last couple of years, there still exist paths on which
> attempts to use ECN after successful ECN negotiation will cause connection
> disruption.
> 
> This draft proposes an experimental approach to determine at runtime
> whether a path is usable, by sending segments after connection startup and
> ECN negotiation with the CE codepoint set, and disabling ECN if all probe
> segments were lost, on a per-flow or per-destination basis.
> 
> Experiments with an implementation of the approach within the Linux kernel
> are underway; we plan to be able to present initial results in Berlin.
> 
> Best regards,
> 
> Mirja and Brian
> 
> [1] Kuehlewind, M., Neuner, S., and Trammell, B., "On the state of ECN and
> TCP Options on the Internet", in Proceedings of the the 2013 Passive and
> Active Measurement Conference, Hong Kong SAR, China, 19 March 2013.
> 
> Begin forwarded message:
> 
> > Subject: New Version Notification for
> > draft-kuehlewind-tcpm-ecn-fallback-00.txt

---

From: Brian Trammell <[email-masked]>
Date: Mon, 22 Jul 2013 17:03:19 +0200

hi Richard,

Thanks for the comments! Inline...

> (I would like to see a diagram visualizing the probing process).

Good suggestion; we'll add one in the next rev.

> In Step 4, the algo sends the next 3 segments after IW with "CE". In addition to the "first hop must not have experienced congestion" scenario, a middlebox might also clear CE if these three segmen…

Yep, this is what we meant, but we need to make this more clear.

> In Step 7, CWR should be sent with the segment (not necessarily data segment) ACKing any of the 1st up to the 3rd probing CE data segment. This could be 1, 2 or 3 segments (the text only hints on a…

hm, that wasn't the intention, but on review that could be clearer, as well. We'll make it so.

> Including the optional ECN Nonce probing (4th segment with ECT1; 4th ACK with NS=1) in the algorithm might be good...

Good suggestion; we were originally thinking of something a little more elaborate, and I do think we need to think about this a bit more, though, since an actual CE mark would obliterate the nonce. I know that's unlikely _now_, but one of the goals of the work is making it less so. :)

Cheers,

Brian

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Mon, 22 Jul 2013 16:58:27 +0000

Hi Brian,

> > Including the optional ECN Nonce probing (4th segment with ECT1; 4th ACK
> with NS=1) in the algorithm might be good...
> 
> Good suggestion; we were originally thinking of something a little more
> elaborate, and I do think we need to think about this a bit more, though,
> since an actual CE mark would obliterate the nonce. I know that's unlikely
> _now_, but one of the goals of the work is making it less so. :)

Indeed... You would want to send the ECT(1) first, and the artificial CEs afterwards; if there is an ACK,ECE for the segment with ECT(1), this would also prove the path is ECN-capable, right?

But I'd like to listen to your elaborate scheme :)

Richard Scheffenegger

> From: Brian Trammell [mailto:[email-masked]]
> Sent: Montag, 22. Juli 2013 17:03
> To: Scheffenegger, Richard
> Cc: [email-masked]; Mirja Kuehlewind
> Subject: Re: [tcpm] New Version Notification for draft-kuehlewind-tcpm-
> ecn-fallback-00.txt
> 
> hi Richard,
> 
> > Having read this draft (I think the rationale for not probing in the IW
> is a good one, but perhaps a little more text in the draft would do good -
> i.e. a short flow will be effectively non-reactive anyway, thus any ECN
> signal would be superfluous), I like it.
> >
> >
> > In Step 4, the algo sends the next 3 segments after IW with "CE". In
> addition to the "first hop must not have experienced congestion" scenario,
> a middlebox might also clear CE if these three segments are pure ACKs, as
> per 3168, pure ACKs should not be marked ECN-capable... also, you expect
> an ACK for these. Thus you need to send 3 "data segments" not simply
> segments, right?
> 
> > In Step 6, you mention an interaction with ECN Nonce... Perhaps having
> an explicit example for an ECN-Nonce enabled session would do well (which
> can detect the removal of the CE-flag in probing data segments by sending
> one ECT(1) data segment).
> >
> > In Step 7, CWR should be sent with the segment (not necessarily data
> segment) ACKing any of the 1st up to the 3rd probing CE data segment. This
> could be 1, 2 or 3 segments (the text only hints on a single such
> segment).
> 
> > Including the optional ECN Nonce probing (4th segment with ECT1; 4th ACK
> with NS=1) in the algorithm might be good...
> 
> Good suggestion; we were originally thinking of something a little more
> elaborate, and I do think we need to think about this a bit more, though,
> since an actual CE mark would obliterate the nonce. I know that's unlikely
> _now_, but one of the goals of the work is making it less so. :)
> 
> Cheers,
> 
> Brian
> 
> 
> >> From: [email-masked] [mailto:[email-masked]] On Behalf
> >> Of Brian Trammell
> >> Subject: [tcpm] Fwd: New Version Notification for
> >> draft-kuehlewind-tcpm- ecn-fallback-00.txt
> >>
> >> Greetings, all,
> >>
> >> We've posted a draft on ECN path probing and fallback, which we'd
> >> like to discuss at the TCPM meeting in Berlin. In a recent work [1],
> >> we found that though the ECN-readiness of popular webservers has
> >> significantly increased even in the last couple of years, there still
> >> exist paths on which attempts to use ECN after successful ECN
> >> negotiation will cause connection disruption.
> >>
> >> whether a path is usable, by sending segments after connection
> >> startup and ECN negotiation with the CE codepoint set, and disabling
> >> ECN if all probe segments were lost, on a per-flow or per-destination
> basis.
> >>
> >> Experiments with an implementation of the approach within the Linux
> >> kernel are underway; we plan to be able to present initial results in
> Berlin.
> >>
> >> Best regards,
> >>
> >> Mirja and Brian
> >>
> >> [1] Kuehlewind, M., Neuner, S., and Trammell, B., "On the state of
> >> ECN and TCP Options on the Internet", in Proceedings of the the 2013
> >> Passive and Active Measurement Conference, Hong Kong SAR, China, 19
> March 2013.
> >>
> >> Begin forwarded message:
> >>
> >>> has been successfully submitted by Mirja Kuehlewind and posted to
> >>> the IETF repository.
> >>>
> >>> Status:          http://datatracker.ietf.org/doc/draft-kuehlewind-
> tcpm-
> >> ecn-fallback
> >>>  Explicit Congestion Notification (ECN) is a TCP/IP extension that
> >>> is  widely implemented but hardly used due to the perceived
> >>> unusablilty  of ECN on many paths through the Internet caused by
> >>> ECN-ignorant  routers and middleboxes.  This document specifies an
> >>> ECN probing and  fall-back mechanism in case ECN has be successfully
> >>> negotiated  between two connection endpoints, but might not be
> >>> usable on the  path.
> >>>
> >>>
> >>>
> >>>
> >>> The IETF Secretariat
> >>

---

From: Brian Trammell <[email-masked]>
Date: Tue, 23 Jul 2013 18:20:15 +0200

hi Richard, all,

> Indeed... You would want to send the ECT(1) first, and the artificial CEs afterwards; if there is an ACK,ECE for the segment with ECT(1), this would also prove the path is ECN-capable, right?

Yep. Would have to make sure we do the right thing if that one ECT(1) packet gets _legitimately_ lost though.

> But I'd like to listen to your elaborate scheme :)

It's basically a generalization of what we submitted (and which covers the case above): the general idea is to send J ECT codepoints (checking the nonce sum) followed by K CE codepoints, falling back any time a problem is detected (as described in the draft). We thought about various combinations before settling on (J,K) = (0,3) for this rev of the draft. This admittedly makes the assumption that middleboxes that would cause connectivity problems when seeing ECT would also do so on CE. On the other hand, we want to keep J+K as small as possible: the more data segments you're sending to probe, the more time you're not getting the full benefit of ECN-signaled CC. 

It seems (J,K) = (1,3) would work for testing nonce functionality, though. It's probably worth thinking a bit about other values, too.

Cheers,

Brian

---

From: Michael Welzl <[email-masked]>
Date: Mon, 29 Jul 2013 11:22:23 +0200

Hi,

I also finally managed to really read this draft. As I said before, I like the idea a lot!

As for the actual algorithm, I'm not so convinced about it - here are a few comments:
- step 4: in addition to Richard's suggestion that these should be data segments, I'd also suggest to require there to be at least 3 more segment to be available in the sender buffer for sending. In case the probe packets are lost, it could be useful to have sent enough extra packets to be avoid an RTO (or at least reduce the risk for an RTO).
- step 6: "In this case, the sender may continue using ECN, because while it may not work for detecting congestion, the use of ECN does not negatively affect connectivity." => I'd rather not have it used if it's not working - if a router marks ECN to CE but the next downstream router clears the codepoint, we ignore congestion, and we'd like to avoid that, no?

The paragraph just after the 8 steps suggests to put ECN probing in the API. I think this is quite bad - if applications need to ask for this to happen, you need to update all applications to move ECN deployment ahead… but this kind of deadlock situation is what we're trying to solve here?  I don't understand why the necessary information about the number of data that are available to send "is only available at the higher layer", can't TCP look into its send buffer to know how much it has available? Surely that must be possible?

Finally, I don't think it's worth including nonce probing. As it stands now, the nonce is not terribly useful anyway, so why probe for it? Anyway nobody uses it AFAIK…

Cheers,
Michael

---

From: Brian Trammell <[email-masked]>
Date: Mon, 29 Jul 2013 13:53:40 +0200

hi Michael,

Thanks! Comments on comments inline...

On 29 Jul 2013, at 11:22, Michael Welzl <[email-masked]> wrote:

> - step 4: in addition to Richard's suggestion that these should be data segments, I'd also suggest to require there to be at least 3 more segment to be available in the sender buffer for sending.

This is a good point, yes. We're assuming that we have information about how long the flow will be, and we're not really specific about how we know that yet...

> In case the probe packets are lost, it could be useful to have sent enough extra packets to be avoid an RTO (or at least reduce the risk for an RTO).

...and requiring segments in the buffer would be a good way of doing this. We didn't want to get into buffer details, in on account of corner cases (in the timing of writes to the buffer) about which we didn't want to think about yet. But if we can ignore those, 

> - step 6: "In this case, the sender may continue using ECN, because while it may not work for detecting congestion, the use of ECN does not negatively affect connectivity." => I'd rather not have i…

The problem in this case is that we don't know if we have an ECN-aware middlebox that is clearing a CE from the end system, on the assumption that it is closer to the end-system than one would expect -- we did see some of this activity in the study detailed in the PAM paper. In this case the CE-clearing is arguably correct (or at least acceptable). 

> 
> The paragraph just after the 8 steps suggests to put ECN probing in the API. I think this is quite bad - if applications need to ask for this to happen, you need to update all applications to move ECN deployment ahead… but this kind of deadlock situation is what we're trying to solve here?

Hm, good point. This probably shouldn't be a per-connection; however, it should be per-endpoint configurable via sysctl or equivalent (as, indeed, ECN is on most end systems today)

>  I don't understand why the necessary information about the number of data that are available to send "is only available at the higher layer", can't TCP look into its send buffer to know how much it has available? Surely that must be possible?

Again, we didn't want to make assumptions about how applications write into the buffer, but yeah, requiring app changes at the API level is much, much worse. So good point, I think I'm convinced, at least, that an approach based on buffer occupancy is worth pursuing.

> Finally, I don't think it's worth including nonce probing. As it stands now, the nonce is not terribly useful anyway, so why probe for it? Anyway nobody uses it AFAIK…

We didn't look at nonce in our measurement study, so it's in there as future work in case it's useful. If it's not, we can certainly leave it out but (measurement geek hat on) I'd like data to support that decision. :)

Again, many thanks, and best regards,

Brian

---

From: Michael Welzl <[email-masked]>
Date: Mon, 29 Jul 2013 14:14:58 +0200

> Again, we didn't want to make assumptions about how applications write into the buffer, but yeah, requiring app changes at the API level is much, much worse. So good point, I think I'm convinced, a…

… seems like we agree now (wow, that was fast  :-)  ).

> The problem in this case is that we don't know if we have an ECN-aware middlebox that is clearing a CE from the end system, on the assumption that it is closer to the end-system than one would expe…

I'm beginning to understand your point. It's about this sentence in the draft:
"If the ECE flag is not set on the ACK segment(s) sent by the
       receiver acknowledging the CE probe segments, the path may or may
       not be usable, as that there might be middleboxes/gateways that
       (arguably correctly) clear CE on segments from end hosts, because
       they assume that congestion can not have occurred up to this
       point on the path."

But here you're assuming that this is the only possible reason for a CE to be cleared, i.e. CE cannot be cleared by a middlebox for any other reason. It might well be that a middlebox makes a mistake when it clears CE…  well, but if that wasn't seen in any of the measurement studies (the IMC one and yours), I'm inclined to accept that it's okay to leave ECN on then.

Cheers,
Michael

---

From: Brian Trammell <[email-masked]>
Date: Mon, 29 Jul 2013 16:58:07 +0200

hi Michael,

inline below...

On 29 Jul 2013, at 14:14, Michael Welzl <[email-masked]> wrote:

<snip>

> But here you're assuming that this is the only possible reason for a CE to be cleared, i.e. CE cannot be cleared by a middlebox for any other reason. It might well be that a middlebox makes a mista…

The question is one of requirements (and therefore one of philosophy): are we trying to guarantee (1) that ECN works, or (2) that attempts to activate ECN do not lead to connectivity problems. Given where we are now (increasing capability in endpoints, but not much activation in routers and clients) we think (2) is the more useful goal: we want to build confidence that TCP stacks could activate ECN negotiation on active open by default without causing connectivity issues that are hard for end-users and admins to debug.

We're not assuming that CE clearing is benign (though it does read that way; we'll need to be more explicit about a few things that aren't as clear as they should be in the -01 rev) -- rather, we're assuming that CE clearing that doesn't result in a dropped packet indicates that ECN can be activated on a path, even if you'll in effect never receive an ECE.

Cheers,

Brian

---

From: Michael Welzl <[email-masked]>
Date: Mon, 29 Jul 2013 12:10:46 -0400

> The question is one of requirements (and therefore one of philosophy): are we trying to guarantee (1) that ECN works, or (2) that attempts to activate ECN do not lead to connectivity problems. Give…

I agree,

> We're not assuming that CE clearing is benign (though it does read that way; we'll need to be more explicit about a few things that aren't as clear as they should be in the -01 rev) -- rather, we'r…

I agree.

Cheers,
Michael