Subject: Re: [tcpm] draft-bensley-tcpm-dctcp: INF or HISTORIC?

From: Bob Briscoe <[email-masked]>
Date: Tue, 8 Apr 2014 17:04:19 +0100

Stephen,

Sorry, I let this thread run dry (I shifted focus from email to deliverables!).

It's not appropriate to just say DCTCP has worked well so far. It's 
as if I had identified a flaw in the emergency braking system of a 
train, and the designer just says "The brakes have been working fine 
so far, so the alleged problem you've identified can't be that bad".

I've identified a flaw that makes DCTCP increasingly more broken the 
more pure ACK losses there are on the reverse channel. That implies 
it could well have been 'working' for years without noticing the 
problem. But in the wrong circumstances, e.g. losing a link that 
causes a lot of traffic to reroute onto other links, competing with 
the reverse channel of other flows. Such a loss situation could 
persist for some time.

In this case, the pre-existing other traffic in the other direction 
would probably not be persistently congested itself, but it would be 
losing control information, so it would be alternately overshooting 
and undershooting without even knowing how badly out of control it had become.

As I show here:
<http://tools.ietf.org/html/draft-ietf-tcpm-accecn-reqs-05#appendix-A>
if one or two pure ACK losses might have occurred, a simple sequence 
of 3 delayed ACKs apparently feeding back no ECE to the sender might 
have meant 0%, 20% or 40% CE feedback when they left the receiver, 
and the sender doesn't know which.

Detailed responses inline...

At 03:27 15/03/2014, Stephen Bensley wrote:
>Hi Bob,
>
>I don't think dropped ACKs are as bad as that. If a drop occurs 
>during an unbroken string of CE or non-CE packets, the congestion 
>estimate is unaffected. If a drop occurs during a transition between 
>CE and non-CE, then yes, the estimate will be less accurate than it 
>would be without drops. However, there are some mitigations:
>- It may be that even with a less accurate congestion estimate, 
>DCTCP still performs better than RFC 3168.

RFC3168 ECN is designed to be robust to any amount of pure ack loss.

>- If the estimation gain is small enough relative to the packet loss 
>rate, the estimate may not be degraded much.

Not knowing whether ECE feedback is 0%, 20% or 40% is not a small 
estimation error.

And the example in Appx A is not a contrived example. Once the sender 
needs to allow for possible pure ACK losses (which it doesn't know), 
it has insufficient information to guess what the ACK stream means: 
in the example given, the sender can guess 0% on the balance of 
probabilities, but it doesn't really know whether 40% is more likely 
to be correct...

...because the sender cannot know the pure ACK loss rate, because 
pure ACKs are not delivered reliably. So, it cannot estimate how much 
pure ACK losses might be altering its estimate of CE probability. 
This is also because the gaps between DCTCP's delayed ACKs depend on 
the unpredictable sequence of CE marks itself, which leads to a catch-22.

>- It may be that packet losses mostly occur under heavy congestion. 
>In which case, most drops occur during an unbroken string of CE packets.

We're talking about ACK losses on the reverse path, which can be 
uncorrelated with congestion on the forward path.

>I'd need data about the distribution of CE events relative to packet 
>drops to analyze the impact, and I don't have this data, so the 
>above is all speculation.

No-one is planning to write artificial intelligence algorithms to 
estimate what the protocol probably intended to mean. We're just 
re-designing the protocol, so it works, including during reverse 
congestion. We're not attached to Microsoft's design of DCTCP like 
its our favourite pet dog. It's just a protocol. If it isn't safe 
when it's most needed, we just kill it and make another one.

>But DCTCP is successfully deployed in large datacenters today, so it 
>works well in at least some environments.

Just because the brakes have always worked on a trian, doesn't mean 
the emergency braking system will always work when its needed, 
especially given a design flaw has been pointed out.

>For the case where one server uses DCTCP and the other implements 
>standard RFC 3168 behavior, Lars had a graduate student who analyzed 
>this situation in detail and proposed some improvements. We will add 
>a reference to this paper in the next version.

Is it anything like our proposal to use differently configured AQM 
for ECN and non-ECN?
<http://www.ietf.org/proceedings/89/slides/slides-89-tsvarea-3.pdf>

>"Won't they conclude that they can use DCTCP as is, given MS is using it?"
>
>I believe they can use DCTCP as is. I'm not opposed to developing 
>something better, but DCTCP seems to work well in practice.

I'm not saying the sky is falling. I am saying, if it does, we don't 
have a way to respond.

Bob

>-----Original Message-----
>From: Bob Briscoe [mailto:[email-masked]]
>Sent: Thursday, March 13, 2014 12:05 PM
>To: Stephen Bensley
>Cc: Scharf, Michael (Michael); Martin Stiemerling; 
>[email-masked]; Richard Scheffenegger; tcpm IETF list
>Subject: RE: [tcpm] draft-bensley-tcpm-dctcp: INF or HISTORIC?
>
>Stephen,
>
>At 14:52 13/03/2014, Stephen Bensley wrote:
> >Hello Bob and others,
> >
> >I'm one of the authors of the draft. I'm new to writing IETF drafts, so
> >I'll defer to my co-authors on the debate over intended status.
> >The current plan is to switch this to Informational in the next version.
> >
> >I agree that we should document any known limitations with DCTCP. I
> >don't have any data on the effect of lost ACKs, but I will mention that
> >dropped ACKs degrade the congestion estimate.
>
>It's worse than that. Even if no ACKS have been dropped, the sender 
>cannot know that no ACKs have been dropped, because every gap 
>between delayed ACKS could represent a pure ACK loss. So the 
>feedback is always highly ambiguous. If losses are extremely rare on 
>ECN traffic, the ambiguity can be assumed not to exist. But as soon 
>as there's a few losses, that assumption collapses and you have a 
>very broken TCP protocol.
>
> >In the deployment issues section, I will make it clear that there is no
> >negotiation. I'll also mention the RTT fairness issue. Anything else
> >specifically that you'd like to see covered?
>
>For me personally, RTT fairness isn't an issue, and if it were for 
>anyone else, they would have to deprecate TCP itself as well.
>
>The only other concern I have is what people will do when they read 
>an RFC that says Microsoft is using DCTCP, altho it has problems, 
>but then the draft doesn't point to any solutions (because we are 
>still working on them). Won't they conclude that they can use DCTCP 
>as is, given MS is using it?
>
>==The no.1 concern is capability negotiation. == There are very few 
>codepoints left for negotiation, so if implementations appear that 
>either burn up the remaining codepoints without agreement, or don't 
>negotiate at all, the TCP wire protocol will become unusable with 
>ECN. So the stakes are very high.
>
>Win8 server already started us down this slippery slope, given it 
>can use a different wire protocol without negotiation, so we need to 
>fix this rapidly.
>
>Superficially it seems safe because:
>- the data centre template has to be turned on in Windows,
>- it has to have negotiated ECN with the other end
>- the RTT has to be below x (5ms from memory).
>
>But what if one of our customers turns on the DC template in the 
>Windows servers in one of the DCs we interconnect with the DCs of 
>other customers? These TCPs will then start talking with other 
>standard TCP implementations in other DCs, but using the ECN flags 
>differently without the other end knowing. Chaos ensues.
>
> >I think Lars has already made this clear, but just to reiterate, our
> >primary goals for documenting DCTCP were to provide a mechanism for the
> >IPR disclosure and to document Windows behavior for the benefit of
> >other stacks that would like to interoperate. There are some sizable
> >DCTCP deployments, and thus, there has been some interest from
> >non-Microsoft vendors to implement DCTCP. We hope to facilitate this by
> >providing quality documentation and a royalty-free license.
>
>This must have involved quite a bit of work internally for no direct 
>gain, so thank you for doing this.
>
>
>
> >If there is interest in doing further congestion control
> >improvements for datacenters, I'm happy to participate, but perhaps
> >this draft isn't the best vehicle for that.
>
>I think you were right to just document what you had done.
>Particularly to make the IPR position clear. And if you make the
>listed problems extremely clear, and point to solutions, I'll be happy.
>
>Do you want me to point you at all the relevant background?
>
>You will see a roadmap of the drafts I think are necessary on slide
>11 of a presentation I gave to TSVAREA about this last week
>(including your draft):
><http://bobbriscoe.net/presents/1403ietf/1403ecn-encap-guidelines.pdf>
>To be clear, this roadmap has greater ambitions than just private 
>data centres.
>
>
>Bob
>
>
> >Thanks for the feedback,
> >Stephen Bensley
> >
> >-----Original Message-----
> >From: tcpm [mailto:[email-masked]] On Behalf Of Bob Briscoe
> >Sent: Tuesday, March 11, 2014 7:26 AM
> >To: Scharf, Michael (Michael); Martin Stiemerling;
> >[email-masked]; Richard Scheffenegger
> >Cc: tcpm IETF list
> >Subject: Re: [tcpm] draft-bensley-tcpm-dctcp: INF or HISTORIC?
> >
> >Gorry, Michael, Martin, Richard,
> >
> >I would be happy with INF as long as the outstanding issues were
> >clearly identified (feedback broken, no negotiation of alternate
> >wire protocol, deployment applicability, etc)
> >
> >The main reason for suggesting HISTORIC was to avoid people getting
> >the impression the protocol as currently implemented is good to go
> >on the public Internet.
> >
> >
> >Bob
> >
> >At 09:28 10/03/2014, Gorry Fairhurst wrote:
> > >This seems sensible to me, get WG comments feedback for an individual
> > >submission (via AD) and propose as INFO. If there is feedback by the
> > >authors they want it to be HISTORIC, that would be OK also.
> > >
> > >To me, it would need careful crafting of the abstract to explain the
> > >current position with respect to other work in this area.
> > >
> > >Gorry
> > >
> > >On 09/03/2014 11:02, Scharf, Michael (Michael) wrote:
> > >>Hi Bob,
> > >>
> > >>I think we should encourage that implementers document widely deployed
> > >>TCP mechanism in the RFC series, if they are willing to discuss with
> > >>the IETF and improve document clarity so that the document is a good,
> > >>informational description that can be used as reference. (This thought
> > >>is actually independent of DCTCP.)
> > >>
> > >>If implementers are responsive to questions, such informational
> > >>documentation (possible as AD-sponsored document) should actually be
> > >>possible rather quickly. This almost likely implies that the RFC
> > >>documents a mechanism that is still in use at the time of publication.
> > >>To me, this is not "historic".
> > >>
> > >>I could imagine that a Proposed Standard produced by the IETF changes
> > >>the status to "Historic" at a later stage. But even then I'd ask
> > >>whether the Proposed Standard indeed describes what is deployed, or
> > >>will be deployed, at large scale in the Internet.
> > >>
> > >>Thanks
> > >>
> > >>Michael (as individual contributor)
> > >>
> > >>
> > >>
> > >>>-----Original Message-----
> > >>>From: tcpm [mailto:[email-masked]] On Behalf Of Bob Briscoe
> > >>>Sent: Thursday, March 06, 2014 3:52 PM
> > >>>To: EGGERT, Lars; Martin Stiemerling
> > >>>Cc: tcpm IETF list
> > >>>Subject: [tcpm] draft-bensley-tcpm-dctcp: INF or HISTORIC?
> > >>>
> > >>>Lars,
> > >>>
> > >>>Everyone seems to agree this should be independent stream. That
> > >>>leaves the question of intended status...
> > >>>
> > >>>Imagine Microsoft now updates DCTCP to fix the feedback flaw. Then do
> > >>>you want to update this draft? I doubt it.
> > >>>
> > >>>So, IMO if you wanted to only document what Microsoft did, even tho
> > >>>it was  broken, then this should be HISTORIC, not INFORMATIONAL.
> > >>>
> > >>>If it's INFORMATIONAL it should at least fix known major problems
> > >>>before it is published.
> > >>>
> > >>>
> > >>>
> > >>>Bob
> > >>>
> > >>>
> > >>>
> > >>>________________________________________________________________
> > >>>Bob Briscoe,                                                  BT
> > >>>
> > >>>_______________________________________________
> > >>>tcpm mailing list
> > >>>[email-masked]
> > >>>https://www.ietf.org/mailman/listinfo/tcpm
> > >>
> > >>_______________________________________________
> > >>tcpm mailing list
> > >>[email-masked]
> > >>https://www.ietf.org/mailman/listinfo/tcpm
> > >
> > >________________________________________________________________
> > >Bob Briscoe,                                                  BT
> >
> >_______________________________________________
> >tcpm mailing list
> >[email-masked]
> >https://www.ietf.org/mailman/listinfo/tcpm
>
>________________________________________________________________
>Bob Briscoe,                                                  BT

________________________________________________________________
Bob Briscoe,                                                  BT

---

From: Stephen Bensley <[email-masked]>
Date: Tue, 8 Apr 2014 18:13:43 +0000

Hi Bob,

I agree that there are almost certainly real world scenarios where DCTCP performs poorly. However, the same is true of the existing congestion control algorithms. That's what motivated the invention of DCTCP.

I also agree with the issues you've identified, and I've documented them in the next version. I too have been side-tracked by deliverables, but I hope to have the next version published soon. Furthermore, as I wrote before, I'm happy to work with you and others to come up with something better.

Please wait for the new version, and see if you're happy with the language.

Thanks again for all the feedback. I do think version 01 is much better than the previous version.

-----Original Message-----
From: Bob Briscoe [mailto:[email-masked]] 
Sent: Tuesday, April 8, 2014 9:04 AM
To: Stephen Bensley
Cc: Scharf, Michael (Michael); Martin Stiemerling; [email-masked]; Richard Scheffenegger; tcpm IETF list
Subject: RE: [tcpm] draft-bensley-tcpm-dctcp: INF or HISTORIC?

Stephen,

It's not appropriate to just say DCTCP has worked well so far. It's as if I had identified a flaw in the emergency braking system of a train, and the designer just says "The brakes have been working fine so far, so the alleged problem you've identified can't be that bad".

I've identified a flaw that makes DCTCP increasingly more broken the more pure ACK losses there are on the reverse channel. That implies it could well have been 'working' for years without noticing the problem. But in the wrong circumstances, e.g. losing a link that causes a lot of traffic to reroute onto other links, competing with the reverse channel of other flows. Such a loss situation could persist for some time.

In this case, the pre-existing other traffic in the other direction would probably not be persistently congested itself, but it would be losing control information, so it would be alternately overshooting and undershooting without even knowing how badly out of control it had become.

As I show here:
<http://tools.ietf.org/html/draft-ietf-tcpm-accecn-reqs-05#appendix-A>
if one or two pure ACK losses might have occurred, a simple sequence of 3 delayed ACKs apparently feeding back no ECE to the sender might have meant 0%, 20% or 40% CE feedback when they left the receiver, and the sender doesn't know which.

At 03:27 15/03/2014, Stephen Bensley wrote:
>Hi Bob,
>
>I don't think dropped ACKs are as bad as that. If a drop occurs during 
>an unbroken string of CE or non-CE packets, the congestion estimate is 
>unaffected. If a drop occurs during a transition between CE and non-CE, 
>then yes, the estimate will be less accurate than it would be without 
>drops. However, there are some mitigations:
>- It may be that even with a less accurate congestion estimate, DCTCP 
>still performs better than RFC 3168.

>rate, the estimate may not be degraded much.

Not knowing whether ECE feedback is 0%, 20% or 40% is not a small estimation error.

And the example in Appx A is not a contrived example. Once the sender needs to allow for possible pure ACK losses (which it doesn't know), it has insufficient information to guess what the ACK stream means: 
in the example given, the sender can guess 0% on the balance of probabilities, but it doesn't really know whether 40% is more likely to be correct...

...because the sender cannot know the pure ACK loss rate, because pure ACKs are not delivered reliably. So, it cannot estimate how much pure ACK losses might be altering its estimate of CE probability. 
This is also because the gaps between DCTCP's delayed ACKs depend on the unpredictable sequence of CE marks itself, which leads to a catch-22.

>In which case, most drops occur during an unbroken string of CE packets.

We're talking about ACK losses on the reverse path, which can be uncorrelated with congestion on the forward path.

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 8 Apr 2014 19:49:33 +0100

Stephen,

Thx for your patience.

I don't think we should be talking about whether DCTCP as a whole is 
'better' or 'worse' than RFC3168. For instance, it performs much 
better, but you can't use it on the public Internet. Is that better or worse?!

For the IETF, I'm essentially advocating that we don't treat DCTCP as 
one inviolable whole. Instead, we build on the whole idea, dividing 
it into component parts, and incrementally improve each one, eg:
- the feedback protocol (accecn)
- the marking algo - co-existence with legacy traffic
- and there is always scope for improving the sender algo too.

Bob

At 19:13 08/04/2014, Stephen Bensley wrote:
>Hi Bob,
>
>I agree that there are almost certainly real world scenarios where 
>DCTCP performs poorly. However, the same is true of the existing 
>congestion control algorithms. That's what motivated the invention of DCTCP.
>
>I also agree with the issues you've identified, and I've documented 
>them in the next version. I too have been side-tracked by 
>deliverables, but I hope to have the next version published soon. 
>Furthermore, as I wrote before, I'm happy to work with you and 
>others to come up with something better.
>
>And the example in Appx A is not a contrived example. Once the 
>sender needs to allow for possible pure ACK losses (which it doesn't 
>know), it has insufficient information to guess what the ACK stream means:
>pure ACKs are not delivered reliably. So, it cannot estimate how 
>much pure ACK losses might be altering its estimate of CE probability.