Subject: [tcpm] draft-bensley-tcpm-dctcp: INF or HISTORIC?

From: Bob Briscoe <[email-masked]>
Date: Thu, 6 Mar 2014 14:51:57 +0000

Lars,

Everyone seems to agree this should be independent stream. That 
leaves the question of intended status...

Imagine Microsoft now updates DCTCP to fix the feedback flaw. Then do 
you want to update this draft? I doubt it.

So, IMO if you wanted to only document what Microsoft did, even tho 
it was  broken, then this should be HISTORIC, not INFORMATIONAL.

If it's INFORMATIONAL it should at least fix known major problems 
before it is published.

Bob

________________________________________________________________
Bob Briscoe,                                                  BT

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Fri, 7 Mar 2014 00:01:43 +0000

Hi Bob,

I tend to disagree (but I don't have a very strong opinion about this) - at least until the time that MS decides to fix the feedback issue (perhaps the RTT fairness issue also?) this one being informational seems reasonable; 

Also, the path for this draft sketched by Martin sounds best to me - improve the draft via WG review/feedback/discussion, but then have it as an individual, AD sponsored submission to the IESG. There is precedence for this, ie Adobe RTMFP / rfc7016, which became much better document that way.

Richard Scheffenegger

> -----Original Message-----
> From: tcpm [mailto:[email-masked]] On Behalf Of Bob Briscoe
> Sent: Donnerstag, 06. März 2014 14:52
> To: Eggert, Lars; Martin Stiemerling
> Cc: tcpm IETF list
> Subject: [tcpm] draft-bensley-tcpm-dctcp: INF or HISTORIC?
> 
> Lars,
> 
> Everyone seems to agree this should be independent stream. That leaves the
> question of intended status...
> 
> Imagine Microsoft now updates DCTCP to fix the feedback flaw. Then do you
> want to update this draft? I doubt it.
> 
> So, IMO if you wanted to only document what Microsoft did, even tho it was
> broken, then this should be HISTORIC, not INFORMATIONAL.
> 
> If it's INFORMATIONAL it should at least fix known major problems before
> it is published.
> 
> 
> 
> Bob
> 
> 
> 
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Sun, 9 Mar 2014 11:02:15 +0000

Hi Bob,

I think we should encourage that implementers document widely deployed TCP mechanism in the RFC series, if they are willing to discuss with the IETF and improve document clarity so that the document is a good, informational description that can be used as reference. (This thought is actually independent of DCTCP.)

If implementers are responsive to questions, such informational documentation (possible as AD-sponsored document) should actually be possible rather quickly. This almost likely implies that the RFC documents a mechanism that is still in use at the time of publication. To me, this is not "historic".

I could imagine that a Proposed Standard produced by the IETF changes the status to "Historic" at a later stage. But even then I'd ask whether the Proposed Standard indeed describes what is deployed, or will be deployed, at large scale in the Internet.

Thanks

Michael (as individual contributor)

> Sent: Thursday, March 06, 2014 3:52 PM
> To: EGGERT, Lars; Martin Stiemerling
> Cc: tcpm IETF list

---

From: Gorry Fairhurst <[email-masked]>
Date: Mon, 10 Mar 2014 09:28:06 +0000

This seems sensible to me, get WG comments feedback for an individual 
submission (via AD) and propose as INFO. If there is feedback by the 
authors they want it to be HISTORIC, that would be OK also.

To me, it would need careful crafting of the abstract to explain the 
current position with respect to other work in this area.

Gorry

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 11 Mar 2014 14:25:34 +0000

Gorry, Michael, Martin, Richard,

I would be happy with INF as long as the outstanding issues were 
clearly identified (feedback broken, no negotiation of alternate wire 
protocol, deployment applicability, etc)

The main reason for suggesting HISTORIC was to avoid people getting 
the impression the protocol as currently implemented is good to go on 
the public Internet.

Bob

At 09:28 10/03/2014, Gorry Fairhurst wrote:
>This seems sensible to me, get WG comments feedback for an 
>individual submission (via AD) and propose as INFO. If there is 
>feedback by the authors they want it to be HISTORIC, that would be OK also.
>
>>I think we should encourage that implementers document widely 
>>deployed TCP mechanism in the RFC series, if they are willing to 
>>discuss with the IETF and improve document clarity so that the 
>>document is a good, informational description that can be used as 
>>reference. (This thought is actually independent of DCTCP.)
>>
>>If implementers are responsive to questions, such informational 
>>documentation (possible as AD-sponsored document) should actually 
>>be possible rather quickly. This almost likely implies that the RFC 
>>documents a mechanism that is still in use at the time of 
>>publication. To me, this is not "historic".
>>
>>I could imagine that a Proposed Standard produced by the IETF 
>>changes the status to "Historic" at a later stage. But even then 
>>I'd ask whether the Proposed Standard indeed describes what is 
>>deployed, or will be deployed, at large scale in the Internet.
>>
>>Thanks
>>

---

From: Stephen Bensley <[email-masked]>
Date: Thu, 13 Mar 2014 14:52:06 +0000

Hello Bob and others,

I'm one of the authors of the draft. I'm new to writing IETF drafts, so I'll defer to my co-authors on the debate over intended status. The current plan is to switch this to Informational in the next version.

I agree that we should document any known limitations with DCTCP. I don't have any data on the effect of lost ACKs, but I will mention that dropped ACKs degrade the congestion estimate. In the deployment issues section, I will make it clear that there is no negotiation. I'll also mention the RTT fairness issue. Anything else specifically that you'd like to see covered?

I think Lars has already made this clear, but just to reiterate, our primary goals for documenting DCTCP were to provide a mechanism for the IPR disclosure and to document Windows behavior for the benefit of other stacks that would like to interoperate. There are some sizable DCTCP deployments, and thus, there has been some interest from non-Microsoft vendors to implement DCTCP. We hope to facilitate this by providing quality documentation and a royalty-free license.

If there is interest in doing further congestion control improvements for datacenters, I'm happy to participate, but perhaps this draft isn't the best vehicle for that.

Thanks for the feedback,
Stephen Bensley

-----Original Message-----
From: tcpm [mailto:[email-masked]] On Behalf Of Bob Briscoe
Sent: Tuesday, March 11, 2014 7:26 AM
To: Scharf, Michael (Michael); Martin Stiemerling; [email-masked]; Richard Scheffenegger
Cc: tcpm IETF list
Subject: Re: [tcpm] draft-bensley-tcpm-dctcp: INF or HISTORIC?

I would be happy with INF as long as the outstanding issues were clearly identified (feedback broken, no negotiation of alternate wire protocol, deployment applicability, etc)

The main reason for suggesting HISTORIC was to avoid people getting the impression the protocol as currently implemented is good to go on the public Internet.

Bob

At 09:28 10/03/2014, Gorry Fairhurst wrote:
>>I think we should encourage that implementers document widely deployed 
>>TCP mechanism in the RFC series, if they are willing to discuss with 
>>the IETF and improve document clarity so that the document is a good, 
>>informational description that can be used as reference. (This thought 
>>is actually independent of DCTCP.)
>>
>>documentation (possible as AD-sponsored document) should actually be 
>>possible rather quickly. This almost likely implies that the RFC 
>>documents a mechanism that is still in use at the time of publication. 
>>To me, this is not "historic".
>>
>>I could imagine that a Proposed Standard produced by the IETF changes 
>>the status to "Historic" at a later stage. But even then I'd ask 
>>whether the Proposed Standard indeed describes what is deployed, or 
>>will be deployed, at large scale in the Internet.
>>
>>Thanks
>>

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 13 Mar 2014 19:05:02 +0000

Stephen,

At 14:52 13/03/2014, Stephen Bensley wrote:
>Hello Bob and others,
>
>I'm one of the authors of the draft. I'm new to writing IETF drafts, 
>so I'll defer to my co-authors on the debate over intended status. 
>The current plan is to switch this to Informational in the next version.
>
>I agree that we should document any known limitations with DCTCP. I 
>don't have any data on the effect of lost ACKs, but I will mention 
>that dropped ACKs degrade the congestion estimate.

It's worse than that. Even if no ACKS have been dropped, the sender 
cannot know that no ACKs have been dropped, because every gap between 
delayed ACKS could represent a pure ACK loss. So the feedback is 
always highly ambiguous. If losses are extremely rare on ECN traffic, 
the ambiguity can be assumed not to exist. But as soon as there's a 
few losses, that assumption collapses and you have a very broken TCP protocol.

>In the deployment issues section, I will make it clear that there is 
>no negotiation. I'll also mention the RTT fairness issue. Anything 
>else specifically that you'd like to see covered?

For me personally, RTT fairness isn't an issue, and if it were for 
anyone else, they would have to deprecate TCP itself as well.

The only other concern I have is what people will do when they read 
an RFC that says Microsoft is using DCTCP, altho it has problems, but 
then the draft doesn't point to any solutions (because we are still 
working on them). Won't they conclude that they can use DCTCP as is, 
given MS is using it?

==The no.1 concern is capability negotiation. ==
There are very few codepoints left for negotiation, so if 
implementations appear that either burn up the remaining codepoints 
without agreement, or don't negotiate at all, the TCP wire protocol 
will become unusable with ECN. So the stakes are very high.

Win8 server already started us down this slippery slope, given it can 
use a different wire protocol without negotiation, so we need to fix 
this rapidly.

Superficially it seems safe because:
- the data centre template has to be turned on in Windows,
- it has to have negotiated ECN with the other end
- the RTT has to be below x (5ms from memory).

But what if one of our customers turns on the DC template in the 
Windows servers in one of the DCs we interconnect with the DCs of 
other customers? These TCPs will then start talking with other 
standard TCP implementations in other DCs, but using the ECN flags 
differently without the other end knowing. Chaos ensues.

>I think Lars has already made this clear, but just to reiterate, our 
>primary goals for documenting DCTCP were to provide a mechanism for 
>the IPR disclosure and to document Windows behavior for the benefit 
>of other stacks that would like to interoperate. There are some 
>sizable DCTCP deployments, and thus, there has been some interest 
>from non-Microsoft vendors to implement DCTCP. We hope to facilitate 
>this by providing quality documentation and a royalty-free license.

This must have involved quite a bit of work internally for no direct 
gain, so thank you for doing this.

>If there is interest in doing further congestion control 
>improvements for datacenters, I'm happy to participate, but perhaps 
>this draft isn't the best vehicle for that.

I think you were right to just document what you had done. 
Particularly to make the IPR position clear. And if you make the 
listed problems extremely clear, and point to solutions, I'll be happy.

Do you want me to point you at all the relevant background?

You will see a roadmap of the drafts I think are necessary on slide 
11 of a presentation I gave to TSVAREA about this last week 
(including your draft):
<http://bobbriscoe.net/presents/1403ietf/1403ecn-encap-guidelines.pdf>
To be clear, this roadmap has greater ambitions than just private data centres.

Bob

>To: Scharf, Michael (Michael); Martin Stiemerling; 
>[email-masked]; Richard Scheffenegger
>Cc: tcpm IETF list
>clearly identified (feedback broken, no negotiation of alternate 
>wire protocol, deployment applicability, etc)
>
>the impression the protocol as currently implemented is good to go 
>on the public Internet.
>
>
>Bob
>

---

From: Stephen Bensley <[email-masked]>
Date: Sat, 15 Mar 2014 02:27:29 +0000

Hi Bob,

I don't think dropped ACKs are as bad as that. If a drop occurs during an unbroken string of CE or non-CE packets, the congestion estimate is unaffected. If a drop occurs during a transition between CE and non-CE, then yes, the estimate will be less accurate than it would be without drops. However, there are some mitigations:
- It may be that even with a less accurate congestion estimate, DCTCP still performs better than RFC 3168.
- If the estimation gain is small enough relative to the packet loss rate, the estimate may not be degraded much.
- It may be that packet losses mostly occur under heavy congestion. In which case, most drops occur during an unbroken string of CE packets. 

I'd need data about the distribution of CE events relative to packet drops to analyze the impact, and I don't have this data, so the above is all speculation. But DCTCP is successfully deployed in large datacenters today, so it works well in at least some environments.

For the case where one server uses DCTCP and the other implements standard RFC 3168 behavior, Lars had a graduate student who analyzed this situation in detail and proposed some improvements. We will add a reference to this paper in the next version.

"Won't they conclude that they can use DCTCP as is, given MS is using it?"

I believe they can use DCTCP as is. I'm not opposed to developing something better, but DCTCP seems to work well in practice.

-----Original Message-----
From: Bob Briscoe [mailto:[email-masked]] 
Sent: Thursday, March 13, 2014 12:05 PM
To: Stephen Bensley
Cc: Scharf, Michael (Michael); Martin Stiemerling; [email-masked]; Richard Scheffenegger; tcpm IETF list
Subject: RE: [tcpm] draft-bensley-tcpm-dctcp: INF or HISTORIC?

Stephen,

At 14:52 13/03/2014, Stephen Bensley wrote:
>Hello Bob and others,
>
>I'm one of the authors of the draft. I'm new to writing IETF drafts, so 
>I'll defer to my co-authors on the debate over intended status.
>don't have any data on the effect of lost ACKs, but I will mention that 
>dropped ACKs degrade the congestion estimate.

It's worse than that. Even if no ACKS have been dropped, the sender cannot know that no ACKs have been dropped, because every gap between delayed ACKS could represent a pure ACK loss. So the feedback is always highly ambiguous. If losses are extremely rare on ECN traffic, the ambiguity can be assumed not to exist. But as soon as there's a few losses, that assumption collapses and you have a very broken TCP protocol.

>In the deployment issues section, I will make it clear that there is no 
>negotiation. I'll also mention the RTT fairness issue. Anything else 
>specifically that you'd like to see covered?

For me personally, RTT fairness isn't an issue, and if it were for anyone else, they would have to deprecate TCP itself as well.

The only other concern I have is what people will do when they read an RFC that says Microsoft is using DCTCP, altho it has problems, but then the draft doesn't point to any solutions (because we are still working on them). Won't they conclude that they can use DCTCP as is, given MS is using it?

==The no.1 concern is capability negotiation. == There are very few codepoints left for negotiation, so if implementations appear that either burn up the remaining codepoints without agreement, or don't negotiate at all, the TCP wire protocol will become unusable with ECN. So the stakes are very high.

Win8 server already started us down this slippery slope, given it can use a different wire protocol without negotiation, so we need to fix this rapidly.

But what if one of our customers turns on the DC template in the Windows servers in one of the DCs we interconnect with the DCs of other customers? These TCPs will then start talking with other standard TCP implementations in other DCs, but using the ECN flags differently without the other end knowing. Chaos ensues.

>primary goals for documenting DCTCP were to provide a mechanism for the 
>IPR disclosure and to document Windows behavior for the benefit of 
>other stacks that would like to interoperate. There are some sizable 
>DCTCP deployments, and thus, there has been some interest from 
>non-Microsoft vendors to implement DCTCP. We hope to facilitate this by 
>providing quality documentation and a royalty-free license.

This must have involved quite a bit of work internally for no direct gain, so thank you for doing this.

Bob