Subject: Re: [tcpm] WG acceptance of draft-bensley-tcpm-dctcp

From: Mirja Kühlewind <[email-masked]>
Date: Mon, 3 Aug 2015 10:26:46 +0200

Hi Michael,

as the doc is in very good shape, we could probably even try to find this by the end of the year… but the authors have to decide here I guess.

Mirja

> Am 24.07.2015 um 13:39 schrieb Scharf, Michael (Michael) <[email-masked]>:
> 
> Hi all,
> 
> In the meeting on Wednesday, we had unanimous support for accepting draft-bensley-tcpm-dctcp as new informational document in TCPM, and we had strong consensus that the document should have the title "Datacenter TCP (DCTCP): TCP Congestion Control for Datacenters". The scope is to accurately document the protocol that has been implemented and deployed.
> 
> This e-mail intends to confirm that it is consensus in the TCPM community to work on this document and to add a corresponding new milestone to the TCPM charter. My suggestion for the charter milestone would be: 
> 
> Apr 2016   Submit document on Datacenter TCP (DCTCP) to the IESG for publication as Informational RFC
> 
> If you have any feedback or suggestions, please speak up in the next two weeks.
> 
> Thanks
> 
> Michael
> 
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: "Eggert, Lars" <[email-masked]>
Date: Mon, 3 Aug 2015 08:31:47 +0000

> as the doc is in very good shape, we could probably even try to find this by the end of the year… but the authors have to decide here I guess.

Agree that we should be able to wrap this up more quickly. Praveen is currently addressing the feedback from Prague. If anyone has other comments, please send them - the doc is otherwise ready from our perspective.

Lars

---

From: "De Schepper, Koen (Koen)" <[email-masked]>
Date: Mon, 3 Aug 2015 16:35:27 +0000

Hi All,

I have reviewed the draft-bensley-tcpm-dctcp-05, and have following questions/remarks:

1.) A general remark is that the draft does not explicitly mention what the response is to drop, and how it interworks with the alpha and other state variables. I think the issues with the Linux implementation justify a clear description.

2.) When exactly is the first window reduction applied? Is it at the end of an observation window, or when the first mark/drop is detected. To implement the "respond to congestion only once per window" requirement, is there a separate mark and drop window state variable? If separate, this means that if both drop and mark are detected the window can be reduced down to 25% in one RTT (as per RFC3168).

3.) Bullet 4 in section 3.3 states "If the sequence number is less than or equal to DCTCP.WindowEnd". Is there a particular reason for the "or equal"? I would assume that if the ACKed sequence number is equal to DCTCP.WindowEnd a window of data is sent?

4.) The last sentence of section 3.3: "The setting of the Congestion Window Reduced (CWR) bit is also exactly as per [RFC3168]". Is there a reason to do this, and how does the receiver respond to this? (I assume a DCTCP receiver ignores it and an RFC3168 receiver needs it to stop echoing CE)

5.) The second paragraph of section 4 explains when DCTCP is activated, but does not explicitly mentions what it means to deactivate DCTCP. If ECN is successfully negotiated (but RTT> 10ms), will it still use ECN in the classic way, or will it stop sending ECT(0) packets?

6.) The last sentence of section 6 states: "Much like standard TCP, DCTCP is biased against flows with longer RTTs. A method for improving the fairness of DCTCP has been proposed in [ADCTCP], but requires additional experimental evaluation". This reads as if DCTCP has an improved bias compared to standard TCP, but if I am correct the DCTCP bias is different than that of standard TCP, and the method in [ADCTCP] aligns it closer (or equal) to that of standard TCP, right?

Regards,
Koen.

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Tue, 4 Aug 2015 08:29:22 +0000

Hi,

regarding to this comment by Koen,

> 
> 4.) The last sentence of section 3.3: "The setting of the Congestion
> Window Reduced (CWR) bit is also exactly as per [RFC3168]". Is there a
> reason to do this, and how does the receiver respond to this? (I assume a
> DCTCP receiver ignores it and an RFC3168 receiver needs it to stop echoing
> CE)

There is actually a gap in the description in RFC3168 for the case that a CWR segment is also CE marked; at least one (significant) TCP stack got that processing in the wrong order in the past, but not anymore. 

The expected processing is to deal with CWR first (clear ECE - for a regular 3168 receiver), and the IP-header CE mark second (set ECE).

Just wondering if this draft want's to describe the expected (proper) behavior for a regular 3168 receiver - or a DCTCP receiver in 3168 mode - somewhere, so that we have this as an RFC?

(But with some luck, all receivers will deploy AccECN anyhow, very soon (TM) :) ).

Best regards,
  Richard

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Thu, 13 Aug 2015 15:51:58 +0000

I can change the milestone to an earlier date. What date would the authors suggest?

For past documents, I have usually tried to pick a date that seemed realistic to me when the doc was adopted in TCPM, and in almost all cases we finally missed the milestone date... Progress in TCPM depends to a large extent on author cycles, which are not trivial to predict...

In case of this specific document, I think have a number of comments to be addressed. And I could imagine further comments will follow once the next version is published. At least I will probably have a look at the next version. When do the authors think that all reviews will be addressed?

Thanks

Michael

-----Original Message-----
From: Eggert, Lars [mailto:[email-masked]] 
Sent: Montag, 3. August 2015 10:32
To: Mirja Kühlewind
Cc: Scharf, Michael (Michael); [email-masked]
Subject: Re: [tcpm] WG acceptance of draft-bensley-tcpm-dctcp

Lars

---

From: Praveen Balasubramanian <[email-masked]>
Date: Tue, 25 Aug 2015 23:41:30 +0000

Thanks Koen for the detailed review. Sorry for the delay. Inline are some comments.

-----Original Message-----
From: tcpm [mailto:[email-masked]] On Behalf Of De Schepper, Koen (Koen)
Sent: Monday, August 3, 2015 9:35 AM
To: [email-masked]
Subject: [tcpm] Review of draft-bensley-tcpm-dctcp

Hi All,

In the Windows implementation, the TCP sender falls back to the NewReno response when there is a loss episode on a DCTCP connection. However it doesn't reset the value of Alpha and the other state variables. We do mention that a configuration knob be provided to reset these state variables in case of a loss episode, as has been done in the Linux implementation. We will improve the text around loss handling in the next revision under work.

ECN alpha value is updated at the end of every send round. The window reduction based on the calculated ECN alpha value is applied when an ECE is processed and a round has been completed. It is correct that if both enough markings happen in a round to inflate alpha to the maximum value and there is also packet loss that the cwnd will be reduced multiple times. The goal of DCTCP is to prevent loss, and if it fails at this goal, then its response is not optimal. If you have suggestions around improving this behavior we would be happy to address them in the "Implementation Issues" section. 

It should actually read " If the acknowledgement number is less than or equal to DCTCP.WindowEnd ". We will make this correction in the next revision. The reason for the "or equal" check is that the Alpha value and the state variables are updated on the receipt of the first ACK beyond the observation window. 

The DCTCP receiver behaves exactly like the RFC3168 receiver when processing CWR. If it receives a packet with CWR set it stops echoing CE until it gets the next packet with CE set. If the same packet has both CE and CWR set, then CE trumps CWR and echoing continues. Adding text in next revision to clarify receiver processing of CWR.

Windows Server 2012+ will always attempt to negotiate ECN for outbound connections by default and accept ECN when requested for inbound connections. So yes for higher RTT connections, classic ECN will continue to work along with an congestion algorithm like CTCP. However be mindful that using RTT and ECN negotiation to decide datacenter versus internet settings is simply the default out-of-box configuration. There are configuration knobs to force any congestion algorithm and TCP settings globally, or on a per interface basis (IP address range) or per TCP port range (say only HTTP traffic). 

Per [ADCTCP] DCTCP does have better RTT fairness when compared to TCP-Drop-Tail but worse than TCP-RED. The method in [ADCTCP] makes it better than TCP-RED. The results therein are based on an ns2 simulation.

Regards,
Koen.

---

From: Praveen Balasubramanian <[email-masked]>
Date: Tue, 25 Aug 2015 23:46:57 +0000

Yes the Windows implementation does follow the order of processing CWR first and then CE in essence allowing CE to override CWR. A DCTCP receiver does not change this behavior. We will add text for CWR handling. Since this DCTCP RFC draft doesn't update RFC 3168, maybe an errata is needed to describe handling of packet with CWR and CE set.

-----Original Message-----
From: Scheffenegger, Richard [mailto:[email-masked]] 
Sent: Tuesday, August 4, 2015 1:29 AM
To: [email-masked]; Praveen Balasubramanian <[email-masked]>; Eggert, Lars <[email-masked]>
Cc: De Schepper, Koen (Koen) <[email-masked]>
Subject: RE: Review of draft-bensley-tcpm-dctcp

Hi,

> reason to do this, and how does the receiver respond to this? (I 
> assume a DCTCP receiver ignores it and an RFC3168 receiver needs it to 
> stop echoing
> CE)

Best regards,
  Richard

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Thu, 27 Aug 2015 10:25:22 +0000

Hi Praveen,

already done so a while ago. See https://www.rfc-editor.org/errata_search.php?rfc=3168 

I believe few actually look into the errata section - and AccECN (which Mirja, Bob and I are working on) are departing from the TCP signaling in 3168 so much, that we can't accommodate that update in our document.

>From my past experiments I knew that the (pre-DCTCP) Windows TCP stack was behaving as expected (thank you), I just thought this might be a chance to put the update into an RFC document earlier than waiting for an 3168bis (which probably won't come anymore).

Best regards,
  Richard

> From: Praveen Balasubramanian [mailto:[email-masked]]
> Sent: Mittwoch, 26. August 2015 01:47
> To: Scheffenegger, Richard; [email-masked]; Eggert, Lars
> Cc: De Schepper, Koen (Koen)
> Yes the Windows implementation does follow the order of processing CWR
> first and then CE in essence allowing CE to override CWR. A DCTCP receiver
> does not change this behavior. We will add text for CWR handling. Since
> this DCTCP RFC draft doesn't update RFC 3168, maybe an errata is needed to
> describe handling of packet with CWR and CE set.
> 
> There is actually a gap in the description in RFC3168 for the case that a
> CWR segment is also CE marked; at least one (significant) TCP stack got
> that processing in the wrong order in the past, but not anymore.
> 
> The expected processing is to deal with CWR first (clear ECE - for a
> regular 3168 receiver), and the IP-header CE mark second (set ECE).
> 
> Just wondering if this draft want's to describe the expected (proper)
> behavior for a regular 3168 receiver - or a DCTCP receiver in 3168 mode -
> somewhere, so that we have this as an RFC?
>