Subject: Re: [tcpm] Review of draft-bensley-tcpm-dctcp-03

From: Praveen Balasubramanian <[email-masked]>
Date: Fri, 10 Jul 2015 16:14:41 +0000

Hi Mirja

We have published a new version of the draft https://tools.ietf.org/html/draft-bensley-tcpm-dctcp-05 which addresses many of your concerns. Inline are some responses to individual feedback.

Thanks

-----Original Message-----
From: tcpm [mailto:[email-masked]] On Behalf Of Mirja Kühlewind
Sent: Thursday, June 18, 2015 6:42 AM
To: Eggert, Lars; Dave Thaler; Stephen Bensley
Cc: [email-masked]
Subject: Re: [tcpm] Review of draft-bensley-tcpm-dctcp-03

One more quick point, an analysis of the problem of the used ECN feedback scheme of DCTCP is also given in the AccECN requirements draft (soon RFC7560) which maybe could be referred here.

Mirja

> Am 18.06.2015 um 15:31 schrieb Mirja Kühlewind <[email-masked]>:
> 
> Hi all,
> 
> as I think draft-bensley-tcpm-dctcp should be an tcpm working group document and because I’m interested in DCTCP, I’ve reviewed this document. This draft documents DCTCP as implemented in Microsoft and is very-well written and easy to understand, therefore I think if adopted by tcpm is can be processed quickly.
> 
> I have two comments which I would like to see addressed; and a few more smaller comments which might help to even further improve the document and might be addressed or not:
> 
> 1) I would like to see the point that DCTP is *only* indented to be used in data center made more strongly and explicit in this document by adding one more small paragraph in the intro or even one more sentence in the abstract. There are two reason why I think this is very important to say more explicitly: a) there is no negotiation that detects if DCTCP is also implemented by the other end (this is mainly important for the ECN feedback). And b) if DCTP is used in parallel to ‚traditional‘ (maybe even loss-based) congestion control, it will starve this competing traffic; I don’t think this point is mentioned anywhere but probably should, at least very briefly.

>> First comment is addressed in the section 1 (Introduction). Second comment of coexistence with non-DCTCP variants of TCP is addressed in section 5 (Deployment issues).

> 
> 2) This document does not describe what the Microsoft implementation does if loss is detected. I think normally the assumption is that there should be no (congestion-based) loss anymore in a data center where all traffic is using DCTCP. However, the draft should still say what to do if loss occurs. If I remember correctly the Linux implementation has a configuration parameter to decide if a loss should be ignored or the window should be halved. Maybe you can again clarify with Daniel Borkmann who implemented this in Linux; I know he has read the draft and is probably willing to review again or even provide some text on this if needed.
> 

>> Windows Server does not adjust the value of Alpha for the round if loss is detected. We are under discussion with Daniel about addressing this in the next revision of the draft under Implementation issues section.

> Other minor comments:
> 
> 1) I would like to see mentioned in the abstract as well as in the intro that the gain in lower latency is achieved by using a very low marking threshold of the AQM. This document is written to mainly describe only the endpoint changes and seem to suggest that DCTCP could work with any ECN-enabled AQM. That might be true or not; at least I have not seems any evaluation results where different AQM schemes are used. I personally would prefer to describe DCTCP as a whole system that implements a certain congestion control scheme (and ECN feedback) as well as assumes a specific parameterization in the bottleneck’s AQM. I don’t think that’s a problem to describe it that way, because DCTCP is clear only intended to be used in data centers where all endpoint and network nodes are under the control of one entity. Actually it should even be mentioned that the proposed parameterization can easily be configured with all switches that implement RED. 

>> This is correct when looking at the system as a whole but the primary goal is to provide guidelines around end host implementation which should not be aware or dependent on any particular AQM scheme. Some text to address RED is added in Section 5 (Deployment issues).

> 
> 2) The second point goes in the same direction: section 3.1 says:
> "However, the actual
>   algorithm for marking congestion is an implementation detail of the
>   switch and will generally not be known to the sender and receiver.
>   Therefore, sender and receiver MUST NOT assume that a particular
>   marking algorithm is implemented by the switching fabric.“ First of 
> all, this informational document should not use normative language here. And second, if DCTCP is used and tested by Microsoft only with a certain configuration, I would remove this sentence and write rather something like:
> „Even though the actual
>   algorithm for marking congestion is an implementation detail of the
>   switch and will generally not be known to the sender and receive, 
>   DCTCP has been design and evaluated to be used with the 
> configuration describe above.“
> 
> 3) First paragraph in the intro says that the problem with low cost switches is high loss. I actually though that the problem with the standard config of low cost switches is high delay because there is too much buffering… can you comment on this?

>> Low cost switches have shallow buffers so burst losses are a problem. 

> 
> 4) Second paragraph says "worker nodes complete at approximately the same time“. I thought that they would even complete at exactly the same time because they often have a timeout..? Can you comment here as well? Maybe also refer the DCTCP paper here because it contains a good analysis of incast.
> 

>> The workloads are partitioned to the same size datasets and assuming a uniform network and server configuration the tasks completing at the same time is highly likely. 

> 5) Paragraph on ECN in the intro says
> "In the presence of mild congestion, it reduces the TCP congestion window too
>   aggressively…"
> ‚It‘ refers here to the ECN mechanism of rfc3168. However, the congestion control reduces the window. Therefore the sentence should rather be:
> "In the presence of mild congestion, traditional TCP congestion 
> control reduces the TCP congestion window too aggressively…“
> 

>> We fixed some text in the introduction. Please review.

> 6) Abstract and intro both say „DCTCP enhances ECN…“; I really don’t think ‚enhance' can be used here so generally; if the congestion control only needs one feedback signal per RTT, there is no enhancement. Maybe say ‚change‘ or ‚improves the accuracy of the ECN feedback signal by signal not only the occurrence of congestion but also its extend‘… or something like this.
> 

>> We altered the language somewhat to make this clearer " Datacenter TCP (DCTCP) improvises upon traditional ECN processing by ...".

> 7) As also mentioned above, I think the intro should say that the more aggressive congestion control of DCTCP should not be used in parallel with traditional congestion as it will starve this traffic.
> 

>> Addressed in section 5 (Deployment issues)

> 8) I’d like to propose to use the list given in section 3 not only to say which elements are ‚involved‘ but also very briefly say what was ‚modified‘, e.g.
> 
> "There are three components that needs to be modified to use DCTCP:
> 
>   o  The switch (or other intermediate device on the network) detects
>      congestion and sets the Congestion Experienced (CE) codepoint in
>      the IP header already when a very short queue builds up. 
> 
>   o  The receiver echoes each occurrence of a the Congestion Experienced (CE) mark
>      back to the sender using the ECN-Echo (ECE) flag in the TCP header.
> 
>   o  The sender reacts to the congestion indication by reducing the TCP
>      congestion window (cwnd) based on the extend of congestion 
> experienced in the last RTT.“
> 
> I think this given a nice high-level overview and helps the reader to easier understand the rest.
> 
> 9) Maybe add  to section 3.2 the state machine image as provided in 
> the paper. I personally would,  in an RFC, also describe the algo 
> first and then give the reasoning but that’s a matter of which style 
> is preferred…
> 

>> Other implementations seem to have got the algorithm correct without needing more explanation here. If you think it's not clear enough, we will revisit adding more details in next revision. 

> 10) Section 3.3: Quick question: Why is the fraction of marked packets called M in the draft while it’s F in the paper?
> 

>> There has been no attempt to reconcile the draft with the paper to use the same terminology. 

> 11) Again Section 3.3: Maybe mention somewhere that the SACK information are not used to calculate BytesAcked because usually it’s assumed to have no loss with the use of DCTCP. However, I guess if a calculation as given in RFC6937 (PRR) for DeliveredData is used, that is probably okay as well and should not be a problem, right?
> 

>> That's correct. Loss handling will have to be addressed in a future revision. The current implementation does not use SACK information.

> 12) Further section 3.3 says "when the sender receives an indication of congestion“. Can you maybe be a little more specify here and say „if the first ECE is received in a RTT“ or something like this. Please also note that this behavior will reduce the congestion window when the marking fraction is low because if will reduce with the first occurrence of ECE and all other ECE markings will occur in the following RTT where the window is not further reduced. Therefore the window will only really be reduced if the congestion stays for more than on RTT (that delays the reaction to congestion a little). That’s okay because this allows to not react to short-bursts of congestion. However, I think this should be mentioned/discussed as it might not be obvious when reading the draft.
> 

>> We fixed some text here so please review again.

> 13) Next sentence is
> "Thus, when no sent byte experienced congestion, DCTCP.Alpha equals
>   zero, and cwnd is left unchanged.“
> I think this sentence is nonsensical as the previous sentence says the 
> congestion window will only be reduces if congestion is signaled…
> 

>> We fixed some text here so please review again.

> 14) As already mention, section 5 should also mention deployment problems if used in parallel to traditional TCP congestion control.
> 

>> Yes already addressed

> 15) Section 6 says:
> "However, if an
>   ACK packet is dropped, it's possible that a subsequent ACK will
>   indeed acknowledge a mix of CE and non-CE segments.“ Either I don’t 
> understand this sentence or it is not true. To my understanding an ACK is either ECE set or not and therefore might ack CE marks or not. However if e.g. only one ACK is ECE marked and this ack gets lost, the following non-ECE marked ACK will accumulatively ack all packets as not-CE marked and this information is completely lost.
> 

>> Yes there is information loss due to ACK loss and that is what section 6 is attempting to explain. Do you have any suggestions for better text?

> Thanks again for writing this document. I hope it can be published soon!
> 
> Mirja
> 
> 
> 
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

_______________________________________________
tcpm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tcpm

---

From: Mirja Kühlewind <[email-masked]>
Date: Mon, 13 Jul 2015 13:24:14 +0200

Hi Praveen,

thanks for updating. See a few comments/responses below.

Mirja

>>> Windows Server does not adjust the value of Alpha for the round if loss is detected. We are under discussion with Daniel about addressing this in the next revision of the draft under Implementati…

Great! I really would like to see loss handling addressed!

>> configuration describe above.“

I’d like to add here that in a datacenter the AQM config might actually be known and that’s the primary use case/deployment scenario of DCTCP. So I don’t really say see why it is so important for you to say that DCTCP works with different kind of AQM. If it is design to be used in datacenters and you have tested it extensively with one AQM config, why don’t you remember this config?
>>> The workloads are partitioned to the same size datasets and assuming a uniform network and server configuration the tasks completing at the same time is highly likely. 

I was basically saying that you could make this point even stronger and say e.g. „works nodes are very likely to complete at the same time“ and remove the ‚approximately‘. But in fact that doesn’t really matter…

>>> We altered the language somewhat to make this clearer " Datacenter TCP (DCTCP) improvises upon traditional ECN processing by …".

Not sure if improvises it the best word… maybe just say ‚extends‘…?

>>> Other implementations seem to have got the algorithm correct without needing more explanation here. If you think it's not clear enough, we will revisit adding more details in next revision. 

I’ve been mention this because I’ve been playing around with the DCTCP implementation provided by Stanford a while ago. At that point of time I was using the DCTCP paper as a reference and I found the diagram super helpful. I know having graphics in ASCII art is not super nice but I guess the state machine would be easy to draw (also it might be possible in future to at real graphic to RFC…).

>>> That's correct. Loss handling will have to be addressed in a future revision. The current implementation does not use SACK information.

Maybe just add one sentence saying „SACK information is not used as it is assumed that there will be no losses if DCTCP is used in a datacenter for all traffic.“ However, if you actually plan to used the SACK information in a future revision that’s fine as well.

>>> We fixed some text here so please review again.

If I’ve seen this correctly, you’ve only added the ECE in brackets? Maybe even say
"when the sender receives the first indication of congestion (ECE) with in one RTT“ ..?

Further I’d like to see some more text (1-2 sentences) stating what I was saying above. With the reception of the first ECE, alpha is usually low (maybe even zero) and only if the congestion stays for more than one RTT it will be updated and reduced more. Maybe you also want to check this in measurements/simulations. Would be interesting to see more data here. But that’s what we’ve seen when we did our first experiments with DCTCP.

>>> We fixed some text here so please review again.

Maybe just say 
„If congestion is very low and DCTCP.Alpha equals zero, and cwnd is left unchanged.“

Because there must be at least one ECE and therefore it can not be the case that 'no sent byte experienced congestion‘.

>> 14) As already mention, section 5 should also mention deployment problems if used in parallel to traditional TCP congestion control.

Thanks!

>>> Yes there is information loss due to ACK loss and that is what section 6 is attempting to explain. Do you have any suggestions for better text?

I’d suggest the following sentence instead to make it more clear:

„However, if an ACK is dropped and the lost ACK was carrying a different marking that the subsequent accumulative ACK, the marking information will be lost.“

---

From: Praveen Balasubramanian <[email-masked]>
Date: Sun, 19 Jul 2015 04:45:54 +0000

Thanks Mirja. Some responses inline prefixed with [PB]. We plan to present the current draft revision 05 in tcpm but we will make sure the remaining feedback is addressed in the next revision.

-----Original Message-----
From: Mirja Kühlewind [mailto:[email-masked]] 
Sent: Monday, July 13, 2015 4:24 AM
To: Praveen Balasubramanian
Cc: Eggert, Lars; Dave Thaler; Stephen Bensley; [email-masked]
Subject: Re: [tcpm] Review of draft-bensley-tcpm-dctcp-03

Hi Praveen,

Mirja

[PB] Looks like we did add a comment about this in implementation section. We would expand upon this a bit more in the next revision. 

>> DCTCP has been design and evaluated to be used with the configuration 
>> describe above.“

[PB] It is important because DCTCP can work with other forms of AQM. AQM support may be different in different commodity switches. The primary goal is to document the changes for the TCP protocol on the sender and receiver so we can achieve interop between multiple implementations of DCTCP. Going forward this would be important as IaaS catches on and datacenters run a mix of multiple OSs.

[PB] Okay we will change this in the next revision. Got the same feedback from Richard.

{PB] Fair enough, we will add the diagram in the next revision.

[PB] This is TBD as I mentioned above. If DCTCP is segregated from traditional TCP, loss should only happen in case of oversubscription such that bursts from multiple flows can overrun the buffer even before the senders can react in a round. 

>>> We fixed some text here so please review again.

If I’ve seen this correctly, you’ve only added the ECE in brackets? Maybe even say "when the sender receives the first indication of congestion (ECE) with in one RTT“ ..?

>> zero, and cwnd is left unchanged.“ I think this sentence is 
>> nonsensical as the previous sentence says the congestion window will 
>> only be reduces if congestion is signaled…
>> 
> 

Maybe just say

Thanks!