Subject: [tcpm] Some comments on draft-ietf-tcpm-dctcp

From: "Scharf, Michael (Nokia - DE)" <[email-masked]>
Date: Mon, 9 May 2016 18:25:14 +0000

Dear authors,

As individual contributor to TCPM, I have read -01 of the DCTCP draft and I've run into some minor issues:

* Section 1: I think the following wording was discussed already in the past:

   It is recommended that DCTCP be deployed in a datacenter environment
   where the endpoints and the switching fabric are under a single
   administrative domain.  This protocol is not meant for uncontrolled
   deployment in the global Internet.

  Reading the document again, I'd prefer a wording that would not generally recommend DCTCP to all single domain datacenter environments. The smallest change I could think of would be to add the word "only":
   
   It is recommended that DCTCP be only deployed in a datacenter environment
   where the endpoints and the switching fabric are under a single
   administrative domain.  This protocol is not meant for uncontrolled
   deployment in the global Internet.

* Section 2+3: This is an informative document so RFC 2119 should be used with care, if at all. My impression is that at least some use of RFC 2119 language could be avoided. Examples:

    Section 3.1: "Therefore, sender and receiver MUST NOT assume that a particular marking algorithm is implemented by the switching fabric."

    => I think an implementation of DCTCP inherently follows this and this does not require RFC 2119 statements.

    Section 3.3: "The congestion estimator on the sender MUST process acceptable ACKs as follows:"

    => This is a sender-side algorithm that relies on internal TCP state variables. I am not sure if an implementation really has to implement the algorithm exactly in this way or if slight variations could yield the same result. A "MUST" on the exact algorithm seems a bit restrictive in that case.  

    (... possibly more ...)

* Section 4: This section discusses both discusses implementation issues for DCTCP as well as further TCP tuning that may be used in combination with DCTCP (lower MinRTO, cwnd idle, ...) but that is not tightly related to the actual DCTCP protocol mechanisms. Also, partly related to this, Section 4 is a relatively long text without subsections. I think that Section 4 could be better structured, e.g., by adding two or three subsection distinguishing DCTCP implementation from other (related) implementation guidance. The actual content may not require rewording, except possibly some reordering of paragraphs.

* Section 5: I am not sure if the following comment on UDP is compatible with draft-ietf-tsvwg-rfc5405bis-12:

   If the datacenter traffic consists
   of such traffic (including UDP), one possible mitigation would be to
   mark IP packets as ECT even when there is no transport that is
   reacting to the marking.

  Also, I don't fully understand the contect, e.g., what marking/drop profiles are here referred to. Maybe some additional explanation could help. As far as I know, this can be quite device-specific, e.g., many routers would allow separate queues for non-TCP/non-IP traffic. In any case, the handling of UDP seems to me rather unrelated to DCTCP, right? If this is considered useful information for potential implementers, Section 5 should also be better structured into DCTCP deployment issues and other related deployment experiences in environments running DCTCP.

Thanks

Michael

---

From: "Eggert, Lars" <[email-masked]>
Date: Mon, 9 May 2016 19:12:21 +0000

Hi,

on this particular point, a comment:

On 2016-05-09, at 20:25, Scharf, Michael (Nokia - DE) <[email-masked]> wrote:
> * Section 2+3: This is an informative document so RFC 2119 should be used with care, if at all. My impression is that at least some use of RFC 2119 language could be avoided.

Understood, but RFC2119 terms are well-understood when specifying protocol behavior, whether it's for standards track or other document levels. So I'd personally like to keep them, because they bring clarity. (Plus, I've frequently seen them even in requirements documents, etc.)

Since the intent of this document is to describe exactly what the Windows, Linux and FreeBSD implementations are doing, IMO such language is helpful.

> Examples:

Not sure. I think it stresses that an implementation that wants to conform to what MS, FreeBSD and Linux are doing in terms of DCTCP cannot be cognizant of what the fabric does. (There are optimizations that are possible in that case, but they are not DCTCP.)

>    => This is a sender-side algorithm that relies on internal TCP state variables. I am not sure if an implementation really has to implement the algorithm exactly in this way or if slight variatio…

But with "slight modifications" it isn't DCTCP as implemented in Windows, Linux and FreeBSD anymore. Sure, if this was an IETF-designed protocol we could have a discussion on whether we want to give implementors leeway here. But if they want to implement what Windows, Linux and FreeBSD are doing, there isn't leeway.

Lars

---

From: "Scharf, Michael (Nokia - DE)" <[email-masked]>
Date: Mon, 9 May 2016 20:05:05 +0000

> >    => This is a sender-side algorithm that relies on internal TCP
> state variables. I am not sure if an implementation really has to
> implement the algorithm exactly in this way or if slight variations
> could yield the same result. A "MUST" on the exact algorithm seems a
> bit restrictive in that case.
> 
> But with "slight modifications" it isn't DCTCP as implemented in
> Windows, Linux and FreeBSD anymore. Sure, if this was an IETF-designed
> protocol we could have a discussion on whether we want to give
> implementors leeway here. But if they want to implement what Windows,
> Linux and FreeBSD are doing, there isn't leeway.

Well, I don't know about Windows and FreeBSD and in general I have not looked at Linux TCP for quite some time.

But since you mention that this is exactly what Linux does, let's check...

>From https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/net/ipv4/tcp_dctcp.c?id=refs/tags/v4.6-rc7

static void dctcp_update_alpha(struct sock *sk, u32 flags)
{
	const struct tcp_sock *tp = tcp_sk(sk);
	struct dctcp *ca = inet_csk_ca(sk);
	u32 acked_bytes = tp->snd_una - ca->prior_snd_una;

	/* If ack did not advance snd_una, count dupack as MSS size.
	 * If ack did update window, do not count it at all.
	 */
	if (acked_bytes == 0 && !(flags & CA_ACK_WIN_UPDATE))
		acked_bytes = inet_csk(sk)->icsk_ack.rcv_mss;
	if (acked_bytes) {
		ca->acked_bytes_total += acked_bytes;
		ca->prior_snd_una = tp->snd_una;

		if (flags & CA_ACK_ECE)
			ca->acked_bytes_ecn += acked_bytes;
	}
...

So, formally, Linux violates the MUST, as far as I can see, since the two "if" statements are not foreseen in the specification, as far as I can see.

I personally believe that this sort of deviation makes sense, but at first sight I fail to see this specific algorithm in the draft and it is not what the MUST asks for.

Also, looking further at the Linux code I believe Linux does not fully implement the following MUST:

   Rather than always halving the congestion window as described in
   [RFC3168], when the sender receives an indication of congestion
   (ECE), the sender MUST update cwnd as follows:

      cwnd = cwnd * (1 - DCTCP.Alpha / 2)

In this case it seems to me that Linux adds a lower bound of 2 (MSS) not foreseen in the specification.

This is nit-picking and this sort of algorithm changes seem to me reasonable implementation choices that will not affect any interoperability. But, at least at first sight, the Linux code does not *exactly* what the specification describes. I believe we should be careful with specifying algorithms.

Note that I neither want to say that the I-D nor that the Linux implementation would have to change. I am just not sure what a RFC 2119 "MUST" should really mean for a sender-side only TCP algorithm that can be implemented in many ways more or less resulting in the same macroscopic behavior.

Michael