Subject: Re: [tcpm] ACK management (was Re:ACKSuppression)

From: Christian Huitema <[email-masked]>
Date: Fri, 9 Oct 2015 20:46:27 +0000

While I agree with many of you that the AQM group has no business standardizing myopic cross layers violation, I would like to switch the focus to "what should TCP do?" 

The "ACK suppression" hacks are actually in response to two issues: ACK compression, and ACK congestion.

ACK compression happens when spaced-out ACKs get somehow delivered back to back. This is not a newly discovered phenomenon, see the SIGCOMM 91 paper by Lixia Zhang, Scott Shenker and Dave Clark: http://web.cs.ucla.edu/~lixia/papers/91SIGCOMM.pdf. It seems to happen a lot in access networks where the MAC provides each station in turn with a long "transmission slot" -- slots that can carry many packets. This includes high speed Wi-Fi networks, DOCSIS modems, and probably more.

ACK congestion happens when the ACK traffic saturates a link. A typical example would be an asymmetric configuration, e.g. a high speed downlink and a very low speed uplink. Suppose for example a downlink capable of 1Gbps. That's about 80,000 1.5KB packets per second, causing 40,000 ACKS per second on the uplink. That's close to 20 Mbps of traffic, potentially enough to saturate a narrow uplink.

Should we modify the end-to-end TCP algorithm to deal with those issues? How?

-- Christian Huitema

---

From: Joe Touch <[email-masked]>
Date: Fri, 9 Oct 2015 14:27:29 -0700

> The "ACK suppression" hacks are actually in response to two issues: ACK compression, and ACK congestion.
...
> Should we modify the end-to-end TCP algorithm to deal with those issues? How?

Assuming other mitigations were already being used (packet compression
in particular), dealing with this inside TCP would require either that a
solution support all link capacity ratios or that TCP would detect and
react to these particular assymetries.

ACK congestion can be detected by tracking gaps in the received ACKs
(i.e., larger than 2*MSS jumps).

ACK compression would be detected by seeing a large number of
back-to-back ACKs.

The trouble lies with:

a) signaling the other end to send fewer ACKs
	not too hard, e.g., using a new TCP option,
	but has to be deployed

b) pacing the local bursts
	TCP can/should already be doing this, e.g., it should
	amortize the ACKs received over the measured RTT and not
	send segments much faster than the average rate

	this ought to be safe to leave active all the time

So the only real question is "when do you know you should send fewer
ACKs?" The side sending those ACKs doesn't have a unilateral way to
deduce this; we'd have to add a new feedback signal into TCP, AFAICT.

The real issue is that TCP is a feedback system. Turning down the
feedback loop makes it more granular, which makes it potentially subject
to wider oscillations - e.g., when a ACK is lost. I'm not sure that's a
good thing to do in general, and I don't think 20Mbps of feedback
unreasonable for a 1Gbps link.

Joe

---

From: [email-masked]
Date: Fri, 9 Oct 2015 17:45:36 -0400 (EDT)

Well, if the forward bottleneck link and the reverse path bottleneck in the TCP feedback loop are asymmetric, one could certainly use ECN marking (or something close to that) as the means to identify ACKs that are encountering congestion (meaning the queue into the link carrying the ACK builds up to an undesirable level).
 
ECN markings on packets that only ACK would be useful to learn about these conditions.  The approach used to handle congestion in the high-traffic direction can be used for ACK management - "if there is congestion, tell the sender to slow down".  Slowing down ACKs would then be an independent function, but the figure of merit is the same - avoided queueing delay on the path carrying the ACKs.
 
I don't know where there are common paths with a high degree of asymmetry.  Most access plant is becoming more symmetric as we know.  But presumably all TCP stacks should evolve to handle this sort of thing eventually.

---

From: Joe Touch <[email-masked]>
Date: Fri, 9 Oct 2015 14:52:15 -0700

That sounds like a plan - basically, if ACKs are being lost (and not
data), then sending fewer ACKs is a gamble that can be taken (or not) at
the discretion of that TCP.

The risk of taking that gamble is higher oscillation with loss, but
that's something the endpoint can figure out too.

Joe

On 10/9/2015 2:45 PM, [email-masked] wrote:
> Well, if the forward bottleneck link and the reverse path bottleneck in
> the TCP feedback loop are asymmetric, one could certainly use ECN
> marking (or something close to that) as the means to identify ACKs that
> are encountering congestion (meaning the queue into the link carrying
> the ACK builds up to an undesirable level).
> 
>  
> 
> ECN markings on packets that only ACK would be useful to learn about
> these conditions.  The approach used to handle congestion in the
> high-traffic direction can be used for ACK management - "if there is
> congestion, tell the sender to slow down".  Slowing down ACKs would then
> be an independent function, but the figure of merit is the same -
> avoided queueing delay on the path carrying the ACKs.
> 
>  
> 
> I don't know where there are common paths with a high degree of
> asymmetry.  Most access plant is becoming more symmetric as we know.
>  But presumably all TCP stacks should evolve to handle this sort of
> thing eventually.
> 
> 
>

---

From: Eric Dumazet <[email-masked]>
Date: Fri, 09 Oct 2015 14:54:07 -0700

> unreasonable for a 1Gbps link.

ACK load should be counted in number of packets, not bandwidth (rarely
an issue). 20 Mbits sounds small, but this is about 40,000 ACK per
second, and this sounds a lot more.

Multiply this load by 10 (10 Gbit link speed)

Is is really needed to get ~ 400,000 ack per second if a flow is sending
at 10Gbits ? What useful info could the sender get from this ACK flood,
given a ridiculous time budget to process them (and continue to
cook/feed tx packets)

We (TCP stacks) spend considerable amount of energy processing ACK
packets all over the world. What a waste.

LRO/GRO aggregates up to 45 MSS per 'super packet', so effectively hosts
are already doing ACK suppression.

That still is about 18,000 ACK per second at 10Gbit. More than enough.

In many cases, one ACK per ms is enough to maintain ACK clocking (look
also at TSO defer at sender side) and proper RTT measures, but this only
matters for flows with a throughput above 25 Mbits (2000 packets per
second)

---

From: Joe Touch <[email-masked]>
Date: Fri, 9 Oct 2015 15:08:07 -0700

On 10/9/2015 2:54 PM, Eric Dumazet wrote:
...
> given a ridiculous time budget to process them (and continue to

If this were from a single flow, then yes, this is silly - the TCP ought
to be sending larger data packets and fewer ACKs. But that's a highly
specialized case and also not necessarily one that needs to be optimized
inside the network...

> packets all over the world. What a waste.

How silly is it that UPS collects signatures for billions of packages a
year when only a very small percentage are ever challenged?

> are already doing ACK suppression.

As I already noted, it's also not all safe.

But aggregating at the receiver is a much different issue than doing so
in the network or at the transmitter. When the receiver aggregates, it
knows that the aggregate result will not have a higher impact if lost
(because it won't be lost).

Joe

---

From: Praveen Balasubramanian <[email-masked]>
Date: Fri, 9 Oct 2015 22:18:35 +0000

LRO/GRO should ideally not coalesce pure ACKs. Also LRO/GRO should be designed in a way that the host TCP/IP stack knows how many segments/bytes were coalesced and can do the right accounting for modifying TCP state variables. What I am saying is that LRO/GRO can be an information preserving packet transformation if so designed. 

I think letting the network suppress ACKs without end host knowledge can have bad consequences. But I am in favor of end host optimization that can reduce ACKs generated at receiver in conjunction with use of ABC at sender. 

-----Original Message-----
From: tcpm [mailto:[email-masked]] On Behalf Of Joe Touch
Sent: Friday, October 9, 2015 3:08 PM
To: Eric Dumazet <[email-masked]>
Cc: Christian Huitema <[email-masked]>; [email-masked]; [email-masked]; [email-masked]
Subject: Re: [tcpm] ACK management (was Re:ACKSuppression)

> Is is really needed to get ~ 400,000 ack per second if a flow is 
> sending at 10Gbits ? What useful info could the sender get from this 
> ACK flood, given a ridiculous time budget to process them (and 
> continue to cook/feed tx packets)

If this were from a single flow, then yes, this is silly - the TCP ought to be sending larger data packets and fewer ACKs. But that's a highly specialized case and also not necessarily one that needs to be optimized inside the network...

> packets all over the world. What a waste.

How silly is it that UPS collects signatures for billions of packages a year when only a very small percentage are ever challenged?

> LRO/GRO aggregates up to 45 MSS per 'super packet', so effectively 
> hosts are already doing ACK suppression.

But aggregating at the receiver is a much different issue than doing so in the network or at the transmitter. When the receiver aggregates, it knows that the aggregate result will not have a higher impact if lost (because it won't be lost).

Joe

_______________________________________________
tcpm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tcpm

---

From: Eric Dumazet <[email-masked]>
Date: Fri, 09 Oct 2015 15:24:07 -0700

> year when only a very small percentage are ever challenged?

It is rare UPS collects 400,000 identical signatures per second from a
single customer ;)

TCP receivers should not send ACK immediately.
They should bet on the immediate future.

I did a ~50 lines prototype patch in linux to arm a 1 ms timer,
instead of immediately sending one ACK.

I did this for SACK packets, trying to cope with reordering fabrics,
(per packet load balancing, not caring on flow hash),
but patch could be extended quite easily with a more fine grained timer
(like min(rtt/10, 1ms))
 
If another packet(s) is(are) coming before timer expires, we avoid
sending all these useless floods of ACK packets, and avoid intermediate
nodes trying to be smart and do their own ACK compression.

---

From: [email-masked]
Date: Sat, 10 Oct 2015 13:23:09 -0400 (EDT)

The idea of "dallying" (waiting a small amount of time before sending an ACK in hope that there will be a piggyback opportunity) was invented, as far as I know, by my colleague Dave Clark in is original expermental TCP implementation on the Xerox Alto (written by Dave in BCPL).  That was done around 1978, shortly after Xerox donated about 20 Altos to MIT's Lab for Computer Science.
 
So it is an old idea, and in some cases it helps with throughput- in particular, when there is a bidirectional flow of data (as happens in protocols like SMTP), or where the destination has the ability to buffer a large number of early TCP segments.
 
However, at high speed, (10 GigE, for example), this "dallying" typically requires the receiving host to take an extra interrupt per ack - basically a timer interrupt needs to be set up to go off after the dally delay expires.  In most OS's, the overhead in timer management code to insert a new event into the queue, operate on the timer, handle the fact that the interrupt might arrive on a different cpu core than the one that originally received the packet, ...  well, you get the idea.
 
Making a statement that "dallying for a millisecond" should always be done doesn't take into account the costs of doing this, vs. the benefit, which is situation dependent.  Most ACKs also adjust the receive window, which reflects into the flow control loop the actual rate at which the receiving process consumes the data.  Keeping the window "constant-sized" and no more than the measured RTT * observed bottleneck rate is also beneficial, because it enables fair usage of the bottleneck link, without oscillations due to receiving process consumption being coupled into the bottleneck link queue size control.   More ACKs can adjust the receive window more gradually.
 
(it should be noted that Dave also named the so-called "silly window syndrome" using his BCPL experimental TCP at about the same time.  Silly window is one of the simplest issues that arise from tuning the receive window to track consumption and avoid intermediate packet buildup)