Subject: Re: [tcpm] [aqm] TCP ACK Suppression

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Wed, 7 Oct 2015 17:58:42 +0000

This thread is may be relevant to TCPM

Michael

-----Original Message-----
From: aqm [mailto:[email-masked]] On Behalf Of Jonathan Morton
Sent: Wednesday, October 07, 2015 7:53 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [aqm] TCP ACK Suppression

> On 7 Oct, 2015, at 20:36, David Collier-Brown <[email-masked]> wrote:
> 
> On 07/10/15 01:19 PM, David Lang wrote:
>> On Wed, 7 Oct 2015, Mikael Abrahamsson wrote:
>> 
>>>> Oh, I hope that this is an exception. Such kind of optimizations may cause a lot of trouble since a link layer device is interfering with transport layer semantics. We all know that exactly these kinds of interference eventually end up in problems with end-to-end transparency and deployment of new protocol options. At least it interferes with the ACK clocking expectation of some congestion control algorithms...
>>> 
>>> Personally, I think you're going to see more and more of this. There are mulitple shared access medium where you're allowed to send only part of the time, and it's someone else who tells you when you may send.
>> 
>> it doesn't even require that someone else tells you when you may send. It can just be waiting for an available transmit timeslot (Wifi for example)
>> 
>> collapsing multiple ACKs that are going to be sent at once is almost always going to be a win.
> 
> I quite agree, but if there is a congestion control implementation "in the wild" that assumes it will get a stream of acks, that one's going to need some work (:-))
> 
> Anyone know if that's the case? The comment above suggest it may be...

It’s not “in the wild”, but this sort of thing is a headache for ELR.  It essentially means that it won’t be able to use AccECN for its feedback (since AccECN doesn’t allow reconstructing a complex 32-packet history of ECN codepoints from a single ACK), but must introduce some other mechanism to feed back the required data to the sender.

I’m leaning towards moving the ELR-aware congestion control algorithm to the receiver, and having the feedback simply be the requested cwnd; it’s the simplest and most compact solution I can think of, and would work for any congestion control scheme, not just ELR.  The sender may of course maintain its own idea of cwnd in the conventional way, and conservatively take the minimum of the two as effective; this would protect against naively greedy receivers.

I note that CUBIC doesn’t care how many acks it gets, since its scaling curve depends only on time so long as the data flow is not application-limited.  That’s good news for Linux, given the lack of ABC.  Do the BSDs and Windows use ABC?

 - Jonathan Morton

_______________________________________________
aqm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/aqm

---

From: Praveen Balasubramanian <[email-masked]>
Date: Wed, 7 Oct 2015 19:03:13 +0000

Thanks Michael, it is indeed very relevant. 

Windows does perform stretch ACKs as an optimization. However, ACK coalescing is limited in the worst case since packets are processed by batching (which is limited to a size N). That is, with a single connection and very small inter packet spacing, Windows will generate one ACK for N segments. In practice however there are multiple flows and packet spacing such that ACKs cover a number much less than N. The receiver ACKing behavior is hence not completely deterministic and depends on packet arrival rate, interrupt and DPC (~ deferred work from interrupts) rates and number of connections etc. We are looking into the impact of this behavior on sender window growth especially during slow start. A possible mitigation is to reduce the size N or introduce a per connection limit M to ensure a single delayed ACK acknowledges no more than M segments. I already provided feedback on the AccECN proposal and it looks like the limit M would need to be 6 when processing CE marked packets which doesn’t seem onerous. 

The ACKing behavior directly ties in to ABC on the sender. Windows does support ABC during both slow start and congestion avoidance and bases the magnitude of the cwnd increase on the number of newly acknowledged bytes (per RFC 3465) up to a fixed factor X of the MSS. All the congestion algorithms supported in Windows like NewReno, CTCP and DCTCP follow this behavior, and CTCP takes advantage the dwnd (delay window) growth as well. It might be useful to have X = M.

Any thoughts on stretch ACKs as an optimization when used in conjunction with ABC? 

Thanks

-----Original Message-----
From: tcpm [mailto:[email-masked]] On Behalf Of Scharf, Michael (Michael)
Sent: Wednesday, October 7, 2015 10:59 AM
To: [email-masked]
Subject: Re: [tcpm] [aqm] TCP ACK Suppression

Michael

-----Original Message-----
From: aqm [mailto:[email-masked]] On Behalf Of Jonathan Morton
Sent: Wednesday, October 07, 2015 7:53 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [aqm] TCP ACK Suppression

 - Jonathan Morton

_______________________________________________
aqm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/aqm
_______________________________________________
tcpm mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/tcpm

---

From: hiren panchasara <[email-masked]>
Date: Wed, 7 Oct 2015 12:05:28 -0700

On 10/07/15 at 05:58P, Scharf, Michael (Michael) wrote:
> It?s not ?in the wild?, but this sort of thing is a headache for ELR.  It essentially means that it won?t be able to use AccECN for its feedback (since AccECN doesn?t allow reconstructing a complex 32-packet history of ECN codepoints from a single ACK), but must introduce some other mechanism to feed back the required data to the sender.
> 
> I?m leaning towards moving the ELR-aware congestion control algorithm to the receiver, and having the feedback simply be the requested cwnd; it?s the simplest and most compact solution I can think of, and would work for any congestion control scheme, not just ELR.  The sender may of course maintain its own idea of cwnd in the conventional way, and conservatively take the minimum of the two as effective; this would protect against naively greedy receivers.
> 
> I note that CUBIC doesn?t care how many acks it gets, since its scaling curve depends only on time so long as the data flow is not application-limited.  That?s good news for Linux, given the lack of ABC.  Do the BSDs and Windows use ABC?
>

FreeBSD uses ABC and is turned on by default.

Cheers,
Hiren

---

From: Joe Touch <[email-masked]>
Date: Thu, 8 Oct 2015 14:27:27 -0700

CC'ing TCPM on my responses to this thread...

On 10/7/2015 8:50 AM, Francini, Andrea (Andrea) wrote:
> How about the effect of ACK suppression on the growth of the
> congestion window, for TCP sources where the trigger for window growth
> is the arrival of an ACK, not the number of bytes it acknowledges?
> 
> The Appropriate Byte Counting (ABC) option
> (https://tools.ietf.org/html/rfc3465) was addressing a similar issue
> with delayed ACKs, but it looks like it is no longer available in Linux
> (http://www.spinics.net/lists/netdev/msg225164.html).

Even with ABC, killing off some of the ACKs would result in sender
bursts because you're interfering with ACK clocking -- unless ABC were
coupled with some sort of rate pacing.

Note also that when you kill off intermediate ACKs you're increasing the
round-trip delay. The sender can't start sending what you ACK until the
ACK arrives; if you kill off earlier ACKs, then the later ACK will (by
definition) arrive later.

This is bad on many levels.

Joe

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Fri, 9 Oct 2015 08:58:05 +0000

Why don't you write up your findings in a short I-D? Based on the length of this thread, it seems that there is some community interest in further discussing pros and cons of stretch ACKs.

Michael

-----Original Message-----
From: Praveen Balasubramanian [mailto:[email-masked]] 
Sent: Wednesday, October 07, 2015 9:03 PM
To: Scharf, Michael (Michael); [email-masked]
Subject: RE: [aqm] TCP ACK Suppression

Thanks

-----Original Message-----
From: tcpm [mailto:[email-masked]] On Behalf Of Scharf, Michael (Michael)
Sent: Wednesday, October 7, 2015 10:59 AM
To: [email-masked]
Subject: Re: [tcpm] [aqm] TCP ACK Suppression

Michael

-----Original Message-----
From: aqm [mailto:[email-masked]] On Behalf Of Jonathan Morton
Sent: Wednesday, October 07, 2015 7:53 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [aqm] TCP ACK Suppression

 - Jonathan Morton

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Fri, 9 Oct 2015 09:21:30 +0000

[Removing AQM, adding TCPM]

Hi Joe,

I don’t find this text in RFC 793, but it is contained in RFC 2581, which is obsolete by RFC 5681.

RFC 5681 states:

4.2.  Generating Acknowledgments

   The delayed ACK algorithm specified in [RFC1122] SHOULD be used by a
   TCP receiver.  When using delayed ACKs, a TCP receiver MUST NOT
   excessively delay acknowledgments.  Specifically, an ACK SHOULD be
   generated for at least every second full-sized segment, and MUST be
   generated within 500 ms of the arrival of the first unacknowledged
   packet.

   The requirement that an ACK "SHOULD" be generated for at least every
   second full-sized segment is listed in [RFC1122] in one place as a
   SHOULD and another as a MUST.  Here we unambiguously state it is a
   SHOULD.  We also emphasize that this is a SHOULD, meaning that an
   implementor should indeed only deviate from this requirement after
   careful consideration of the implications.  See the discussion of
   "Stretch ACK violation" in [RFC2525] and the references therein for a
   discussion of the possible performance problems with generating ACKs
   less frequently than every second full-sized segment.

Thus, according to our current standards deviation from this SHOULD is possible ”after careful consideration of the implications”.

Michael

From: aqm [mailto:[email-masked]] On Behalf Of Yuchung Cheng
Sent: Thursday, October 08, 2015 11:52 PM
To: David Lang
Cc: LAUTENSCHLAEGER, Wolfram (Wolfram); Greg White; [email-masked]; Joe Touch
Subject: Re: [aqm] TCP ACK Suppression

On Thu, Oct 8, 2015 at 2:31 PM, David Lang <[email-masked]<mailto:[email-masked]>> wrote:
On Thu, 8 Oct 2015, Joe Touch wrote:
On 10/7/2015 12:42 AM, LAUTENSCHLAEGER, Wolfram (Wolfram) wrote:
...
Is this topic addressed in some RFC already?

It's a direct violation of RFC793, which expects one ACK for every two
segments:

4.2 Generating Acknowledgments

  The delayed ACK algorithm specified in [Bra89] SHOULD be used by a
  TCP receiver.  When used, a TCP receiver MUST NOT excessively delay
  acknowledgments.  Specifically, an ACK SHOULD be generated for at
  least every second full-sized segment, and MUST be generated within
  500 ms of the arrival of the first unacknowledged packet.

actually, this is only a violation of the SHOULD section, not the MUST section.

And if the Ack packets are going to arrive at wire-speed anyway (due to other causes), is there really an advantage to having 32 ack packets arriving one after the other instead of making it so that the first ack packet (which arrives at the same time) can ack everything?

I would like to add that GRO also cause stretched ACKs (acking up to 64KB), and GRO is crucial to reduce cycles for +10Gbps transfers on some architectures.

And if there is such an advantage, does it outweight the disadvantages that the extra ack packets cause by causing highly asymmetric links to be overloaded and drop packets?

David Lang

_______________________________________________
aqm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/aqm

---

From: Wesley Eddy <[email-masked]>
Date: Fri, 9 Oct 2015 09:44:31 -0400

> which is obsolete by RFC 5681.

To confuse matters further, there's also 1122 text with requirements
language:

            A TCP SHOULD implement a delayed ACK, but an ACK should not
            be excessively delayed; in particular, the delay MUST be
            less than 0.5 seconds, and in a stream of full-sized
            segments there SHOULD be an ACK for at least every second
            segment.

This probably belongs in 793bis.  In my view, normative acknowledgement
behavior belongs in the base spec rather than particular congestion
control descriptions, since there may be more than CC that depends
upon it or makes assumptions about it.

-- 
Wes Eddy
MTI Systems

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Fri, 9 Oct 2015 15:21:23 +0000

Yes, I have realized today myself that 793bis lacks that section. I agree that ACK generation belongs into 793bis.

793bis actually may open an opportunity to rethink the exact wording that explains this SHOULD. Personally, I believe replacing this SHOULD is not an option for a bis document.

But if deviations from that SHOULD are widely deployed in the meantime, 793bis could possibly comment on that. Also, having some informational reference could perhaps be an option. 

Michael

-----Original Message-----
From: Wesley Eddy [mailto:[email-masked]] 
Sent: Friday, October 09, 2015 3:45 PM
To: Scharf, Michael (Michael); Joe Touch
Cc: LAUTENSCHLAEGER, Wolfram (Wolfram); Greg White; [email-masked]; David Lang
Subject: Re: [tcpm] [aqm] TCP ACK Suppression

On 10/9/2015 5:21 AM, Scharf, Michael (Michael) wrote:
> I don't find this text in RFC 793, but it is contained in RFC 2581, 
> which is obsolete by RFC 5681.
> 
>  

This probably belongs in 793bis.  In my view, normative acknowledgement behavior belongs in the base spec rather than particular congestion control descriptions, since there may be more than CC that depends upon it or makes assumptions about it.

--
Wes Eddy
MTI Systems

---

From: Joe Touch <[email-masked]>
Date: Fri, 9 Oct 2015 10:32:50 -0700

> which is obsolete by RFC 5681.

I think I pasted it from 1122, but had a tab open to 793 and mistook it
for being there as well.

...
...
> Thus, according to our current standards deviation from this SHOULD is
> possible ”after careful consideration of the implications”.

Agreed. And "careful consideration" does not consist of "I deployed it
silently, didn't track what happened, and didn't hear whether anyone had
any problems I wasn't listening for".

Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 9 Oct 2015 10:34:37 -0700

On 10/9/2015 8:21 AM, Scharf, Michael (Michael) wrote:
> 793bis actually may open an opportunity to rethink the exact wording
> that explains this SHOULD. Personally, I believe replacing this
> SHOULD is not an option for a bis document.

Yes, and as I noted, any SHOULD needs to include an explanation of the
kind of situation that was prevented picking MUST or the conditions
under which the SHOULD might be ignored.

Otherwise, SHOULD becomes equivalent to MAY, or - in the current
environment - "MIGHT, UNTIL IT INTERFERES WITH MY REVENUE MODEL".

Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 9 Oct 2015 10:58:50 -0700

Dave in particular wanted some specific reasons this is a bad idea as
presented. Here is my summary.

The rest of this discussion should happen on TCPM.

Joe

---

1) *you* shouldn't be using a mechanism that destroys information for others

	whether the timestamps or ACK stream spacing has any
	meaning is for the receiver - not you - to decide

2) *you* don't know where your mechanism will have an impact

	those clumped ACKs might be spaced out further
	downstream

	even if you have a rule of "I'll only gather 3 ACKs",
	you can't know if another box - including yours -
	downstream might gather those composite ACKs further

3) you claim this might be safe *if* AQM is widely deployed

	but *you* don't appear to be making that determination
	*before* deploying your approach

	also, AQM is in the opposite direction, and unless
	you deploy this with enough state to track the fact
	that your box is seeing traffic in both directions,
	you shouldn't be turning it on at all

As others have noted, the SHOULD of ACK requirements can be exceeded,
but only with careful consideration of the potential impact. That
careful consideration does not consist of "I turned it on and I didn't
hear anyone scream".

---

---

From: Joe Touch <[email-masked]>
Date: Fri, 9 Oct 2015 11:58:39 -0700

On 10/9/2015 11:52 AM, David Lang wrote:
> On Fri, 9 Oct 2015, Joe Touch wrote:
> 
> The temporal spacing is already lost.

You don't know that won't be restored downstream.

> this discussion started in the context of AQM. AQM is needed in both
> directions, it's not a one-direction only thing.

Needed, but you don't *know* it's deployed on both path directions. The
reverse path could be taking a different route.

> that's not the only consideration here. There's also the consideration
> that the ACK stream is already badly distorted.

It's not distorted at all - it's queued up and waiting its turn.

> As it turns out (as discussed on the other thread), the RFC actually
> recommends what we was proposed (with some additional nuances)

Assuming that you know what you can't know and assuming that you don't
care whether your actions affect what others have a right to assume.

Joe

---

From: Mark Allman <[email-masked]>
Date: Fri, 09 Oct 2015 16:04:19 -0400

> 3) you claim this might be safe *if* AQM is widely deployed

tl;dr summary: myopia is why we can't have nice things

allman

--
http://www.icir.org/mallman/

---

From: Joe Touch <[email-masked]>
Date: Fri, 9 Oct 2015 13:07:20 -0700

> tl;dr summary: myopia is why we can't have nice things

Actually, it's "be conservative in what you do, be liberal in what you
accept".

Joe

---

From: Greg White <[email-masked]>
Date: Fri, 9 Oct 2015 22:02:02 +0000

As I said earlier in this thread, I'm listening (and it seems others are
too).  While I can't take credit for the initial deployments in DOCSIS
close to 15 years ago, I did have a hand in adding the recommendations in
the DOCSIS 3.0 spec nearly 10 years ago.  I am genuinely interested in
specific problems that have occurred as a result of this practice, so that
their impact can be weighed against the established benefits.  So far, it
seems the objections have been mostly philosophical, which doesn't
necessarily negate them, but it is harder to weigh philosophy against
measurable benefit.

There have been some inflammatory (and uninformed) statements made in this
thread about cable ISPs and the motivations for this practice, but I think
the truth is simply that engineers try to make things work better.  This
is a case where a fairly simple optimization really appears to make things
work better for the observable universe.  If there are specific cases
where things are made worse, then perhaps some further guidance can be
given to implementers in a RFC3449bis in order to avoid those problems.

In terms of the deleterious impact that this has on the world, I'm unclear
where we stand.  If intelligently discarding superfluous ACKs resulted in
worse performance for the TCP session, it would never have been deployed
(so I need more help understanding the arguments about it destroying
information that TCP needs).

Delivery of IP packets in real networks is rarely guaranteed, and
transports need to deal with missing packets.  The status quo until
recently was that a single drop-tail queue with buffering equivalent to
the expected worst case Internet RTT (or I've even heard some vendors
doubling that for good measure) was considered the right thing to do.
>From an end-to-end perspective, that seems good, simple, and predictable.
Things work. They may not work really well, but they work.  TCP (per RFC)
isn't perfect. In fact it has contributed significantly to the problems
with network performance, and is very difficult to change, so the
equipment industry has tried to find ways to make the network work better
(ACK prioritization, ACK suppression, AQM, flow queuing, etc.), largely by
being more intelligent about dropping packets.  AQM adds intelligence and
decision making into the network that some find objectionable (and,
despite what RFC7567 says, all AQMs make an implicit assumption that the
majority of traffic causing a standing queue is TCP that will respond to
loss in a certain way), but today it is widely considered to be better
than the alternative.

In the big picture this is definitely a tradeoff (simple and predictable
network vs performant network).  But for device vendors (and network
operators) good pragmatic solutions that provide high performance are
likely going to win out over philosophical stances that don't.  If TCP
gets fixed so it only sends on the order of 1 ACK/ms, or it doesn't bloat
queues, then the need for these sorts of pragmatic solutions goes away and
people stop doing them, but until then?

-Greg

On 10/9/15, 11:32 AM, "Joe Touch" <[email-masked]> wrote:

>> I don¹t find this text in RFC 793, but it is contained in RFC 2581,
>> possible ²after careful consideration of the implications².
>

---

From: Greg White <[email-masked]>
Date: Fri, 9 Oct 2015 22:02:24 +0000

>tl;dr summary: myopia is why we can't have nice things

Too true.  DOCSIS would have been much cleaner if we didn't have to deal
with the fallout from the myopic TCP designers.  :-P

---

From: Joe Touch <[email-masked]>
Date: Fri, 9 Oct 2015 15:10:17 -0700

> with the fallout from the myopic TCP designers.  :-P

Wouldn't it have been cleaner with more appropriate network provisioning?

Why are you blaming TCP?

A DOCSIS router ought to route. If it left the packets alone, IMO we'd
all be better off.

Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 9 Oct 2015 15:24:19 -0700

On 10/9/2015 3:16 PM, David Lang wrote:
...
> "more appropriate network provisioning" is not always going to result in
> more bandwidth the way you want it to.
> 
> If there is established infrastructure that can handle X in one
> direction and 100X in the other direction, but "appropriate network
> provisioning" requires that the ratio never be more than 3x, it's not
> going to magically increase bandwidth in one direction, all it can do is
> cap bandwidth in the other direction (throwing away capacity)

Sure, but we're dealing with a problem that arises when the ratio can't
support 40:1. That's quite an asymmetry except in extreme cases where we
already know extreme measures are required (e.g., satcom with telco
backchannels).

>> Why are you blaming TCP?
> 
> first off, didn't you see the smiley?
> 
> But treating your question as serious, it's because as RFC3449 shows,
> highly assymetric networks or networks with other traffic on them don't
> work with TCP the way you want it to work. So the option is to modify
> TCP or tinker with the packets to leverage less commonly used features
> of TCP (or features that are implicit in the design rather than explicit)

The former is always on the table - that's what TCPM is for.

The latter has always been problematic, exactly because it's subject to
tragedy-of-the-commons. TCP is a contract between the endpoints, and
whenever a middlebox starts modifying those packets, IMO it becomes a
party to that contract that needs to be taken very seriously (smiley
faces aside ;-)

> So you would rather have people getting slower downloads than have
> network equipment modify packets?

I would rather people get slower downloads than break TCP in ways we
won't see for 5 years or inhibit the extension of TCP in new ways that
might be even more beneficial.

> Those are the real-world choices.

There were other cable modem companies that tried similar "tricks" in
the past and got in a lot of hot water for making a box that went a LOT
faster for *their* customers.

TCP works when everyone plays by the same conservative rules. There has
always been a choice to break those rules for individual gain, and there
has always been a reason we try very hard to avoid that.

Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 9 Oct 2015 15:39:10 -0700

On 10/9/2015 3:31 PM, David Lang wrote:
> that's not the only situation we're talking about here.
> 
> Also, keep in mind that your 40:1 ratio assumes that there is no traffic
> going the other direction.

FWIW, I wasn't - I was assuming that 40:1 is fine if what you intend to
support is browsing only.

> If you have one person trying to watch streaming video while another
> person is uploading pictures to facebook, you can run into trouble at
> much more even ratios.

Restated, you run into trouble because you sold a service you didn't
provision for ;-)

(or, more to the point, you took a gamble that you could sell a useful
service with a particular assumption about traffic ratios, and that
assumption no longer holds)

Again, IMO TCP isn't to be tampered with merely to support a business
model, and that's all I see so far.

Joe

---

From: "Richard Scheffenegger" <[email-masked]>
Date: Sat, 10 Oct 2015 00:52:54 +0200

This might also be an opportunity to go more into the reasoning for the 
500ms maximum...

Many implementations interpret  "the delay MUST be less than 0.5 seconds" to 
trigger the delayed ACK timer at 500ms (with this time a fixed, compile-time 
value) - while it has been shown, that making that dynamic (or a based on 
RTT and pps might be a more efficient approach (especially during loss at 
end-of-stream), depending on the environment [1].

[I thought to have read a recent paper on tuning the delayed ACK timer down 
to 1ms in datacenter environments, but cann't find that now].

Best regards,
  Richard

[1] Altman, Eitan, and Tania Jiménez. "Novel delayed ACK techniques for 
improving TCP performance in multihop wireless networks." Personal Wireless 
Communications. Springer Berlin Heidelberg, 2003.

----- Original Message ----- 
From: "Scharf, Michael (Michael)" <[email-masked]>
To: "Wesley Eddy" <[email-masked]>; "Joe Touch" <[email-masked]>
Cc: "LAUTENSCHLAEGER, Wolfram (Wolfram)" 
<[email-masked]>; "Greg White" 
<[email-masked]>; <[email-masked]>; "David Lang" <[email-masked]>
Sent: Friday, October 09, 2015 5:21 PM
Subject: Re: [tcpm] [aqm] TCP ACK Suppression

> 793bis actually may open an opportunity to rethink the exact wording that 
> explains this SHOULD. Personally, I believe replacing this SHOULD is not 
> an option for a bis document.
>
> But if deviations from that SHOULD are widely deployed in the meantime, 
> 793bis could possibly comment on that. Also, having some informational 
> reference could perhaps be an option.
>
> Michael
>
>
> control descriptions, since there may be more than CC that depends upon it 
> or makes assumptions about it.
>
>
> --
> Wes Eddy
> MTI Systems
>

---

From: Praveen Balasubramanian <[email-masked]>
Date: Fri, 9 Oct 2015 23:25:49 +0000

Agreed that delayed ACK timeout should be a function of RTT than a hardcoded value. Windows 8 onwards this is the behavior for high latency connections. Windows Server uses a 1 msec delayed ACK timeout for low latency (datacenter) connections. 

And while we are on the topic of delayed ACKs, the bad interaction between Nagling and delayed ACKs is well known. John Nagle's comment in HN which suggests a per connection logic to throttle delayed ACKs:
"If you're going to do that, it's a good time to fix delayed ACK as well. As is well known, the interaction between delayed ACKs and the Nagle algorithm is awful. This is a historical accident; I implemented one, the Berkeley guys implemented the other, and by the time I found out I was out of networking and involved with a small startup called Autodesk. So that never got fixed. Here's how to think about that. A delayed ACK is a bet. You're betting that there will be a reply, upon with an ACK can be piggybacked, before the fixed timer runs out. If the fixed timer runs out, and you have to send an ACK as a separate message, you lost the bet. Current TCP implementations will happily lose that bet forever without turning off the ACK delay. That's just wrong. The right answer is to track wins and losses on delayed and non-delayed ACKs. Don't turn on ACK delay unless you're sending a lot of non-delayed ACKs closely followed by packets on which the ACK could have been piggybacked. Turn it off when a delayed ACK has to be sent."

-----Original Message-----
From: tcpm [mailto:[email-masked]] On Behalf Of Richard Scheffenegger
Sent: Friday, October 9, 2015 3:53 PM
To: Scharf, Michael (Michael) <[email-masked]>; Wesley Eddy <[email-masked]>; Joe Touch <[email-masked]>
Cc: LAUTENSCHLAEGER, Wolfram (Wolfram) <[email-masked]>; [email-masked]; David Lang <[email-masked]>
Subject: Re: [tcpm] [aqm] TCP ACK Suppression

This might also be an opportunity to go more into the reasoning for the 500ms maximum...

Many implementations interpret  "the delay MUST be less than 0.5 seconds" to trigger the delayed ACK timer at 500ms (with this time a fixed, compile-time
value) - while it has been shown, that making that dynamic (or a based on RTT and pps might be a more efficient approach (especially during loss at end-of-stream), depending on the environment [1].

[I thought to have read a recent paper on tuning the delayed ACK timer down to 1ms in datacenter environments, but cann't find that now].

Best regards,
  Richard

[1] Altman, Eitan, and Tania Jiménez. "Novel delayed ACK techniques for improving TCP performance in multihop wireless networks." Personal Wireless Communications. Springer Berlin Heidelberg, 2003.

---

From: Joe Touch <[email-masked]>
Date: Fri, 9 Oct 2015 16:30:03 -0700

On 10/9/2015 4:25 PM, Praveen Balasubramanian wrote:
> And while we are on the topic of delayed ACKs, the bad interaction
> between Nagling and delayed ACKs is well known.

Nagle should be disabled except for very specific protocols such as
telnet using ASCII, i.e., those that
	a) generate large numbers of sub-MTU packets
	b) do NOT use multi-byte interactions (which could be
	split across segments and delayed due to the interaction
	referred to above)

Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 9 Oct 2015 17:01:19 -0700

FWIW, I'm using "you" metaphorically, both here and in other posts.

I.e., whomever is pushing this solution is supporting a business model
that's broken. I don't much care whether the proponents are employed by
that business or not.

There is always tension between commercial interests (and proponents)
and the model of the Internet. The Internet presents a dangerous
opportunity for incredible gain without responsibility.

The tragedy of the commons is very real, and this is a good example of
it. It doesn't matter whether the solutions are being pushed by those
profiting or by those who see only the local, short-term benefit rather
then the potential long term impact.

As has been discussed on TCPM, there may be ways to reduce the impact of
ACKs on the net without having an intermediate device interfere. Let's
have that discussion there and work together on a solution that benefits
everyone - avoiding unnecessary load on the net AND preserving the E2E
semantics of TCP so it can continue to evolve for future capabilities.

Joe

On 10/9/2015 4:31 PM, David Lang wrote:
> Where did you get the idea that I work for an ISP, let alone a cable ISP
> or had any influence in designing or building such networks. I don't.
> 
> You are assigning motives here that are clearly false.
> 
> As I keep re-stating, this is not a cable/DOCSIS issue, it applies to
> all sorts of things, with Wifi and Cellular being the newest (and
> arguably biggest) sources of the network flow getting quantomized.
> 
> David Lang

---

From: Yuchung Cheng <[email-masked]>
Date: Fri, 9 Oct 2015 17:18:40 -0700

On Fri, Oct 9, 2015 at 4:25 PM, Praveen Balasubramanian <[email-masked]
> wrote:

> Agreed that delayed ACK timeout should be a function of RTT than a
> hardcoded value. Windows 8 onwards this is the behavior for high latency
> connections. Windows Server uses a 1 msec delayed ACK timeout for low
> latency (datacenter) connections.
>
> And while we are on the topic of delayed ACKs, the bad interaction between
> Nagling and delayed ACKs is well known. John Nagle's comment in HN which
> suggests a per connection logic to throttle delayed ACKs:
> "If you're going to do that, it's a good time to fix delayed ACK as well.
> As is well known, the interaction between delayed ACKs and the Nagle
> algorithm is awful. This is a historical accident; I implemented one, the
> Berkeley guys implemented the other, and by the time I found out I was out
> of networking and involved with a small startup called Autodesk. So that
> never got fixed. Here's how to think about that. A delayed ACK is a bet.
> You're betting that there will be a reply, upon with an ACK can be
> piggybacked, before the fixed timer runs out. If the fixed timer runs out,
> and you have to send an ACK as a separate message, you lost the bet.
> Current TCP implementations will happily lose that bet forever without
> turning off the ACK delay. That's just wrong. The right answer is to track
> wins and losses on delayed and non-delayed ACKs. Don't turn on ACK delay
> unless you're sending a lot of non-delayed ACKs closely followed by packets
> on which the ACK could have been piggybacked. Turn it off when a delayed
> ACK has to be sent."
>
Well .. Mr. Nagle will be glad to learn that Linux has implemented his (or
similar) idea for years. It's called QUICK_ACK. The decision window is a
bit short and can use some tuning. But it is not "never" fixed.

Agree on that 500ms is ridiculous.

> To: Scharf, Michael (Michael) <[email-masked]>; Wesley
> Eddy <[email-masked]>; Joe Touch <[email-masked]>
> Cc: LAUTENSCHLAEGER, Wolfram (Wolfram) <
> [email-masked]>; [email-masked]; David Lang <
> [email-masked]>
> Many implementations interpret  "the delay MUST be less than 0.5 seconds"
> to trigger the delayed ACK timer at 500ms (with this time a fixed,
> compile-time
> [I thought to have read a recent paper on tuning the delayed ACK timer
> down to 1ms in datacenter environments, but cann't find that now].
>
> Best regards,
>   Richard
>
>
>
>

---

From: Joe Touch <[email-masked]>
Date: Fri, 9 Oct 2015 17:39:32 -0700

On 10/9/2015 5:22 PM, David Lang wrote:
> You don't want to acknowlege it, but TCP is broken in the face of
> excessive buffering. 

Arguably, buffering was broken and failed to provide the feedback to TCP
(see next paragraph)

> TCPM isn't fixing that, grassroots efforts are
> developing the fixes and AQM is formalizing the results.

TCPM fixed it in 2001 by providing the flags for ECN, which was enabled
by default in Windows since 2012. ALTQ support for ECN has been around
for nearly that long.

What's changed? Not the TCP reaction (except in extreme cases such as
for datacenters) but the router algs. And getting the router algs into
routers - esp. home devices.

> TCP is also broken in the face of current and within-the-next-year
> future wifi technologies. AQM helps some here, and there is again
> outside efforts to address the problem. TCPM hasn't done anything that
> I've heard of other than possibly say that the network technology is
> broken and shouldn't be used.

You might start tracking the TCPM list. We've been talking today about
how to reduce the ACKs by direct action of the source.

Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 9 Oct 2015 21:19:15 -0700

On 10/9/2015 7:14 PM, David Lang wrote:
> The problem with proposals like that is just like RFC3449 says, the
> source can't know what the network looks like to decide if there is a
> benefit to reducing ACKs.

To understand your position, is your conclusion that if it benefits a
single place in the network, then that's enough of a view to know that
it's safe and beneficial throughout the network?

Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 9 Oct 2015 23:24:10 -0700

On 10/9/2015 10:58 PM, David Lang wrote:
> RFC3449 doesn't completely address the current situation, but it
> provides a very good place to start, and it seems to me that the
> solitions it explores to address the conerns that it (and you) raise are
> actually being addressed pretty completely. There are still some areas
> to talk about (ECN interaction for example) and we wshould be talking
> about those issues rather than arguing that the proposal violates holy
> writ.

I'm OK with the recommendations for the endpoints to modify their ACK
behavior, but still am concerned that stripping out ACKs in the middle
of the network is problematic for a number of reasons, only some of
which are addressed in RFC3449 (which also recommends against deploying
them in the general Internet, so if that's your justification then it's
very clearly in direct opposition).

Joe

---

From: Joe Touch <[email-masked]>
Date: Sat, 10 Oct 2015 17:56:30 -0700

On 10/10/2015 4:06 PM, Dave Taht wrote:
> I just wanted to say that I'm with dlang here... aggregated and
> asymmetric transmits in wifi, cell, and cable, are here to stay. Deal
> with it.

Right until they're encrypted, then we'll be back to square 1.

Time to deal with the case where the net isn't involved in tampering
with E2E connections, IMO.

Joe

---

From: Joe Touch <[email-masked]>
Date: Sat, 10 Oct 2015 22:13:46 -0700

On 10/10/2015 8:45 PM, David Lang wrote:
> But I actually view this as a tragedy of the commons situation. Sending
> the extra packets doesn't hurt you, but it does mean that you are using
> more airtime than you need, and that hurts everyone (including,
> eventually, you)

So, basically, you appreciate the tragedy of the commons when it's your
tragedy and their commons, but not the other way around?

(using 'sed')

s/sending the extra/altering the transport/

s/using more airtime/altering E2E semantics/

Joe

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Sun, 11 Oct 2015 14:41:20 +0000

As far as I recall (Wes can correct me if I am wrong) we have an open issue in 793bis regarding that issue.

See draft-ietf-tcpm-rfc793bis-01 Section 4:

   11.  look at possible mention of draft-minshall-nagle (e.g. as in
        Linux)

The challenge is that for 793bis we would need a Proposed Standard RFC specifying how to sort out this interaction, unless we extend the scope of 793bis beyond what we agreed upon (i.e., only clarification, no protocol changes).

I have personally never looked at how that logic is actually implemented in different stacks, and I actually don’t know if there has ever been community consensus on how the exact solution to that interaction should look like.

But if we have a gap to what is widely implemented, and if it was realistic to publish a PS RFC before completing 793bis (I guess it could be a 2-page RFC), I’d argue in favor of a “fast track” PS publication. (Well, with ‘fast’ being relative to our normal PS publication speed.)

Michael

From: Yuchung Cheng [mailto:[email-masked]]
Sent: Saturday, October 10, 2015 2:19 AM
To: Praveen Balasubramanian
Cc: Richard Scheffenegger; Scharf, Michael (Michael); Wesley Eddy; Joe Touch; LAUTENSCHLAEGER, Wolfram (Wolfram); David Lang; [email-masked]
Subject: Re: [tcpm] [aqm] TCP ACK Suppression

On Fri, Oct 9, 2015 at 4:25 PM, Praveen Balasubramanian <[email-masked]<mailto:[email-masked]>> wrote:
Agreed that delayed ACK timeout should be a function of RTT than a hardcoded value. Windows 8 onwards this is the behavior for high latency connections. Windows Server uses a 1 msec delayed ACK timeout for low latency (datacenter) connections.

And while we are on the topic of delayed ACKs, the bad interaction between Nagling and delayed ACKs is well known. John Nagle's comment in HN which suggests a per connection logic to throttle delayed ACKs:
"If you're going to do that, it's a good time to fix delayed ACK as well. As is well known, the interaction between delayed ACKs and the Nagle algorithm is awful. This is a historical accident; I implemented one, the Berkeley guys implemented the other, and by the time I found out I was out of networking and involved with a small startup called Autodesk. So that never got fixed. Here's how to think about that. A delayed ACK is a bet. You're betting that there will be a reply, upon with an ACK can be piggybacked, before the fixed timer runs out. If the fixed timer runs out, and you have to send an ACK as a separate message, you lost the bet. Current TCP implementations will happily lose that bet forever without turning off the ACK delay. That's just wrong. The right answer is to track wins and losses on delayed and non-delayed ACKs. Don't turn on ACK delay unless you're sending a lot of non-delayed ACKs closely followed by packets on which the ACK could have been piggybacked. Turn it off when a delayed ACK has to be sent."
Well .. Mr. Nagle will be glad to learn that Linux has implemented his (or similar) idea for years. It's called QUICK_ACK. The decision window is a bit short and can use some tuning. But it is not "never" fixed.

-----Original Message-----
From: tcpm [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Richard Scheffenegger
Sent: Friday, October 9, 2015 3:53 PM
To: Scharf, Michael (Michael) <[email-masked]<mailto:[email-masked]>>; Wesley Eddy <[email-masked]<mailto:[email-masked]>>; Joe Touch <[email-masked]<mailto:[email-masked]>>
Cc: LAUTENSCHLAEGER, Wolfram (Wolfram) <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; David Lang <[email-masked]<mailto:[email-masked]>>
Subject: Re: [tcpm] [aqm] TCP ACK Suppression

Best regards,
  Richard

----- Original Message -----
From: "Scharf, Michael (Michael)" <[email-masked]<mailto:[email-masked]>>
To: "Wesley Eddy" <[email-masked]<mailto:[email-masked]>>; "Joe Touch" <[email-masked]<mailto:[email-masked]>>
Cc: "LAUTENSCHLAEGER, Wolfram (Wolfram)"
<[email-masked]<mailto:[email-masked]>>; "Greg White"
<[email-masked]<mailto:[email-masked]>>; <[email-masked]<mailto:[email-masked]>>; "David Lang" <[email-masked]<mailto:[email-masked]>>
Sent: Friday, October 09, 2015 5:21 PM
Subject: Re: [tcpm] [aqm] TCP ACK Suppression

> From: Wesley Eddy [mailto:[email-masked]<mailto:[email-masked]>]
> Cc: LAUTENSCHLAEGER, Wolfram (Wolfram); Greg White; [email-masked]<mailto:[email-masked]>; David
> Lang
> [email-masked]<mailto:[email-masked]>

_______________________________________________
tcpm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/tcpm

_______________________________________________
tcpm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/tcpm

---

From: Yuchung Cheng <[email-masked]>
Date: Sun, 11 Oct 2015 09:28:53 -0700

On Sun, Oct 11, 2015 at 7:41 AM, Scharf, Michael (Michael)
<[email-masked]> wrote:
> The challenge is that for 793bis we would need a Proposed Standard RFC
> specifying how to sort out this interaction, unless we extend the scope of
> 793bis beyond what we agreed upon (i.e., only clarification, no protocol
> changes).
>
>
>
> I have personally never looked at how that logic is actually implemented in
> different stacks, and I actually don’t know if there has ever been community
> consensus on how the exact solution to that interaction should look like.
>
>
>
> But if we have a gap to what is widely implemented, and if it was realistic
> to publish a PS RFC before completing 793bis (I guess it could be a 2-page
> RFC), I’d argue in favor of a “fast track” PS publication. (Well, with
> ‘fast’ being relative to our normal PS publication speed.)
I am not sure what your concern is: are you concerned Minshall's
revision to Nagle's algorithm on the sender would interact badly with
Mr. Nagle's idea to improve delayed-ACKs at the receiver?

> Cc: Richard Scheffenegger; Scharf, Michael (Michael); Wesley Eddy; Joe
> Touch; LAUTENSCHLAEGER, Wolfram (Wolfram); David Lang; [email-masked]
> On Fri, Oct 9, 2015 at 4:25 PM, Praveen Balasubramanian
> <[email-masked]> wrote:
>
> Agreed that delayed ACK timeout should be a function of RTT than a hardcoded
> value. Windows 8 onwards this is the behavior for high latency connections.
> Windows Server uses a 1 msec delayed ACK timeout for low latency
> (datacenter) connections.
>
> "If you're going to do that, it's a good time to fix delayed ACK as well. As
> is well known, the interaction between delayed ACKs and the Nagle algorithm
> is awful. This is a historical accident; I implemented one, the Berkeley
> guys implemented the other, and by the time I found out I was out of
> networking and involved with a small startup called Autodesk. So that never
> got fixed. Here's how to think about that. A delayed ACK is a bet. You're
> betting that there will be a reply, upon with an ACK can be piggybacked,
> before the fixed timer runs out. If the fixed timer runs out, and you have
> to send an ACK as a separate message, you lost the bet. Current TCP
> implementations will happily lose that bet forever without turning off the
> ACK delay. That's just wrong. The right answer is to track wins and losses
> on delayed and non-delayed ACKs. Don't turn on ACK delay unless you're
> sending a lot of non-delayed ACKs closely followed by packets on which the
> ACK could have been piggybacked. Turn it off when a delayed ACK has to be
> sent."
>
> similar) idea for years. It's called QUICK_ACK. The decision window is a bit
> short and can use some tuning. But it is not "never" fixed.
>
>
>
> Cc: LAUTENSCHLAEGER, Wolfram (Wolfram)
> <[email-masked]>; [email-masked]; David Lang
> <[email-masked]>

---

From: David Lang <[email-masked]>
Date: Sun, 11 Oct 2015 14:18:28 -0700 (PDT)

> s/using more airtime/altering E2E semantics/

I actually disagree that there is a significnt difference in the actual E2E 
semantics between sending a single stretched ACK and sending the full number of 
ACKs back-to-back. The "transmit burst" issue will happen in either case.

But you will notice that in both cases, I am in favor of reducing the number of 
packets on the wire. Packets not send can't interfere with other traffic.

If ACK packets are delayed to the point that additional ACK packets have caught 
up with them, they are not providing the rate signalling signal that you are 
looking for.

Question: Does anyone know if current server TCP stacks implement the TCP byte 
counting that RFC3449 talks about?

David Lang

---

From: Joe Touch <[email-masked]>
Date: Sun, 11 Oct 2015 14:28:27 -0700

On 10/11/2015 2:18 PM, David Lang wrote:
> But you will notice that in both cases, I am in favor of reducing the
> number of packets on the wire. Packets not send can't interfere with
> other traffic.

The degenerate case of this approach is a circuit. Take a look at
Morris's 1997 ICNP paper to see what happens when you end up with one -
or sometimes less than one - packet per round trip time.

One of the reasons we use packets is to provide more timely,
fine-grained feedback between endpoints. Aggregating them at the source
or in the network (rather than at the receiver) can amplify reaction to
packet loss (when the aggregate ACK is lost) and increases
discretization effects in the TCP algorithms.

Joe

---

From: David Lang <[email-masked]>
Date: Sun, 11 Oct 2015 15:49:25 -0700 (PDT)

On Sun, 11 Oct 2015, Joe Touch wrote:

> packet loss (when the aggregate ACK is lost)

the issue of amplified reaction to packet loss is a valid one, but packet loss 
is still pretty rare, so the proposal of putting a duplicate stretched ack into 
the queue for the next transmit window when you aggregate packets in the current 
transmit window should largely mitigate this.

In addition, Since cablemodems have been doing this for 15 years, we should be 
able to get people to look at real-world traffic and tell us how much of a 
problem this is.

> and increases

This is where I disagree with you. The ACK packets are already collapsed 
time-wise. Whatever timing they had when they were generated, the fact that they 
are all sitting in the same queue at the same time means taht that timing has 
been lost, Nobody is adding latency by trying to recreate the time spacing, so 
the discretization (or as I've been calling it quantitization) of the traffic is 
happening even if every ACK packet is sent.

My contention is that since this is already happening, consolidating the ACK 
packets into stretched ACKs doesn't make this any worse, and it saves network 
bandwidth (and decreases latency to the extent that data is acked faster than 
waiting for the entire chain or original ACKs to get through, especially if that 
would take multiple transmit windows). As a result, thinning the ACKs is kinder 
to the network.

I am in no way advocating delaying any packet that could be sent, just combining 
those that have already been delayed.

We're talking about ACK packets here, but any protocol that sends short TCP 
packets is potentially fair game (TCP/SSH for example). If the packets are all 
sitting in the same queue, they should be able to be combined. This happens on 
the sending server if multiple writes to the socket happen before the packet has 
been sent, it should be able to happen on the network as well (further reducing 
overhead by sending the same data with fewer packets)

David Lang

---

From: David Lang <[email-masked]>
Date: Sun, 11 Oct 2015 17:10:37 -0700 (PDT)

On Mon, 12 Oct 2015, Jonathan Morton wrote:

>> My contention is that since this is already happening, consolidating the
> ACK packets into stretched ACKs doesn't make this any worse, and it saves
> network bandwidth (and decreases latency to the extent that data is acked
> faster than waiting for the entire chain or original ACKs to get through,
> especially if that would take multiple transmit windows). As a result,
> thinning the ACKs is kinder to the network.
>
> I agree, *iff* they are not DupACKs signalling packet loss.  Do existing
> and future cable modems take that subtle distinction into account?

I don't know, and RFC3449 section 5.2.1 lists a couple other conditions to 
watch out for. But this discussion started with what AQM algorithms can/should 
do when they see multiple ACKs in their queue (and I added wifi as part of 
make-wifi-fast). So we are talking about what should be put in future code. The 
cable modems are just an example we can draw from of a past implementation of 
similar ideas.

hopefully future cable modems will use a more generic AQM algorithm instead of 
the hard-coded optimizations they've had in the past. It's almost always a win 
when a generic solution can replace a specific solution :-)

David Lang

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Mon, 12 Oct 2015 12:04:52 +0000

Ah, sorry, I mixed up the two problems. 

My general concern is to avoid that 793bis is already outdated at the time of publication. draft-minshall-nagle may or may not be one example in which TCPM lacks a small PS/BCP document. As I said already, I have not checked this particular area myself so far.

Out-of-my-head, I don't recall any recent RFC that really deals with DelACK tuning as discussed below. If that sort of heuristic is done by most stacks these days, it could perhaps make sense to document that, if the community could e.g. agree on certain high-level constraints (e.g., does it make sense to reduce the timer below 1ms?).

But this is just brainstorming, and my view as individual engineer only...

Michael

-----Original Message-----
From: Yuchung Cheng [mailto:[email-masked]] 
Sent: Sunday, October 11, 2015 6:29 PM
To: Scharf, Michael (Michael)
Cc: Praveen Balasubramanian; Wesley Eddy; [email-masked]
Subject: Re: Delayed ACKs vs. Nagle (was: RE: [tcpm] [aqm] TCP ACK Suppression)

On Sun, Oct 11, 2015 at 7:41 AM, Scharf, Michael (Michael) <[email-masked]> wrote:
> specifying how to sort out this interaction, unless we extend the 
> scope of 793bis beyond what we agreed upon (i.e., only clarification, 
> no protocol changes).
>
>
>
> I have personally never looked at how that logic is actually 
> implemented in different stacks, and I actually don’t know if there 
> has ever been community consensus on how the exact solution to that interaction should look like.
>
>
>
> But if we have a gap to what is widely implemented, and if it was 
> realistic to publish a PS RFC before completing 793bis (I guess it 
> could be a 2-page RFC), I’d argue in favor of a “fast track” PS 
> publication. (Well, with ‘fast’ being relative to our normal PS 
> publication speed.)
I am not sure what your concern is: are you concerned Minshall's revision to Nagle's algorithm on the sender would interact badly with Mr. Nagle's idea to improve delayed-ACKs at the receiver?

> hardcoded value. Windows 8 onwards this is the behavior for high latency connections.
> between Nagling and delayed ACKs is well known. John Nagle's comment 
> in HN which suggests a per connection logic to throttle delayed ACKs:
> "If you're going to do that, it's a good time to fix delayed ACK as 
> well. As is well known, the interaction between delayed ACKs and the 
> Nagle algorithm is awful. This is a historical accident; I implemented 
> one, the Berkeley guys implemented the other, and by the time I found 
> out I was out of networking and involved with a small startup called 
> Autodesk. So that never got fixed. Here's how to think about that. A 
> delayed ACK is a bet. You're betting that there will be a reply, upon 
> with an ACK can be piggybacked, before the fixed timer runs out. If 
> the fixed timer runs out, and you have to send an ACK as a separate 
> message, you lost the bet. Current TCP implementations will happily 
> lose that bet forever without turning off the ACK delay. That's just 
> wrong. The right answer is to track wins and losses on delayed and 
> non-delayed ACKs. Don't turn on ACK delay unless you're sending a lot 
> of non-delayed ACKs closely followed by packets on which the ACK could 
> have been piggybacked. Turn it off when a delayed ACK has to be sent."
>
> similar) idea for years. It's called QUICK_ACK. The decision window is 
> a bit short and can use some tuning. But it is not "never" fixed.
>
>
>
> To: Scharf, Michael (Michael) <[email-masked]>; 
> Wesley Eddy <[email-masked]>; Joe Touch <[email-masked]>
> <[email-masked]>; [email-masked]; David 
> Lang <[email-masked]>
> Many implementations interpret  "the delay MUST be less than 0.5 
> seconds" to trigger the delayed ACK timer at 500ms (with this time a 
> fixed, compile-time
> value) - while it has been shown, that making that dynamic (or a based 
> on RTT and pps might be a more efficient approach (especially during 
> loss at end-of-stream), depending on the environment [1].
>
> [1] Altman, Eitan, and Tania Jiménez. "Novel delayed ACK techniques 
> for improving TCP performance in multihop wireless networks." Personal 
> Wireless Communications. Springer Berlin Heidelberg, 2003.
>
>
>> Yes, I have realized today myself that 793bis lacks that section. I 
>> agree that ACK generation belongs into 793bis.
>>
>> But if deviations from that SHOULD are widely deployed in the 
>> meantime, 793bis could possibly comment on that. Also, having some 
>> informational reference could perhaps be an option.
>>
>> Michael
>>
>>
>> This probably belongs in 793bis.  In my view, normative 
>> acknowledgement behavior belongs in the base spec rather than 
>> particular congestion control descriptions, since there may be more 
>> than CC that depends upon it or makes assumptions about it.
>>
>>
>> --
>> Wes Eddy
>> MTI Systems
>>

---

From: Mikael Abrahamsson <[email-masked]>
Date: Mon, 12 Oct 2015 16:13:45 +0200 (CEST)

> with the fallout from the myopic TCP designers.  :-P

So I agree that most likely, it's beneficial to have fewer ACKs.

What I think people arguing against this practice are these kinds of 
issues:

http://blog.dan.drown.org/sb6183-dropping-ipv6-traffic/

I don't think there is a solution that we all can agree on, all approaches 
have their benefits and drawbacks. I think the above article just shows 
how things can go wrong in very subtle ways.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: "Bless, Roland (TM)" <[email-masked]>
Date: Mon, 12 Oct 2015 17:00:37 +0200

Hi,

On 12.10.2015 at 16:13 Mikael Abrahamsson wrote:
> I don't think there is a solution that we all can agree on, all
> approaches have their benefits and drawbacks. I think the above article
> just shows how things can go wrong in very subtle ways.

Surely, broken or non-prudent implementations can easily break
things. My other issue with this is that *protocol evolution*
suffers. New options may not be understood by such optimization,
so their deployment is inhibited, or, the ACKing strategy changes
and the previously greatly working optimization is totally
counter-productive. I guess that the MPTCP WG work has suffered
a lot from several kinds of "built-in optimization" network gear.
Moreover, if people are starting to use newer transports other
than TCP, they may complain about poor performance then, because
the optimization doesn't work in these cases...

Regards,
 Roland

---

From: David Lang <[email-masked]>
Date: Mon, 12 Oct 2015 09:09:04 -0700 (PDT)

On Mon, 12 Oct 2015, Mikael Abrahamsson wrote:

> What I think people arguing against this practice are these kinds of issues:
>
> have their benefits and drawbacks. I think the above article just shows how 
> things can go wrong in very subtle ways.

no question that things can go wrong in subtle ways. If network protocol design 
was easy, we would have had a couple protocols designed back in the early days 
of computing and there wouldn't be the recent rash of new protocol designs :-)

But too much fear that "something may go wrong somewhere" can prevent any 
progress.

David Lang

---

From: David Lang <[email-masked]>
Date: Mon, 12 Oct 2015 09:11:04 -0700 (PDT)

> the optimization doesn't work in these cases...

New protcols should have similar optimizations built in and not require network 
assistance ;-)

Or, if the improvement is really something that can only take place at the 
network layer (possibly due to the local knowledge of the queue state for 
example), this will just be one disadvantage of the new protocol until similar 
optimizations can be deployed for the new protocol.

David Lang

---

From: Jeremy Harris <[email-masked]>
Date: Mon, 12 Oct 2015 17:44:07 +0100

On 11/10/15 23:49, David Lang wrote:
> packet loss is still pretty rare,

Not in my day-job it isn't.

>  The ACK packets are already collapsed
> time-wise. Whatever timing they had when they were generated, the fact
> that they are all sitting in the same queue at the same time means taht
> that timing has been lost,

Unless they carried timestamps?  What are you doing with TCP options
(and ECN marks)?
-- 
Cheers,
  Jeremy

---

From: Greg White <[email-masked]>
Date: Mon, 12 Oct 2015 23:08:41 +0000

> My contention is that since this is already happening, consolidating the ACK packets into stretched ACKs doesn't make this any worse, and it saves network bandwidth (and decreases latency to the extent that data is acked faster than waiting for the entire chain or original ACKs to get through, especially if that would take multiple transmit windows). As a result, thinning the ACKs is kinder to the network.

I agree, *iff* they are not DupACKs signalling packet loss.  Do existing and future cable modems take that subtle distinction into account?

I presume that they are not discarding DupACKs or corrupting SACK in some way (the whole point of specialized ACK handling is to ensure good TCP performance after all), but since the algorithms are proprietary, for existing cable modems I don't know.  I will contact the developers as well as run some tests on a few modems to get a better idea (and maybe Mikael already has some data from his recent testing he could share?).   For future modems, I can add an explicit reference to RFC3449 in the DOCSIS 3.1 spec just in case implementers aren't aware for some reason.

---

From: Joe Touch <[email-masked]>
Date: Tue, 13 Oct 2015 07:54:56 -0700

David,

On 10/11/2015 3:49 PM, David Lang wrote:
> the issue of amplified reaction to packet loss is a valid one, but

But you claim to need to drop the ACKs to avoid dropping other packets.
What's to say that 1) the ACKs won't end up getting dropped in your
system or 2) the ACKs won't get dropped further downstream?

Again, *you* don't know.

...
> time-wise.

Time is only one element of the discretization. The other is fate
sharing - dropping one aggregate ACK has the effect of dropping the
entire set of un-aggregated ones. The stream of unaggregated ACKs gives
TCP a robustness to drops, but you're now forcing a large burst of drops
for a specific packet type. That's the problem.

We've gone round this issue for a lot of messages, but the key point is
that this isn't a new idea, nor are the numerous considerations and
concerns.

Ultimately, if you have too many packets from a protocol that reacts to
congestion, the correct solution is to provide that feedback and let the
ends figure out how to respond. That's the only solution that will work
when (not if) new messages or encrypted headers enter the picture.

These kinds of in-network hacks are what's really killing the ability of
the Internet to evolve.

Joe

---

From: David Lang <[email-masked]>
Date: Tue, 13 Oct 2015 12:09:12 -0700 (PDT)

> system or 2) the ACKs won't get dropped further downstream?

combining ACKs further downstream should not hurt anything (as long as the 
RFC3449 criteria are covered)

_dropping_ ACKs without combinging them is packet loss, pure and simple. With 
all the implications of that. The aggregated ACK is more significant that one of 
the other ACKs that were combined into it, but reducing the number of packets 
reduces the chances of needing to drop a packet. And the suggestion that was 
made to dup the aggregated ACK into the next transmit slot will greatly reduce 
the impact of the aggregated ACK being lost.

> for a specific packet type. That's the problem.

this is why the suggestion to dup the aggregated ACK into the next transmit slot 
is such a great idea (I'd have to go back through this thread and dig up who 
suggested it). It almost entirely mitigates the problem of the ACK packet 
getting dropped.

> These kinds of in-network hacks are what's really killing the ability of

If people are faced with existing protocols not working well, or making local 
modifications that make things work, but limit the ability for the "Internet to 
eveolve" the local modifications are going to win every time. This is the 
Internet evolving, just not in the way that you want.

David Lang

---

From: David Lang <[email-masked]>
Date: Tue, 13 Oct 2015 12:36:01 -0700 (PDT)

On Mon, 12 Oct 2015, Greg White wrote:

>> waiting for the entire chain or original ACKs to get through, especially if 
>> that would take multiple transmit windows). As a result, thinning the ACKs is 
>> kinder to the network.
>
> I agree, *iff* they are not DupACKs signalling packet loss.  Do existing and 
> future cable modems take that subtle distinction into account?
>
> I presume that they are not discarding DupACKs or corrupting SACK in some way 
> (the whole point of specialized ACK handling is to ensure good TCP performance 
> after all), but since the algorithms are proprietary, for existing cable 
> modems I don't know.  I will contact the developers as well as run some tests 
> on a few modems to get a better idea (and maybe Mikael already has some data 
> from his recent testing he could share?).  For future modems, I can add an 
> explicit reference to RFC3449 in the DOCSIS 3.1 spec just in case implementers 
> aren't aware for some reason.

It should also say that AQM is needed on the downstream side to protect against 
transmit bursts.

The suggestion was posted early in this thread (but not in someting I can 
quickly find) that to harden the connection against the aggregate ACK being 
lost, it should be sent again in a separate transmit window.

I latched onto that and think the right thing to do is to duplicate the 
aggregate ACK that's created and put it at the beginning of the queue that 
remains after the current transmit window.

If there are more ACKs in the next transmit window, it will get aggregated into 
them and not cause any additional traffic.

If there are not any more ACKs in the next transmit window, and the first ACK 
was lost, it will replace it with the minimum of delay

If there are not any more ACKs in the next transmit window, and the first ACK 
was not lost, we are now transmitting a duplicate ACK, which may be interpreted 
as a signal to speed up tranmission (depending on the exact timing of the 
arrivials of these two ACKs). While this is not good, we already need to be 
protected from transmit bursts, so this is just another possible cause for such 
bursts.

David Lang

---

From: David Lang <[email-masked]>
Date: Tue, 13 Oct 2015 12:41:16 -0700 (PDT)

> Unless they carried timestamps?  What are you doing with TCP options

This is an area that should be discussed more.

RFC3449 says "uitable algorithms to support IPSec authentication, SACK, and ECN 
remain areas of ongoing research."

I don't know what research has been done with this since 2002.

The cablemodem folks have practical experience doing _something_. We should 
probably nail down what they are doing with SACK, ECN, etc and work through the 
issues that they could be causing so that we can expand on RFC3449 and say what 
should be done in those cases.

David Lang

---

From: "Richard Scheffenegger" <[email-masked]>
Date: Wed, 14 Oct 2015 20:30:18 +0200

Well, for the Timestamps case, getting all of them (plus the typically 
broken EDWA factors) will tend to increase your RTTvar, and thus the RTO.

I doubt that it would be smart to keep the TSecr of the oldest, removed ACK 
and replace the TSecr value of the ACK that is to be transmitted with that 
old value though...

OTOH, TSval often has a granularity of only 10ms (or higher), so chances are 
that multiple ACKs do actually contain the very same TSval/TSecr values 
before the thinning.

In general, I would think any ACK suppression should take all options into 
consideration, and only iff they all are equal, earlier ACKs may be 
discarded while in the queue...

Best regards,
  Richard