Subject: Re: [tcpm] [aqm] ACK Suppression

From: "Bless, Roland (TM)" <[email-masked]>
Date: Thu, 8 Oct 2015 11:44:54 +0200

Hi David,

Am 07.10.2015 um 21:52 schrieb [email-masked]:
> Nonetheless I am troubled by the very fact of the discussion taking
> place, for two reasons:
> 
> 1) TCP ACKs are TCP's business only.  It's not a gateway or router's job
> to get involved in or to understand end-to-end protocols, *even if* the
> router thinks it knows exactly what the endpoints' goals are.  And the
> router cannot know that for every protocol, not even the many higher
> level protocols on top of TCP, which use TCP quite differently.  The
> idea that routers can be omniscient, merely by looking at packets and
> taking the designers' personal prejudices into account, seems
> ridiculous. TCP endpoints on both ends of a connection can reduce the
> number of ACKs they send if they want. If ACKs are filling up buffers in
> intermediate routers, just drop them or mark them to notify that they
> are contributing to congestion.  The endpoints have to slow down
> something, and they can slow down ACKs by mutual agreement.

+1

> 2) The hypothetical that there will be a sufficiently long sequence of
> ACKs for a single end-to-end flow buffered in a single router queue may
> seem plausible, *until it becomes clear that in the big picture, having
> so many packets in a queue means that the network is extremely congested
> by that point*.  In other words, in order for this "optimization" to
> apply, you would have to operate the network at an unacceptable
> operating point! It's like saying that when a highway has slowed to a
> crawl, we can load all the cars going to a particular destination onto
>  single "car carrier" to save gas. Far better to insure that queues are
> not built up!  The purpose of queueing is to absorb bursts that can't be
> anticipated, not to build up congestion in order to have enough data to
> perform a dubious optimization that can best be done at the source of
> traffic in cooperation with the destination.

+1
Maybe an incentive for some people to think about alternative link
layer access schemes that will be better suited for such kind of
Internet traffic. As already pointed out the specific optimization will
be useless for newer transport protocols as well as for tunneled or
encrypted traffic or advanced TCP features, including ECN feedback.

> It's said that in committees the amount of time spent on trivialities
> far exceeds the time spent on important issues.  That seems to be true
> as I watch the discussion on this list.

I think the "discussion" seems to be necessary since Mikael's original
question was:
>> Now, this kind of mechanism, how should it be treated when it comes
>> to AQM? This mechanism is basically done at de-queue, when a number of
>> packets are emptied from the queue at one time, which is then allowed to
>> fill up again until the next transmit opportunity arises.

I really hate it if the IETF tries to work around "broken"
(short-sighted, cross-layer optimized, and so on) behavior
of middleboxes or other devices. So I don't think that the AQM
WG should take into account this particular optimization
of specific link layer boxes. Otherwise, we'll make the situation
only worse for transport protocol evolution. We've got enough
examples for ossification due to non-farseeing implementations
of middlebox stuff. It's quite perverted if we start to design
mechanisms according to such kind of "special/broken" behavior.

Regards,
 Roland

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Thu, 8 Oct 2015 18:22:52 +0000

I'm not sure why this discussion is happening on aqm@ instead of tcpm@... I have added cpm to the cc line, and would recommend that anyone responding to this thread do the same and remove aqm@.

On Oct 7, 2015, at 2:13 PM, David Lang <[email-masked]> wrote:
> So things that reduce the flow of acks can result in very real benefits to users.

Dumb question of the month. What would it take to see wide deployment of RFC 5690? That would result in the data/ack ratio being reduced, on average, to whatever amount had been negotiated.

Summary for those that haven't read it - TCP implementations today generally ack every other packet, with caveats for isolated or final data packets. This proposal allows consenting adults to change that ratio, acking every third or fourth packet, or every tenth.

If ack reductions are so very valuable, what's the chance of doing that on an end to end basis instead of in the network?

> On Oct 7, 2015, at 2:13 PM, David Lang <[email-masked]> wrote:
> On Wed, 7 Oct 2015, Jonathan Morton wrote:
>>> On 7 Oct, 2015, at 23:40, Agarwal, Anil <[email-masked]> wrote:
>>>> Since the cable modem link will lead to clumped ACKs the difference between sending 100 ACKs vs. 1 ACK is probably not that big... (except w.r.t. reliability).
>>> The difference may not be big in the spacing of new packets that a sender will send, unless the sender implements some sort of pacing or if the return link is very thin.
>> 
>>> But with ABC, there will be a difference in the amount of cwnd increase at the sender.
>> 
>> There is also a potential difference for detecting packet loss in the forward direction.  It’s entirely possible that thinning would cause a DupAck condition to be recognised only after three MAC grants in the reverse direction have elapsed, rather than one.  Receivers are REQUIRED to send an ack for every received packet under these conditions, but that would be subverted by the modem.  AckCC would not induce this effect, because the receiver would still produce the extra acks as required.
>> 
>> Packet loss causes head-of-line blocking at the application level, which is perceived as latency and jerkiness by the end-user, until the lost packet is retransmitted and actually arrives.  Hence the addition of two MAC grant delays (60ms?) may make the difference between an imperceptible problem and a noticeable one.
> 
> and excessive ack traffic causes congestion and results in packet loss on real-world hightly asymmetric links.
> 
> So things that reduce the flow of acks can result in very real benefits to users.
> 
> David Lang_______________________________________________
> aqm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/aqm

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Thu, 8 Oct 2015 19:22:16 +0000

> On Oct 8, 2015, at 11:34 AM, David Lang <[email-masked]> wrote:
> 
> a little less than the chance of  shutting off IPv4  :-)

Hey, we can dream, can't we? :-)

> requiring changes to all the software on each end to enable this is wishful thinking. The major servers could get the update pretty promtly, but updating the client side?? not for a long time.

Yes, of course. That said, consider trends like

http://www.zdnet.com/article/latest-os-share-data-shows-windows-still-dominating-in-pcs/
http://www.netmarketshare.com/operating-system-market-share.aspx?qprid=11&qpcustomb=0
https://www.pinterest.com/pin/137148751123435826/

What that basically says is that corporations tend to pick a version and install every update after testing it. The folks still on XP are very likely people who installed a (potentially pirated) release in 2003 and have probably never updated it. If we can get this into the next MacOSX 10.11 and Windows 10 update cycle, roughly 50% of computers will have it within six months, maybe three. If we can get it into common Linux distributions, we can probably cover the servers as well. 100% deployment - that might take until the XP computers actually all fry. Getting it significantly deployed - I don't see the cause of the pessimism.

You're correct that one has to update set-top-boxes and smart TVs. They get updates too, or at least mine do. It doesn't require buying a new TV, it requires getting the updated software.

---

From: [email-masked]
Date: Fri, 9 Oct 2015 04:03:41 +0100

RFC3449 pointed to some of the reasons & problems with manipulating TCP
ACKs, and could provide a useful background if people needed to get up to
speed on the history...

Gorry

> I'm not sure why this discussion is happening on aqm@ instead of tcpm@...
> I have added cpm to the cc line, and would recommend that anyone
> responding to this thread do the same and remove aqm@.
>
> Dumb question of the month. What would it take to see wide deployment of
> RFC 5690? That would result in the data/ack ratio being reduced, on
> average, to whatever amount had been negotiated.
>
> Summary for those that haven't read it - TCP implementations today
> generally ack every other packet, with caveats for isolated or final data
> packets. This proposal allows consenting adults to change that ratio,
> acking every third or fourth packet, or every tenth.
>
> If ack reductions are so very valuable, what's the chance of doing that on
> an end to end basis instead of in the network?
>
>>>>> Since the cable modem link will lead to clumped ACKs the difference
>>>>> between sending 100 ACKs vs. 1 ACK is probably not that big...
>>>>> (except w.r.t. reliability).
>>>> The difference may not be big in the spacing of new packets that a
>>>> sender will send, unless the sender implements some sort of pacing or
>>>> if the return link is very thin.
>>>
>>> There is also a potential difference for detecting packet loss in the
>>> forward direction.  It’s entirely possible that thinning would cause
>>> a DupAck condition to be recognised only after three MAC grants in the
>>> reverse direction have elapsed, rather than one.  Receivers are
>>> REQUIRED to send an ack for every received packet under these
>>> conditions, but that would be subverted by the modem.  AckCC would not
>>> induce this effect, because the receiver would still produce the extra
>>> acks as required.
>>>
>>> Packet loss causes head-of-line blocking at the application level,
>>> which is perceived as latency and jerkiness by the end-user, until the
>>> lost packet is retransmitted and actually arrives.  Hence the addition
>>> of two MAC grant delays (60ms?) may make the difference between an
>>> imperceptible problem and a noticeable one.
>>
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm
>

---

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 9 Oct 2015 08:43:44 +0200 (CEST)

On Thu, 8 Oct 2015, David Lang wrote:

> Ok, so it turns out that what we are talking about here is talked about 
> explicitly in section 5.2.1 of RFC3449. It warns of increased sender 
> burst sizes, but I really question if that's a valid concern in the 
> cases that we are talking about here. The choice in these cases isn't to 
> send more acks spaced out, but rather to send a lot of acks back to back 
> in one transmit slot vs one stretch ack per transmit slot.

Exactly. And in my specific case, we're talking 20kPPS of data down, and 
10kPPS of ACKs, over a medium that allows the modem to send a bunch of 
packets only once every few milliseconds.

So there are two alternatives:

You get approx 10-60 ACKs as a "train of ACKs" within 0.3 milliseconds, 
every 1-4 milliseconds.

You get 1-3 ACKs within 0.1ms, every 2-3 milliseconds. These ACKs will 
acknowledge 16-32 packets at a time.

By the nature of the beast, I'd say that this kind of medium implicitly 
will be able to handle the "bursts" we're talking about here, if you ACK 
24-48kilobyte of data per ACK (when a middle box does ACK Suppression to 
remove intermediate ACKs), it's implicit that the medium designers know 
that they can handle a wirespeed burst of data that is the result of the 
ACKing of these fairly large number of bytes.

But again, I question the wisdom in ACKing every 2 packets when the 
congestion window is in the hundreds of kilobytes and you're sending tens 
of thousands of packets per second.

I suggested in another part of the thread that perhaps the host itself 
should do ACK suppression and don't send ACKs more closely together than 
once per millisecond or 1/10th RTT, whatever is lower. Perhaps the RTT 
value needs to be even lower, 1/25 of RTT or something.

In my specific test, using once per millisecond or 1/25 of RTT would cut 
down the number of ACKs to around 2kpps instead of the 10kpps ACKs I see 
now, but it wouldn't cut it down to the around 500-1000 ACKs per second 
that the DOCSIS modem is cutting it down to...

I think that as bandwidths go up, there are changes to TCP that can be 
done to make it work better, and reducing the frequency of ACKs sent 
depending on RTT and different window sizes might be a way forward. I 
fully believe there are cases where ACKing every 2 packets is exactly the 
right way to go, but when you're sending 1 gigabit/s over 100ms RTT and 
the medium itself clumps the packets together because that's what it does, 
then these ACKs don't carry the same information anymore and some of them 
could be optimized away (preferrably by the host stack) to spare upstream 
bandwidth and transmit opportunities.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Fri, 9 Oct 2015 17:21:41 +0000

> On Oct 9, 2015, at 10:17 AM, Daniel Havey <[email-masked]> wrote:
> 
> This sounds like an awful idea.  What if there were congestion marks in those ACKs?  What if the ACK has data in it?

ECN is an issue. I would expect a piggy-backed ack is treated as a data message.

> From: "[email-masked]" <[email-masked]>
> To: Fred Baker (fred) <[email-masked]>
> Cc: David Lang <[email-masked]>; Jonathan Morton <[email-masked]>; "[email-masked]" <[email-masked]>; "[email-masked]" <[email-masked]>; "[email-masked]" <[email-masked]>
> Sent: Thursday, October 8, 2015 8:03 PM
> Subject: Re: [aqm] [tcpm] ACK Suppression
> 
> > I'm not sure why this discussion is happening on aqm@ instead of tcpm@... <mailto:tcpm@...>
> > responding to this thread do the same and remove aqm@. <mailto:aqm@.>
> >
> > On Oct 7, 2015, at 2:13 PM, David Lang <[email-masked] <mailto:[email-masked]>> wrote:
> >> On Oct 7, 2015, at 2:13 PM, David Lang <[email-masked] <mailto:[email-masked]>> wrote:
> >>>> On 7 Oct, 2015, at 23:40, Agarwal, Anil <[email-masked] <mailto:[email-masked]>>
> >>>> wrote:
> >>> forward direction.  Itâ€™s entirely possible that thinning would cause
> >> [email-masked] <mailto:[email-masked]>
> >> https://www.ietf.org/mailman/listinfo/aqm <https://www.ietf.org/mailman/listinfo/aqm>
> >
> > [email-masked] <mailto:[email-masked]>
> > https://www.ietf.org/mailman/listinfo/tcpm <https://www.ietf.org/mailman/listinfo/tcpm>
> 
> 
> 
> >
> 
> 
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/aqm <https://www.ietf.org/mailman/listinfo/aqm>
>

---

From: Mark Allman <[email-masked]>
Date: Fri, 09 Oct 2015 13:29:40 -0400

I am not following along much, but why would that stop me?! :)

>     This sounds like an awful idea. What if there were congestion
>     marks in those ACKs? What if the ACK has data in it?
> 

(1) The point of delayed ACKs is to piggyback ACKs on data packets.
    So, when a host has data to send and would naturally do so then
    the ACK will get sent out with that data.

(2) At least RFC 5681 (and previous versions) have been careful to
    note that when there is a hole in the sequence space of the
    arriving packets that ACKs should not be delayed.  I.e.,
    duplicate ACKs or ACKs with SACK information should be sent
    immediately as they are a possible indication of congestion.

(3) I would think the spirit of (2) would also apply to ECN, but I
    am not sure that is written down anywhere (perhaps the ECN
    spec?).  I.e., reflecting congestion marks should not be
    delayed.

allman

--
http://www.icir.org/mallman/

---

From: "Fred Baker (fred)" <[email-masked]>
Date: Fri, 9 Oct 2015 17:36:33 +0000

> On Oct 9, 2015, at 10:29 AM, [email-masked] wrote:
> 
> 

I think the question here relates to DOCSIS specifications, which basically encourage Cable Modems to separately queue acks from data traffic, and delete an older untranmsitted ack from the queue when a subsuming ack is enqueued. When the modem gets a transmission grant, it sends all of the data it has but, as a result, a potentially-small subset of the acks. The intended effect is to change the data:ack ratio from 2:1 to many:1 in high transmission rate contexts.

I'm not convinced that the two algorithms mesh perfectly. (understatement)

---

From: "Agarwal, Anil" <[email-masked]>
Date: Fri, 9 Oct 2015 17:59:24 +0000

I presume, that "AKC Suppression", assuming it is a good thing to do, requires some rules for -
	Handling multiple ACK segments some of which may contain data, timestamps, TCP-ECN info, 
	    FIN, SACK, different receive window sizes, AO, ...
	Handling multiple ACK segments interspersed with large data packets of other TCP connections, in the same queue.
	Presumably, the DOCSIS folks have designed such rules.

And the performance implications of doing so, besides the obvious benefit of reducing return-link congestion.

Regards,
Anil

---

From: Joe Touch <[email-masked]>
Date: Fri, 9 Oct 2015 11:03:41 -0700

Not to mention what to do if you see TCP option fields you don't
understand (i.e., not to simply copy them).

Please don't forget that anything that tampers with a TCP segment is
taking on the responsibility of RFC1122 - i.e., it becomes a complete
proxy for the correct operation of TCP.

Joe

---

From: [email-masked]
Date: Fri, 9 Oct 2015 14:42:23 -0400 (EDT)

It's interesting.  I think the term "bikeshedding" has started to apply here.  A couple minor comments:
 
1. The idea that one should put *new* features into routers because of unmaintained/unmaintainable endpoint stacks seems like an argument that is very stretched.  Why do people who can't afford to update their endpoint gear for whatever reason buy new routers instead?  Since that old gear was probably designed for slow networks (< 1 Mb/sec end-to-end max) why would they generate a large enough percentage of traffic to be worth special detection, etc.?  I'd like to see the actual data that such situations exist and that new routers would be used there.
 
2. The search for "features" to put into routers has always been driven by some kind of vague "value add" concept. But it has rarely been a path to successfully making the Internet experience better.  I could give a list of cases where it was "plausible" that you could sell a few routers that solve a hypothetical issue where it was observed that "hey, we could do this thing at the router level, sometimes".  There are probably Ethernet switches that claim to reduce email spam, for example, by detecting SMTP, reading the messages, using a bayesian classifier, and sending RST messages to both endpoints.  (actually there are, a company called Sandvine sells stuff like that).  These are rarely mainstream opportunities to make the Internet better, and it's worth wondering why they don't get adopted. The basic answer is that they don't work well, and there are better ways to achieve the goal, usually by rethinking the problem consistently with the design principles of the Internet.

On Friday, October 9, 2015 2:03pm, "Joe Touch" <[email-masked]> said:

> > I think the question here relates to DOCSIS specifications, which basically
> encourage Cable Modems to separately queue acks from data traffic, and delete an
> older untranmsitted ack from the queue when a subsuming ack is enqueued. When the
> modem gets a transmission grant, it sends all of the data it has but, as a result,
> a potentially-small subset of the acks. The intended effect is to change the
> data:ack ratio from 2:1 to many:1 in high transmission rate contexts.
> >

---

From: Yuchung Cheng <[email-masked]>
Date: Fri, 9 Oct 2015 15:59:14 -0700

>     spec?).  I.e., reflecting congestion marks should not be
Gorry pointed out earlier: https://tools.ietf.org/html/rfc3449#section-5.2.1

As Wes and Michael pointed out, this ack policy is discussed in too many
RFCs (5681, 1112, 3449).

---

From: Mark Allman <[email-masked]>
Date: Fri, 09 Oct 2015 23:06:00 -0400

> this ack policy is discussed in too many RFCs (5681, 1112, 3449).

And his answer was to put it in one more! :)

Everything about TCP is discussed in too many RFCs.  Who can keep
track?!

allman