Subject: Nagle -- again

From: Greg Minshall <[email-masked]>
Date: Thu, 17 Jun 1999 19:27:13 -0700

All and sundry,

We last talked about this in March.  The I-D i had submitted last fall has 
either expired or is about to expire.  So...

My plan is to submit the updated version of my I-D (that i sent to this list a 
long time ago -- Fri, 26 Feb 1999) to the Internet Drafts directory (soon, 
given that the cut-off for the Oslo IETF is coming up, and i don't want it to 
be "out of print" that long).

There was, at the end of our previous discussions, still some amount of 
controversy.  I guess that will flare up again, and we will talk about it.  
After that, my hope is to submit the I-D as an experimental RFC (but, we will 
see!).

By the way, since our discussions in March, i (and some others) have published 
a short paper on one of the scenarios that made me feel like a modified Nagle 
would be a "good thing".  A postscript version of that paper is online at

	http://www.cc.gatech.edu/fac/Ellen.Zegura/wisp99/papers/minshall.ps

(it was presented at the "workshop on internet server performance" at the 
beginning of May).  Note that the "implementation of a modified version of 
Nagle" reported in the paper -- which didn't give all that great results -- 
was *NOT* an implementation of the modified version of Nagle as suggested in 
the I-D.  However, the paper gives both a good explanation of how application 
developers may not be doing the right buffering, and thus see many more 
"Nagle+delayed ACK" delays than they need have seen, as well as of how, even 
with "perfect buffering", applications may have the trailing packet of some 
responses delayed; the first motivates us to educate application developers, 
the second motivates us (me, at least) to modify Nagle slightly to remove this 
class of delay.

Cheers,  Greg

---

From: Joe Touch <[email-masked]>
Date: Wed, 23 Jun 1999 21:19:09 -0700 (PDT)

> From: [email-masked] (Sam Manthorpe)
> Subject: Re: Nagle -- again
> To: [email-masked] (Joe Touch)
> Date: Wed, 23 Jun 1999 20:57:58 -0700 (PDT)
> Cc: [email-masked], [email-masked]
> 
> Joe Touch wrote:
> > 
> > One observation - the paper indicates that there _must_ be at least
> > one ACK every two packets. This is not strictly required, at least
> > last time I checked. 
> > 
> 
> I've been meaning to bring that up for a while.  The `ack every other
> segment' algorithm recommended in RFC-1122 is bad news for high bandwidth,
> low packet size media.  In particular I'm thinking of gigabit Ethernet
> without jumbograms.  When we get an ack for every 2 packets, i.e. one
> ack for every ~2920 bytes of application data, then if we want to obtain a 
> TCP goodput of, say, 600 Mbits/sec on a single connection, then our 
> sending machine will be receiving approx. 26000 ACKs per second which 
> places a substuntial load on the sending hosts's stack (interrupt 
> processing, IP packet processing, TCP processing).

This is indeed problem. However, it appears that a significant part
of the issue is the lack of a reasonable sized MTU, i.e., sticking
with non jumbograms. Relying on ACK reduction alone won't fix that. 

The key issue is whether ACK clocking is important. It is implicit,
but only if the ACK frequency is high enough. If ACKs are less
frequent, the source becomes more bursty (allowed to send K packets
when an ACK arrives).

>     (a) the SHOULD in RFC-1122 should remain a SHOULD and not become
> 	a MUST, IMO

It seems useful to require _either_ a minimum acceptable ACK frequency
or a separate clocking mechanism that spaces transmission. This could
be something negotiated on a per-pair basis.

>     (b)	I'm sure there is a clever algorithm to be found that decides
> 	when the optimal time to send an ACK is; anybody got any ideas?

Clever, _and_ runs as needed for a gigabit, if you please :-) 

Joe

---

From: Vern Paxson <[email-masked]>
Date: Wed, 23 Jun 1999 21:43:27 PDT

>    (a)	the SHOULD in RFC-1122 should remain a SHOULD and not become

RFC 2581 makes it clear that this is a SHOULD and not a MUST (for precisely
the reason you raise, that there are some scenarios where an implementation
will have a good reason to ack less often).

		Vern

---

From: Joe Touch <[email-masked]>
Date: Thu, 24 Jun 1999 14:03:07 -0700 (PDT)

> From [email-masked] Thu Jun 24 13:04:33 1999
> Date: Thu, 24 Jun 1999 12:24:17 -0700
> From: "Eric A. Hall" <[email-masked]>
> X-Accept-Language: en
> To: Joe Touch <[email-masked]>
> CC: [email-masked]
> > > My thinking: the receiver should ACK when enough data has been received
> > > to fill 50% of the outstanding receive window. No more, no less.
> > 
> > This would encourage bursts of 1/2 the window size.
> 
> I prefaced this in terms of half-duplex links. Half/half is better than
> 2/2/2/2/2/etc in terms of actual utilization.

How does a system know the links are half duplex?

Also, half half isn't better than 2/2/2/2, where power is concerned.
Running at 1/2 speed all the time consumes less power than running full speed
for half the time. For some half-duplex links, where line speed can be
configured, this could affect power consumption.

Joe

---

From: Joe Touch <[email-masked]>
Date: Thu, 24 Jun 1999 14:07:18 -0700 (PDT)

> From [email-masked] Thu Jun 24 13:44:17 1999
> To: [email-masked] (Sam Manthorpe)
> cc: [email-masked] (Joe Touch), [email-masked]
> Date: Thu, 24 Jun 1999 13:04:54 -0700
> From: Greg Minshall <[email-masked]>
> 
> i've been *told* that Linux will let exactly one ACK live in the i/f output 
> queue, updating it with new acknowledgments and window updates.  this can 
> reduce the number of ACKs (though breaking the clock).
> 
> however, if your host is generating 52,000 *data* packets per second, i wonder 
> if getting 26,000 ACKs per second *should* be a problem.  (i don't doubt that 
> it *is* a problem, but *should* it be a problem?)

It will always be more computation than processing less ACKs.
However, it should be possible to have a network interface
where incoming interrupts are suppressed for a time (maybe
for all packets, maybe for solo ACKs, ???), such that the
source scans the input as part of its packet output routine.

I.e., the normal 'inner loop' could be:

	send a packet
	process an ACK

(or one every other, etc).

This would cut the number of input packet interrupts. 

I would claim that processing ACKs at this rate is part
of the price of playing nice, allowing the source to 
avoid bursts. If you want to negotiate around it for
sources that can pace, that's even better. But the 
default can be 'play nice'.

Joe

---

From: Vernon Schryver <[email-masked]>
Date: Thu, 24 Jun 1999 16:15:43 -0600 (MDT)

> it *is* a problem, but *should* it be a problem?)

Yes, it *should* be a problem.  If your host can handle 78K p/s, then it
probably doesn't care whether they contain 48, 64, 1500, or even 4K bytes
(assuming you don't finally hit the wall on raw medium speed).  If you
spend 30% of your maximum p/s rate on acks, then your throughput will be
limited to 60% of your max p/s times your MSS, and similarly if you spend
50% or 10% of your p/s rate on acks.

And then there are the odd details of various media where it every
turn-around of the medium costs as much or more bandwidth than a TCP ACK.
Examples of such media include CSMA/CD as implemented by the classic AMD
LANCE.

Vernon Schryver    [email-masked]

---

From: [email-masked]
Date: Thu, 24 Jun 1999 16:38:33 -0700

*> 
  *> Yes, it *should* be a problem.  If your host can handle 78K p/s, then it
  *> probably doesn't care whether they contain 48, 64, 1500, or even 4K bytes
  *> (assuming you don't finally hit the wall on raw medium speed).  If you
  *> spend 30% of your maximum p/s rate on acks, then your throughput will be
  *> limited to 60% of your max p/s times your MSS, and similarly if you spend
  *> 50% or 10% of your p/s rate on acks.
  *> 

I am not sure I believe in the significance of this reasoning.  VJ TCP
deliberately chose an approach (e.g., ACK clocking) that gives great
robustness and adaptability to a wide range of speeds and delays, while
not being strictly optimum in any particular circumstance.  Personally,
I believe this is still the right approach; optimization has a cost and
is likely to limit robustness and adaptability, and the latter are the
more important.

Bob Braden

  *> And then there are the odd details of various media where it every
  *> turn-around of the medium costs as much or more bandwidth than a TCP ACK.
  *> Examples of such media include CSMA/CD as implemented by the classic AMD
  *> LANCE.
  *> 
  *> 
  *> Vernon Schryver    [email-masked]
  *>

---

From: Vernon Schryver <[email-masked]>
Date: Thu, 24 Jun 1999 19:13:06 -0600 (MDT)

> From: [email-masked]

> is likely to limit robustness and adaptability, and the latter are the

I was not advocating or opposing changes to ACK clocking.  I was talking
about the cases that I think Sam was talking about.  In those cases, ACK
clocking is a bad thing, as are most of the other the TCP improvements
including slow start.  When you are going full blast over a local network,
you have not hit the wall on raw medium speed, and you are dealing with
small packets (now <=4 KBytes), then your hosts are limited (to a good
first approximation) by their raw packet per second rates.  To predict
your best throughput on media faster than your CPU's, multiply your p/s
rate by 60% of your MSS, or to predict your maximum p/s rate, divide your
throughput by 0.6 of your MSS. 

I think the 30% of packets that are Acks does far worse to TCP throughput
than merely make it not strictly optimum.  That factor of 0.6 in the formula
*hurts*.  I also don't have any wonderful ideas for reducing the number
of Acks only when it would not be naughty.

   ...........

] From: Joe Touch <[email-masked]>

] ...
] It will always be more computation than processing less ACKs.
] However, it should be possible to have a network interface
] where incoming interrupts are suppressed for a time (maybe
] for all packets, maybe for solo ACKs, ???), such that the
] source scans the input as part of its packet output routine.

] I.e., the normal 'inner loop' could be:
] 	send a packet
] 	process an ACK
]
] (or one every other, etc).
]
] This would cut the number of input packet interrupts. 

For more than 10 years, Sam's employer has been shipping systems that do
more sophisticated kinds of "interrupt rate limiting," "interrupt
throttling," "interrupt merging," or "interrupt bursting" than that loop,
using timers, counters, and other stuff in hardware, software, and
firmware.  Other UNIX vendors have also been in that market for a long
time.  My experience was that TCP goes fastest at about 0.1
interrupts/packet.  Fewer interrupts/packet allow the inevitable,
occassional miss in the game of patty-cake among hardware, board software,
and host software to take so long to be recovered by an interrupt that
throughput starts to decline.  If your ratio of CPU cycle rate to medium
bit rate is more than 10X from 1:1, I suspect that rule of thumb would
break down, and you'd want more or less than 0.1 interrupts/packet.

Some media, notably FDDI if you build the hardware right, enforce such
interrupt rate limiting.  Once you grab the token and start transmitting,
you cannot get any input or input interrupts.  The half-duplex nature of
a token ring caused bursts of ACKs to arrive at 20+ Kp/s.  Today FDDI is
slow, but 10 years ago you had to work very hard (and foolishly) to have
as many as one host interrupt per Ack.

Because of the way the BSD IP input queue and tcp_output() work, if your
host has cycles to spare while running at 100% of the medium, then
something like the loop described above is exactly what you get, just as
at slow speed over WANs.  That is the whole point of Ack clocking.  On
the other hand, when the limit is CPU cycles, I doubt it is good to have
the interrupt service routine doing a lot of fiddling with windows and
sequence numbers that will be duplicated by tcp_input() and tcp_output().
At 78 Kp/s, you have only 13 usec for absolutely everything you are going
to do with a packet.  Even at 500 MHz, and even if you reduce interrupts,
that's not a lot of cycles.

Many people have proposed various things, such as TCP agglomeration
(segment assembly) in firmware or hardware, but there are almost always
good reasons to not implement them.

] I would claim that processing ACKs at this rate is part
] of the price of playing nice, allowing the source to 
] avoid bursts. If you want to negotiate around it for
] sources that can pace, that's even better. But the 
] default can be 'play nice'.

It seems hard to figure out how to reduce the number of ACKs on a fast
LAN without being naughty in other important cases, which is why none of
the UNIX vendors (as far as I know) have shipped anything that doesn't
always play nice by default, even when it would help a lot (1/0.6) to cheat.

---

From: Joe Touch <[email-masked]>
Date: Fri, 25 Jun 1999 11:42:23 -0700 (PDT)

> From [email-masked] Thu Jun 24 18:18:31 1999
> cc: [email-masked]
> Date: Thu, 24 Jun 1999 18:18:35 -0700
> Joe,
> no, i think this is important:
> 
> >> the keys points are: 1) turn off Nagle and you may become a very *bad* 
> >> netizen;
> 
> >I don't follow that - it seems like it requires "and implement 
> >applications poorly", in which case you're bad anyway, no?
> 
> the point of the paper (understated) and of much of my experience is that it 
> is *very hard* to implement applications ``right'' (with respect to 
> buffering).  most people are going to get it wrong.  and, even when it is done 
> right, minor changes (bug fixes, adding features) are likely to make it wrong 
> again (in a sense, "proper buffering" is an unstable point, whereas "improper 
> buffering" is a stable point).  Nagle protects the net *not* from "poorly 
> written" applications, but rather from applications that aren't "very well 
> written".  this may seem like a subtle point, but i think the distinction is 
> very important.

Agreed. I would hope that better libraries would be
the place to put this collective wisdom, rather than
encumbering the protocol with it, though :-)

Joe