Subject: internet draft on suggested mod to the Nagle algorithm

From: Greg Minshall <[email-masked]>
Date: Mon, 01 Feb 1999 18:07:11 -0800

Tcp-impl people,

Just in case you didn't see the following, i'm forwarding the announcement 
along.  I'd love to hear any thoughts people might have, both on the concept 
and on how to implement it.  (I need to do at least one modification to add in 
how Sam Manthorpe/SGI has addressed this issue.)

Thanks,  Greg Minshall
------- Forwarded Message

Message-Id: <[email-masked]>
Mime-Version: 1.0
Content-Type: Multipart/Mixed; Boundary="NextPart"
To: IETF-Announce: ;
From: [email-masked]
Reply-to: [email-masked]
Subject: I-D ACTION:draft-minshall-nagle-00.txt
Date: Mon, 01 Feb 1999 11:53:39 -0500
Sender: [email-masked]
X-UIDL: 2d7829f9365bbfc5454f7c0d6a61f975

- --NextPart

A New Internet-Draft is available from the on-line Internet-Drafts directories.

	Title		: A Suggested Modification to Nagle's Algorithm
	Author(s)	: G. Minshall
	Filename	: draft-minshall-nagle-00.txt
	Pages		: 4
	Date		: 29-Jan-99
	
   The Nagle algorithm is one of the primary mechanisms which protects
   the internet from poorly designed and/or implemented applications.
   However, for a certain class of applications (notably,
   request-response protocols) the Nagle algorithm interacts poorly
   with delayed acknowledgements to give these applications poorer
   performance.

   This draft is NOT suggesting that these applications should disable
   the Nagle algorithm.

   This draft suggests a fairly small and simple modification to the
   Nagle algorithm to preserve Nagle as a means of protecting the
   internet while at the same time giving better performance to a
   wider class of applications.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-minshall-nagle-00.txt

Internet-Drafts are also available by anonymous FTP. Login with the username
"anonymous" and a password of your e-mail address. After logging in,
type "cd internet-drafts" and then
	"get draft-minshall-nagle-00.txt".

A list of Internet-Drafts directories can be found in
http://www.ietf.org/shadow.html 
or ftp://ftp.ietf.org/ietf/1shadow-sites.txt

Internet-Drafts can also be obtained by e-mail.

Send a message to:
	[email-masked].
In the body type:
	"FILE /internet-drafts/draft-minshall-nagle-00.txt".
	
NOTE:	The mail server at ietf.org can return the document in
	MIME-encoded form by using the "mpack" utility.  To use this
	feature, insert the command "ENCODING mime" before the "FILE"
	command.  To decode the response(s), you will need "munpack" or
	a MIME-compliant mail reader.  Different MIME-compliant mail readers
	exhibit different behavior, especially when dealing with
	"multipart" MIME messages (i.e. documents which have been split
	up into multiple messages), so check your local documentation on
	how to manipulate these messages.

Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.

- --NextPart
Content-Type: Multipart/Alternative; Boundary="OtherAccess"

- --OtherAccess
Content-Type: Message/External-body;
	access-type="mail-server";
	server="[email-masked]"

Content-Type: text/plain
Content-ID:	<[email-masked]>

ENCODING mime
FILE /internet-drafts/draft-minshall-nagle-00.txt

- --OtherAccess
Content-Type: Message/External-body;
	name="draft-minshall-nagle-00.txt";
	site="ftp.ietf.org";
	access-type="anon-ftp";
	directory="internet-drafts"

Content-Type: text/plain
Content-ID:	<[email-masked]>

- --OtherAccess--

- --NextPart--

------- End of Forwarded Message

---

From: "Eric A. Hall" <[email-masked]>
Date: Wed, 03 Feb 1999 13:03:33 -0800

> Just in case you didn't see the following, i'm forwarding the
> announcement along. I'd love to hear any thoughts people might have,
> both on the concept and on how to implement it.

I'll bite.

Q: If Nagle is a problem for my application, I can just disable it
altogether. What benefit is there in changing Nagle so that it affects
ALL transactions instead of just the ones I'm troubled by?

IE, by changing Nagle, you're going to be changing everything, and not
just those applications that are request/response-intensive. Can you
prove that your theory is always better with every application?
Otherwise I'd say just fix the apps that are causing you problems.

I'm interested in hearing your answer.

-- 
Eric A. Hall                                            [email-masked]
+1-650-685-0557                                    http://www.ehsco.com

---

From: "Eric A. Hall" <[email-masked]>
Date: Wed, 03 Feb 1999 13:13:24 -0800

> altogether.

One related question: Does everybody support applications disabling
Nagle on a per-circuit basis, as mandated by 4.2.3.4 in RFC 1122?

---

From: [email-masked] (Sam Manthorpe)
Date: Wed, 3 Feb 1999 14:32:43 -0800 (PST)

> Nagle on a per-circuit basis, as mandated by 4.2.3.4 in RFC 1122?

IRIX does.

-- Sam

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Sam Manthorpe, SGI.  tel: (650)933-2856 fax: (650)932-2856  [email-masked]

---

From: [email-masked] (Sam Manthorpe)
Date: Wed, 3 Feb 1999 14:30:57 -0800 (PST)

Hi Eric,

> ALL transactions instead of just the ones I'm troubled by?

Because you just have to make one change to the kernel code instead
of many changes to all applications which do bulk data transfer.  Moreover, 
in some cases, you will have to introduce some special intelligence
in the application so that it can decide _whether_ it is doing bulk
transfer or not.  Why not just put this extra intelligence in the
kernel.

> just those applications that are request/response-intensive. 

The request/response-intensive applications should notice no difference,
except where the responses consist of significant amounts of data.  I
can't think of any cases where fixing this in the kernel would cause
problems with other applications or for network utilisation; can anyone
else?

-- Sam

---

From: [email-masked] (Alan Cox)
Date: Wed, 3 Feb 1999 23:57:32 +0000 (GMT)

> > Nagle on a per-circuit basis, as mandated by 4.2.3.4 in RFC 1122?

Linux does

---

From: Rick Jones <[email-masked]>
Date: Wed, 03 Feb 1999 15:29:56 -0800

> of many changes to all applications which do bulk data transfer.  Moreover,

Back-up a step here - I thought that the problem being fixed here was
for request/response applications. From the original rfc txt:

begin quote:

The interaction of delayed ACKs and Nagle

   If a TCP has more application data to transmit than will fit in one
   packet, but less than two full-sized packets' worth of data, it
   will transmit the first packet.  As a result of Nagle, it will not
   transmit the second packet until the first packet has been
   acknowledged.  On the other hand, the receiving TCP will delay
   acknowledging the first packet until either i) a second packet
   arrives (which, in this case, won't arrive), or ii) approximately
   100ms (and a maximum of 200ms) has elapsed.

   When the sending TCP receives the delayed ACK, it can then transmit
   its second packet.

   In a request-response protocol, this second packet will complete
   either a request or a response, which then enables a succeeding
   response or request.

   Note two (related) bad results of the interaction of delayed ACKs
   and the Nagle algorithm in this case: the request-response time may
   be increased by up to 400ms (if both the request and the response
   are delayed); and, the number of transactions per second is
   substantially reduced.

end quote.

Perhaps my definition of bulk data transfer application is differnet,
but I take that to mean a very large (>>> MSS) quantity of data
transfered unidirectinally (eg FTP). Given that, I'm not sure that there
is any problem with such an application and the current Nagle algorithm
is there? Enough data accumulates from the small sends to make full-MSS
segments and away they go.

Now, in separate email I was discussing that first quoted paragraph with
Greg. I'll try to paraphrase it here :) First, the paragraph again:

   If a TCP has more application data to transmit than will fit in one
   packet, but less than two full-sized packets' worth of data, it
   will transmit the first packet.  As a result of Nagle, it will not
   transmit the second packet until the first packet has been
   acknowledged.  On the other hand, the receiving TCP will delay
   acknowledging the first packet until either i) a second packet
   arrives (which, in this case, won't arrive), or ii) approximately
   100ms (and a maximum of 200ms) has elapsed.

My understanding of "the way things are supposed to work" :) is that if
that one-and-a-fraction MSS-worth of data was presented to the TCP in a
single "send" that all of it should go out without delay. And yes, that
could mean a series of 1460,1,1460,1 segments traversing the network. I
demonstrated that being the case with a quick netperf TCP_RR test with
"-r 1461,1" - if the paragraphs description was correct, I should have
only gotten ~5 transactions per second (200 ms delayed ACK), but I got
several hundred.

My further understanding (going further in the quoted text above) was
that if the application presented the data to the TCP in two (or more)
"sends" and that the remote could only reply when all the data arrived,
that the application was essentially broken - it was presenting
logically associated data to the TCP in separate "send" calls when it
should have been presented in a single send (perhaps a gathering send).

Depending on how that application structured its sends. I'm not sure the
proposed change: 

   The proposed Nagle algorithm modifies this as follows:

        If a TCP has less than a full-sized packet to transmit,
        and if any previous less than full-sized packet has not
        yet been acknowledged, do not transmit a packet.

would actually "fix" things. For instance, a (broken IMO) application
that sent a 2048 byte "request" as a pair of 1024 byte "sends" would
still encounter the delayed ACK. As I am guessing would an application
dribbling data out through stdio (say a web plug-in or something)

Now, there may still be sufficient cases of reasonable applications that
are not broken, but still "suffer" from Nagle, but I'm not sure (yet -
I'm sure someone else will point some out :) what they are. _Maybe_
something to do with pipelined http requests generating sub MSS
responses?

The idea presented in the draft about an explicit flush mechanism was
interesting, though it would be a coupled change in both the
application(s) and the kernel.

rick jones
-- 
today, an ACK is just as expensive to the server as a data segment...
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to email, or post, but please do not do both...
my email address is raj in the cup.hp.com domain...

---

From: Kacheong Poon <[email-masked]>
Date: Wed, 3 Feb 1999 18:14:37 -0800 (PST)

> only gotten ~5 transactions per second (200 ms delayed ACK), but I got

This is interesting.  So are you saying that the receiver did not delay acks
in your test case?  Assuming MSS is 1460 bytes.  After the sender sends the
first 1460 bytes of the 1461 bytes request (in a single send call), it cannot
send any more because of Nagle.  And if the receiver implements delay ack and
ack every other segment, it will not send back an ack immediately if the 1460
bytes segment happen to be the first of the 2 segments.  Have you looked at
the transfer trace to see what is going on?

> something to do with pipelined http requests generating sub MSS

Well, in your test case, Nagle is supposed to kick in and reduce the
RR rate.  Are you sure the stack you tested with does not have some form
of optimization for this?

							K. Poon.
							[email-masked]

---

From: Dave Marquardt <[email-masked]>
Date: 04 Feb 1999 08:56:29 -0600

> Nagle on a per-circuit basis, as mandated by 4.2.3.4 in RFC 1122?

AIX does.

I suspect most implementations based on BSD do.  It's the TCP_NODELAY
socket option.

-Dave

---

From: Rick Jones <[email-masked]>
Date: Thu, 04 Feb 1999 09:42:14 -0800

> the transfer trace to see what is going on?

No, I'm saying that the last byte did not wait for the ack to go out -
the Nagle check was applied to the "send" as a whole, not the individual
segments that made up the "send."

rick jones

---

From: Kacheong Poon <[email-masked]>
Date: Thu, 4 Feb 1999 10:53:02 -0800 (PST)

> segments that made up the "send."

So this is an optimization your stack does.  For TCP, it just looks at
how much data there is to send.  It does not care if it is a single send
or not.  I infer from what you said that two segments, one 1460 bytes
and one 1 byte, would go out immediately regardless of Nagle because it
was a single send.

I think this is another variant of the proposed change.  It seems that
people are not against the change.  Maybe the author can think of another
way to phrase it.

							K. Poon.
							[email-masked]

---

From: Rick Jones <[email-masked]>
Date: Thu, 04 Feb 1999 11:55:43 -0800

Kacheong Poon wrote:
> So this is an optimization your stack does.  

I guess I've been so used to this behaviour over the last N years that I
thought it was present in all stacks :) Silly me :)

rick

---

From: [email-masked]
Date: Fri, 5 Feb 1999 07:55:05 -0500

> thought it was present in all stacks :) Silly me :)

BSD (at least since 4.4BSD--I haven't checked the earlier code)
does the same thing.  Since Kacheong reports that Solaris also does
this, it looks as if this is common behavior.

Jon Snader

---

From: Greg Minshall <[email-masked]>
Date: Fri, 05 Feb 1999 11:46:45 -0800

All and sundry,

I'm going to try to respond to several people in one (or more?) message(s).

Eric Hall and Joe Touch both ask, quite reasonably, why not just go ahead and 
disable Nagle and be done with it.

The reason is that Nagle is very effective at protecting the network from 
hordes of small packets.  One of the main reasons i am proposing this 
modification is to keep people from being tempted to disable Nagle.

I've seen a lot of cases where people wanted to turn off Nagle, but most of 
them have been because of not really understanding what is going on, 
especially with respect to buffering.

As an example, some colleagues of mine recently turned off Nagle on a Usenet 
server.  They saw article retrieval latency drop substantially, which was a 
good thing.  However, at first they did not notice that the number of packets 
being transmitted (for the same request stream) increased an order of 
magnitude (with the average packet size dropping from, basically, MSS to 140 
bytes!).  (It turned out that when they went back in, turned Nagle *back on*, 
and improved the buffering of the Usenet server, they got the reduced 
latencies *without* introducing any more load into the network.)

Thus, turning off Nagle is a very dangerous thing to do.  I wouldn't want lots 
of applications running with Nagle turned off; in fact, i wouldn't want any 
more applications turning off Nagle than already do.

Again, my hope is that by introducing this change, there will be less of a 
tendency for application writers to think "bad performance?  oh, i know, i'll 
disable Nagle!".

Eric Hall also asks (basically) could changing the Nagle algorithm in this way 
either make things worse for existing applications, or increase the load on 
the network for existing applications.  Here is about all i know how to say 
about that:  if an application is dribbling out data to TCP at a little over 1 
MSS worth of data per RTT, then with the original Nagle (and assuming the 
timing is exactly right) you might see one ``small'' packet, then a stream of 
big packets (about one per RTT), whereas with the modified Nagle (again, 
assuming the timing is exactly right) you might see one ``small'' packet and 
one big packet each RTT.

In a later mail, you say:

> the idea here is to avoid waiting for a timer right? specifically, the
> standalone ACK timer.

Actually, in some cases the RTT that Nagle imposes may be as significant as 
the delayed ACK timer.

> that would seem to indicate that if one allowed up to 2MSS of unacked,
> "small" sends and _then_ stopped one could be reasonably sure that an
> ACK would be arriving within one RTT of the 2MSSth byte being sent.
> 
> it would allow a "pathological" application to put up to 2MSS packets
> out there, but I'm guessing it would be cleaner than simply disabling
> nagle entirely?

This would probably be cleaner than disabling Nagle entirely, but i don't 
think this is a good idea.  Imagine 2*MSS of 1 byte packets!

The point is to get the entire request or response to the other side with as 
few packets as possible (each as large as possible).  Once it is there, then 
the other side can respond or send another request.

Greg Minshall

---

From: Greg Minshall <[email-masked]>
Date: Fri, 05 Feb 1999 11:47:10 -0800

Rick,

> Now, there may still be sufficient cases of reasonable applications
> that are not broken, but still "suffer" from Nagle, but I'm not sure
> (yet - I'm sure someone else will point some out :) what they are.
> _Maybe_ something to do with pipelined http requests generating sub
> MSS responses? 

If a response (could equally well be a request) sends an odd number of MSS 
packets, and then wants to send one small packet, Nagle will block until all 
the MSS packets have been acknowledged.  Because the response stream was an 
odd number of packets, the remote TCP will probably decide not to ACK for 
100ms or so, thus preventing the response from making it through.  So, the 
response will take RTT+100ms (approximately) longer than it could have taken.

So, this is the application where there is an argument to be made that Nagle 
(especially when coupled with delayed ACKs) is slowing things down somewhat.

With the proposed modification, the small packet will be allowed to follow the 
MSS packets (assuming the window, congestion stuff, etc.).

Greg

---

From: Rick Jones <[email-masked]>
Date: Fri, 05 Feb 1999 12:45:08 -0800

> (especially when coupled with delayed ACKs) is slowing things down somewhat.

But if that application is presenting that data in separate send calls,
and it was indeed data that was really one unit, is that application
broken or not? The whole issue here is for request/response applications
right? Which means that there is all the data of the "request" and then
all the data of the "response."

> MSS packets (assuming the window, congestion stuff, etc.).

I guess part of my hangup is considering any transport that did not
interpret Nagle on a per"send" basis rather than a per segment basis as
broken to begin with - perhaps then I'm in violent agreement with the
spirit of the proposed change :) And with the sonofNagle, apps which are
broken and sending logically associated data as two sub-MSS sends are
still broken and still going to either get fixed by the app developer
(unlikely but good) or disable Nagle (likely and bad) anyway. 

I'm guessing (common and bad :) that most apps that disable nagle today
are like that - successive sub-mss sends of logically-associated data
synchronously awaiting a response/request. Those apps are only going to
be fixed with apps mods - either by getting all (the major) TCP's to
include the explicit flush mechanism, or by getting the app to do a
gthering write. If we are going to have to outreach to get apps modified
anyway, might as well get all the "rightness" into them we can and get
them onto gathering writes right?

rick jones

---

From: Greg Minshall <[email-masked]>
Date: Fri, 05 Feb 1999 15:52:04 -0800

Rick,

I see what you are saying:

> But if that application is presenting that data in separate send
> calls, and it was indeed data that was really one unit, is that
> application broken or not? The whole issue here is for request/
> response applications right? Which means that there is all the data of
> the "request" and then all the data of the "response." 

I think it is reasonable to have the application buffer reasonable amounts of 
data, but i'm not sure it is reasonable to say it must buffer 50Kbytes or 
100Kbytes.  It seems to me (but, i agree, there is room here to differ) that 
if the application is passing down its response (or request) in 4096 byte 
chunks (to pick a popular size), TCP should "work well".

Greg

---

From: Rick Jones <[email-masked]>
Date: Fri, 05 Feb 1999 18:20:38 -0800

> data, but i'm not sure it is reasonable to say it must buffer 50Kbytes or

I will go along with the statement that (while desirable :) it is not
always reasonable to expect an application with a 50K response/request
to provide that to TCP in one call. 

I will point-out thought that I regularly see (ok, ok, in benchmarking
:) web servers handing the transport as much data as it will take in a
"send" call and that being more than 50K often enough depending on the
"SO_SNDBUF" setting. 

By the time we are talking about 50K request/responses are we really
talking request/response any more, or are we back to (short)
unidrectional streams? Another way to ask that is is there really a
problem with applications sending across the net 50KB of data and then a
single byte?

> chunks (to pick a popular size), TCP should "work well".

I would probably say "work well enough" :) I think that if data is given
in 4096 byte chunks that well enough is (assuming a 1460 MSS) 1460,
1460, 1176. I see that today in netperf TCP_RR tests over some
transports, with no nagle/delayedack induced delays:

$ uname -a
HP-UX loiter B.10.20 A 9000/735 2004221734 two-user license
$ remsh hpisrdq "uname -a"
HP-UX hpisrdq A.09.05 E 9000/725 2007119367 8-user license
$ ./netperf -t TCP_RR -H hpisrdq -- -r 1,4096 -S 32K -s 32K
TCP REQUEST/RESPONSE TEST to hpisrdq
Local /Remote
Socket Size   Request  Resp.   Elapsed  Trans.
Send   Recv   Size     Size    Time     Rate         
bytes  Bytes  bytes    bytes   secs.    per sec   

32768  32768  1        4096    10.00     151.95   
32768  32768 
$ ./netperf -t TCP_RR -H hpisrdq -- -r 4096,1 -S 32K -s 32K
TCP REQUEST/RESPONSE TEST to hpisrdq
Local /Remote
Socket Size   Request  Resp.   Elapsed  Trans.
Send   Recv   Size     Size    Time     Rate         
bytes  Bytes  bytes    bytes   secs.    per sec   

32768  32768  4096     1       10.00     150.17   
32768  32768 

here's some to include HP-UX 11 as a sender and to show it at something
that might not be MCLBYTES in size :)

# ./netperf -H hpisrdq -t TCP_RR -- -r 4097,1 -S 32K -s 32K
TCP REQUEST/RESPONSE TEST to hpisrdq
Local /Remote
Socket Size   Request  Resp.   Elapsed  Trans.
Send   Recv   Size     Size    Time     Rate         
bytes  Bytes  bytes    bytes   secs.    per sec   

32768  32768  4097     1       10.01     107.13   
32768  32768 
# uname -a
HP-UX hpntc11a B.11.00 A 9000/778 2011481613 two-user license

back to the 10.20 box...
$ ./netperf -t TCP_RR -H hpisrdq -- -r 4097,1 -S 32K -s 32K
TCP REQUEST/RESPONSE TEST to hpisrdq
Local /Remote
Socket Size   Request  Resp.   Elapsed  Trans.
Send   Recv   Size     Size    Time     Rate         
bytes  Bytes  bytes    bytes   secs.    per sec   

32768  32768  4097     1       10.00     146.46   
32768  32768 

I'm even happy if that was 8192 bytes in two 4096 byte sends - unless of
course I'm talking over a link with a 9000 byte MTU...which takes us
into other territory :) 

I guess I would be in the camp that feels that interpreting nagle send
by send is sufficient and the way to go and that we need to make sure
app writers understand just what they are doing when they set
TCP_NODELAY. A "mantra" along the lines of "provide as much of the
logically associated data as you can to the transport at one time" or
something...

rick jones

---

From: Kacheong Poon <[email-masked]>
Date: Fri, 5 Feb 1999 19:08:02 -0800 (PST)

> logically associated data as you can to the transport at one time" or

Is there such a document available somewhere?  If not, maybe we should write
up an informational draft to describe this simple but not widely understood
guidelines in writing network applications using TCP.

I just checked Steven's "UNIX Network Programming" book.  It does suggest
people to use writev().  But I don't think it puts enough emphasis on the
adverse effect of disabling Nagle algorithm by default.  It is this point
that we need to bring to the application programmers.

							K. Poon.
							[email-masked]

---

From: Greg Minshall <[email-masked]>
Date: Mon, 08 Feb 1999 16:53:38 -0800

Rick,

> logically associated data as you can to the transport at one time" or

I think two things.

One, i don't think application writers can always guarantee that even if they 
get the write buffer size, all the data will be exempt from Nagle.  For 
example, in the BSD stack at least, if you pass too much data for the current 
(congestion or whatever) window in your send() call, subsequent packets may be 
subject to Nagle (thus susceptible to bad interactions with delayed ACKs at 
the remote side).  Check out, for example, John Heideman's article in the 
April, 1997, ACM SIGCOMM Computer Communications Review, or Henrik Nielsen, et 
al, from SIGCOMM 1997, for example of fairly well-crafted applications that 
still run afoul of Nagle/delayed ACK interactions.

Two, i think that making this change may have the effect of preventing people 
from wholesale disabling of Nagle.

(I also think i'm happy to have the change apply to "send()s", as is quite 
often current practice, rather than to the output of TCP.  I think, though, 
that it should be *written* as if it applies to the output of TCP, just as 
RFC896.)

Does that make sense?

Cheers,  Greg

---

From: Kacheong Poon <[email-masked]>
Date: Mon, 8 Feb 1999 18:05:16 -0800 (PST)

> One, i don't think application writers can always guarantee that even if
> they  get the write buffer size, all the data will be exempt from Nagle. 
> For  example, in the BSD stack at least, if you pass too much data for the
> current  (congestion or whatever) window in your send() call, subsequent
> packets may be  subject to Nagle (thus susceptible to bad interactions with
> delayed ACKs at  the remote side).

If we agree that Nagle algorithm should be applied on a per send basis, not on
a per segment basis, then change the BSD code so that it will send out the
remaining data.  And I think Dave has pointed out that his BSD code does that,
more or less.  Please correct me if I misunderstood that.

> Two, i think that making this change may have the effect of preventing
> people  from wholesale disabling of Nagle.

I don't understand.  We are trying to discourage people from disabling Nagle
by default, right?  Do I misunderstand your sentence above?

							K. Poon.
							[email-masked]

---

From: Rick Jones <[email-masked]>
Date: Mon, 08 Feb 1999 18:52:05 -0800

> still run afoul of Nagle/delayed ACK interactions.

OK, so one of the problems reported was (IMO) a bug in Apache (sending
the headers separate from the data and yes, prompted by a browser bug
:), so I'm not sure that is not a priori cause to modify TCP.

The second biggie there - the small last segment is at least in part
solved by applying nagle on a per-send basis. That does indeed leave-out
the case where an application with 4097 bytes of data sends it as 4096
+1, and that last "1" gets delayed because it was a separate send. I'm
still not sure that is not an application bug for not presenting the
4097 bytes of data in a writev() or the like.

Where that magic constant changes I do not know, and I can see that as
an argument for some explicit flush mechanism between the app and TCP -
ioctl or send() flag I suppose. I think though that an application aught
to be able to present up to SO_SNDBUF bytes of a > SO_SNDBUF bytes
request or response. 

Regardless, it is hard for me to not believe that the experiements in
the paper were not simply conducted with buggy implementations.

> from wholesale disabling of Nagle.

People are already wholesale disabling Nagle, the only question is
whether or not we can reverse that trend.

> that it should be *written* as if it applies to the output of TCP, just as

If the updated RFC is written as if it applies to the the output of TCP,
there will be someone who really wants to pick nits with implementations
that are still doing the right/sufficient thing, just not implementing
it precisely the same way. That of concerns me.

Also, I'm not sure what part of RFC896 is stating that the algorithm
should be impemented on a per-segment basis rather than per-send. For
example, the second paragraph of the second "The solution to the small
packet problem" reads:

#begin quote
The solution is to inhibit the sending of new TCP  segments  when
new  outgoing  data  arrives  from  the  user  if  any previously
transmitted data on the connection remains unacknowledged.   This
inhibition  is  to be unconditional; no timers, tests for size of
data received, or other conditions are required.   Implementation
typically requires one or two lines inside a TCP program.
#end quote

Notice how it says "new data arrives from the user" - that sounds much
more like someone calling "send()" than TCP sending a segment. Two
paragraphs later it reads:

#begin quote
When a user process writes to a TCP connection, TCP receives some
data.   It  may  hold  that data for future sending or may send a
packet immediately.  If it refrains from  sending  now,  it  will
typically send the data later when an incoming packet arrives and
changes the state of the system.  The state changes in one of two
...
Thus, when we omit sending data on arrival from the
user,  we  are  simply  deferring its transmission until the next
message arrives from the distant host. 
#end quote

again "a user process writes to a TCP connection" and "when we omit
sending data on arrival from the user" sounds like a call to "send()" 

if I then switch to RFC1122, I find this in 4.2.2.14 I find:

#begin quote
                 Another important TCP performance issue is that some
                 applications, especially remote login to character-at-
                 a-time hosts, tend to send streams of one-octet data
                 segments.  To avoid deadlocks, every TCP SEND call from
                 such applications must be "pushed", either explicitly
                 by the application or else implicitly by TCP.  The
                 result may be a stream of TCP segments that contain one
                 data octet each, which makes very inefficient use of
                 the Internet and contributes to Internet congestion.
                 The Nagle Algorithm described in Section 4.2.3.4
                 provides a simple and effective solution to this
                 problem.  It does have the effect of clumping
#end quote

with the bit about "every TCP SEND call from such applications..."
wouldn't that be the send() socket call? That is the call the
application makes. And it says that each send should be "pushed" modulo
4.2.3.4 so:

#begin quote
         4.2.3.4  When to Send Data
 
            A TCP MUST include a SWS avoidance algorithm in the sender.
 
            A TCP SHOULD implement the Nagle Algorithm [TCP:9] to
            coalesce short segments.  However, there MUST be a way for
            an application to disable the Nagle algorithm on an
            individual connection.  In all cases, sending data is also
            subject to the limitation imposed by the Slow Start
            algorithm (Section 4.2.2.15).
 
            DISCUSSION:
                 The Nagle algorithm is generally as follows:
 
                      If there is unacknowledged data (i.e., SND.NXT >
                      SND.UNA), then the sending TCP buffers all user

Internet Engineering Task Force                                [Page 98]

 
RFC1122                  TRANSPORT LAYER -- TCP             October 1989

                      data (regardless of the PSH bit), until the
                      outstanding data has been acknowledged or until
                      the TCP can send a full-sized segment (Eff.snd.MSS
                      bytes; see Section 4.2.2.6).
#end quote

So, one might interpret the "regardless of the PSH" bit as being counter
to the "interpret on each send()" that I am espousing, but 1122 was
expanding upon 896 anyway (since 896 was an unconditional wait, not tied
to MSS) so I would argue that interpreting per-send is either a return
to (the possible) intent of 896 (it talked of user sends, not tcp_output
calls) or simply a further refinement of 1122 (I'm sure there is more
chapter and verse out there...). 

Short of an explicit end of message indicator, or an explicit flush from
the user, the send() boundary is the best bet TCP has for guessing if
there is going to be any more data to even be able to make-up an
MSS-sized segment in a request/response situation. 

We are already (or aught to be) telling apps writers that bigger sends
are better. The bigger the sends become, the more Nagle+delACK becomes a
race condition or less of a concern yes? 

Similarly for a per-send interpretation of Nagle on a bulk-data
application - if the application gets ahead of the cwnd (how often does
a bulk-data app remain behind cwnd anyway?) cwnd is going to do the
aggregating for us anyway. 

Make SO_SNDBUF just a skosh larger (perhaps simply as a matter of when
one decides it is full...) than the TCP window and you get the same
thing when you do stay ahead of cwnd - for instance, HP-UX TCP's from
HP-UX 9 up through 10.20 allow a full MCLBYTES to be queued to the
socket even if there is only 1 byte left. Fewer icky, small mbuf issues
then, and it gives you a better shot of buffering up data in bulk
transfers and not having small segments.

I'm not sure which is easier to implement or the most logical (I have an
opinion on the latter :) - per-send or tracking that sndnxt-snduna is
not an MSS multiple. It sounds like many stacks have already gone
per-send. That (and the proposed change) still leaves all those "series
of sub-mss send" apps wanting to disable Nagle, and I'm betting they are
the ones we really need to get "fixed" - which is only going to be fixed
with the explicit flush. (I do believe that explicit flush appears
easier to implement than CORK...)

It is interesting (at least to me) to consider what the packets would
look like had HTTP been implemented on top of UDP - packet generation
would have been on a per-send basis as well...and we would probably be
encouraging the http apps types to be presenting as much data to the
transport at one time as possible and using writev() and such to avoid
long series of sub-MTU-sized datagrams...

rick
-- 
today, an ACK is just as many CPU cycles as a data segment...
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to email, or post, but please do not do both...
my email address is raj in the cup.hp.com domain...

---

From: Greg Minshall <[email-masked]>
Date: Thu, 11 Feb 1999 11:14:22 -0800

K. Poon,

> If we agree that Nagle algorithm should be applied on a per send
> basis, not on a per segment basis, then change the BSD code so that it
> will send out the remaining data.  And I think Dave has pointed out
> that his BSD code does that, more or less.  Please correct me if I
> misunderstood that. 

1.  I may well have misunderstood what Dave said.  However, if the final 
send() of a response is for 33 bytes, say, and there is unacknowledged data 
(from "full sized" send() calls), then the current Nagle would block the 33 
bytes until all previous data has been acknowledged.  The modified Nagle would 
send the 33 bytes immediately (modulo windows, etc.).

> I don't understand.  We are trying to discourage people from disabling
> Nagle by default, right?  Do I misunderstand your sentence above?

Yes, that is what we are trying to do.  I'm sorry that my sentence was unclear.

By the way, there are a couple of ways that you could apply "Nagle on a per 
send basis".

The way the TCP stack on my machine works (FreeBSD 2.2.x) is that when 
tcp_output is called, if there is no unacknowledged data, it will send *all* 
the available data (modulo windows), even if the last segment (possibly only 
segment) it transmits is less than a full MSS.  In *this* case, data that is 
*not* transmitted in this call to tcp_output because of window restrictions 
may be subject to being delayed by Nagle the next time tcp_output is called 
(because of a returning ACK, say).

It sounds like Dave has made sure that the entire send()'s amount of data gets 
at least this treatment (rather than on a per-call-to-tcp_output() basis as is 
true in my stack).  But, again, it may be the case that in his, or other's, 
stacks, the "residual" data not transmitted this time around may be subject to 
being delayed by Nagle later in the connection.

Greg

---

From: Greg Minshall <[email-masked]>
Date: Thu, 11 Feb 1999 11:35:31 -0800

Rick,

> solved by applying nagle on a per-send basis. That does indeed
> leave-out the case where an application with 4097 bytes of data sends
> it as 4096 +1, and that last "1" gets delayed because it was a
> separate send. I'm still not sure that is not an application bug for
> not presenting the 4097 bytes of data in a writev() or the like. 

Let's say a web server (httpd) is reading data from a CGI script, buffering 
it, and passing it down to TCP to send out over an ethernet.  Buffering it in 
4096 byte chunks seems reasonable.  Let's say the CGI script writes 4096 bytes 
(or, 5000 bytes, or ...).  Then the question is "should the trailing bytes be 
delayed by Nagle?".

I think the answer is "no".  The application, in this case, is doing an 
adequate job of buffering (larger than the MSS), and the trailing data should 
be sent "immediately" (modulo windows stuff).

> whether or not we can reverse that trend.

I think making this change is a necessary, though perhaps not sufficient, step 
in reversing that trend.

> If the updated RFC is written as if it applies to the the output of
> TCP, there will be someone who really wants to pick nits with
> implementations that are still doing the right/sufficient thing, just
> not implementing it precisely the same way. That of concerns me. 

I think this is a valid point (as you have pointed out multiple times, and as 
many other people have pointed out again and again).  I will try to re-write 
the draft in a way that allows either behavior.  (I think i will say that some 
implementations apply Nagle on application send(), and that that is perfectly 
reasonable; i don't think i will say "that is preferred", since that seems to 
open a can of worms.  Thoughts from the assembled multitude?)

> The bigger the sends become, the more Nagle+delACK becomes a

My experience is that the more you close the window for a race condition, the 
more it seems to occur!

I think that by modifying the Nagle algorithm in the manner presented in the 
draft, we can improve the performance for a percentage of the request/response 
interactions for many internet applications (including the biggie, http) in a 
way that is transparent to application developers.  I think this is a win.

Greg (who'd like to move a revised draft into the RFC process, possibly as 
experimental)

---

From: Rick Jones <[email-masked]>
Date: Thu, 11 Feb 1999 12:32:09 -0800

> be sent "immediately" (modulo windows stuff).

Well, unless 2*MSS is an integral divisor (?) of the blessed application
buffering size, the second, small send is going to be delayed anyway, in
either nagle case - original or updated right? (and I suspect we do not
want to go down the rathole of having apps getsockopt(TCP_MAXSEG)?-)

send(4096)
  transmit 1460
  transmit 1460
  transmit 1176  
send(904)
  wait for ack

The only thing that will force that second, small send out would be an
explicit flush, and if there is an explicit flush, is there any real
need to change the interpretation of nagle?

Perhaps the thing to focus on and codify then is the explicit flush? I
suspect that the explicit flush would be sufficient to reverse the trend
of TCP_NODELAY usage. The app will always know better than TCP just what
is and is not the end of a message no matter what naglesque heuristic we
use, and even if apps started flushing on every send the behaviour on
the net would be no worse than it is today with TCP_NODELAY.

> way that is transparent to application developers.  I think this is a win.

I do not think that just the interpretation change alone will make much
of a difference because the internal application buffering such as you
discuss will not align nicely with 2*MSS, or the applications will make
a pair of sub-mss sends.

Also, it sounds like we have a number of stacks out there already which
are implementing something very close to the proposed change to Nagle,
and we still have rampant use of TCP_NODELAY.

So, we could go ahead with the codification of the change to nagle as
being per-send/the draft, but unless explicit flush becomes reality I
don't think there will be a significant impact on the use of
TCP_NODELAY.

rick

---

From: Kacheong Poon <[email-masked]>
Date: Thu, 11 Feb 1999 12:55:54 -0800 (PST)

> it, and passing it down to TCP to send out over an ethernet.  Buffering it
> in  4096 byte chunks seems reasonable.  Let's say the CGI script writes 4096
> bytes  (or, 5000 bytes, or ...).  Then the question is "should the trailing
> bytes be  delayed by Nagle?".

I think most implementations support the socket option TCP_MAXSEG.  If a
well written application is going to buffer data, I think it is better
to buffer data in multiples of SMSS (sending MSS) bytes size.  Then the
problem mentioned above should not be there, assuming PMTU does not change...
Should this, application buffering in chunks of SMSS bytes buffers, be
mentioned in the draft as a best practise?

> draft, we can improve the performance for a percentage of the
> request/response  interactions for many internet applications (including the
> biggie, http) in a  way that is transparent to application developers.  I
> think this is a win.

Yes, I think we all agree with the draft's good intention.

							K. Poon.
							[email-masked]

---

From: "Eric A. Hall" <[email-masked]>
Date: Thu, 11 Feb 1999 13:42:31 -0800

> Greg (who'd like to move a revised draft into the RFC process,
> possibly as experimental)

I'm all for expiremental.

I'd also like to see some sort of wording that says "developers should
only disable Nagle if they know for certain that they will only be
generating data that is cumulatively smaller than two full-sized
segments, as many writes of small blocks of data will negatively impact
the network, possibly to the point that performance degradation occurs
for all applications (including the disabler)" or something similar.
This helps to get the message out.

---

From: Greg Minshall <[email-masked]>
Date: Thu, 11 Feb 1999 14:23:16 -0800

Rick,

In the case you are doing Nagle on send() (rather than on segments leaving 
tcp_output()), then the send(904) would *not* be delayed by the modified Nagle 
algorithm.

Greg

---

From: Rick Jones <[email-masked]>
Date: Thu, 11 Feb 1999 15:06:47 -0800

> tcp_output()), then the send(904) would *not* be delayed by the modified Nagle

OK, I'm losted then :) Or perhaps I missed a clarification:

So, the application doing internal buffering in 4096 byte chunks trying
to send 5000 bytes goes

send(4096) - this is > MSS so it all goes out right away
 xmit 1460
 xmit 1460
 xmit 1176
send(904)
 
the second send call is < MSS, and there is an unacked, less than
full-sized segment on the network, so the send(904) waits.

Is the clarification I'm missing that in essence _all_ instances of
"packet" were replaced by "send?" If a TCP is presented with less than
an MSS-sized send to transmit, and if any previous less than MSS-sized
send has not been acknowledged, do not transmit the send. Is that it?

I've been assuming something a triffle simpler (presumeably) to
implement - if the send is larger than MSS send all of it now (windows
willing). If the send is less than MSS, only send it if it and unsent
data combines to form a >=MSS-sized segment.

Which have you been assuming?

rick

PS - since everyone in the discussion is on the mailing list, might as
well trim the reply to just tcp-impl - that will serialize the
discussion through the list server, and cut-down on the number of
messages some of us receive :)

---

From: Rick Jones <[email-masked]>
Date: Thu, 11 Feb 1999 15:56:06 -0800

One thought that just struck me is that we've all been bantering about
with a 1460 MSS thinking primarily about 1500 byte MTU networks and how
4096 is big enough for a 1500 byte MTU network and all that.

However, once someone takes that nice Apache (etc) application up to
some (yes, possibly rare, but not unknown) larger MTU intranet (ATM,
JumboFrame Ethernet, 802.5, etc) that "4096 bytes of application
buffering is fine" thinking falls apart and we again have even sends
that are sub-MSS.

Which, I think is yet another reason why if we codify anything it aught
to be the explicit flush mechanism. That is the only thing that seems to
have any chance of putting a serious dent in the use of TCP_NODELAY.

rick jones

---

From: Kacheong Poon <[email-masked]>
Date: Fri, 12 Feb 1999 17:24:22 -0800 (PST)

> buffering is fine" thinking falls apart and we again have even sends

What should be the size of small segment in Nagle algorithm?  If MSS can be
as large as, say 4056 bytes, then should a 1500 bytes segment be considered as
small?  I will say no.  While we are talking about modification to Nagle
algorithm, maybe we should also think about what a small segment really 
means.  To define small as less than 1 SMSS bytes may not be good any more
with today's technology.  BTW, although you think it is a rathole, I guess
doing a getsockopt(TCP_MAXSEG) is useful, especially in this case.

							K. Poon.
							[email-masked]

---

From: Greg Minshall <[email-masked]>
Date: Fri, 12 Feb 1999 17:45:17 -0800

Rick,

Sorry.  You are right.

> send has not been acknowledged, do not transmit the send. Is that it?

This *is* what i am assuming.

> 4096 is big enough for a 1500 byte MTU network and all that.

While i don't think it is reasonable to assume that an application tracks the 
actual MSS (given path MTU, etc.), i think it *is* reasonable that an 
application know *something* (such as the largest MTU on a connected, 
non-loopback interface).

Greg

---

From: "Eric A. Hall" <[email-masked]>
Date: Sat, 13 Feb 1999 19:53:27 -0800

> What should be the size of small segment in Nagle algorithm?  If MSS
> can be as large as, say 4056 bytes, then should a 1500 bytes segment
> be considered as small?  I will say no.

Nagle should stay as it is, at least wrt the defintion to "small
segments." Think of an org using Token Ring everywhere. They benefit
from having large frames everywhere mostly by being able to fill them,
and here you are wanting to take that away from them. Tsk tsk.

---

From: Greg Minshall <[email-masked]>
Date: Wed, 24 Feb 1999 08:06:16 -0800

K. Poon,

> I think most implementations support the socket option TCP_MAXSEG.  If
> a well written application is going to buffer data, I think it is
> better to buffer data in multiples of SMSS (sending MSS) bytes size.
> Then the problem mentioned above should not be there, assuming PMTU
> does not change... Should this, application buffering in chunks of
> SMSS bytes buffers, be mentioned in the draft as a best practise?

Three things:

1.  I don't know how to guarantee an application a way of tracking Eff.snd.MSS 
(rfc1122 terminology).  They can get a snapshot and, most times, that is 
sufficient but, as you mention, this number can change.

2.  Even so, i've added a parenthetical remark to the draft: "(ideally a 
multiple of Eff.snd.MSS)".

3.  Even if the application *does* buffer response (say) data in multiples of 
Eff.snd.MSS, there will quite possibly be a trailing portion of data that is 
less than Eff.snd.MSS.  The modified algorithm is designed to ensure timely 
transmission of that trailing portion.

Greg

---

From: Kacheong Poon <[email-masked]>
Date: Fri, 26 Feb 1999 15:14:47 -0800 (PST)

> 1.  I don't know how to guarantee an application a way of tracking
> Eff.snd.MSS  (rfc1122 terminology).  They can get a snapshot and, most
> times, that is  sufficient but, as you mention, this number can change.

This may not be a problem.  Suppose the application gets SMSS right after the
connection is established.  This SMSS is the maximum SMSS this connection can
ever use because PMTUd will only make SMSS smaller.  If the app buffers, say
2 times of the max SMSS, data and the TCP stack applies Nagle on a per send
basis, no delay of sending will happen.

Actually, as long as an app buffers data in chunks greater than 1 SMSS and the
TCP stack applies Nagle on a per send basis, no delay of sending will happen. 
So maybe we should answer this question first, is the original idea of Nagle
algorithm based on a per send basis, not per segment basis as implemented in
original BSD code?  If so, maybe we should change the draft to clarify this.

							K. Poon.
							[email-masked]

---

From: Greg Minshall <[email-masked]>
Date: Fri, 26 Feb 1999 17:16:26 -0800

K. Poon,

Yes, you are right, the MSS at the beginning is the largest a session will 
see, so that may be sufficient.  I had forgotten that it can't grow (since, as 
you point out, the PMTU may change, but the negotiated MSS will not grow).

However:

> Actually, as long as an app buffers data in chunks greater than 1 SMSS and
> the TCP stack applies Nagle on a per send basis, no delay of sending will 
> happen. 

If the *final* application write is < 1 SMSS, it may be delayed by the current 
algorithm.  The modification would allow that last application write to be 
transmitted.

Greg

---

From: Kacheong Poon <[email-masked]>
Date: Fri, 26 Feb 1999 18:28:29 -0800 (PST)

> K. Poon,
> If the *final* application write is < 1 SMSS, it may be delayed by the
> current  algorithm.  The modification would allow that last application
> write to be  transmitted.

Hmmm, it may be a little bit more complicated than this.  Suppose the last but
one send is a little bit bigger than 1 SMSS.  With your proposed change, TCP
will send the small tail segment right away.  Then the final send you
mentioned above will also be delayed because that small tail segment of the
previous send has not been acked.  So an app must write in exaclty SMSS size,
which we know may not be possible.  It seems to me that your proposed change
may be equivalent to applying Nagle on a per send basis in most situations.

Are we going around a circle here (-:  Let me ask this question again.  I
think many implementators also want to know the answer.  Is there any
objection that Nagle algorithm "should" be applied on a per send basis, as
opposed to what RFC 1122 section 4.2.3.4 says?

							K. Poon.
							[email-masked]