Subject: Re: internet draft on suggested mod to the Nagle algorithm

From: David Borman <[email-masked]>
Date: Mon, 1 Mar 1999 14:09:32 -0600 (CST)

> Date: Fri, 26 Feb 1999 18:28:29 -0800 (PST)
> From: Kacheong Poon <[email-masked]>
> ...
> Is there any
> objection that Nagle algorithm "should" be applied on a per send basis, as
> opposed to what RFC 1122 section 4.2.3.4 says?

Greg and I have exchanged a couple of messages off-line, and in the
course of that, it became clear to on me that "applying Nagle on a
per-send basis" has some ambiguity that needs to be cleared up.

This message is a bit longer that I'd like, but giving examples
is the clearest way for me to make my points.

Assume 2 4K and 1 500 byte writes, over ethernet.  What I implemented
in BSD/OS will do (assuming no window limitations...):

	write(4096)
	   so_send()
	      tcp_output(2048, moretosend=1)
		send(1460)
		<defer trailing 588, more to send>
	      tcp_output(2048, moretosend=0)
		send(1460 (588+872))
		<no more to send, no outstanding unacked data from
		 before this write, send the trailing piece>
		send(1176)

	write(4096)
	   so_send()
	      tcp_output(2048, moretosend=1)
		send(1460)
		<defer trailing 588, more to send>
	      tcp_output(2048, moretosend=0)
		send(1460 (588+872))
		<no more to send, outstanding unacked data from
		 before this write, defer data>

	write(500)
	   so_send()
		tcp_output(500, moretosend=0)
		send(1176+284);

	recv(ACK 2920); pkts 1-2
	recv(ACK 7016); pkts 3-5
	<next ACK is delayed...>
	recv(ACK 8476); pkt 6
		<all outstanding data is acked, send last piece>
		send(216)

So, in this instance, Nagle is applied to the write as a whole,
allowing the trailing piece of the initial write to be sent out
immediatly.  This was the goal, so that a single write of a request
that exceeded the MTU would be sent out immediatly, rather than
having the last part delayed waiting for an ACK.  After that, as
long as there is outstanding unacked data, the trailing tiny piece
is not sent out until all outstanding data is acked.  (In our test
case, it was a request of ~2K that now gets sent out immediatly in
two packets)

Now, taking it one step farther, another way to apply Nagle to the
entire write() could be:

	write(4096)
	   so_send()
	      tcp_output(2048)
		send(1460)
		<defer trailing 588, more to send>
	      tcp_output(2048)
		send(1460 (588+872))
		<trailing piece of the initial "big" write, send it>
		send(1176)

	write(4096)
	   so_send()
	      tcp_output(2048)
		send(1460)
		<defer trailing 588, more to send>
	      tcp_output(2048)
		send(1460 (588+872))
		<trailing piece of a "big" write, no outstanding
		 unacked data from a "small"write, send it>
		send(1176)

	write(500)
	   so_send()
		tcp_output(500)
		<A "small" write, and no outstanding unacked data from
		 a previous "small" write, so send it.
		send(500);

In a sense, just a single write > MTU can get all its big packets
and one little packet sent out,  applying the same logic to a
sequence of writes would allow all the big writes followed by
one little write to get sent out immediatly.

Of course, the problem with this (which is also what you can get
if you set TCP_NODELAY) is that you don't get a stream of full
sized packets followed by one small packet,
	send(1460); send(1460); send(1176);
	send(1460); send(1460); send(1176);
	send(500);

As opposed to the former which gives you:
	send(1460); send(1460); send(1176);
	send(1460); send(1460);
	send(1460);
	send(216);

And in reality, with slow-start, the BSD/OS code actually generates
(with sufficently large RTT):
	write(4096)
	   so_send()
	      tcp_output(2048, moretosend=1)
		send(1460)
		<defer trailing 588, more to send>
	      tcp_output(2048, moretosend=0)
		send(1460 (588+872))
		<no more to send, no outstanding unacked data from
		 before this write, but CW=2*1460; defer 1176>

	write(4096)
	   so_send()
	      tcp_output(2048, moretosend=1)
		<congestion window full, no send>
	      tcp_output(2048, moretosend=0)
		<congestion window full, no send>

	write(500)
	   so_send()
		tcp_output(500, moretosend=0)
		<congestion window full, no send>

	recv(ACK 2920); pkts 1-2
		<CW=3*1460, all data acked>
		send(1460); send(1460); send(1460);
	recv(ACK 5840); pkts 3-4
		<due to Nagle, defer final 1392 bytes>
	<delay for next ACK>
	recv(ACK 7300);
		<all outstanding data ACKed, send final piece>
		send(1392);

			-David Borman, [email-masked]

---

From: Greg Minshall <[email-masked]>
Date: Tue, 02 Mar 1999 19:09:52 -0800

All,

In a message yesterday, David Borman pointed out that "applying Nagle on a 
per-send basis" is somewhat ambiguous.  I'd love to hear how various people 
interpret that statement (in terms, i guess, of how their own stack implements 
this).

I think the question of whether the proposed modification proposes anything 
new, or simply codifies existing practice, is probably wrapped up in this 
issue.

Thanks,  Greg

---

From: Rick Jones <[email-masked]>
Date: Wed, 03 Mar 1999 11:12:28 -0800

> interpret that statement (in terms, i guess, of how their own stack implements

How does this interpretation sound:

If an application makes an API call providing for transmission a
quantity of data greater than or equal to the MSS of the associated TCP
connection, all data provided to the API call is immediately transmitted
by TCP, subject to the constraints of the receiver and congestion
windows.

This is a slightly formalized version of "If an application calls send()
with at least an MSS's worth of data, its transmission will not be
delayed by the Nagle algorithm."

rick jones
-- 
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to email, or post, but please do not do both...
my email address is raj in the cup.hp.com domain...

---

From: Bill Fenner <[email-masked]>
Date: Wed, 3 Mar 1999 13:53:55 PST

I think a lot of the question of what applying Nagle on a per-send
basis means includes what happens when data is delayed due to cwnd
or the receive window.  My imagination says that you keep another
variable in the TCB that's the highest sequence number that has
passed Nagle (in order to not delay the last packet of a large
write), but I haven't particularly thought through exactly how this
variable is updated or used except in the obvious case.  In
particular, I don't know how it should be updated when you perform
a send() and there is still data outstanding.

  Bill