Subject: Re: internet draft on suggested mod to the Nagle algorithm

From: Joe Touch <[email-masked]>
Date: Wed, 3 Feb 1999 14:48:35 -0800 (PST)

> To: [email-masked]
> Subject: internet draft on suggested mod to the Nagle algorithm
> Date: Mon, 01 Feb 1999 18:07:11 -0800
> From: Greg Minshall <[email-masked]>
> 
> 	Title		: A Suggested Modification to Nagle's Algorithm
> 	Author(s)	: G. Minshall
> 	Filename	: draft-minshall-nagle-00.txt
> 	Pages		: 4
> 	Date		: 29-Jan-99
> 	
>    The Nagle algorithm is one of the primary mechanisms which protects
>    the internet from poorly designed and/or implemented applications.
>    However, for a certain class of applications (notably,
>    request-response protocols) the Nagle algorithm interacts poorly
>    with delayed acknowledgements to give these applications poorer
>    performance.
> 
>    This draft is NOT suggesting that these applications should disable
>    the Nagle algorithm.

Why not?

Nagle was a solution to char-at-a-time remote logins, and
is discouraged for transactional systems, even ones with
bursts as small as a few characters, e.g., X11.

See:

	John Heidemann. Performance Interactions Between P-HTTP 
	and TCP Implementations. ACM Computer Communication Review, 
	27 2, 65-73, April, 1997.

	http://www.isi.edu/~johnh/PAPERS/Heidemann97a.html

Joe

---

From: David Borman <[email-masked]>
Date: Wed, 3 Feb 1999 17:22:41 -0600 (CST)

Eric,

> One related question: Does everybody support applications disabling
> Nagle on a per-circuit basis, as mandated by 4.2.3.4 in RFC 1122?

Most BSD systems support this via:

	int on = 1;
	setsockopt(s, IPPROTO_TCP, TCP_NODELAY, &on, sizeof(on));

			-David Borman, [email-masked]

---

From: Vernon Schryver <[email-masked]>
Date: Wed, 3 Feb 1999 16:41:39 -0700 (MST)

> IRIX does.

] Linux does.

Let's not have a brust of 10,000 messages listing implementations
that do support TCP_NODELAY.  Instead, let's wait for the few, if
any, announcements of those that don't.

You would expect that the TCP_NODELAY setsockopt() would work on most BSD
derived TCP stacks, including IRIX, BSD/OS, HP-UX, Ultrix and the other
(OSF?) DEC system, System V Release 4 and beyond, AIX, and the zillions
of others I can't think of in 3 seconds.  TCP_NODELAY also seems to work
in some other popular, not exactly BSD-derived systems, notably at least
some "winsock" libraries for WIN32 platforms, including WIN95, WIN98, and
NT 4.0.  (I've rather limited confidence that they do everything right,
but at least those Microsoft system do not complain when my code makes
the TCP_NODELAY request.)

I bet there are more systems that do not support the Nagle algorithm at
all than those that support the Nagle algorithm but cannot turn it off
(except, of course, systems that cannot be changed to want to turn it off,
such as embedded code burned into ROM).

Is TCP_NODELAY somewhere in POSIX?  I bet so.  
And also DLPI?  Ditto.

Vernon Schryver    [email-masked]

---

From: Bernie Volz <[email-masked]>
Date: Wed, 3 Feb 1999 18:48:10 -0400

Folks:

Perhaps this is a dumb question and has been discussed before, but is
there any value in having TCP know a bit about what the application is
doing? This might be considered a protocol layering violation. Has
anyone implemented this kind of scheme?

I'm thinking one could have logic that says "delay ACK unless all data
delivered to application". I don't mean moved into the queue, but a read
was actually waiting and all data delivered or on its way to the
application. Of course, you would ONLY not delay if you received a small
segment (in the case where you believe the sender may be doing the Nagle
algorithm). One could also consider including an explicit "full" window
update with this immediate ACK.

A small change to this logic might be to only avoid delaying the ACK
if the application can accept more data (ie, all has been delivered and
there is still a read outstanding) - you know the application wants more
data!

One could also implement this that when a read is queued by an
application, if a delayed ACK is pending on the connection (and the last
segment was small), send the ACK now.

This of course is wasteful if your round trip time is long (since the
delay ACK time is insignificant in that case). So, perhaps this would
only be turned on if the round-trip estimate is "short".

However, the cost is also small as if all data was read, the application
must be ready for the data and why not expidite delivery of more. Also,
for a fast network, it is unlikely that an application could keep up
with network speeds for long and hence this wouldn't trigger that often.

- Bernie Volz
  Process Software

---

From: Art Shelest <[email-masked]>
Date: Wed, 3 Feb 1999 16:07:28 -0800

The distinction between "full" and "small" segments is not certain
whenever PMTU discovery is used. There may be other ways to detect
that specific segment is at the end of user's buffer and accelerate 
an ACK.

-----Original Message-----
<snip>

<snip>

- Bernie Volz
  Process Software

---

From: Vern Paxson <[email-masked]>
Date: Wed, 03 Feb 1999 19:51:30 PST

An observation: some of the comments on this thread make me wonder if some
contributors are speculating about the I-D before having read it.  If you
haven't read it, please do, it's short and a very simple change.

		Vern

---

From: der Mouse <[email-masked]>
Date: Wed, 3 Feb 1999 23:12:34 -0500 (EST)

>> [foo] does.
>> [bar] does.

I suspect this won't work very well.  I feel quite sure that having a
representative on tcp-impl correlates positively, and probably fairly
strongly, with having a relatively decent implementation.  Thus, the
people who could truthfully chime in as you request - even assuming
they'd be willing to publicly admit their failing - are the people who
probably won't see the question.

The only way I expect to see "<foo> doesn't do this right" here is from
someone who wrote code that tried to disable Nagle (eg, TCP_NODELAY)
and then discovered in a packet trace that it was happening anyway.

					der Mouse

			       [email-masked]
		     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B

---

From: Jonathan Webb <[email-masked]>
Date: Thu, 4 Feb 1999 11:27:01 -0000

SCO UnixWare7 and OSR5 do.

- Jonathan

---

From: Vernon Schryver <[email-masked]>
Date: Thu, 4 Feb 1999 11:06:19 -0700 (MST)

> From: David Borman <[email-masked]>

> ...
> I don't like SO_EXPLICITPUSH, because that means that the kernel
> is trusting the application to remember to tell it when it is
> done sending the data.  If the application forgets, then it could
> adversly affect the performance below what would have happened
> had SO_EXPLICITPUSH not been set.

Good point.

>                                    I'd rather have the kernel
> continue to be as aggressive as normal in sending out data, but
> take the MSG_PUSH bit as a suggestion that it should be more
> aggessive and transmit any delayed data, since no more data will
> be written at this time.

MSG_PUSH sounds interesting, but

  - you would definitely need to pick some other name, since otherwise
     you would have hordes of experts up in arms when they saw the PUSH
     bit in the TCP header set or not set in concert with the MSG_PUSH
     flag in the system call.  Recall the continuing popularity of their
     complaints and questions about the PUSH bit as sent by the BSD code.
     They would think that something in the API named *PUSH would be a
     direct control on the TCP header bit, and never mind how much sense
     that makes.

  - I bet that most applications that now need to turn set TCP_NODELAY
     would still need to use TCP_NODELAY.  In other words, what could you
     do with MSG_PUSH that you could not do more efficiently (i.e. with
     fewer CPU cycles) with writev()?

---

From: [email-masked]
Date: Thu, 4 Feb 1999 10:29:59 -0800

*> 
  *> As for sending multiple requests in a row, I think the additional
  *> idea about SO_EXPLICITPUSH could be used to fix the delay of the
  *> final packet.  I've bounced this idea around before, but never
  *> followed through on it.  My suggestion is to just add a MSG_PUSH
  *> flag that would be used on the last send to indicate that the
  *> application doesn't intend to send anymore data at this time.
  *> 
  *> I don't like SO_EXPLICITPUSH, because that means that the kernel
  *> is trusting the application to remember to tell it when it is
  *> done sending the data.  If the application forgets, then it could
  *> adversly affect the performance below what would have happened
  *> had SO_EXPLICITPUSH not been set.  I'd rather have the kernel
  *> continue to be as aggressive as normal in sending out data, but
  *> take the MSG_PUSH bit as a suggestion that it should be more
  *> aggessive and transmit any delayed data, since no more data will
  *> be written at this time.
  *> 
  *> 			-David Borman, [email-masked]
  *> 

Dave,

Yes. Your definition of MSG_PUSH seems to exactly match the PUSH bit in
the TCP spec (FINALLY!!!!!!!).

Bob

---

From: David Borman <[email-masked]>
Date: Thu, 4 Feb 1999 13:59:21 -0600 (CST)

> Date: Thu, 4 Feb 1999 11:06:19 -0700 (MST)

The name doesn't matter, e.g. MSG_NDELAY would work as well.

>      fewer CPU cycles) with writev()?

First, the combination of modifying Nagle and adding MSG_PUSH
is what I would recommend, not just MSG_PUSH.  The combination
of the two would eliminate most of the need for TCP_NODELAY.
Secondly, yes, sending the transaction in a single writev() is
by far the prefered method.

The usual need for using TCP_NODELAY is to get around Nagle, due
to it interacting poorly with delayed acks.

1) With the modification to Nagle, then for a single request sent
   in a single writev(), no matter how large the request, there would
   be no need for TCP_NODELAY.

2) If a request is sent via multiple writes, then using MSG_PUSH
   on the last write will eliminate the need for TCP_NODELAY.

3) If an application is sending multiple requests with a separate
   write for each request, with or without the Nagle change, the
   trailing data will get delayed.  Using MSG_PUSH on the last write
   will eliminate that.

The point is that rather than bypassing Nagle in all instances,
using MSG_PUSH you can bypass Nagle at the specific points where
you need to.

---

From: Vernon Schryver <[email-masked]>
Date: Thu, 4 Feb 1999 13:21:33 -0700 (MST)

] From: [email-masked]

] Yes. Your definition of MSG_PUSH seems to exactly match the PUSH bit in
] the TCP spec (FINALLY!!!!!!!).

As I tried to say earlier, it's almost 20 years too late for such
considerations.  The PUSH bit irrevocably means what it now means,
as useless as that might be.  Or in other words, you can't starting
tommorrow saying that a write without the MSG_PUSH option will be
arbitrarily delayed.  It might have been really swell to have the
MSG_PUSH bit in ~1981 when sockets were defined, but since none of
us are of the Wishful Thinking School of Engineering, ...

And one writev() is vastly better than two writes, no matter how many
bits you add to the API.   You could consider the BSD definition of
the PUSH bit as a stupid tax.  People who use two write()'s when one
is possible are likely to to have designs and code that won't notice
the additional slow down from Nagle+delayed Ack.  (From my experience,
that's emperically true.)

 ....................

> of the two would eliminate most of the need for TCP_NODELAY.

Maybe so, but I've doubts.   My current application would still need
TCP_NODELAY or would set MSG_NDELAY on every write, which I think amount
the same thing on the wire.

> by far the prefered method.

I agree that the MSG_NDELAY bit would be a nice supplement for writev(),
when where it is hard to get all of the buffers together for a writev().
It would also be a better way than TCP_NODELAY to patch code from the many
people that don't realize why write-write is a bad idea.   TCP_NODELAY is
a big hammer unless you really know there are no write-write sequences in
which the first write should be delayed.

> using MSG_PUSH you can bypass Nagle at the specific points where

I really like the Nagle algorithm, but how many applications really need
it today?  Among carefully written applications (i.e. use writev or at
least never more than 1 write/request) that might use MSG_NDELAY, how
many would not get exactly the same results by TCP_NODELAY?

The biggest trouble with MSG_NDELAY is the same as with writev().
You'd have it where you don't really needed it (e.g. BSD systems)
but not where you do need it (e.g. winsock).

---

From: David Borman <[email-masked]>
Date: Thu, 4 Feb 1999 17:00:01 -0600 (CST)

> Date: Thu, 4 Feb 1999 13:21:33 -0700 (MST)
> arbitrarily delayed.

I think there is a mis-understanding here.  Setting MSG_PUSH means
push this data through (and set the PUSH flag when it gets to TCP).
Lack of the MSG_PUSH does not mean that the data is not sent, nor
that the PUSH flag will not be set.  At least that's my thinking.
But this may just be support for your agument to call it something
else than MSG_PUSH.

> the same thing on the wire.

Well, I've tossed around the idea of MSG_PUSH off and on for many years,
and the fact that I haven't done anything about it sort of says that I
haven't even convinced myself that it is really needed.

I do think the change to Nagle is good (supported by the fact that
I implemented it in BSD/OS over 2 years ago).

> which the first write should be delayed.

The question is are there enough real-world situations where this
would be worth-while.

> many would not get exactly the same results by TCP_NODELAY?

A other question is, if you have the modification to Nagle, how
many of those carefully written applications that use TCP_NODELAY
could drop use of it?  Or would they still need TCP_NODELAY/MSG_NDELAY
because they send multiple requests without waiting for a response
to the first request?

> but not where you do need it (e.g. winsock).

Patient: Doctor, it hurts when I do this.
Doctor: Then don't do that.
:-)

---

From: Vernon Schryver <[email-masked]>
Date: Thu, 4 Feb 1999 16:30:54 -0700 (MST)

> But this may just be support for your agument to call it something

I think we agree.

> ...
> > Maybe so, but I've doubts. ...

> haven't even convinced myself that it is really needed.

I had thoughts along those lines, but forgot to be offensive by mentioning
them.

> I implemented it in BSD/OS over 2 years ago).

Yes.
Although I'm not convinced that BSD/OS sees really high traffic or as
many installations as other systems.  Yes, BSD/OS is used on lots of
web servers (and others such as this box), but I think the really big
servers use boxes bigger than you can get with 80*86 CPU's.  The HP
report is the overwhelming endorsement.

> The question is are there enough real-world situations where this

Attempts to change to ancient, extremely popular API's are awfully painful
for eveyrone, and often unsuccessful in various ways.  Winsock 2.0 is an
example of how more or less necessary changes can get out of hand.  (I've
found the complications around select, non-blocking I/O, and WIN32 messages
in winsock 2.0 ... engrossing.)

> because they send multiple requests without waiting for a response

Maybe I don't understand the proposed modification, but I have the
impression that the answer to that question is "most of them."  It
seems the modification helps only two sets of cases:

    1. bad code that does write-write-read-write-write-read-...
    2. odd cases such as write(>1 segment but not a MBbyte)

If you have an application that involves lots of small random writes mixed
with random reads, then TCP_NODELAY/MSG_NDELAY is unavoidable.  Perhaps
X clients, as someone said.  Certain the code I'm currently bashing.

> Doctor: Then don't do that.

"If wishes were horses, then beggars would ride"

---

From: David Borman <[email-masked]>
Date: Fri, 5 Feb 1999 14:09:25 -0600 (CST)

Rick,

> PS where can I find the original "chapter and verse" on Nagle?

You can find it in:
	"Congestion Control in IP/TCP," J. Nagle, RFC-896, January 1984.
and section 4.2.3.4 of RFC-1122.

---

From: Vernon Schryver <[email-masked]>
Date: Fri, 5 Feb 1999 15:32:20 -0700 (MST)

> From: Art Shelest <[email-masked]>

> > How would you implement that?  
>
> It's a valid question, however I'd suggest that benefit of the
> algorithm is discussed before perceived implementation complexity.

In general, I strongly disagree.  Implementation costs are never merely
perceived.  Good ideas are a subset of those that have reasonable
implementations.  Without a good implementation, talk aobut the
wonderfulness of any idea is no better than useless usenet jabber, of
which there is no shortage.  Besides, the name of this mailing list
is relevant.  That said, all that's necessary to make it reasonable to
discuss an idea is a plausible hint that it can be reasonably implemented.

> If the recommended value for N is 2, the implementation is trivial,
> consumes extra 2-4 bytes* per TCB and the overhead is 1-2 extra
> instructions per send.
>
> *One can store beginning sequence number of the last packet (4 bytes),
> or the size of the last segment (2 bytes). If storing last segment
> size, then Nagle check would change:
> N=1 (today):	SND.NXT > SND.UNA
> N=2:			SND.NXT -SND.SIZ > SND.UNA 
> where SND.SIZ is last segment size.

"Configurable" implies that sometimes N=1 and other times N=2.  That means
that the time costs are at least the following additional instructions,
  1   fetch of the switch
  2   test of the switch
  3   branch on the switch's value
  4   fetch SND.SIZ
  5   subtract 

Depending on the CPU, you might combine #1, #2, and #3 into 2 instructions,
such as a test and a branch, or a fetch and a branch-zero.  The largest
costs are probably not the instructions but the primary cache misses and
dirtying.  The secondary tertiary cache misses probably don't matter since
the entire TCB is likely to be faulted in and dirtied.  Overall, that
sounds fairly cheap to me.

> Back to the real issue: would allowing N=2 unacknowledged segments 
> resolve effects of Nagle/Delayed ACK interaction?

How is this idea better than the idea of the draft?  

This idea would effectively turn off Nagle for the most common (in my
experience) bad code, which loves write-write-read-write-write-read....
It would double the number of packets on the wire for such bad
applications.  I think that would be Not Good.  Would relaxing Nagle for
2 send requests be much different from turning it off entirely?

Do we agree that I misread the following or it was not tightly phrased?
I read it as a proposal for letting the application vary the value of N
from 1 to some large number.

> > Why not parametrize the Nagle?
> >
> > Instead of "allow only one unacknowledged segment"
> > allow "allow N unacknowledged segments/bytes" and 
> > specify recommended range of values.
> >
> > The N=2 would be similar to current proposal and also
> > fix the case where application follows the 
> > "small write, small write, read" operation, 
> > which is not addressed by the proposed draft.
> >
> > The N=2 seems like a good choice, because receivers
> > typically ACK every other packet.

---

From: Vernon Schryver <[email-masked]>
Date: Fri, 5 Feb 1999 16:34:06 -0700 (MST)

> In the case above, present or draft-proposed Nagle will both generate 2
> packets because first will be sent immediately. The performance, however,
> will improve from 2 packets per 200ms to 2 packets per RTT (10+ times on
> many LANs)

True, Nagle does not reduce the number of packets per transaction from
bad applications, but it reduces or at least strongly limits the number
of packets per second from them.  As far as the rest of the network
is concerned, the effect of Nagle is to reduce the number of packets.

> I don't agree with notion that W-W-R applications come from bad programmers,
> application programmer should not be required a TCP expert when they simply
> treat
> socket as a file handle. For all they know, it may or may not be a network
> device.
> If W-W-R works well with fwrite, why shouldn't it work with send?

NO!

  - W-W-R does NOT work well with fwrite(), compared to a single write().

  - when confronted with W-W-R, fwrite() will make as single write() 
      system call, in the relevant, tiny-write cases.  That's why
      fwrite() and the rest of stdio were invented.

  - mixing stdio writes and reads without any intervening fseek()s is not
     for the faint of heart.  Are such combinations now legal, eg. in POSIX?

  - any programmer who doesn't design for the target media is incompetant.
     Competant programmers do what is necessary to make their code work
     well on all of their design-targeted media.  What works well on
     network streams often does not work on other media, and vice versa.
     No amount of CASE/OOP snake oil will make rewind work on a TCP stream,
     nor does select() generally make much sense on disk files or tapes.

> Large sends is a more complex case. Both Nagle(2) and modified Nagle will
> create 
> extra packets with sends slightly bigger than MSS, which is bad for bulk
> transfers, 
> but good for interactive operations.

Not so.  Extra packets are bad for interactive operations.  Extra packets
cost both time and CPU cycles.  It is also true that longer delays while
the wire is idle are worse for interactive applications.  More over, large
transmissions are generally not involved in interactive applications, for
obvious reasons.

> It almost sounds like Nagle needs it's own timer rather than being driven by
> receiver's delayed ACK timer or RTT. This way, Nagle(T) could read:
> "Delay sending data until there's full MSS, but no longer than period T".
> This will guarantee predictable Nagle delay, will not inhibit bulk
> transfers,
> improve interactive performance and not depend on the receiver's delayed ACK
> policy.
>
> Now we have 4 possible solutions: Nagle, modified Nagle, Nagle(N) and
> Nagle(T)
> it would be interesting to compare their behavior for the cases previously
> described, from telnet to bulk transfer to MSS+1 sends.

Adding yet another timer to TCP is an non-starter, since it fails the
reasonable-implementation criterion.  The several timers that are now
needed for every TCP stream are already a major burden in TCP
implementations that care about speed, especially speed when dealing with
lots of simultaneous streams, as on a large web server.  The gyrations
required to make a large, fast multi-processor deal with the current
zillions of TCP timers are painful to contemplate.

---

From: David Borman <[email-masked]>
Date: Mon, 8 Feb 1999 10:02:54 -0600 (CST)

> From [email-masked] Mon Feb  8 01:48:27 1999
> Date: Fri, 05 Feb 1999 10:08:03 -0800
> From: "Eric A. Hall" <[email-masked]>
> Organization: EHS Company
> X-Accept-Language: en
> To: David Borman <[email-masked]>
> CC: [email-masked]
> > Also, I might not have said it, but this example was to show
> > 3 separate transaction requests, not 1 transaction done with
> > 3 writes.  If it was one request, it should really be done with
> > a single 4500 byte write, yeilding:
>
> You're not showing any response activity so it's hard to guess what's
> really going on.

Ok.  My example was for an application that is request/response
oriented, but it does not necessarily wait for the preceeding
response before sending the next request.  The point being that
the first two requestes will get sent out, but there will be a
delay in the 3rd request.

With the current Nagle code:
	1st request:	send 1460 bytes  (defer 40 bytes)
			(pause for some short period of time)
	2nd request:	send 40 + 1420 = 1460 bytes (defer 80 bytes)
			(pause for some arbitrary period of time)
	3rd request:	send 80 + 1380 = 1460 bytes  (defer 120 bytes)
			(wait for ACK of 4380 bytes or timeout)
			send 120 bytes

With the new Nagle code:

	1st write:      send 1460 bytes
			send 40 bytes
			(pause for some short period of time)
	2nd write:      send 1460 bytes (defer 40 bytes)
			(pause for some short period of time)
	3rd write       send 40 + 1420 = 1460 bytes (defer 80  bytes)
			(wait for ACK of first 1500 bytes or timeout)
			send 80 bytes

The difference is that with the new Nalge code, the whole initial
request gets sent immediatly, whereas with the old code it is only
the addition of the 2nd request that actually pushes out the rest
of the 1st request.

Additionally, note that I've clarified that there is a difference in
when the final piece gets sent, relative to when the ACKs arrive.
With the old Nagle code, it will be waiting for an ACK of all the
outstanding data before sending the final 120 bytes.  With the new
Nagle code, once the first 40 byte packet has been acked, the trailing
80 byte packet can go out because there is no longer any previous
unacked small packets.

Also, you said, wrt to this example and the New Nagle code:
> Five sends. Better.
And the old Nagle:
> Four sends. Best.

Not necessarily.  Though it is usually best to send as few packets
as possible, strict packet count is not the only criteria.  In this
situation we are also looking at how quickly the whole requests get
sent out, and what artificial delays are introduced as an artifact
of interaction between the Nagle algorithm and delayed acks.

---

From: Vernon Schryver <[email-masked]>
Date: Mon, 8 Feb 1999 19:34:40 -0700 (MST)

>                                              ... if i'm getting better at 
> explaining it! ...

The best possible explanation should be put into anything that is to
eventually become an RFC.

In other words, should the proposal be advanced, the text needs to be
sharpened.   For example, it would be quite wrong to leave issues such as
whether the algorithm is to be applied to application requests or the
output of the segmenter to the archives of this mailing list, since it
seems the results are on the wire differ.  A standard must support a

---

From: Joe Touch <[email-masked]>
Date: Thu, 11 Feb 1999 13:38:10 -0800 (PST)

> Date: Thu, 11 Feb 1999 16:29:19 -0500
> To: Joe Touch <[email-masked]>, [email-masked], [email-masked]
> From: Henrik Frystyk Nielsen <[email-masked]>
...
> Instead, added control at the application layer would be of tremendous help
> as the app is likely to know what it going on. In other words, I like the
> SO_EXPLICITPUSH flag idea a lot and think that it would solve at least our
> needs.
> 
> Henrik

I agree with this. It isn't clear how the current efforts
to generalize Nagle will help this; if anything, they 
potentially lull the application programmer into assuming
TCP will "do the right thing".

The right thing, in this case, is complicated semantically,
and hard to infer simply from send calls or message sizes.
The explicit signalling seems like a much more useful solution.

Joe

---

From: David Borman <[email-masked]>
Date: Mon, 15 Feb 1999 14:21:43 -0600 (CST)

I just remembered one very important item.  The whole issue of
delayed acks vs Nagle, and applying Nagle to the whole send()
doesn't help at all if you have an intial congestion window of
only 1*MSS, and the request is >1*MSS.  Case in point, an initial
2K request sends 1440 bytes and defers the rest not due to Nagle,
but due to the congestion window.  The other side only gets 1 packet,
so it delays the ack.  And you have the same problem.

The fix is to have at least a 2*MSS initial window, so you need
to implement RFC 2414 "Increasing TCP's Initial Window".

---

From: Vern Paxson <[email-masked]>
Date: Tue, 16 Feb 1999 01:34:52 PST

> to implement RFC 2414 "Increasing TCP's Initial Window".

Or 2001.bis, which should pop out as an RFC quite soon - it also allows an
initial window of two segments.

		Vern

---

From: Barney Wolff <[email-masked]>
Date: Wed, 17 Feb 1999 17:03 EST

At the risk of looking silly, don't I remember that setting NODELAY
has to be done before the socket is connected?  That means that a
server has to make its choice before it knows who's asking and what
the request is, unless it can listen on different ports.  To me,
that's the advantage of the flush bit.  Same problem with setting
SND/RCVBUF.

Barney Wolff  <[email-masked]>

> Date: Wed, 17 Feb 1999 14:03:41 -0700 (MST)
> Second, for an application I'm hacking, I've been trying to see how I
> would use a flag or the proposed change or something else.  As far as I
> can tell, nothing does better than simply turning off Nagle, and I bet
> the same applies to HTTP.  It's not that you could not use a flush bit,
> but that the packet traces and everything else would be essentially the
> same as with Nagle off.  So why bother writing code that only works on
> those platforms with the flush bit implemented?  Why not forget the
> flush bit even when available and just turn off Nagle?

---

From: Vernon Schryver <[email-masked]>
Date: Wed, 17 Feb 1999 17:27:42 -0700 (MST)

> ...
> Yes, I too do application level buffering with all the problems of figuring
> out what a good buffer size is (I ended up using 1K as a compromise). In
> most cases this mean I don't need Nagle at all.
>
> But, from an architectural viewpoint, I think it makes a lot more sense if
> I didn't have to do that at the application layer and that I instead could
> keep the services separated. This means that I need enough control to be
> able to indicate what I want - an explicit flush is one of those hooks.
> Today, I don't have that so I have to repeat a lot of the functionality in
> a layer above and short-circuit things like Nagle in lower layers.

That's a fair argument, but I don't see how it actually applies here.

Case 0: no buffering, Nagle off, use flush bit
Case 1: no buffering, Nagle off, don't use flush bit
   same packets on the wire in either case...far too many!

Case 2: you don't buffer, leave Nagle on, and use the flush bit.  
    Your writes of slow data generally go out immediately, because you
    send them when things are idle.  When you have fast data, you use the
    flush bit, so they also go out immediately instead of possibly being
    delayed for an Ack of slow data.  The result on the wire is too many
    packets, almost as bad as #0 and #1.  The only improvement in packet
    counts over #0 and #1 is when you happen to send slow data within
    100 ms after receiving fast data or send 2 slow bursts within 100 ms.

Case 3:  you buffer, leave Nagle on, and use the flush bit
Case 4:  you buffer, turn Nagle off, and do not use the flush bit.
    Both cases have the similar application code complexity and the
    same, minimal count of packets on the wire.

Case 5-7:  boring.

Do you agree that #0 #1 are bad, and so you pick either #3 or #4?

Given that #4 works everywhere today and forever, why would you ever bother
to code #3?  For the architectual purity seal of approval?

What's the difference between a flush bit in the send() system call, and
explicit setsockopt()'s turning off TCP_NODELAY off before a send() and
on afterwards?  No, I don't like system call overhead--I'm asking what
happens on the wire.

---

From: Joe Touch <[email-masked]>
Date: Wed, 17 Feb 1999 17:16:30 -0800 (PST)

> Date: Wed, 17 Feb 1999 17:27:42 -0700 (MST)
...
> on afterwards?  No, I don't like system call overhead--I'm asking what

I don't think there is an answer to this. The problem is that
there isn't really a required correlation between send() calls
and segments, regardless of NODELAY (or anything else I can find,
excepting _only_ the proposed flush bit, which might be defined as
guaranteeing that the last byte of the write is the last byte
of a segment).

e.g, Nagle waits for a timeout or enough data for a full-sized
segment. However, there is no requirement that this data is sent 
in a single segment, per se.

Granted, this may seem broken, but given it holds, it may be a bigger
hole in the spec than can be fixed by tweaking Nagle... 

Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 17 Feb 1999 17:35:10 -0800 (PST)

> Date: Wed, 17 Feb 1999 19:01:40 -0500
> To: Vernon Schryver <[email-masked]>, [email-masked]
...
> a layer above and short-circuit things like Nagle in lower layers.

TCP currently lacks a few things that would certainly be useful
to applications that want something other than a byte-stream, 
including (but not limited to):

	in-band EOF
		separate "close" and "disconnect" system calls, e.g.

	flush
		implementing the intended semantics of the PUSH bit,
		i.e., an application signal to avoid intermediate
		buffering and emit whatever data the lower layers have
		ASAP

	record mark
		a way to indicate the last byte of a segment, i.e.,
		to force a (minor) correlation between segment and app 
		message boundaries

		note, this is different from FLUSH, since the system
		could be emitting segments as fast as possible while
		a subsequent send() could add more data

	aggregate-for-me
		a way to ask the system to aggregate packet emissions
		if they're not "MSS", by stalling writes when there
		is less than 1 MSS to send.

		this isn't Nagle, which stalls sending if there
		is any outstanding unack'd segment

		this isn't Minshall's recent proposal, which stalls
		sending if there is any outstanding unack'd PARTIAL
		segment (even if there is currently a full segment to 
		be sent).

		i.e., use a flag to indicate "SENT_TINY";
			on emit,
				SENT_TINY = "segment < MSS"
			on attempted non-timer emit:
				if "current segment < MSS" &&
				"SENT_TINY == 1",
					wait
			on timer-based emit:
				ignore SENT_TINY

The most useful of these, for message-based systems, require
changes to the semantics of the send() API, however...

Joe

---

From: der Mouse <[email-masked]>
Date: Wed, 17 Feb 1999 21:23:07 -0500 (EST)

> TCP currently lacks a few things that would certainly be useful to
> applications that want something other than a byte-stream, [...]

Um, yeah.  I'm not quite sure what your point is.  This sounds like
"TCP isn't all things to all applications", which doesn't really sound
to me like something that is possible to fix, nor do I think it would
be desirable to try.

> 		separate "close" and "disconnect" system calls, e.g.

Um, isn't that what the FIN bit is all about?  shutdown(2) in the
socket API, with second argument 1?  Or have I misunderstood?  (In this
vein, I really think TCP needs a way to implement shutdown(...,0), that
is, to push "no more receives" back to the sender in a way that doesn't
break data flow in the other direction.)

> 		implementing the intended semantics of the PUSH bit,

This sounds a lot like one of the things proposed in this Nagle thread.
I think it would be a good thing to have, but it's an API issue, not
really a TCP issue, as far as I can see.

					der Mouse

---

From: der Mouse <[email-masked]>
Date: Wed, 17 Feb 1999 21:33:27 -0500 (EST)

>> What's the difference [on the wire] between a flush bit in the
>> send() system call, and explicit setsockopt()'s turning off
>> TCP_NODELAY off before a send() and on afterwards?

None that I can see.

> The problem is that there isn't really a required correlation between
> send() calls and segments, regardless of NODELAY (or anything else I
> can find, excepting _only_ the proposed flush bit, which might be
> defined as guaranteeing that the last byte of the write is the last
> byte of a segment).

That's not how I read it.  Suppose there is no send window available,
MSS 1500, 500 bytes buffered waiting for window, and then, without any
network traffic occurring during it, the application does
write-with-flush 10 bytes then ordinary write 10 bytes more.  I don't
think any of the flush-bit proposals would require the next segment to
hold 510 bytes rather than 520 (assuming that when the window opens, it
opens at least 520 bytes' worth).

					der Mouse

---

From: Joe Touch <[email-masked]>
Date: Wed, 17 Feb 1999 18:57:48 -0800 (PST)

> From [email-masked] Wed Feb 17 18:43:40 1999
> Date: Wed, 17 Feb 1999 21:23:07 -0500 (EST)
> From: der Mouse  <[email-masked]>

Restated, TCP lacks some features that appear to be consistent
with a general reliable byte-stream model, but have not yet been
designed in.

> Um, isn't that what the FIN bit is all about? 

What I mean is to be able to send "EOF" more than once in a stream.
You can for a file - i.e., you can read the EOF, then rewind, then
get an EOF again. But you can't for a stream. Once you send EOF, you
can't send any more data.

Joe

---

From: der Mouse <[email-masked]>
Date: Wed, 17 Feb 1999 23:44:32 -0500 (EST)

> What I mean is to be able to send "EOF" more than once in a stream.

I'm not sure this really makes sense.  Once you do this, it isn't a
byte-stream any longer; it's a (byte-or-EOF)-stream, and will have a
*real* EOF when there are no more (byte-or-EOF)s possible.  You can
then repeat the construction as many times as you like...but what's the
point of doing it at all?

> get an EOF again.  But you can't for a stream.

You can't rewind a byte-stream.  They also don't support a lot of other
things files do, like random access (of which rewinding is really just
a special case).  I don't see this as a problem; it is the nature of
byte-streams to be byte-streams.

If an application wants to send more than 256 different things over a
stream, I am inclined to let the application deal with encoding them.
Different applications will have different tradeoffs (such as how
frequent different values are and thus how to represent them)....

"If you want pseudo-terminals, you know where to find them."

					der Mouse

---

From: Vernon Schryver <[email-masked]>
Date: Wed, 17 Feb 1999 22:38:10 -0700 (MST)

> >> TCP_NODELAY off before a send() and on afterwards?

I couldn't think of one either, but the issue is slippery.

> > defined as guaranteeing that the last byte of the write is the last

That sides steps my intended question.  As with previous rounds in this
discussion of the Nagle algorithm, what we think is a correct exegesis of
the (or a) standard is basically irrelevant.  Does it matter whether RFC
896 talked about send() system calls, passages through tcp_output(), or
cycles of the RFC 793 state machine?  NO!--What really matters is what
happens in representative, currently deployed implementations.  First
figure out the problem, and after that choose an architecture, a layering,
and a religion.  That order is what made TCP, and the opposite is what
made TP0-TP4/CONS-CLNS.

] From: Joe Touch <[email-masked]>
]
] > > TCP currently lacks a few things that would certainly be useful to
] > > applications that want something other than a byte-stream, [...]
] > 
] > Um, yeah.  I'm not quite sure what your point is.  This sounds like
] > "TCP isn't all things to all applications", which doesn't really sound
] > to me like something that is possible to fix, nor do I think it would
] > be desirable to try.

I was horrified by some of the ideas.  EOF and record markers particularly
bother me because they are so trivially implemented by the application
and there are so many more kinds of markers than any of us would dream
of putting into a kernel (or so I would like to think).  You would need
at the very least all of ANSI magnetic tape markers, including "my summer
vacation" naratives optionally associated with every mark.

On the other hand, the notion of "delay sending these low priority bits
until I find some more or you really have nothing better to do" and
"send this glob" immediately sound very enticing.  I think they might
finally be an improvement on Nagle.

On the third hand, was in it this mailing list where Joe Touch and others
tonight mentioned Gbit/sec TCP for clusters?  Every minor optional feature
that you add to a protocol hurts enormously at the boundaries, on very
large and fast and on very small systems.  It wouldn't take more than a
smidgeon more improvements to transform the TCP/IP of 1985 that buried
TP0-TP4/CONS-CLNS into a monster that makes those poor guys look agile
and svelte.  The necessary but so far small and subtle changes including
header prediction, fast retransmission, fast recovery, slow start, double
initial MSS, window scaling and SAK, etc. are a long way down that road
paved with good intentions.  Start adding neat knobs to the API, and TCP
will instantly flash under the lintel with the damning motto.

---

From: Joe Touch <[email-masked]>
Date: Thu, 18 Feb 1999 10:16:42 -0800 (PST)

...
> and a religion.  That order is what made TCP, and the opposite is what

AOK - stated differently, current implementations do not guarantee
any correlation between send() calls and segments.  There are more
deployed implementations than just BSD. It makes more sense, as Sally Floyd
has said in other threads, to keep the requirements as open and flexible
as possible. We might be better less concerned with "what would 
OS X.Y.Z do with this mod" than "what's the best and worst a 'compliant' 
system could do, given these requiremetns".

> bother me because they are so trivially implemented by the application

Reliability is also trivially implemented at the app layer, but TCP/IP
is preferred to TCP/UDP/IP. Some things are more effectively, efficiently,
or consistently implemented in the transport layer. Granted, it's open
to debate what that list is; I was giving examples, not requirements.

> On the third hand, was in it this mailinglist where Joe Touch and others
> large and fast and on very small systems. 

Not always - some can be disabled (such as disabling windowing for
connections on the same subnet), others do not come into play unless
triggered (e.g., some of the markings proposed could be implemented
in options; there's only one check to see if there are "no options", and
it has to be performed anyway).

Joe

---

From: Vernon Schryver <[email-masked]>
Date: Thu, 18 Feb 1999 12:41:58 -0700 (MST)

> From: Joe Touch <[email-masked]>

> system could do, given these requiremetns".

So what are we supposed to do with the standards based non-answer?
Of course there is no guaranteed correlation between send() and segments,
and there could not be even with an explicit flush bit (eg. window space
& MSS).  Of course there are more than BSD implementations, and even among
BSD implementations there are differences.  We can nevertheless talk about
typical, representative, useful, and/or counter-productive behaviors.

Let me try to state a pedantic proposition that does not admit standards
exegesis equivocation:
  - in RFC 793, 896, & 1122 TCP conformant implementations which allow
     dynamic switching of the Nagle algorithm, turning off Nagle before
     a send() request and on afterwards MAY have identical results in
     segments on the wire as the proposed explicit flush bit.  Moreover,
     all existing implementations that allow dynamic control of the Nagle
     algorithm would have such identical results if the flush bit were
     added to their send() API's.

Now, is that proposition true or false?
I do not know the answer; if I did, I'd say so.  I do suspect it's true.

sheesh--will I have to write standards language to specify Dave Borman's
flush bit?  Or will we now have an argument about the definitions of truth,
models, satisfiability, and the rest of basic mathematical logic?
Rejecting a statement about the integers on the grounds that nonstandard
models of arithmetic also satisfy the Piano Axioms but falsify the
statement is fine, but this mailing list is named "tcp-impl".

> . ..
> is preferred to TCP/UDP/IP.

If you'd step aside from the heat of the rhetorical battle, I hope you'd
not like that statement.  Reliability requires timers and buffering for
retransmissions, but markers need require no more than a byte count at
the start of the file, record, or block, and never more than simple pattern
recognition (e.g. escaping) if you don't have the file, record or block
size a priori.  You must know that even ANSI tape labels are not in the
same league as any kind of reliability.  I know you've heard of slow start,
fast retransmission, and fast recovery.  The reasons those necessary
complications of reliability exist have nothing to do with convenient
abstractions like "layers."  You must recall and surely agree with the
contempuous remarks in this mailing list about the persistent proposals
to replace TCP/IP with simplistic retransmitting of UDP/IP.

>                             Some things are more effectively, efficiently,

hmmm...I wrote "in the application" while you wrote "at the app layer."
I've noticed people with .edu addresses on the Southern Calif. coast tend
to have strong views on the sanctity of Architecture and Network Layers.
I view both as mere tools, as handy ways to talk, or more commonly not
talk, about things.  Suppressing momentarily irrelevant details is necessary
for effective thinking, but those curtains to hide details have no virtues
of their own.  The statement that something is more effective in a layer
makes little sense to me.  Network layers and architecture do not exist
independent of real, concrete systems.  There is no such thing as the
transport layer except when you are talking about implementations or
writing an academic (pejorative sense) paper.  Computer architecture
independent of implementations makes as little sense as theoretical civil
architecture.

> it has to be performed anyway).

NO!  ALWAYS!  In the common case, "disabling" involves code in the critical
exection path.  That code is minimally an executed fetch, test and a whole
gob of unexecuted bloating code that spreads out the executed code and so
by I-cache bloat slows the code that is executed.  Even without cache
effects, that fetch and test have costs.  As you know, TCP can be done
with a few dozen instructions (not counting byte copies or checksum, which
can be done with hardware).  Each optional feature would noticably increase
those <<few dozen>> instructions even if not used.

In some cases, "disabling" involves conditional compilation, and so no
run-time cost, but the maintanance costs of that solution are usually even
higher than the run time costs of the other tactic.  There is a kernel of
truth to the claims that TCP is too heavyweight for this or that (currently
"clusters").  That kernel is the intellectual baggage like slow-start that
comes with TCP.  You couldn''t make TCP go fast or fit tiny processors if
all of your programmers run screaming when you mention it...proof: the
ISO OSI suite could be made to go almost as fast as TCP or be as small as
TCP, but no one really tried.  Most competant people ran screaming from
the specs, forseeing a hell of maintenance programming for the rest of
their lives because of all of that OSI flexibility.

---

From: Joe Touch <[email-masked]>
Date: Thu, 18 Feb 1999 14:09:06 -0800 (PST)

Vern et al,

Here are some clarifications; perhaps those who would like to continue
the discussion can circulate remaining comments to Vernon and myself
"off-line"... 

> Of course there is no guaranteed correlation between send() and segments,

We appear to be tuning Nagle based on some very specific
behaviors which are particular to an implementation.

Granted, this is (or was) tcp-impl, but that refers to "places where
implementations have bugs or stray from the desired behavior". It does
not necessarily mean "constrained to behave like a particular implementation".

> I do not know the answer; if I did, I'd say so.  I do suspect it's true.

This is precisely where we differ. While I suspect it may be true
for MOST implementations, I firmly claim that an entirely conformant
TCP implementation could have different results, and that it would be
inappropriate to claim either result as uniquely correct.
Adapting Postel's motto and Floyd's position, I am advocating that
(excepting bugs, which was where this WG started), we do not 
overspecify the behavior of a TCP implementation.

> size a priori.

If I don't know the sizes ahead of time, I have to do pattern recognition,
which is computationally intensive (must examine each byte), and can
cause the packet size to increase by up to 50% in the worse cases.
Out-of-band indicators, i.e., in the transport protocol, avoid that.

The differences between out of band and in-band, application vs. kernel,
are all just optimizations. Few, if any of the ones we've described
are necessarily of one flavor or another.

> > or consistently implemented in the transport layer. 
> 

OK. I stepped on a religion button somewhere. We're talking about
the same thing - in the application. 

> to have strong views on the sanctity of Architecture and Network Layers.

Yes, we are minimalists about architecture, and some would claim
that has helped, rather than hurt. 

> makes little sense to me. 

Would it be better to refer to it as in-band vs. out-of-band? Or "restricted
to a common, central implemenation" vs "per application program"? These are
the relevant differences to which I, and others using the term 'layer', usually
refer.

> exection path. 

See the Fast Sockets work at Berkeley. You can make almost all your decisions 
at connection setup time, and run completely different implementations
if you prefer. Even those decisions that are made while an implementation is
running need not be made on a per-packet basis, necessarily.

Joe

---

From: Vernon Schryver <[email-masked]>
Date: Thu, 18 Feb 1999 15:50:03 -0700 (MST)

> behaviors which are particular to an implementation.

We are talking about proposed changes to TCP, specifically Minshall's
proposed change to the Nagle algorithm and alternatives including explicit
flush and go-slow bits in the application send-request API.  Which is the
"very specific behavior particular to an implementation," that which all
except hypothetical implementations already do, or that which none now do
and which would require a new state variable that would have to checked
and updated on most send()'s?

> overspecify the behavior of a TCP implementation.

SHEESH!  Why do you think I care about the answer?  Give me more credit
than care about getting your hypotetical, probably non-existent
implementations declared non-conformant.  If, as you say, "MOST
implementations" statisfy my pedantic proposition, then instead of
defining yet another API bit, we can talk about the heretofore ill
defined results of turning Nagle off and on.  Most of us would agree
that adding yet another flag to the standard send() API is more of a
constraint on or over-specification of implementations than specifying
what happens when you toggle Nagle, particularly when the latter would
require no changes to any existing implementation.

> that has helped, rather than hurt. 

No, just the opposite of "minimalist architecture".  (Never mind how
rewindable streams might be called minimalist anything.)  The tendency
I've noticed over there is to support surprising positions by appeals
to the Oracles of Network Architecture and the Sacred Network Layering.

> running need not be made on a per-packet basis, necessarily.

DOUBLE SHEESH!...I heard about the Berkeley (or Livermore) squashed stacks
at least 6 and I think 8 years ago.  Didn't you notice my reference to a
"few dozen instructions"?  For that matter, what do you think the far
older header-prediction amounts to, except moving optional stuff out of
the fast path?

The problem with "making almost all of your decisions at connection setup"
or otherwise out of the performance path is that it puts you in a worse
mess then compile-time option selection.  You still have zillions of lines
of bug prone code that does not get well tested, and that sends competant
programmers screaming and running away.  It's a worse mess because you
don't only have the optional code in the source tree, but also sitting in
your kernel, ready for execution when you least expect it, and bloating
your systems minimum memory footprint.   Do you think that kernel memory
is free?  Yes, you might make your kernel TCP code pagable, but that's
not without large costs.

THERE IS NO SUCH THING AS A FREE OPTION!  Not in any real implementations,
that is.  Implementations that exist only on standards committee overhead
projectors and management whiteboards are not so limited.

---

From: Kacheong Poon <[email-masked]>
Date: Wed, 3 Mar 1999 12:35:52 -0800 (PST)

>Of course, the problem with this (which is also what you can get
>if you set TCP_NODELAY) is that you don't get a stream of full
>sized packets followed by one small packet,
>	send(1460); send(1460); send(1176);
>	send(1460); send(1460); send(1176);
>	send(500);
>
>As opposed to the former which gives you:
>	send(1460); send(1460); send(1176);
>	send(1460); send(1460);
>	send(1460);
>	send(216);

In this particular example, the number of packets is the same, I don't see a
problem.  Yes, we can construct an example which with the "one step further
definition," tcp will send out more packets.  But if an app is well written
and does appropriate buffering, I think that is OK.

>	recv(ACK 2920); pkts 1-2
>		<CW=3*1460, all data acked>
>		send(1460); send(1460); send(1460);
>	recv(ACK 5840); pkts 3-4
>		<due to Nagle, defer final 1392 bytes>
>	<delay for next ACK>
>	recv(ACK 7300);
>		<all outstanding data ACKed, send final piece>
>		send(1392);

I think this last 1392 bytes should not be delayed, given that it is part
of a send which should not be delayed.  If the sequence of packets are
like

-->	PKT3 (1460); PKT4 (1460); PKT5 (1460)
<--	ACK (4)

Solaris will not delay sending

-->	PKT6 (1392)

Vernon brought up an interesting point in an earlier mail.  Apply Nagle
either on send requests or on segments, but not both.  That means TCP
will not apply Nagle check on sending packet if it is because of an ACK.
When we consider the meaning of Nagle on per send basis, I think we should
also think about that in the definition.

							K. Poon
							[email-masked]

---

From: Vernon Schryver <[email-masked]>
Date: Wed, 3 Mar 1999 14:15:22 -0700 (MST)

> From: Rick Jones <[email-masked]>

> ...
> If an application makes an API call providing for transmission a
> quantity of data greater than or equal to the MSS of the associated TCP
> connection, all data provided to the API call is immediately transmitted
> by TCP, subject to the constraints of the receiver and congestion
> windows.
>
> This is a slightly formalized version of "If an application calls send()
> with at least an MSS's worth of data, its transmission will not be
> delayed by the Nagle algorithm."

I prefer the perhaps less formalized version.  It says what we all seem
to mean and want (with the quibble that "MSS's worth" is not good).  The
more formal versions invites other questions, such as whether silly window
avoidance mechanisms, sender operating system scheduling or buffer
management, and a myriad of other things might delay the data, with or
without interaction with the Nagle algorithm.

I'd vote for the less formal but wordy "If an application calls send()
with at least 512 bytes or if it has turned off the Nagle algorithm, the
transmission of the data will not be delayed by the Nagle algorithm either
immediately or after other delays such as those caused by receiver or
congestion windows."

Given something like the MORETOSEND hack in recent BSD code, don't the
only undesirable Nagle delays occur after window delays?

---

From: Vernon Schryver <[email-masked]>
Date: Wed, 3 Mar 1999 17:00:59 -0700 (MST)

> From: Bill Fenner <[email-masked]>

> I think a lot of the question of what applying Nagle on a per-send
> basis means includes what happens when data is delayed due to cwnd
> or the receive window.  My imagination says that you keep another
> variable in the TCB that's the highest sequence number that has
> passed Nagle (in order to not delay the last packet of a large
> write), but I haven't particularly thought through exactly how this
> variable is updated or used except in the obvious case.  In
> particular, I don't know how it should be updated when you perform
> a send() and there is still data outstanding.

I think that's still confusing applying the algorithm per-send with
applying it per-Ack, and that is a Bad Thing(tm).  If you apply the
algorithm per-send, then future Acks are irrelevant, and there is no need
to save any state.

Applying it per-send is the equivalent of:

	link_new_data_into_socket_buffer();
	if (!idle || !NO_DELAY) {
		tcp_output();
	}

and that's all; no code whatsoever anywhere or anyplace else.

"delaying per-send" does not mean "delay per-send at first than randomly
delay arbirary segments at random future times as determined by the
vagaries of interface and MTU changes."

If you are delaying-per-send, then there should be at most one opportunity
for the entire blob of application data to be delayed.  If some of the
data is not delayed at first, then none of it should ever be delayed at
all.  If any of it is delayed, of course), then all of it should be
delayed.  Later (e.g. when an Ack arrives), all of the data should be
sent, just as it is now sent in a system with TCP_NODELAY=1.

P.S. I'm sending to [email-masked] instead of @grc.nasa.gov because
  that's how the return address arrived here.