Subject: Re: Nagle -- again

From: Joe Touch <[email-masked]>
Date: Wed, 23 Jun 1999 18:34:55 -0700 (PDT)

> From [email-masked] Thu Jun 17 20:10:54 1999
> To: [email-masked]
> Subject: Nagle -- again
> Date: Thu, 17 Jun 1999 19:27:13 -0700
> From: Greg Minshall <[email-masked]>
> 
> All and sundry,
> 
> We last talked about this in March.  The I-D i had submitted last fall has 
> either expired or is about to expire.  So...
> 
> My plan is to submit the updated version of my I-D (that i sent to this list a 
> long time ago -- Fri, 26 Feb 1999) to the Internet Drafts directory (soon, 
> given that the cut-off for the Oslo IETF is coming up, and i don't want it to 
> be "out of print" that long).
...

One observation - the paper indicates that there _must_ be at least
one ACK every two packets. This is not strictly required, at least
last time I checked. 

Presuming that, would you propose to allow K-1 outstanding partial
packets, given one ACK per K packets?

Some other notes:

	- the paper uses analysis based on ethernet MSS's,
		how do things change if Internet default MSSs are assumed?

	- Table 3 doesn't match the other tables (either) for original nagle

Joe

---

From: [email-masked] (Sam Manthorpe)
Date: Wed, 23 Jun 1999 20:57:58 -0700 (PDT)

> one ACK every two packets. This is not strictly required, at least

I've been meaning to bring that up for a while.  The `ack every other
segment' algorithm recommended in RFC-1122 is bad news for high bandwidth,
low packet size media.  In particular I'm thinking of gigabit Ethernet
without jumbograms.  When we get an ack for every 2 packets, i.e. one
ack for every ~2920 bytes of application data, then if we want to obtain a 
TCP goodput of, say, 600 Mbits/sec on a single connection, then our 
sending machine will be receiving approx. 26000 ACKs per second which 
places a substuntial load on the sending hosts's stack (interrupt 
processing, IP packet processing, TCP processing).

In other, lower bandwidth, scenarios the ACK every other packet scheme 
can improve performance by keeping a steady flow of ACKs that stop
the source from hitting the end of its congestion window.

So, two things:

    (a)	the SHOULD in RFC-1122 should remain a SHOULD and not become
	a MUST, IMO

    (b)	I'm sure there is a clever algorithm to be found that decides
	when the optimal time to send an ACK is; anybody got any ideas?

Cheers,
Sam.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Sam Manthorpe, SGI.  tel: (650)933-2856 fax: (650)932-2856  [email-masked]

---

From: "Eric A. Hall" <[email-masked]>
Date: Thu, 24 Jun 1999 09:58:24 -0700

I also think ACK-for-every-2nd is bad, although I've thought about this
in terms of half-duplex links where ACKs just mean the sender can't keep
streaming data.

>      when the optimal time to send an ACK is; anybody got any ideas?

My thinking: the receiver should ACK when enough data has been received
to fill 50% of the outstanding receive window. No more, no less.

If the reciever publishes an 8k rwin, then it should ACK once half of
the window (or a near quantity of MSS segments) is received. If the rwin
advertisement drops to 4k for some reason, the the recipient should ACK
at 2k. Etc.

This would obviously be very problematic with slow start and congestion
avoidance. There is also a problem of the receiver not being aware of
RTT changes in a bulk-transfer environment, and they should not use 50%
if the sender will fill rwin before the ACK arrives. Imbalanced links
(satellite for sends but modems for ACKs) are also problematic.

I don't think the scheme is usable until these issues are resolved.
Given all of those issues, the only thing that can be done in the
current model is to ACK every other segment.

Having a negotiated virtual circuit (with explicit markings for link
characteristics) would allow for a more accurate ACK strategy.

-- 
Eric A. Hall                                            [email-masked]
+1-650-685-0557                                    http://www.ehsco.com

---

From: "Eric A. Hall" <[email-masked]>
Date: Thu, 24 Jun 1999 10:06:02 -0700

> current model is to ACK every other segment.

Another possible solution to the problem is the use of a Mega ACK, where
the recipient ONLY sends an ACK after (A) a discontiguous segment is
received or (B) no more data arrives.

(B) is the killer, since it would introduce delay at the end of a
successful transfer unless the receiver knew how much data was being
transferred (which is not possible except with some bulk transfers). It
may be that a MACK option would be useful for bulk transfer apps.

(B) would also introduce a LOT of delays in the event of a lossy link.
This could be resolved with the use of an explicit NACK though, and also
illustrates that a MACK option should be limited to applications that
can tell the recipient exactly how much data is coming.

---

From: Greg Minshall <[email-masked]>
Date: Thu, 24 Jun 1999 13:04:54 -0700

i've been *told* that Linux will let exactly one ACK live in the i/f output 
queue, updating it with new acknowledgments and window updates.  this can 
reduce the number of ACKs (though breaking the clock).

however, if your host is generating 52,000 *data* packets per second, i wonder 
if getting 26,000 ACKs per second *should* be a problem.  (i don't doubt that 
it *is* a problem, but *should* it be a problem?)

cheers,  Greg M.

---

From: Greg Minshall <[email-masked]>
Date: Thu, 24 Jun 1999 13:01:55 -0700

Joe,

> one ACK every two packets. This is not strictly required, at least

yes, sorry.

> packets, given one ACK per K packets?

no.

>		how do things change if Internet default MSSs are assumed?

well, i suspect that ethernet MSS ~== Internet default MSS, however if you 
mean 512 or 576 bytes, i'm not sure.  i'll (possibly) get back to you/the list 
on this.

>	- Table 3 doesn't match the other tables (either) for original nagle

yes, table 1 is from one run on one kernel; table 3 is from one run on a 
kernel that had could switch between the two implementations.  also, note our 
(rueful) disclaimer:

	We intend to do a further study using a careful statistical
	analysis of a large set of trials (and apologize for the use
	of single trials in this paper)

;-)

Greg

---

From: [email-masked] (Sam Manthorpe)
Date: Thu, 24 Jun 1999 18:01:43 -0700 (PDT)

> it *is* a problem, but *should* it be a problem?)

Hmm, not sure what you mean here. `should' according to which principle?

Do you mean that ACK packets should be cheap compared to data packets 
because they are so much smaller?  The relationship between packet size
and processing time is generally far from linear, especially in systems 
which have zero-copy stacks.  It is possible to modify your network stack 
to reduce the overhead of a large number of ACKs (e.g. poll for packets 
instead of relying interrupts).

Cheers,
Sam.

---

From: Greg Minshall <[email-masked]>
Date: Thu, 24 Jun 1999 18:27:56 -0700

Sam,

"should" from the principle of how expensive (oh lord, this really isn't 
anything worth getting into, huh?) interrupts should be on a system.

jumbograms, reduced numbers of ACKs, etc., are all based on the premise that 
it is *very* expensive to process a single packet (interrupt time, context 
switch, etc.).

it is the case that per-packet processing costs dominate (ref lots of papers 
on the issue, with at least one [1993] SIGCOMM paper pointing out that data 
costs, also), but it would be nice (principle) if system designers made sure 
this cost less onerous rather than more onerous over time.

i can (will) be accused of being somewhat utopian.  i accept this as being 
basically valid; i understand the growing disparity between CPU cycle times 
and memory access times.  but i think it is worth spending time pushing back.

(you, and Joe T., are right in saying that sometimes polling can be used to 
reduce the per-packet costs.)

Greg

---

From: Rick Jones <[email-masked]>
Date: Tue, 29 Jun 1999 6:48:27 PDT

> however, if your host is generating 52,000 *data* packets per
> second, i wonder if getting 26,000 ACKs per second *should* be a
> problem.  (i don't doubt that it *is* a problem, but *should* it be
> a problem?)

depends on ones definition of "problem." with copy-avoidance and
checksum offload, the processing costs of an ACK are not that much
less than a data packet, so 26k ACKs would probably be about 25 to 30
percent of the CPU consumed.

one of those lifetime employment discussion areas - save CPU cycles,
or send lots of acks to avoid possible burst problems... :)

rick jones