Subject: 'Slow-Start Restart after Idle' draft

From: [email-masked] (Raghu Mallena)
Date: Thu, 22 Oct 1998 11:36:20 -0700 (PDT)

I have a couple of questions on draft-ietf-tcpimpl-restart-00.txt
(Issues in TCP Slow-Start Restart After Idle) and I was hoping
that either the authors(Amy Hughes, Joe Touch, John Hiedemann)
or someone else on the tcp-impl group who read it or implemented
it can clarify this for me. 

My main question is about the Congestion Window Monitoring (CWM)
solution proposed to avoid bursts.   The draft talks about
replacing the following lines in tcp_output():

	idle = (snd_max == snd_una);
	if (idle && now - lastrcv >= rto)
		cwnd = 1; 

with
	maxwin = 4 + snd_nxt - snd_una;
	if (cwnd > maxwin)
		cwnd = maxwin;

This proposal does burst control as expected,  but I wanted to make 
sure that it does not undercut the regular slow-start algorithm 
during normal start up.  It would be really bad if it did  because 
exponential window growth is highly desirable for high speed LAN 
connections so that the stable max window size can be reached 
as soon as possible.

I took some traces and noticed that the initial slow-start behaviour
will not be compromised *if* Delayed Ack works as expected.
i.e  as long as we have an ACK coming in for every other packet
sent out (or one ACK per every packet if Delayed Ack is disabled),
then "maxwin" above is always more than "cwnd" (which is the cwnd
computed using Slow Start when the last ACK came in) and we will
still be in the exponential growth mode.   This is because
"snd_nxt - snd_una" roughly corresponds to "cwnd - 2" (2 comes
from the fact that snd_una would have moved by 2 in tcp_input()
when the last delayed ACK came in) and the comparison of 
'cwnd > maxwin' always fails.  

But what if Delayed Ack does not work as well and you start getting
one Ack for every N packets with N>2?   In this case,  the CWM
proposal could lower the cwnd below the existing value and then slow
start starts pushing it up again resulting in an osciallting
behaviour.   In a pathelogical case,  you could get one ACK
for all the outstanding unacknowledged packets which will
make snd_una=snd_nxt and cwnd will now be set all the way down to
4 from whatever its current value is.
 
One reason I can think of why the smooth 2-for-1 Ack flow
might not happen is because of the way some of the BSD-derived
stacks are implemented.   In Header Prediction mode, tcp_input() 
routine does not call tcp_output() directly which means that
an ACK will not be sent out for the second packet immediately.
Instead, the input routine wakes up the socket and the
thread that wakes up then calls tcp_output() after removing
the data from the socket buffer.   In the mean time,  if there
were a bunch of other packets that came in,  all of them are
ACKed *at once* in the output routine.   This could happen
quite often on high speed networks (even more so, if combined
with a relatively slower cpu).

Any comments?  The fact that CWM can backfire if the
ACKs dont come in smoothly is quite subtle.   Should the draft
talk about this dependence?   Any real world experience from
implementors who have incorporated CWM into their stack?

Thanks
Raghu

---

From: Joe Touch <[email-masked]>
Date: Thu, 22 Oct 1998 13:22:36 -0700 (PDT)

> From [email-masked] Thu Oct 22 12:54:49 1998
> From: [email-masked] (Raghu Mallena)
> Subject: Re: 'Slow-Start Restart after Idle' draft
> To: [email-masked] (Joe Touch)
> Date: Thu, 22 Oct 1998 12:53:20 -0700 (PDT)
> Cc: [email-masked], [email-masked], [email-masked],
>         [email-masked]
> 
> > 
> > > stacks are implemented.  
> > Have you seen this behavior, or is this a thought-experiment?
> 
> Not a thought-experiment.  I have seen this happen very often while
> working on Gigabit Ethernet driver and TCP/IP performance
> tuning over it.    If I recollect correctly,   some traces 
> showed one ACK for about every 10 packets or so at times.   This
> was with no packet loss of any sort.  

This might be an effect of the bundled-transmission of packets
that gigabit ethernet uses.

> > 'delayed ACKS' would require that there were at least one ACK
> > for every two full segments received correctly, regardless of
> > what other data arrived in the meantime. This would result in 
> > an ACK burst.
> 
> Host Requirements RFC (rfc1122) that describes Delayed Ack says
> that "there SHOULD be an ACK for at least every second segment".
> Note that its a SHOULD and not a MUST.  So its not really

This is called "stretch ACK violation"  in draft-ietf-tcpimpl-prob-04.txt,
and there it's listed as a MUST.

It appears that's a bug in the draft, unless it has been
upgraded to a MUST elsewhere (??)

Joe

---

From: Scott Bradner <[email-masked]>
Date: Thu, 22 Oct 1998 17:15:49 -0400 (EDT)

> that gigabit ethernet uses.

Gigabit Ethernet only uses bundled-transmission if its running in 
half duplex mode (as far as I know) and there are few if any 
products that support halk-duplex gigabit Ethernet

Scott

---

From: Vernon Schryver <[email-masked]>
Date: Thu, 22 Oct 1998 15:36:00 -0600 (MDT)

I doubt the intrinsic burstiness of Gbit Ethernet is responsible for the
reduced ACKs reported here this morning.  While the Ethernet driver might
know that a bunch of IP packets arrived not merely during the servicing
of a single interrupt but were a concatenated burst, the upper layers know
only that a bunch of TCP segments arrived in a short time.  Such things
happen for many causes, including the system being busy dealing with other
work.

> From: Joe Touch <[email-masked]>

> ...
> Sending reduced numbers of ACKs can generate line-rate bursts,
> ...

Line-rate bursts can be bad, for example when you are talking to a LAN
that is the first hop in a long and slow link.

However, line-rate bursts are necessary, desirable, and inevitable on LANs
if you want to approach the 100% utilization.  That is obvious on token
rings such as FDDI.  If your system has not had the token for a while, it
has received more than one Ack while it was waiting, and your system is
not amone the lame and halt, then it will send a bunch of TCP segments in
a burst at a somewhat more than 98 Mbit/sec.  (The speed of light for
TCP/IP/FDDI on a small ring with 4K segments and 2:1 segments:ACKs is 98
Mbit/sec of user data.)

Between a pair of hosts on <= 100MHz Ethernet, similar effects exist,
although they are perhaps less obvious.  Ignoring those effects, and with
BEB instead of BLAM, you really want line-rate bursts that are long but
not too long, and you want to reduce the number of ACKs to about one per
burst in order to avoid or minimize the effects of the Ethernet capture
effect.

Traffic that consists of alternating line-rate bursts of several dozen
full sized segments in one direction and bursts of half that many ACKs in
the other direction not only looks stupid on a monitor, but is measurably
slower than the obvious alternative.

Because of the line-rate burst facts of life on LANs, any standard that
tries to outlaw line-rate bursts is hopeless in LANs, even if implementors
do not choose to ignore it to improve the benchmark numbers that matter
most.

Vernon Schryver    [email-masked]

---

From: Vern Paxson <[email-masked]>
Date: Thu, 22 Oct 1998 15:29:58 PDT

> I take it the benefit of frequent ACKs is most pronounced when timing is
> a problem. What is the benefit to increasing the ACK frequency if the
> timing is solid?

Along with the points Joe made, two other problems are that infrequent acks
open the congestion window more slowly during slow start, and if acks are
rare then if there's loss along the reverse path, connections are more
likely to suffer from timeouts because they're starved for acks.

		Vern

---

From: Vernon Schryver <[email-masked]>
Date: Thu, 22 Oct 1998 17:13:11 -0600 (MDT)

> The caveat is that these mechanisms, like slow-start restart,
> should be disabled on a LAN. We hadn't put that in the latest
> draft, but it is in the revision. It's the same check as slow-start -
> if the net prefixes match, ignore the code.

That's a big improvement and I wish it were enough.  I don't think it is.

A pair of hosts can tell only that they are on the same IP network by
looking at prefixes, and cannot even get that right with some flavors of
variable length subnet masking.

If they assume there are no boxes acting as forwarders between them and
turn off the usual mechanisms, then they might be right and they might be
wrong.  Consider the literally millions of hosts that are either bridged
over very slow links (e.g. ISDN TA's) or that have intentionally bogus
network masks and rely on prox-ARP or other hacks at the "PPP server" side
of a modem or ISDN PPP link.   Those cases need slow-start more than many
others on the Internet, and probably more than most of the tests rigs in
which such things are tested, but their network prefixes will tell them
to turn off slow-start when talking to hosts (e.g. DNS, NNTP, and email
servers) at their ISP.

Then consider a pair of hosts that are on different <=100 Mbit/sec LANs,
but where the LANs are interconnected with fast mechanisms of one sort or
another (>= 100 Mbps, e.g. ATM, short-haul SONET, FC, or HIPPI), and where
they should be in fast-break-the-rules mode.

There needs to be a test that detects a super fast, almost zero delay,
practically zero loss link between a pair of hosts to turn off slow-start
etc., but I think those examples show that network prefixesis not good
enough.

Instead of matching network prefixes, why not look at the RTT?  Regardless
of prefixes, if the RTT is less than 10 milliseconds, it's probably a good
idea enter go-fast mode.  If the RTT is more than 10 ms, go-fast mode
won't gain anything and slow-start is a good idea, again regardless of
prefixes.

---

From: Joe Touch <[email-masked]>
Date: Thu, 22 Oct 1998 18:05:27 -0700 (PDT)

I agree that there is a problem with intra-LAN links, and
the detection thereof.

Whatever that solution, it should be applied equally to
avoiding slow-start and slow-start restart.

The current code makes an assumption, that the network
address indicates intra-LAN if it coincides with an
existing interface address. That assumption generates false
positives and negatives.

Can anyone suggest a general solution?

(in the meantime, I'd consider the "is_local" a placeholder
for a more correct function - and otherwise not address it
separately in the slow-start-restart doc...)

Joe

---

From: Vernon Schryver <[email-masked]>
Date: Thu, 22 Oct 1998 19:35:50 -0600 (MDT)

> Can anyone suggest a general solution?

What about what I just said?

You want to avoid slow-start and slow-restart and relax the 2:1
segment:ACK ratio exactly when the hosts are close to each other,
which to say, when the RTT is small.

10 milliseconds is about the right dividing line, since round trips
on reasonable LANs are less than 10 ms, and any long haul link with
an RTT of less than 10 is forced by the speed of light to be fast
and short enough to treat like a LAN.

The only problem with using the measured RTT to know when to drop fancy
stuff is that a new TCP connection does not know it immediately.  To fix
that, use the mechanisms that have been suggested for priming the RTT on
long haul lines.  Cache it however you cache the discovered path MTU.

You might use the round trip time on the 3-way handshake instead of the
smoothed RTT.  Hosts that should start with slow-start off are likely
to receive the SYN-ACK within 10 ms of sending the SYN, and the first
data within 10 ms of sending the SYN-ACK.

---

From: Joe Touch <[email-masked]>
Date: Fri, 23 Oct 1998 00:56:25 -0700 (PDT)

> From: Vernon Schryver <[email-masked]>
> To: [email-masked]
...
...
> which to say, when the RTT is small.

RTT may be correlated (it is debatable how closely); hopcount is
much more meaningful. If you can get there in 0 hops, then there
is no router, and you shouldn't be doing these tricks. That would 
require a traceroute-like test, out-of-band to the regular TCP
packets.

Closeness is not an issue - if I have a direct satellite
link, the RTT can be 500 ms but I still want to do the bursting,
since there is no router to overload.

Joe

---

From: Sean Doran <[email-masked]>
Date: Fri, 23 Oct 1998 12:29:34 +0200 (CEST)

Vernon Schryver wrote:

| A pair of hosts can tell only that they are on the same IP network by
| looking at prefixes, and cannot even get that right with some flavors of
| variable length subnet masking.

Well, yes and no: ARP or ES-IS usually breaks when there
is a serious mask mismatch.

The bigger concern is LISes which are bridged.   

Unfortunately, there is stuff like that out in the world,
and so making assumptions based only on the IP address seems
pretty dangerous.

One might also consider the case where a large server is
on the "outside" of a NAT, and is sending data to a host
on the "inside" (where, for example, it has an RFC-1918 address).
If the server and the NAT are in the same LIS (or on the same LAN)
then disabling congestion-avoidance mechanisms on the assumption
that no non-local infrastructure will be involved is a mistake,
despite the fact that the receiver address, from the perspective
of the "outside" server, is the NAT itself.

Finally, as Vernon notes, even where a LAN really is a LAN and
retransmissions are cheap,  it seems fair to wonder if it's smart
to dump traffic into them in an non-congestion-avoiding way for
the sake of performance.  After all, the medium may be shared by
other users, and they may see a performance decrease as the result
of traffic being blasted into the LAN.

| There needs to be a test that detects a super fast, almost zero delay,
| practically zero loss link between a pair of hosts to turn off slow-start
| etc., but I think those examples show that network prefixesis not good
| enough.

The only thing that seems to make sense would be recent observations
of performance between the two things that may be super-fast-losslessly-
connected to each other.

	Sean.

---

From: Sean Doran <[email-masked]>
Date: Fri, 23 Oct 1998 13:03:00 +0200 (CEST)

Vernon Schryver wrote:

| Instead of matching network prefixes, why not look at the RTT?  Regardless
| of prefixes, if the RTT is less than 10 milliseconds, it's probably a good
| idea enter go-fast mode.  If the RTT is more than 10 ms, go-fast mode
| won't gain anything and slow-start is a good idea, again regardless of
| prefixes.

At which point do you take the RTT measurement?

Obviously, you also want to rapidly transition into normal mode
the moment you detect any increase in RTT or the moment you see loss.

Hopefully, if there's a mistake and a bunch of congestion is
created by an assumption that it might be safe to be in go-fast
mode, it will remain local and on cheap infrastructure and won't
interfere too badly with other traffic.

Maybe I'm a little too conservative, but 10ms seems pretty long to me.
I can get well past the border and beyond Tele2's router in Malmö
Sweden in less time than that, queues willing (which is thankfully 
fairly often).   Fast hosts blasting traffic to there would be very painful.

	Sean.

---

From: Sean Doran <[email-masked]>
Date: Fri, 23 Oct 1998 13:25:31 +0200 (CEST)

Vernon Schryver wrote:

| 10 milliseconds is about the right dividing line, since round trips
| on reasonable LANs are less than 10 ms, and any long haul link with
| an RTT of less than 10 is forced by the speed of light to be fast
| and short enough to treat like a LAN.

You have to explain this one to me.

Bear in mind that some links are quiescent most of the time.

Bear in mind also that mileage charges are meaningless when
a border is being crossed.  A circuit crossing the Canada/US
border is going to be roughly 3x the cost of the same length
national circuit on either side of the border.   The factor
of three rapidly becomes a factor of five to twenty in many
other parts of the world.

Treating short expensive links like a LAN is perhaps a bad idea
in some cases.  I wonder if heavily multiplexed but usually fairly
quiet fast lines could lose badly is multiple senders suddenly tried
to treat such a line as being "short and fat enough to treat like a LAN".

A classical example might be a 45Mbps line connected on either end to a
multiplexing router with a half dozen or so 10Mbps ethernet ports,
where the hosts on the ethernets randomly do bulk transfers amongst
themselves.  

	Sean.

---

From: Joe Touch <[email-masked]>
Date: Fri, 23 Oct 1998 06:02:05 -0700 (PDT)

> From [email-masked] Fri Oct 23 05:50:50 1998
> Date: Fri, 23 Oct 1998 14:45:25 +0200
> From: Andi Kleen <[email-masked]>
> On Fri, Oct 23, 1998 at 02:25:28PM +0200, Sean Doran wrote:
> > Joe Touch wrote:
> > 
> > | RTT may be correlated (it is debatable how closely); hopcount is
> > | much more meaningful. If you can get there in 0 hops, then there
> > | is no router, and you shouldn't be doing these tricks. That would 
> > | require a traceroute-like test, out-of-band to the regular TCP
> > | packets.
> > 
> > This seems like a good approach, but traceroute-like
> > tests may not be so simple -- consider the NAT case again:  you
> > send traffic to the "outside" address of the NAT itself, which may
> > be hopcount=0, although the traffic passing through the NAT will
> > be hopcount > 0.

Agreed.
 
> Also it doesn't work for things like LANE over ATM or MPOA. 

Here I'd argue it works just fine - if you want to make your ATM/MPOA
look like a LAN, then it looks like a LAN, bursts and all. Rate limits
at the ATM level would be required to avoid problems, however.

Joe

---

From: Vernon Schryver <[email-masked]>
Date: Fri, 23 Oct 1998 08:38:37 -0600 (MDT)

> From: Sean Doran <[email-masked]>

> is a serious mask mismatch.

Proxy-ARP is amazingly common--amazing considering what a kludge it is.

> The bigger concern is LISes which are bridged.   

Yes, bridges are fatal for either hop-counting or netmask checks for 
turning on go-fast mode.

> of traffic being blasted into the LAN.

Agreed.
The Ethernet capture effect is a good (i.e. bad) case to consider.
It need not involve two hosts, but a group of winners and another
group of losers.

> of performance between the two things that may be super-fast-losslessly-

Exactly!  The average (not smoothed), the recent worst-case RTT, or the
time required for the 3-way handshake are good indications of the common
cases where you can use go-fast mode, with 10 ms being a reasonable cut-off.

Note that a router or bridge hop or two likely to push the RTT or 3-way
turn-around above 10 ms.  Or maybe 3-5 ms.  The RTT for small packets on
a quiet 10 MHz Ethernet is less than 2 ms., but the maximum minimum RTT
approaches 3 ms for full sized packets.  5 ms on FDDI tells you that you
have 0 hops and the ring is not eggregiously saturated.

---

From: Vernon Schryver <[email-masked]>
Date: Fri, 23 Oct 1998 09:28:31 -0600 (MDT)

(I appologize for sending 3 separate messages instead of checking
everything in my mailbox before replying.)

> From [email-masked] Fri Oct 23 05:03:07 1998

> At which point do you take the RTT measurement?

Why not during the 3-way handshake?
Or during the previous connection, cached along with your path-MTU
measurement?

> the moment you detect any increase in RTT or the moment you see loss.

Agreed.
I think it would go-fast mode would best leave slow-restart enabled at
all times, and only turn off initial-slow-start and enable ACK compression
(or disable ACK pacing).  With the big windows you need to get reasonable
throughput on a LAN, fast-retransmission is quite effective for the rare
loss (due e.g. excessive Ethernet collisions) and mitigates the slow-dwon
from slow-restart.

> Maybe I'm a little too conservative, but 10ms seems pretty long to me.

Ok, how about 5 ms?
Or 10 ms divided by the half ratio of the local interface speed to 10 MHhz.
E.g. 10 ms on a 10 MHz Ethernet, 2 ms on FDDI or 100 MHz Ethernet.
weeell, that breaks on HIPPI, which definitely wants go-fast mode.

] From [email-masked] Fri Oct 23 05:25:37 1998

] ...
] | 10 milliseconds is about the right dividing line, since round trips
] | on reasonable LANs are less than 10 ms, and any long haul link with
] | an RTT of less than 10 is forced by the speed of light to be fast
] | and short enough to treat like a LAN.
]
] You have to explain this one to me.

My calculator broke.  10 ms is ~2000 miles, not the 2-20 I was (not)thinking.

On the other hand, each forwarding hop at bit rates of less than 45 Mbps
tends to add 1 or more ms., and that tends to weed out the metro-sized
fast WANs.  The wire-occupancy of a 1500 byte packet at 10 MHz is 1.2 ms,
, so 10 ms is not far from the watershed.

] Bear in mind that some links are quiescent most of the time.

And that is fundamental to why go-fast mode is a good thing on LANs.
Conversely, a saturate LAN has a larger RTT and probably wants slow-start.

] Bear in mind also that mileage charges are meaningless when
] a border is being crossed.  A circuit crossing the Canada/US
] border is going to be roughly 3x the cost of the same length
] national circuit on either side of the border.   The factor
] of three rapidly becomes a factor of five to twenty in many
] other parts of the world.

What do monetary charges have to do with go-fast mode?

] Treating short expensive links like a LAN is perhaps a bad idea
] in some cases.

I would have thought that short, expensive links are even more important
candidates for go-fast mode, because their bits are more expensive.  What
matters is whether go-fast mode works better than the now standard way
of pushing bits.

] in some cases.  I wonder if heavily multiplexed but usually fairly
] quiet fast lines could lose badly is multiple senders suddenly tried
] to treat such a line as being "short and fat enough to treat like a LAN".
]
] A classical example might be a 45Mbps line connected on either end to a
] multiplexing router with a half dozen or so 10Mbps ethernet ports,
] where the hosts on the ethernets randomly do bulk transfers amongst
] themselves.  

It depends on the inter-Ethernet traffic.  As long as there are not more
than 4.5 inter-Ethernet bulk transfers happening, you want go-fast mode.

In fact, even when the 45 Mbps interconnect is saturated, you want to turn
off the 2:1 segment:ACK rule to reduce bad results of the Ethernet capture
effect on the individual Ethernets.
The Ethernet capture effect reduces TCP/IP/Ethernet throughput from
the TCP speed of light of about 93% of the raw bit rate to about 68%.
Lots of ACKs are a Very Bad Thing on Ethernets.

---

From: Vernon Schryver <[email-masked]>
Date: Fri, 23 Oct 1998 12:45:39 -0600 (MDT)

> From: Vern Paxson <[email-masked]>
>
> Please back this up with further analysis.  I've put a tcpdump trace in
>
> 	ftp://ftp.ee.lbl.gov/.vp-100mbps.trace
>
> This is an rcp along a 100 Mbps fast Ether link, with sufficiently large
> windows to fill it.  The receiver for the most part acks every-second,
> sometimes acking every third, and there's a periodic glitch in which it
> generates stretch acks (doesn't affect the throughput).  Once the connection
> reaches steady-state, it sustains upwards of 11 MB/s.

I cannot find a file there.

However, I bet that at least one of the following is true, any one of
which is sufficent to sidestep the problem:
  - you used a TCP window size of less than 16 K
  - you used half-duplex MAC chips with BLAM (except that I didn't
     think any were commercially available)
  - you used full-duplex EThernet, which is not really Ethernet since
     it has nothing more to do with CSMA/CD than 100VG-AnyLAN, 802.5,
     HIPP, ATM, FC, or FDDI.
  - you used half-duplex MAC chips that violate 802.3 as the old AMD 7990
     LANCE did, by being "polite" and not transmitting immediately after
     deferring.

If you use a TCP window of >= 32K, 802.3 compliant MAC chips with BEG
instead of BLAM, and fast hosts, you'll get about 8 MByte/sec instead of
11 MByte/sec.

The problem is that if both systems are transmitting about the same
number of packets, and they are using 802.3 compliant MACs with BEB and
not BLAM, then one of them will capture the Ethernet, and the other will
binary-exponential-backoff into the weeds and even suffer excessive
collision errors.  What usually happens is that the TCP sender blasts a
full window, starves for ACKs because the TCP receiver has BEB'ed into
waiting for 512 slot times, and goes to sleep.  Eventually the TCP receiver
wakes up, sends one of its queued ACK's, and everything starts rolling
again, for a while.  If you use a small TCP window, the hiccups still
exist, but they are much smaller.  If the TCP sender does capture the wire
with an 8K window, it runs out of transmit window before the TCP receiver
has backed off to death, and so the TCP receiver is able to get its ACK
through without having the wire being idle for most of 1024 slot times.

If you abuse the 4.3BSD-Reno TCP code to not send as many ACKs, the problem
goes away, at least for TCP windows of 61,440.

I say this based on years of glaring at a <<LOT>> of traces, as well as
having personally abused 4.3BSD-Reno.  I could also tell you about custom
silicon in MAC chips intended by design to violate IEEE 802.3 to deal with
this problem, but it's supposed to be secret.

There is also Mart Molle's extensive writings on the issue.

---

From: Vern Paxson <[email-masked]>
Date: Fri, 23 Oct 1998 12:24:42 PDT

> I cannot find a file there.

Some browsers don't like hidden files - if so, ordinary FTP ought to
fetch it.

>   - you used a TCP window size of less than 16 K

Receiver offered 17K, sender had about 16K in flight.

>      think any were commercially available)

I don't know, but we're running stock hardware.

>   - you used full-duplex EThernet, which is not really Ethernet since

No, we have that on one of our machines, but not this one.

>      LANCE did, by being "polite" and not transmitting immediately after

I don't know - it's fairly new hardware, though.

> collision errors.

It would appear that acking frequently could actually help with this.  The
window moves in small increments, so there are lulls in the sender's
transmission, during which more acks can be sent.  It seems that infrequent
acking could rather aggravate the capture effect: it leads to large bursts of
back-to-back data packets, so a lot more opportunity for the receiver's
collision timer to back off into the weeds.  I'd be interested in
understanding what's missing from this analysis.

		Vern

---

From: Scott Bradner <[email-masked]>
Date: Fri, 23 Oct 1998 15:46:40 -0400 (EDT)

>      think any were commercially available)  

I had not heard that such chips were in products - has BLAM even been
endorsed by IEEE?

Scott

---

From: Vernon Schryver <[email-masked]>
Date: Fri, 23 Oct 1998 14:56:16 -0600 (MDT)

> Some browsers don't like hidden files - if so, ordinary FTP ought to

Netscape delivered garbage, so I tried `ls` via FTP, which denied the file
existed.  Having more than once received large (many MByte) traces by
email, I was a little shy about fetching it blindly.  I just now did
anyway, and found it was < 200KBytes, but it's some kind of binary.  I
guess I could try `tcpdump -r`...yes, I see that produces things.

But you answer the point 

> Receiver offered 17K, sender had about 16K in flight.

That's close enough to reduce the problem to tolerable levels.

You might also have BLAM in one of the MAC chips.  Now that I think
back, I think I heard rumors one of the 100 MHz chips has BLAM.

> understanding what's missing from this analysis.

In summary, back-to-back packets are not the problem.  The problem is the
collison between to long trains of packets.  If the TCP receiver is not
sending ACKs frequently, then it is less likely to collide with the TCP
sender, and so it's less likely to back-off into the weeds.

If the TCP receiver responds to the second TCP segment with an ACK (and
is fast), it is certain to collide with the third segment.  If it is
unlucky and loses the first 2 or 3 collisions while trying to send that
first ACK, it will probably back-off into the weeds (backoff counter=10)
or wait until the TCP sender stops sending, whichever comes first.  TCP
sender will continue sending and occassionally colliding, as the receiver's
timer occassionally and less and less frequently expires, until it runs
out of window.

With your TCP window of ~17K, the TCP sender will never send more than
~11 packets before starving for an ACK.  That means the receiver will
never suffer more than 8, and at 100 MHz if it is not super fast, probably
not more than 5 or 6 consecutive collisions on any ACK, so its back-off
counter will be only 5, so it will wait only about 16 slot times, which
is less than a packet-time.

Note that the probability of the TCP receiver losing the wire is less than
87% on each ACK.  More than 87% of the time, it will win the first
collision or one of the early subsequent collisions.  (Specifically, on
each packet, it will not be locked out with probability .5+.25+.125+...)

Now consider a TCP window of 60*1024.  That's a collision a train of 40
packets with a train of 20 ACKs under normal rules.  Either the TCP sender
or the TCP receiver is likely to capture the wire, and the other is like
to have a back-off counter equal to 10, and so wait up to 1024 slot times
or the equivalent of 65KBytes of data after the other has stopped sending.
Worse, there's a good chance that the loser will suffer 16 collisions and
drop a packet, and if the loser is the TCP sender, that really hurts.

Now consider reducing the ACK ratio by always delaying ACKs, and not just
after partial segments.  If the TCP receiver does send its first ACK for
the arriving train of segments until 10 or 15 segments have arrived, it
is less likely to be locked off the wire by its own back-off counter. 
Or if its train of ACKs is only 5 intead of 20, it is not nearly as certain
to have one of those ACKs get unlucky and run up its back-off counter.

Note that the point Rick Jones made is also valid, but less dramatic than
the 30% throughput loss due to the Ethernet Capture Effect.  Host
performance is a generally function of packet counts, so more packets
always costs speed, even if they cost no bandwidth, as with full-duplex
media.

---

From: Vernon Schryver <[email-masked]>
Date: Fri, 23 Oct 1998 15:02:47 -0600 (MDT)

> From: Scott Bradner <[email-masked]>

> I had not heard that such chips were in products - has BLAM even been

What does "endorsed by the IEEE" mean?

The last I heard from Mart Molle was quite a while ago and that 802.3v
(u? x? whatever) was chugging along, but that the 10 MHz chip vendors were
not happy with the prospect of either fixing their ancient designs or
shipping slower chips.  Never mind that the really ancient AMD LANCE beats
them all, except when you turn on their various really nasty (e.g. linear
backoff) work-arounds or in a case or two, their emulation of the LANCE
bug.

On reflection, I think I was told at my former job that one of the 100
MHz MACs in one or another of the products would eventually have BLAM.
However, don't take that as gospel, since I was never directly involved,
never even saw a 100 MHz or GHz chip spec, and besides, it might be a secret.

---

From: Vernon Schryver <[email-masked]>
Date: Fri, 23 Oct 1998 16:37:14 -0600 (MDT)

> From: Venkat Padmanabhan <[email-masked]>

> ...
> Regarding your second point, yes, if there is a forwarder in the
> path so that the sender does not have know the load on a LAN
> segment on the other side of a bridge than itself, then it should 
> do slow start.

Given the near universal installation of bridges, aren't you saying that
by your reconning, no go-fast switch is possible?  Recall, for example,
that every 100/10BASE-T hub is necessarily a bridge.

> ...
> Of course, as we have agreed, there can still be a problem due to
> slowness of the receiver or sender. That is why in my first email,
> I said that a 0-hop path was an *only if* requirement but not an
> *if and only if* one.

We disagree.  You keep saying that a 0-hop path is almost sacred but
others are vulgar and cannot use a go-fast mode.  Consider a private,
three-segment campus or metro network where the local links are 10 MHz
Ethernet and the long-haul segment is as fast as necessary to guarantee
no slow down, say FDDI or T3.  Such things do exist in real life for
economically sound reasons.  (Many Silicon Valley outfits lease dark
fiber under public streets to connect their buildings).  Please say why
that case is more likely to experience flash crowds than the
128.32.33.00/24 near your desk, and also say how it would be easier for
mercenary.CS.Berkeley.EDU to detect such problems.  (I'm guessing that
there is a "smart hub" or "bridge" on the 128.32.33.00/24 Ethernet, given
the many well known and presumably sometimes busy hosts on it.)  My guess
is that the only way congestion can be detected in either network is by
TCP timeouts or duplicate ACKs, and that congestion is far more likely on
your local Ethernet then on my not so hypothetical network.

> ...
> Yes, I mean it can change, and quite unpredicatably in the worst case.
> In the LAN case, as I have explained, the sender knows through *local*
> information that the LAN has a high load, so can slow down its
> sending rate without aggravating the congestion in the LAN.

Please say how a sender on a LAN can tell that its local LAN has suddenly
gone bad, given the installed base of millions of bridges that we all know
will never look at any new bits in proposed revisions of TCP headers.
There is no hope among those of us who've experienced previous generations
of bridges that they'll even honor old bits they're required by standards
and common sense (e.g. FDDI-Ethernet and IP DF).

Also as I keep saying, the "switch" sales critters have entirely broken
MAC-layer flow control.  They actually use that fact as one of their
selling points, and their suckers--oops actually think it is a good thing.

> ...
> > Anything involving "dropping" seems irrelevant to the issue at hand.
> > 
> That's a strong statement! I thought we were discussing congestion
> avoidance and control, and given that, I don't see how we can ignore
> the possibility of packets getting dropped (however undesirable that
> may be).

We are not discussing new ideas ofr congestion avoidance or management.
We are talking about a sneaky-smart switch in hosts that turns off the
standards-required ACK pacing and/or slow-(re)start.  Yes, packets can be
dropped, but the idea of the go-fast switch is ONLY for cases where packet
losses are low or at least not due to congestion, and only when congestion
is being managed with slow-(re)start, with the need of slow-(re)start for
well smoothed RTT's with requirement from that for the 2:1 ACK:segment
ratio.

> ...
> The operative part of what I said was "learn from past experience".
> Here's the outline of a simple scheme. It makes no assumptions
> about hop counts, netmasks, etc.
>
> 1. If doing a transfer to another host for the first time in x minutes
> (x TBD), do slow start.
>
> 2. If you've done a transfer in the past x minutes, start with window size
> equal to where you left off the last time. Use timer to smooth out bursts.
> Mark all packets, except for the first one, as "low priority". In the best
> case, all the packets will go through fine and you will experience the
> benefit of fast start. In a worse case (such as the networking having become
> much more congested since the last time you used it), priority dropping
> with shield (to an extent) other traffic from the ill-effects of your
> aggressiveness.
> ...

In which bit of the current TCP or IPv4 headers is your low priority
mark?

Given a free hand, a lot of things other than Van Jacobson's old but
extremely cool old kludge of watching for packet loss and guessing about
congestion could be tried.  ECN is an obvious and currently popular idea,
after years of quiet.  However, that's all entirely irrelevant to this
discussion and possibly to this mailing list.  (Recall it's title is
something about TCP implementation).  We are not talking about TCPng, but
how to mitigate the new, stronger language about ACK pacing and
slow-(re)start.

Again, given things that the host that needs to set its own go-fast switch
for traffic to a specified other host, what do you suggest we do?  If you
have a real alternative to looking at the RTT or the netmask, please
describe it.

---

From: Scott Bradner <[email-masked]>
Date: Fri, 23 Oct 1998 20:28:39 -0400 (EDT)

> What does "endorsed by the IEEE" mean?  

passed as a standard

Scott

---

From: Vernon Schryver <[email-masked]>
Date: Mon, 26 Oct 1998 08:01:42 -0700 (MST)

> From: Michael Ridley <[email-masked]>

> >   - RTT's (e.g. from a previous connection or measured during the
> >      3-way handshake) are imperfect, possibly producing false positives
> >      on, e.g. OC-12 metro links, or on bursty LANs.   Still, what 
> >      alternative is there?

> I think I must be missing something here, but why do we care if it's
> an OC-12 link, for instance?  Is that a false positive?  The goal
> here is to bypass slow start when appropriate and send the full
> advertised window so as to maximize use of non-congested links, correct?
> Assuming that I haven't gotten really confused in all of this, then
> I would think you'd want to send the max window on the OC-12 if it's RTT
> was low (often a faster link even than your LAN on both ends).
>
> By stating it's a false positive, you make me think that you'd want slow
> start in such a case- why?

Yes, I phrased things badly there.

My general claim is that it does not matter whether there is one hop or
many, or whether some of the hops are longer or shorter.  All that matters
is that you want a go-fast mode that violates some of the proposed, even
stronger language enforcing the TCP congestion control and avoidance
mechanisms when there are sufficiently few, opportunites for congestion
at the current instant.  It would be nice if there the go-fast mode and
how and when it is invoked were documented, for all of the standard reasons
standards exist.

I was trying to concede in those words that a fairly short, fast, but
somewhat congested link can appear to have a very small RTT, when the RTT
is measured with a single pair of packets, such as during the 3-way
handshake.

The point of a go-fast mode is the extreme case of what started the
discussions in this mailing list (I think it was here) weeks or months
ago about increasing the initial window for HTTP.  If you have a
semi-private 1 Gbit/sec link connecting hosts using 500KByte windows and
1432 byte MSS to move blocks of data, you really do not want to wait for
175 ACKs before you are using the full window.  You use such large windows
because your bandwidth*delay product is that large or larger, and you are
going at speeds where much of the delay comes from inside your hosts.
For example, if your operating system wakes up your application to dump
to a screen buffer 10 or 20 ms, you want enough data on hand to make it
worth while.  Or consider moving files; most UNIX file systems do not
deliver or sink GBytes/sec, but some do, and there soon will be more.
Given today's cheap Gbit Ethernet and $2000 PC's with 100's of MByte of
RAM, such scenarios are no longer found only among people with $30M CPU's,
or at least with $B atomic particle detectors.  Competative pressures will
at least entice implementors to discard slow-start in such situations.
If they are not conservative in how they turn off slow-start, their code
will also turn it off in the classic cases where it is needed.