Subject: Re: 'Slow-Start Restart after Idle' draft

From: Vernon Schryver <[email-masked]>
Date: Fri, 23 Oct 1998 14:20:57 -0600 (MDT)

> From: Venkat Padmanabhan <[email-masked]>

> ...
> I guess I am entirely missing what you are saying here...

My point is that classic, Van Jacobson TCP congestion avoidance and control
should be applied only where it does more good than harm.

> What I am saying is that in the 0-hop case, the regulation would be
> done by the MAC. TCP needn't. TCP would still have a flow-control
> window, though, but that's a function of the socket buffer size set
> by applications, and hence orthogonal.

That is false in both directions.  First is the case where there is a
bottleneck within a host between it and its LAN that benefits from
slow-start etc.  Second is the case of today's typical bridged network.
If you insist on slow-start whenever there is a forwarder in the path,
then you insist on slow-start practically everywhere.  Host-to-host
MAC-layer flow-control has been almost universally destroyed by so called
"smart hubs" and "switches."

> ...
> I was saying that there is *fundamental* difference between 0-hop and
> n-hop where n > 0.  It is dangerous to assume that a path is lightly
> loaded. Since the path is presumably shared by many hosts and no one
> host has a complete picture of the load imposed by other hosts, what
> is lightly loaded at one instant could be heavily loaded at another
> (for instance, due to "flash crowds").

Those words apply equally to a WAN as to a LAN.  "Flash crowds" happen
on LANs, although they are vastly less frequent than the "switch"
salescritters and trade-rag consultants would like suckers to believe.

>                                        In other words, it is difficult
> (in fact, fundametally flawed) to characterize a path as being
> "without significant congestion or bottlenecks" given the shared
> nature of the network. 

When a path is not "without significant congestion or bottlenecks", then
it obvious is not, and when it is, it is.  How in the world can it be
"fundamentally flawed" to charactize a path as it is?  Do you mean that
the characterization can change?  If so, that problem applies even more
to single-hop LANs than to WANs.  The instant a bulk transfer starts, with
or without go-fast mode, a single hop LAN is no longer "without significant
congestion or bottlenecks" to other traffic.

> However, in the 0-hop case, the MAC protocol
> gives a host exclusive access to the channel for a certain length of
> time. There is no competition above the MAC layer, except between
> multiple connections on a host. Of course, there is competition
> at the MAC layer and we would want it to be resolved fairly, etc.,
> but that is an orthogonal issue.

Those words do not apply to no LAN that I can think of.  They obviously
do not apply to any LAN that involves a shared medium.  "Flash crowds" on
on LANs are familiar to anyone who has looked at an Ethernet or token
ring, not mention implemented applications that did things like update
files on many systems at the stroke of 8 a.m.  Access to the Ethernet
carrier or the 802.5 or FDDI token is just as vulnerable to contention as
access to a leased line or the buffer in a router talking to a leased
line.  Even LANs that are based enirely on non-blocking switches, such as
ATM and HIPPI do not fit your model, because of congestion in the host
interfces.

How do you resolve congestion at the MAC layer without slowing something
down?  If the MAC layer simply slows down but your TCP code keeps pumping,
what must happen?  Hint: my point about what happens with
`ttcp -b 1048576 -l 1048576` between a pair of hosts was not theoretical.
Or consider NFSv3/TCP/IP; it does not take many NFS connections with the
default 32K block size of NFSv3 to overwhelm the buffering in most drivers.

> As a performance optimization one certainly might want to take advantage
> of times when a multi-hop network is in fact lightly loaded. But I believe such
> aggressive use of the network must happen only in conjunction with 
> mechanisms that avoid/alleviate disaster when the "lightly loaded"
> assumption is incorrect. One possible mechanism is priority dropping
> which I discuss in the "TCP fast start" chapter of my thesis:
> http://www.cs.berkeley.edu/~padmanab/phd-thesis.html

Anything involving "dropping" seems irrelevant to the issue at hand.

I favor turning on go-fast mode only when it appears from recent RTT
measurements that it is reasonable, and giving up on lossy networks by
limiting go-fast mode to only turning off slow-start and ACK pacing and
retaining slow-restart.  Turning off slow-start but keeping slow-restart
(which I take to mean standard congestion avoiding TCP/IP but with an
initial full-sized transmit window) has the nice property of responding
appropriately if a "flash crowd" appears.  It has the bad property of
making flash crowds worse, and so should not be used unless there is
evidence things are not bad.  The only way I see to get that evidence is
by looking at the recent RTT.

> ...
> > A per-host, purely manual, violate-the-standard-and-go-fast flag is always
> > possible, but is also outside the scope of of a standard.
>
> A per-host flag need not be set manually. A host can learn from past 
> experience and set it automatically.

Exactly how the host learns and so automatically sets the per-host flag
is the whole point!  The etiology of 0- vs. multi-hop networks might be
interesting, but the issue is how to set that per-host flag automatically!

So far, netmasks, hop counting, and  RTT's have been suggested.  Do
you have another way?

  - netmasks do not work because proxy-ARP, bridges, and VLSM would
     cause false positives with dire consequences on literally millions
     of networks (PPP stubbs connected to home computeres).

  - hop counting is simply impossible.  How does TCP discover how many
     hops are in the path?  Does it delay sending the SYN-ACK until it
     probes with `traceroute`?  How does that detect bridges?
     How can you detect >1 hop paths except with the RTT?

  - RTT's (e.g. from a previous connection or measured during the
     3-way handshake) are imperfect, possibly producing false positives
     on, e.g. OC-12 metro links, or on bursty LANs.   Still, what 
     alternative is there?

Vernon Schryver    [email-masked]

---

From: Venkat Padmanabhan <[email-masked]>
Date: Fri, 23 Oct 1998 14:58:32 -0700

> "smart hubs" and "switches."

Regarding your first point, you are basically saying that congestion
is can happen at the sender (a point you made in your previous email
and I agreed with in my previous one). But the problem is a local
one that is easy to resolve, and in fact there is a reasonable fix in the
BSD stack. If the interface queue fills up, TCP's call to ip_output()
returns an error code and the sender slows down ("quench") instantaneously.
When there are multiple connections originating from a host, they
should coordinate their actions so that if one of them experiences
a "buffer full" condition, the others should learn from it. (In my
thesis, I worked on doing this in certain special cases.)

Experiencing a "buffer full" condition on your local stack is much
easier to deal with than having your packets lost elsewhere in the
network. For one thing, there is no uncertainty about whether the
buffer is in fact full (and hence dropping packets) or whether there is
only a temporary RTT spike (i.e., large delay but no drops).

Regarding your second point, yes, if there is a forwarder in the
path so that the sender does not have know the load on a LAN
segment on the other side of a bridge than itself, then it should 
do slow start.

> salescritters and trade-rag consultants would like suckers to believe.

You are absolutely right. But the difference I was pointing out was
not that flash crowds happen in one but not in the other. It was that
in the 0-hop case, the MAC guarantees the sender access to the wire
from itself to the receiver. No such guarantees are made for an
n-hop path. 

Of course, as we have agreed, there can still be a problem due to
slowness of the receiver or sender. That is why in my first email,
I said that a 0-hop path was an *only if* requirement but not an
*if and only if* one.

> congestion or bottlenecks" to other traffic.

Yes, I mean it can change, and quite unpredicatably in the worst case.
In the LAN case, as I have explained, the sender knows through *local*
information that the LAN has a high load, so can slow down its
sending rate without aggravating the congestion in the LAN.

> ATM and HIPPI do not fit your model, because of congestion in the host

Perhaps you misunderstood what I meant by a "certain length of time".
I was referring to the time granularity of MAC allocations, not minutes
or hours. An example may clarify the issue. Suppose that host A has a single 
TCP connection to host B, and both A and B are on the same LAN segment.
Assuming that the end hosts are not bottlenecks and that TCP flow control
does not kick in, A should be sending to B as fast as its MAC is able
to grab the channel. Of course, at times such as "the stroke of 8 a.m.",
A's MAC will not be able to grab the channel very much because many
others will also be trying to grab the channel, so ... <on to the next point>.

> default 32K block size of NFSv3 to overwhelm the buffering in most drivers.

My original characterization of TCP and the MAC as being orthogonal was
incorrect. As I have mentioned above, BSD TCP has a "quench" mechanism
that allows TCP to slow down instantly when its local interface queue
is full (for instance, because the MAC is not able to grab the channel very
much). I hope you will agree that dealing with such local
congestion is a whole lot easier than the general problem of dealing
with congestion "somewhere" in the network. I think it makes little
sense to use a general probing algorithm such as slow start is the only 
capacity that you are trying "match" is that of your local link/buffer. 

> Anything involving "dropping" seems irrelevant to the issue at hand.
That's a strong statement! I thought we were discussing congestion
avoidance and control, and given that, I don't see how we can ignore
the possibility of packets getting dropped (however undesirable that
may be).

I'll try to explain how I use priority dropping in my answer to your
next question.
 
> So far, netmasks, hop counting, and  RTT's have been suggested.  Do

The operative part of what I said was "learn from past experience".
Here's the outline of a simple scheme. It makes no assumptions
about hop counts, netmasks, etc.

1. If doing a transfer to another host for the first time in x minutes
(x TBD), do slow start.

2. If you've done a transfer in the past x minutes, start with window size
equal to where you left off the last time. Use timer to smooth out bursts.
Mark all packets, except for the first one, as "low priority". In the best
case, all the packets will go through fine and you will experience the
benefit of fast start. In a worse case (such as the networking having become
much more congested since the last time you used it), priority dropping
with shield (to an extent) other traffic from the ill-effects of your
aggressiveness.

In my work, I also developed some special loss detection/recovery algorithms 
for use during the fast start phase. If your are interested, you should
look at my thesis.

-Venkat

---

From: Michael Ridley <[email-masked]>
Date: Sun, 25 Oct 1998 20:35:53 -0800

> Vernon Schryver    [email-masked]

I think I must be missing something here, but why do we care if it's
an OC-12 link, for instance?  Is that a false positive?  The goal
here is to bypass slow start when appropriate and send the full
advertised window so as to maximize use of non-congested links, correct?
Assuming that I haven't gotten really confused in all of this, then
I would think you'd want to send the max window on the OC-12 if it's RTT
was low (often a faster link even than your LAN on both ends).

By stating it's a false positive, you make me think that you'd want slow
start in such a case- why?

Sorry about the late reply on this...

-- 
Michael Ridley
[email-masked]