Subject: Re: flows stuff

From: [email-masked] (Noel Chiappa)
Date: Sat, 3 Dec 94 05:31:56 -0500

<Since we're trying to characterize flows here, I'm going to stop
 inundating N mailing lists with this thread, and try and keep it just
 on "flows" after this message...>

    From: Curtis Villamizar <[email-masked]>

    If at 2 orders of magnitude more traffic (200 Mb/s) there were 2
    orders of magnitude more new flows per second ... would something on
    the order of 10,000 new flows per second and a cache of 1,000,000
    entries be a problem.  

As I indicated, of those 10K source-destination pairs active in the average
second (which is really what those counts are, *not* flows), most are from
applications that would prefer the datagram service mode (e.g. 19% of all
active SDP's are DNS), and of the remaining "real" flows, not all would need
to be in the cache for good performance (somehow I doubt TELNET connections
needs to be in a high-speed cache). So a cache of 1M entries is not a
reasonable projection.

    Noel- keep in mind that if 90% of this data is TCP

Well, 90% of the data might be TCP, but 19% of the flows were DNS-UDP...

    and you are suggesting that 90-95% of it should not require flows setup at
    all, then you are suggesting that we should not be setting up flows on a
    majority of TCP connections. Just wanted to clarify that.

It's a little more complicated than that. A lot of those TCP connections seem
to be transferring very little data. I'm not sure what the exact minimum
number of packets is to open and close a TCP connection; probably there are
implementations which aren't pushing things to the max (e.g. sending data in
the same packet as a SYN, or making sure that the data of a response to a
query goes in the same packet as the ACK of the query, etc, etc, etc).
Simplemindedly, for a simple query, it looks like about 11 packets, viz:

	SYN->
	<-SYN, SYN-ACK
	SYN-ACK->
	DATA->
	<-DATA-ACK
	<-DATA
	DATA-ACK->
	FIN->
	<-FIN-ACK
	<-FIN
	FIN-ACK->

This is a lot of packets when for simple data retrieval two (or a few more, if
the response won't fit into one packet) would do. (The overhead of all this
makes the overhead of setting up a flow look small! :-) That something like
this is happening can be seen from HWB's data (looking at some of the top TCP
users):

  Prt SPort DPort flow pkts byts secs	|	ave-pkt	ave-byt	ave-size

  6     0    80    2    2    4    4	|	13	720	54
  6    80     0    3    4    1    3	|	7	2874	386
WWW (32% of unidirectional "flows", 20% of all packets) - average of 20
packets per connection
  6     0    25    4   10   10    8	|	8	1050	130
  6    25     0    5    8   11   11	|	9	523	58
SMTP (11%, 5%) - average of 17 packets per connection
  6    70     0    6   12    5   12	|	7	2588	356
  6     0    70    7    9   13   15	|	13	575	42
Gopher (7%, 4%) - average of 20 packets per connection

Looking at the two "data retrieval" applications (SMTP is more complex,
because you want assured delivery, but it could probably still be improved a
lot), together they add up to 39% of all the unidirection "flows" measured,
and 24% of all the packets in the network. Probably 3/4 of those packets, i.e.
16% of all network traffic, is useless overhead. Major applications like that
probably shouldn't use something with as much overhead as TCP for simple data
retrieval transactions. Even if the data being retrieved takes more than one
packet, rerunning the entire transaction if a packet is lost probably is more
efficient, on average.

So, to the extent that they are things that are using TCP, but don't look
like long-lived clients, or clients that transfer a lot of data, no I don't
think we should be setting up flows for them.

    If so, we are in agreement. Some TCP will want flow setup. For some flow
    setup doesn't make sense and I'm not sure it will scale well enough anyway.

In general, yes, I don't think there's a one-one mapping from TCP connections
to flows. However, I see nothing in this data, when you really look at it, to
warrant the conclusion that there's an explosive growth in flows that won't
scale.

	Noel

---

From: [email-masked]
Date: Sat, 03 Dec 94 10:22:36 -0500

Before you all disappear to 'flows' let me offer one piece of fairly
sound data and two as-yet-unanswered questions that must be answered
to know if my data point is significant.

In a previous job I collected a large amount of statistical data about
SMTP behavior as experienced on a couple of dozen Arpanet and Milnet
hosts.  Unfortunately the data files are now inaccessible or lost and
they were starting to disappear even some years ago.  Thus I am working
from personal memory here.  Over the period 1/1/83 to ??/87 or 88 I
gathered this data on every email entering or leaving the hosts in
question.  The number of RFC822 messages was between 1 and 2 million.
The number of TCP connection attempts was almost exactly two times the
number of TCP connections on which a successful delivery occurred.
All distinguishable failure causes were counted separately and over 90%
were due to failure to get the TCP connection ESTABLISHED, i.e.,
destination did not answer or was unreachable through the network.

Implication --> ~50% of the TCP connections that my SMTP intended to be
flows never became useful flows.

I'd have to go back and review source code to figure out how many SYNs
happened before it gave up but think it was 5-10.

SMTP whizzes will say that my result is very sensitive to the retry
interval for failed messages.  The answer is not simple because I built
a complex retry algorithm after seeing this behavior.  My retry used
progressive backoff from a few minutes to a few hours in about 10 stages
over something like 20 hours.  This is much more clever than sendmail.
Goal was to get mail through quickly if net is having a routing flap
but don't waste resources trying to talk to hosts that are really dead.
I'd guess that the short-retries and the long-retries might offset each
other to a large extent.

Unanswered questions:

1. This data point is application-specific.  What about other
applications?  What is the pdf of persistence of screwups in general?

2. What is the significance of the answer to question 1 in terms of
impact on flow caching or size of flow ID space or any of the other
burning questions of our day?

If this is all well understood by the wizards then I apologize for
using your bandwidth, but I hadn't heard it discussed and I remember
a hullabaloo some years ago when it came to light that the DNS root
servers were spending much of their time responding to typing errors.
Just trying to learn from experience...

/Bill Barns

---

From: Curtis Villamizar <[email-masked]>
Date: Sat, 03 Dec 1994 13:58:58 -0500

> number of TCP connections on which a successful delivery occurred.

Do you think it is fair to compare MILNET in 1987 and 1988 to the
current Internet?

> destination did not answer or was unreachable through the network.

This is not surprising.  I don't have direct experience, but from what
I've heard from others MILNET was not very reiable for a long time.

> flows never became useful flows.

I don't think the figures are anywhere close to this today.  I run a
home system and often run sendmail -q -v (smtp is set as an expensive
mailer and my -q is set to 2h) and I don't see anything near 50%
failing to connect.

Some people set DNS MX records to go direct to the user's workstaion
and fall back to a server that is always up.  This can cause that type
of problem, but I don't think it is that common and that you get
bounced due to the ws being powered off very often.

> happened before it gave up but think it was 5-10.

Sounds about right.  I think it might be 12.

> I'd guess that the short-retries and the long-retries might offset each

Your results will vary depending on who you are talking to.  You could
probably duplicate those results by sending lots of mail to third
world countries that just brought up their first Internet connections.

> applications?  What is the pdf of persistence of screwups in general?

I would gues that it is much much higher than 50% completion of the
TCP handshake.  This is something that could be gleaned from Hans
Werner Brauns's raw data (what percentage TCP SYNs show later evidence
that the connection was completed).

> burning questions of our day?

Don't send mail to the third world?  ;-)

MX records and replacing full host from addresses with a more
canonical user at domain name should help this, relative to 1987,8.
Today you more often forward to a presumably more reliable
intermediary (a central server which hopefully isn't overloaded)
rather than forwarding to someones workstation that may have been
powered off.  Network availability is also much better.

> Just trying to learn from experience...

That's still a real DNS problem.  As far as I know, the only technical
improvement that helps is the ability to cache negative responses.

> /Bill Barns

IMO: The things you bring up had more to do with 1) the Internet was
considerably less reliable 6-7 years ago, and 2) we've all got a
little better at administering or mail (and DNS) systems.  I doubt
that TCP handshakes are completing at a low rate today.

Curtis

---

From: Curtis Villamizar <[email-masked]>
Date: Sat, 03 Dec 1994 13:09:37 -0500

> Well, 90% of the data might be TCP, but 19% of the flows were DNS-UDP...

yes.

> this is happening can be seen from HWB's data (looking at some of the top TCP

This is critical!  The overhead on the router is simply packets to
forward and not state change in the router.  Routers with RISC
processors on each card can forward 10kpps per interface per direction
easily and routers with help from silicon can hit 200 kpps (per the
whole box).  These same routers can only deal with 100s of state
changes per second before falling over and dying (they handle peaks
nicely by deferring work, but sustained state change will kill them).

TCPs state change overhead is handled entirely by the end systems that
are not handling 100s of new flows per second (unless NCSA/UIUC only
has one host:).

>   Prt SPort DPort flow pkts byts secs	|	ave-pkt	ave-byt	ave-size

I disagree.  When you connect to port 80, you don't know if you will
be retrieving a big file or a small one.  If it is small, you eat a
few RTTs and go part way up the slow start ramp before the transfer is
over.  If it is larger, you get bounced off the slow start ramp and go
into congestion avoidance and you are being a godd citizen.  Use UDP
and you are stuck with either poor performance because you guessed too
small one the window size or data rate, or poor performance because
you guessed too high on the data rate and don't have the the
congestion avoidance.

Of course, if you can keep the congestion avoidance but reduce the
conection startup and tear down overhead, that is probably a good
thing.  Sounds like we just argued in favor of T/TCP.

> think we should be setting up flows for them.

We agree on something.  ;-)  It has happenned before.

>     setup doesn't make sense and I'm not sure it will scale well enough anywa

And T/TCP might be a good idea?

> warrant the conclusion that there's an explosive growth in flows that won't

I agree that things will scale well enough as long as we don't set up
flows for everything.  If we do take a pure connection oriented
approach and try to set up a flow with separate QoS for every
connection (some people used to advocate TCP and UDP), or even every
TCP, or every host-host pair, we see a much greater explosion in state
size and state change at the points where high speed forwarding is
taking place.  Didn't we just do CIDR because the amount of state and
state change experienced by routers in WANs was too great?

Curtis