Subject: flows stuff

From: [email-masked] (Hans-Werner Braun)
Date: Fri, 2 Dec 94 10:56:13 PST

Given some discussions here, it seemed interesting to run a flows test
based on more current traffic, but along the lines outlines in the
flows.ps pager on ftp.sdsc.edu (/pub/sdsc/anr/papers). I don't mean to
engage in the discussion here, so just use this as noise from the side
lines?

So, incidentally, around 7AM pacific time (right after I saw some of this
email I am responding to) for about an hour I ran a packet trace at the
interface between the NSFNET backbone and the attached UIUC/NCSA/...
FDDI ring, just filtering on the NSFNET backbone node's MAC level
address. This location is historically a very heavily used traffic hub.
I am using the natural Flow ID that I find in packets already, namely
address/port quadruples for TCP and UDP traffic, and address pairs for
the remaining stuff. I am using a 64 second timeout, which seems ok
from our earlier studies, though a little on the long site, I would
think.

Unfortunately the SGI Indigo R4000 I am using is not able to keep up
with the traffic volume and reports alot of losses per second at the
following percentiles:

         Min           0.000
         05p          35.885
         10p          39.477
         15p          42.046
         20p          44.104
         25p          45.740
         30p          47.248
         35p          48.536
         40p          49.737
         45p          51.010
         50p          52.051
         55p          53.222
         60p          54.322
         65p          55.435
         70p          56.426
         75p          57.659
         80p          59.159
         85p          60.719
         90p          62.555
         95p          65.703
         Max          77.422

so, what I all send here should be more used as indications, rather
than reflecting all the packets exchanged. I guess a high end DEC Alpha
could keep up with the current volume, from what I have seen and heard,
but I don't have any to test it with. Though MCI, as part of the vBNS
project, is trying to help out there.

---

In the 3983 seconds measured I saw 474121 unique flows.

Ok, now, measuring the flows on a second  by second basis, as known to
the "cache," we see a distribution of:

        Mean	   10649.010
      Median	   10680.000
    Midpoint	    6276.500
   Geometric	   10605.443
    Harmonic	   10501.868
         Min	     994.000
         05p	   10031.000
         10p	   10301.000
         15p	   10357.000
         20p	   10394.000
         25p	   10458.000
         30p	   10503.000
         35p	   10546.000
         40p	   10591.000
         45p	   10636.000
         50p	   10680.000
         55p	   10720.000
         60p	   10762.000
         65p	   10830.450
         70p	   10920.200
         75p	   10988.000
         80p	   11045.000
         85p	   11093.050
         90p	   11179.000
         95p	   11354.000
         Max	   11559.000
          SD	     719.818
    QuartDev	     265.000
       Range	   10565.000
      SEmean	      11.406
        Skew	      -7.030
    Kurtosis	      72.561
   Null Mean	       0.000
           t	     933.666
    prob (t)	       0.000
           F	  871731.843
    prob (F)	       0.000
  lowerbound	   10619.617
  upperbound	   10678.403
  Conf Level	      99.000
    Accuracy	       5.000
    #Samples	       9.891

and looking at newly created flows per second, which could be used for
an indication of the churn (though a ratio of new/total would probably
be better), we see:

        Mean	     119.014
      Median	     118.000
    Midpoint	     533.000
   Geometric	     117.751
    Harmonic	     116.654
         Min	      72.000
         05p	      95.000
         10p	     100.000
         15p	     103.000
         20p	     105.000
         25p	     108.000
         30p	     110.000
         35p	     112.000
         40p	     114.000
         45p	     116.000
         50p	     118.000
         55p	     120.000
         60p	     122.000
         65p	     124.000
         70p	     126.000
         75p	     129.000
         80p	     131.000
         85p	     134.000
         90p	     138.000
         95p	     145.000
         Max	     994.000
          SD	      21.936
    QuartDev	      10.500
       Range	     922.000
      SEmean	       0.348
        Skew	      17.057
    Kurtosis	     648.188
   Null Mean	       0.000
           t	     342.414
    prob (t)	       0.000
           F	  117247.042
    prob (F)	       0.000
  lowerbound	     118.118
  upperbound	     119.910
  Conf Level	      99.000
    Accuracy	       5.000
    #Samples	      73.539

Uh, the 994 is in the first second, when every flow is a new one.

Obviously, the lower the timeout, the worse the churn, but the better
(less) the amount of state that has to be kept for flows.

For the packet/byte/duration percentiles I foun:

Packets per flow:

         Sum     8017637.000
        Mean	      16.911
      Median	       3.000
    Midpoint	   27645.000
   Geometric	       3.741
    Harmonic	       2.231
         Min	       1.000
         05p	       1.000
         10p	       1.000
         15p	       1.000
         20p	       1.000
         25p	       1.000
         30p	       2.000
         35p	       2.000
         40p	       2.000
         45p	       3.000
         50p	       3.000
         55p	       4.000
         60p	       4.000
         65p	       5.000
         70p	       6.000
         75p	       7.000
         80p	       9.000
         85p	      11.000
         90p	      16.000
         95p	      36.000
         Max	   55289.000
          SD	     183.818
    QuartDev	       3.000
       Range	   55288.000
      SEmean	       0.267
        Skew	     101.596
    Kurtosis	   21242.567
   Null Mean	       0.000
           t	      63.345
    prob (t)	       0.000
           F	    4012.628
    prob (F)	       0.000
  lowerbound	      16.223
  upperbound	      17.598
  Conf Level	      99.000
    Accuracy	       5.000
    #Samples	  255781.872

Bytes per flow:

         Sum	998647024.000
        Mean	    2106.313
      Median	     218.000
    Midpoint	 2773067.000
   Geometric	     288.601
    Harmonic	     135.351
         Min	      28.000
         05p	      40.000
         10p	      56.000
         15p	      62.000
         20p	      72.000
         25p	      80.000
         30p	     111.000
         35p	     123.000
         40p	     152.000
         45p	     177.000
         50p	     218.000
         55p	     280.000
         60p	     360.000
         65p	     464.000
         70p	     592.000
         75p	     712.000
         80p	     924.000
         85p	    1267.000
         90p	    2054.000
         95p	    4916.000
         Max	 5546106.000
          SD	   26324.236
    QuartDev	     316.000
       Range	 5546078.000
      SEmean	      38.231
        Skew	      88.260
    Kurtosis	   12566.857
   Null Mean	       0.000
           t	      55.095
    prob (t)	       0.000
           F	    3035.451
    prob (F)	       0.000
  lowerbound	    2007.837
  upperbound	    2204.788
  Conf Level	      99.000
    Accuracy	       5.000
    #Samples	  338123.489

Duration of flows in seconds:

         Sum	11988958.227
        Mean	      25.287
      Median	       1.495
    Midpoint	    1991.897
   Geometric	       1.000
    Harmonic	         inf
         Min	       0.000
         05p	       0.000
         10p	       0.000
         15p	       0.000
         20p	       0.000
         25p	       0.000
         30p	       0.103
         35p	       0.391
         40p	       0.687
         45p	       1.033
         50p	       1.495
         55p	       2.115
         60p	       2.941
         65p	       3.825
         70p	       5.527
         75p	       8.251
         80p	      13.358
         85p	      24.113
         90p	      47.980
         95p	     105.344
         Max	    3983.794
          SD	     125.371
    QuartDev	       4.126
       Range	    3983.794
      SEmean	       0.182
        Skew	      15.496
    Kurtosis	     351.182
   Null Mean	       0.000
           t	     138.880
    prob (t)	       0.000
           F	   19287.555
    prob (F)	       0.000
  lowerbound	      24.818
  upperbound	      25.756
  Conf Level	      99.000
    Accuracy	       5.000
    #Samples	   53213.454

Yeah, averaging them over all kinds of traffic is, ahem, subobtimal,
and we talked to that in the flows paper, but this email is getting
too long already. Let me just attach a little of a summary of the
flows to this note, which is a next step of collapsing the flow traces
that were derived from packet traces.

I also have some visualization program that does some of this in real
time or off a file, but it requires the SGI GL library to work.

Hans-Werner

---

SDSC-ANR:MP2PPCOLL on file 'A' from 0 to 9999999 seconds

Total triplets: 474121
Total packets:  8017637
Total bytes:    998647024
Total duration: 11988958

Itemized by protocols:
Prot  Packets  Fract       Bytes Fract   Duration  Count
==== ======== ======= ========== ======= ======== =======
   1    61920 0.00772    3973752 0.00398   365314   15062
   2     2738 0.00034    1209656 0.00121     5767      25
   4     3156 0.00039     500511 0.00050     5231      15
   6  7329355 0.91415  920324014 0.92157 10178144  344113
   8      458 0.00006      15832 0.00002    11905      44
  17   619545 0.07727   72564478 0.07266  1419415  114798
  93      402 0.00005      51196 0.00005     2418      44
  94       60 0.00001       7516 0.00001      698      19
  98        3 0.00000         69 0.00000       61       1

Itemized by TCP/UDP ports:
Total number of unique port references:  107196 (for TCP)    5483 (for UDP)
Port references after <1000 collapse:     14588 (for TCP)    2107 (for UDP)
612 ports used in below 1000 range 

Find multiple (3) port references for port pairs:

Port  8001 is found paired   1081 times.
Port  8080 is found paired   1028 times.
Port  6969 is found paired    783 times.
Port  6667 is found paired    719 times.
Port  7997 is found paired    611 times.
Port  8000 is found paired    395 times.
Port  5042 is found paired    391 times.
Port  2347 is found paired    371 times.
Port  7999 is found paired    357 times.
Port  2592 is found paired    281 times.
Port  5563 is found paired    234 times.
Port  1525 is found paired    226 times.
Port  6000 is found paired    210 times.
Port  4320 is found paired    193 times.
Port  2627 is found paired    180 times.
Port  4000 is found paired    179 times.
Port  4324 is found paired    167 times.
Port  3000 is found paired    148 times.
Port  7777 is found paired    146 times.
Port  7000 is found paired    140 times.
Port  4570 is found paired    134 times.
Port  8337 is found paired    126 times.
Port  8888 is found paired    125 times.
Port  5007 is found paired    108 times.
Port  8002 is found paired    107 times.
     =====                 ======
For   2665 ports and        27008 references

Collapsing with the added information
Detailed information sorted by traffic volume (bytes):

Prt SPort DPort kPkts P-Fract  kByt   B-Fract Size  Duratn D-Fract Count C-Fract

  6    80     0   554 0.069    214197 0.214    386  976259 0.081  74510 0.157
  6    20     0   428 0.053    201852 0.202    470  454533 0.038  14685 0.031
  6     0    23  1391 0.174     57741 0.058     41 2190717 0.183  10295 0.022
  6     0    80  1047 0.131     56346 0.056     53  953368 0.080  78241 0.165
  6    70     0   144 0.018     51338 0.051    355  271574 0.023  19832 0.042
  6     0    20   791 0.099     44177 0.044     55  409140 0.034  13197 0.028
  6    23     0   420 0.052     35107 0.035     83 1066045 0.089   6839 0.014
  6     0   119   152 0.019     33882 0.034    222  209881 0.018   1021 0.002
 17    53    53   286 0.036     30795 0.031    107  913410 0.076  89208 0.188
  6     0    25   213 0.027     27817 0.028    130  357435 0.030  26492 0.056
  6    25     0   233 0.029     13466 0.013     57  314360 0.026  25720 0.054
  6   119     0   139 0.017     10890 0.011     78  132044 0.011    843 0.002
  6     0    70   214 0.027      8920 0.009     41  157641 0.013  15506 0.033
  6    21     0    88 0.011      7649 0.008     85  335623 0.028  10303 0.022
  6     0   175    34 0.004      6346 0.006    183   77865 0.006    174 0.000
  6   500     0    10 0.001      5614 0.006    559    2360 0.000     35 0.000
  6   175     0    28 0.004      5225 0.005    185   64777 0.005     70 0.000
  6     0    21   110 0.014      5131 0.005     46  327203 0.027   9206 0.019
  6  6667     0    34 0.004      4924 0.005    142   40164 0.003    222 0.000
  6     0   388    11 0.001      4088 0.004    361   10330 0.001     91 0.000
  6  6667  2199    11 0.001      2609 0.003    219    7945 0.001      2 0.000
  6  2822  6667    12 0.002      2603 0.003    210    3982 0.000      1 0.000
  6  2861  6667    14 0.002      2323 0.002    158    2816 0.000      2 0.000
  6   514     0    57 0.007      2306 0.002     40    3568 0.000     20 0.000
  6   513     0    27 0.003      2207 0.002     81   45089 0.004    384 0.001

6247 remaining port pair references were found

Sorted by reference count:
Prt SPort DPort kPkts P-Fract  kByt   B-Fract Size  Duratn D-Fract Count C-Fract

 17    53    53   286 0.036     30795 0.031    107  913410 0.076  89208 0.188
  6     0    80  1047 0.131     56346 0.056     53  953368 0.080  78241 0.165
  6    80     0   554 0.069    214197 0.214    386  976259 0.081  74510 0.157
  6     0    25   213 0.027     27817 0.028    130  357435 0.030  26492 0.056
  6    25     0   233 0.029     13466 0.013     57  314360 0.026  25720 0.054
  6    70     0   144 0.018     51338 0.051    355  271574 0.023  19832 0.042
  6     0    70   214 0.027      8920 0.009     41  157641 0.013  15506 0.033
  6    20     0   428 0.053    201852 0.202    470  454533 0.038  14685 0.031
  6     0    20   791 0.099     44177 0.044     55  409140 0.034  13197 0.028
 17  7001     0    22 0.003      1514 0.002     67   22823 0.002  11848 0.025
  6    21     0    88 0.011      7649 0.008     85  335623 0.028  10303 0.022
  6     0    23  1391 0.174     57741 0.058     41 2190717 0.183  10295 0.022
  6     0    21   110 0.014      5131 0.005     46  327203 0.027   9206 0.019
  6    23     0   420 0.052     35107 0.035     83 1066045 0.089   6839 0.014
  6   113     0     7 0.001       341 0.000     43    3766 0.000   4462 0.009
 17   123   123     7 0.001       573 0.001     76  201205 0.017   4231 0.009
  6     0   113     7 0.001       316 0.000     42    9758 0.001   4186 0.009
 17     0    53     3 0.000       190 0.000     62    4974 0.000   2007 0.004
  6     0    79     8 0.001       345 0.000     41   11165 0.001   1855 0.004
 17    53     0     2 0.000       316 0.000    122    1500 0.000   1832 0.004
  6    79     0     6 0.001       930 0.001    133   10502 0.001   1816 0.004
  6     0   119   152 0.019     33882 0.034    222  209881 0.018   1021 0.002
  6   119     0   139 0.017     10890 0.011     78  132044 0.011    843 0.002
  6     0   105     3 0.000       141 0.000     43    2340 0.000    609 0.001
  6   105     0     2 0.000       317 0.000    116    1520 0.000    540 0.001

Sorted by flow frequency and displaying ranks
Prt SPort DPort flow pkts byts secs	|	ave-pkt	ave-byt	ave-size

 17    53    53    1    7    9    5	|	3	345	108
  6     0    80    2    2    4    4	|	13	720	54
  6    80     0    3    4    1    3	|	7	2874	386
  6     0    25    4   10   10    8	|	8	1050	130
  6    25     0    5    8   11   11	|	9	523	58
  6    70     0    6   12    5   12	|	7	2588	356
  6     0    70    7    9   13   15	|	13	575	42
  6    20     0    8    5    2    6	|	29	13745	471
  6     0    20    9    3    6    7	|	59	3347	56
 17  7001     0   10   23   36   24	|	1	127	67
  6    21     0   11   15   14    9	|	8	742	86
  6     0    23   12    1    3    1	|	135	5608	41
  6     0    21   13   14   18   10	|	11	557	47
  6    23     0   14    6    7    2	|	61	5133	84
  6   113     0   15   51  121  106	|	1	76	44
 17   123   123   16   52   76   14	|	1	135	76
  6     0   113   17   54  129   36	|	1	75	42
 17     0    53   18  114  187   53	|	1	94	62
  6     0    79   19   48  119   31	|	4	186	41
 17    53     0   20  134  128  451	|	1	172	123
  6    79     0   21   58   50   33	|	3	512	134
  6     0   119   22   11    8   13	|	149	33185	222
  6   119     0   23   13   12   16	|	164	12918	78
  6     0   105   24  103  243  202	|	5	232	43
  6   105     0   25  121  127  438	|	5	587	116

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 02 Dec 1994 23:22:42 -0500

> from our earlier studies, though a little on the long site, I would

Hans-Werner,

Thanks for the very detailed data point.

Just some observations.

You list 998647024 total bytes over 3983 seconds.  That is only 2
Mb/s.  Even at that level I noticed median about 10,000 flows in cache
and a median turnover of about 120 new flows per second.  About 90% of
this was TCP.  It looks like 344113 TCP flows or 86 per second on
average at 2 Mb/s.  It would be interesting to know what the peak is.

It seems that this sample might not be "typical" since NCSA/UIUC is
one of the most heavily used WWW server sites, accounting for the 90%
TCP.  In terms of new flows per second, I don't know which way it
would be skewed if at all.

If at 2 orders of magnitude more traffic (200 Mb/s) there were 2
orders of magnitude more new flows per second (it can be argued that
transfer will also get bigger making this less), would something on
the order of 10,000 new flows per second and a cache of 1,000,000
entries be a problem.  

Currently we see <20,000 CIDR routes (total) and typically 10 route
changes per second (though peaks are much higher).  Neither of these
increase much as traffic increases.  In fact the number of routes has
remained fairly constant for the last half year due to the benfits of
CIDR aggregation while traffic almost doubled and the number of
registered networks more than doubled.

Curtis

btw- Noel- keep in mind that if 90% of this data is TCP and you are
suggesting that 90-95% of it should not require flows setup at all,
then you are suggesting that we should not be setting up flows on a
majority of TCP connections.  Just wanted to clarify that.  If so, we
are in agreement.  Some TCP will want flow setup.  For some flow setup
doesn't make sense and I'm not sure it will scale well enough anyway.