Subject: Re: flows stuff

From: [email-masked] (Noel Chiappa)
Date: Fri, 2 Dec 94 16:22:26 -0500

From: [email-masked] (Hans-Werner Braun)

First, thanks for providing some modern data!

    not able to keep up with the traffic volume and reports alot of losses per
    second ...  I all send here should be more used as indications, rather than
    reflecting all the packets exchanged.

Hmm, if we assume the drops are fairly random (bad assumption?) this shouldn't
affect things too much.

    I am using the natural Flow ID that I find in packets already, namely
    address/port quadruples for TCP and UDP traffic, and address pairs for
    the remaining stuff.

Appearance of a unique quadruple is perhaps not the best definition of a flow,
but I realize it was one that was easy to work with. (I'd have included a
minimum length, since nobody seriously suggests setting up a flow between two
hosts to exchange one packet in each direction.)

    Ok, now, measuring the flows on a second by second basis, as known to
    the "cache," we see a distribution of:

Not quite. That's total "flows"; if a "flow" exchanges only one packet in that
second, it doesn't need to be cached for good performance, but you counted it.
To calculate cache hit ratios you actually have to simulate a cache size,
replacement strategy, etc and then draw hit ratio %'s for various sizes, or
alternatively, the size you need to get a given hit %. (E.g. 99%, if you assume
misses are ten time as expensive as hits, and you want only a 10% performance
drop).

    Packets per flow:
        Mean	      16.911
      Median	       3.000

This is pretty informative right here. If half the "flows" have three packets
or less, your caclulations above as to number of flows active in any second
needs to be seen in that light. I'm not sure what I'd consider worth setting
up a flow for, but I imagine it would be *at least* 10 packets (it depends on
the overhead to set up a flow, something it's hard to guess).

(Speaking of which, explicit flows are a better cache management strategy than
something like local LRU which is based only on observation of incoming data.
If I send one packet, never intending to send another, it will be put in an
LRU cache. Since the originator has knowledge of the future, it can make a
better prediction for what is worth "caching", i.e. flow-setup, than any
switch can locally.)

         Min	       1.000
         85p	      11.000
         90p	      16.000
         95p	      36.000
         Max	   55289.000

If I'm reading this table right, 80% of all those "flows" have less than 11
packets? So, if we assume no flow setup for less than 10 packets (just cart
them around on the datagram service, in presetup flows), now we're down to
roughly 2000+ active flows per second total. (The count of DMF's is pretty
small, in the noise compared to that.) A cache to provide a reasonable hit
ratio is presumably somewhat smaller than that.

    Duration of flows in seconds:
        Mean	      25.287
      Median	       1.495
         45p	       1.033
         55p	       2.115
         65p	       3.825
         70p	       5.527
         75p	       8.251
         80p	      13.358
         85p	      24.113
         90p	      47.980
         95p	     105.344

Interesting. Given 2000+ active flows (of 10 packets or more) per second,
which is only achieved at the 85% mark of all "flows", this says flows of that
length are probably taking at least 20+ seconds or so of realtime. (This is a
guess, since there's no definite correlation between length in packets and
length in time.)

If correct, this would work out to a packet every second or so on average for
75% of those, since 75% of these flows (percentiles 85, 90, and 95) are
between 11 and 35 packets long. (I'm sure these calculations are off, since
I'm making all sorts of assumptions, but they are ballpark.)

It's hard to see how putting them in a cache is going to be useful. (I admit
it's not plausible to to figure out a strategy to keep them out of the cache,
but if the cache is sized correctly so that an slow flow is dropped in a
fraction - e.g. 25% - of the average interpacket time on slow flows, LRU
should do OK provided the cache is large enough to contain all fast flows,
plus the slow flows that are being aged. Binomial distribution of interpacket
times for flows will help this sorting process and allow reduction of the
excess cache size, of course.) This brings the flow cache down to 800 entries
or so (500 for the fast flows, assuming the top 5% of "flows" are all fast
flows, plus 300 to hold the slow flows being aged if 25% is OK).

I'll look at the detailed data some more and send conclusions to "flows".
Again, thanks very much for providing this information.

	Noel

---

From: [email-masked] (Hans-Werner Braun)
Date: Fri, 2 Dec 94 15:03:37 PST

>hosts to exchange one packet in each direction.)

Sure, that's an easy thing to add. I may just do it and resend.

>misses are ten time as expensive as hits, and you want only a 10% performance

Yeah, we had done that in a different study a year or so ago. What I
sent was just some quick and dirty stuff. Still might have the old
dusty software for the cache stuff somewhere.

>the overhead to set up a flow, something it's hard to guess).

Right, part of the issue here is to find ways to discriminate between
worthy flows and stuff that should be dumped onto some background
traffic substrate. The "minimum packet" filter you mentioned earlier
may be a good first step.

>better prediction for what is worth "caching", i.e. flow-setup, than any

Sure, I agree. This is just exploratory as to what the profile of the
current traffic is. There are other applications for this stuff,
though, like network accounting, for example, where you can just
create a pile of buckets.

When we started working on this, "flows" was more of a detail, we were
more interested in generally understanding traffic signatures. Flows
are just a model to do so.

>packets? So, if we assume no flow setup for less than 10 packets (just cart

Yes.

>ratio is presumably somewhat smaller than that.

I can try to reduce the data set to flows 10+ packets.

Hans-Werner

---

From: [email-masked] (Hans-Werner Braun)
Date: Fri, 2 Dec 94 15:19:10 PST

This is no based only on the subset of the flows that have at least 10
packets:

SDSC-ANR:MP2PPCOLL on file 'A10plus' from 0 to 9999999 seconds

Total triplets: 88095
Total packets:  6796475
Total bytes:    858394464
Total duration: 8626549

Itemized by protocols:
Prot  Packets  Fract       Bytes Fract   Duration  Count
==== ======== ======= ========== ======= ======== =======
   1    35373 0.00520    2315122 0.00270   196089     488
   2     2723 0.00040    1209164 0.00141     5767      10
   4     3148 0.00046     499896 0.00058     5216      14
   6  6305670 0.92779  798125069 0.92979  7809233   85777
   8      334 0.00005      11382 0.00001     8709      11
  17   448957 0.06606   56207563 0.06548   599590    1785
  93      247 0.00004      21671 0.00003     1915       9
  94       23 0.00000       4597 0.00001       24       1

Itemized by TCP/UDP ports:
Total number of unique port references:   43339 (for TCP)     250 (for UDP)
Port references after <1000 collapse:      3595 (for TCP)     208 (for UDP)
181 ports used in below 1000 range 

Find multiple (3) port references for port pairs:

Port  6667 is found paired    251 times.
Port  6969 is found paired    213 times.
Port  8080 is found paired    185 times.
Port  2592 is found paired    176 times.
Port  6000 is found paired    130 times.
Port  4000 is found paired    124 times.
Port  8001 is found paired     99 times.
Port  7997 is found paired     85 times.
Port  4570 is found paired     82 times.
Port  5555 is found paired     77 times.
Port  8888 is found paired     69 times.
Port  8000 is found paired     58 times.
Port  3000 is found paired     49 times.
Port  5000 is found paired     46 times.
Port  2347 is found paired     44 times.
Port  9000 is found paired     37 times.
Port  7000 is found paired     30 times.
Port  7777 is found paired     30 times.
Port  7999 is found paired     29 times.
Port  1701 is found paired     28 times.
Port  3500 is found paired     28 times.
Port  9999 is found paired     25 times.
Port  2222 is found paired     25 times.
Port  5190 is found paired     23 times.
Port  2001 is found paired     23 times.
     =====                 ======
For    536 ports and         4398 references

Collapsing with the added information
Detailed information sorted by traffic volume (bytes):

Prt SPort DPort kPkts P-Fract  kByt   B-Fract Size  Duratn D-Fract Count C-Fract

  6    20     0   391 0.058    195196 0.227    498  368719 0.043   3193 0.036
  6    80     0   322 0.047    159544 0.186    495  398073 0.046  10451 0.119
  6     0    23  1380 0.203     57221 0.067     41 2129981 0.247   6942 0.079
  6     0    20   762 0.112     42846 0.050     56  375801 0.044   3921 0.045
  6    70     0    84 0.012     41507 0.048    490   83320 0.010   2262 0.026
  6     0    80   785 0.116     36708 0.043     46  529655 0.061  22074 0.251
  6     0   119   151 0.022     33794 0.039    223  206632 0.024    658 0.007
  6    23     0   408 0.060     33510 0.039     81  965567 0.112   3582 0.041
  6     0    25   117 0.017     18678 0.022    158  136916 0.016   7500 0.085
 17    53    53   160 0.024     18304 0.021    114  363487 0.042   1347 0.015
  6  6667     0    76 0.011     10958 0.013    144   94597 0.011    131 0.001
  6   119     0   137 0.020     10827 0.013     78  129790 0.015    548 0.006
  6    25     0   152 0.023      8550 0.010     55  159336 0.018   9549 0.108
  6     0    70   158 0.023      6411 0.007     40   85965 0.010   2869 0.033
  6     0   175    34 0.005      6327 0.007    184   76436 0.009     76 0.001
  6     0  6667    70 0.010      5869 0.007     82   72624 0.008    132 0.001
  6   500     0    10 0.001      5609 0.007    560    2310 0.000     26 0.000
  6   175     0    28 0.004      5205 0.006    184   64237 0.007     51 0.001
  6    21     0    59 0.009      4752 0.006     79  240196 0.028   2809 0.032
  6     0   388    11 0.002      4050 0.005    362    8525 0.001     53 0.001
  6     0    21    86 0.013      4030 0.005     46  266393 0.031   3555 0.040
  6  6667  2199    11 0.002      2609 0.003    219    7945 0.001      2 0.000
  6  2861  6667    14 0.002      2323 0.003    158    2816 0.000      1 0.000
  6   514     0    57 0.008      2305 0.003     40    3558 0.000     14 0.000
  6   513     0    26 0.004      2160 0.003     80   40381 0.005    191 0.002

2295 remaining port pair references were found

Sorted by reference count:
Prt SPort DPort kPkts P-Fract  kByt   B-Fract Size  Duratn D-Fract Count C-Fract

  6     0    80   785 0.116     36708 0.043     46  529655 0.061  22074 0.251
  6    80     0   322 0.047    159544 0.186    495  398073 0.046  10451 0.119
  6    25     0   152 0.023      8550 0.010     55  159336 0.018   9549 0.108
  6     0    25   117 0.017     18678 0.022    158  136916 0.016   7500 0.085
  6     0    23  1380 0.203     57221 0.067     41 2129981 0.247   6942 0.079
  6     0    20   762 0.112     42846 0.050     56  375801 0.044   3921 0.045
  6    23     0   408 0.060     33510 0.039     81  965567 0.112   3582 0.041
  6     0    21    86 0.013      4030 0.005     46  266393 0.031   3555 0.040
  6    20     0   391 0.058    195196 0.227    498  368719 0.043   3193 0.036
  6     0    70   158 0.023      6411 0.007     40   85965 0.010   2869 0.033
  6    21     0    59 0.009      4752 0.006     79  240196 0.028   2809 0.032
  6    70     0    84 0.012     41507 0.048    490   83320 0.010   2262 0.026
 17    53    53   160 0.024     18304 0.021    114  363487 0.042   1347 0.015
  6     0   119   151 0.022     33794 0.039    223  206632 0.024    658 0.007
  6   119     0   137 0.020     10827 0.013     78  129790 0.015    548 0.006
  6     0   513    38 0.006      1588 0.002     41   40458 0.005    213 0.002
  6   513     0    26 0.004      2160 0.003     80   40381 0.005    191 0.002
  6     0  6667    70 0.010      5869 0.007     82   72624 0.008    132 0.001
  6  6667     0    76 0.011     10958 0.013    144   94597 0.011    131 0.001
  6     0  4000    21 0.003       899 0.001     42   64301 0.007     97 0.001
  6     0  6000    14 0.002      1572 0.002    104   13783 0.002     93 0.001
  6     0    79     1 0.000        48 0.000     40    2031 0.000     92 0.001
  6     0  6969     3 0.001       157 0.000     41    8067 0.001     90 0.001
  6  6000     0    23 0.004      1545 0.002     64   12728 0.001     84 0.001
  6     0   175    34 0.005      6327 0.007    184   76436 0.009     76 0.001

Sorted by flow frequency and displaying ranks
Prt SPort DPort flow pkts byts secs	|	ave-pkt	ave-byt	ave-size

  6     0    80    1    2    6    3	|	35	1662	47
  6    80     0    2    6    2    4	|	30	15265	495
  6    25     0    3    9   13   11	|	16	895	56
  6     0    25    4   12    9   12	|	15	2490	159
  6     0    23    5    1    3    1	|	198	8242	41
  6     0    20    6    3    4    5	|	194	10927	56
  6    23     0    7    4    8    2	|	114	9355	82
  6     0    21    8   13   21    8	|	24	1133	47
  6    20     0    9    5    1    6	|	122	61132	498
  6     0    70   10    8   14   15	|	55	2234	41
  6    21     0   11   17   19    9	|	21	1691	80
  6    70     0   12   14    5   16	|	37	18349	491
 17    53    53   13    7   10    7	|	118	13588	114
  6     0   119   14   10    7   10	|	229	51358	223
  6   119     0   15   11   12   13	|	251	19758	78
  6     0   513   16   19   31   21	|	181	7458	41
  6   513     0   17   22   25   22	|	139	11310	81
  6     0  6667   18   16   16   18	|	536	44463	83
  6  6667     0   19   15   11   14	|	580	83654	144
  6     0  4000   20   26   51   19	|	219	9274	42
  6     0  6000   21   29   32   31	|	161	16904	105
  6     0    79   22  273  509  219	|	13	526	40
  6     0  6969   23   85  220   41	|	42	1752	41
  6  6000     0   24   23   33   32	|	284	18394	65
  6     0   175   25   20   15   17	|	450	83256	185