Subject: Re: Don't let Delay eat up all TOS bits.

From: Ed Ellesson <[email-masked]>
Date: Tue, 02 Dec 1997 14:06:43 -0500

Hyogon, 

Sorry for delay in responding to this.  I had to collect some data from
some people who were on vacation in order to give you some numbers. 

At 10:56 PM 11/19/97 -0500, you wrote:
>
>thanks for reminding me of his posting. Yes, it might be a 'successful'
>example for use of multiple levels of priorities. But I wonder how much
>successful example is that for Diff-Serv Internet? (Frankly I am not as old
>a guy as to know SNA well enough to argue anything for or against it ;-)
>Maybe Mr. Ed Ellesson could tell us how big was his SNA network in terms
>of the diameter (propagation delay, number of hops, etc.), 
>number of customers, how scalable is the method to distribute bandwidth
>to priorities, what was the difference in delays between priorities that 
>so predictably set priorities apart? From a sketchy reading, I couldn't see 
>any numbers there which apparently convinced you to believe that
>that example's plausible enough to work in Internet as well.

Some numbers regarding SNA networks, compiled by Tom Nolle of CIMI Corp (a
consultant who works with alot of our customers:

-Number of SNA networks: over 40,000 worldwide (I am told that this is just
the mainframe-style SNA nets, and that it does not include AS/400 SNA nets,
but I have not yet confirmed that....there are alot of those also.)

-Largest single-enterprise SNA network: something like 1 million devices,
with 435,000 active sessions simultaneously.  

-Experience with deployed SNA networks using queuing precedence:  approx.
20 years (There are still a few folks around here who remember the first
shipments, and they tell me it was in the mid-70's, with the queuing
priority introduced sometime in the late 70's.).  

-Ongoing experience: 4.7 million SNA clients shipped in 1995
                     5.38 million SNA clients shipped in 1996

-Regarding how big they are:  many SNA corporate nets are worldwide...I
don't have the numbers at hand.  Even more are continent-wide.   I am told
that most SNA sessions are probably three "hops", and many are four, but
remember this is at the SNA level, and does not take into account the
routers that are traversed when SNA is encapsulated in IP using TN3270 or
Data Link Switching, and run over a corporate IP backbone.  

So, when I say "hops" in the context of SNA, I mean links traversed among
host 
processors, communication controllers, cluster controllers and workstations.

It often looks like this: 

                  a                         b                       c 
mainframe (vtam)-----Comm Controller (ncp)------cluster controller-----work
stn
            |  |   e                   |
          d |   -------             f  |
            |     g    |               |    h                       i  
mainframe (vtam)-----Comm Controller (ncp)------cluster controller-----work
stn

(If my ascii art didn't come thru very well, let me know, and I will re-do
it).

The mainframe SNA implementation is called "VTAM" (Virtual
Telecommunications Access Method).  The communication controllers run "NCP"
(Network Control Program), as their SNA implementation.

So a three-hop session might go a-b-c, or g-h-i, or e-h-i, for example.

A four-hop session, might go d-g-h-i, or a-f-h-i, and so on. 

This is just a small example to illustrate the hop concept. A large corporate 
network would have many more mainframes, network nodes and workstations, of
course.     

When running over an IP backbone, you can replace any or all of these SNA
links with an IP network consisting of as many router hops as it takes to
get across that network.  Using the IBM internal network, I am told it is
not unusual to traverse eight router hops going from a LAN interface on one
site to a LAN interface at another site.    

As I mentioned in earlier postings, SNA uses four priority levels. By
default, the network control flows run at the highest priority.  Most SNA
networks use at least two additional priorities: "high", for interactive
traffic, and "low", for batch traffic, like print jobs, for example.  I am
told that the majority of users run their SNA networks with these three
priorities, but that maybe 25% use all four priorities (this is an educated
guess, based on experience with customers' networks, but not a hard number).  

The multi-level delay priority has been found to be most important on slow
speed lines (eg. 56kb).  In some commercial networks, especially banking
and finance, where response times of 2 seconds or less are very important,
turning off the queuing priority has proven disastrous in the past.
Response times went to multiple tens of seconds, and customer transactions
could not be entered fast enough to keep up with demand, and business was
lost as a result. Sometimes network administrators not only lose their
bonuses, sometimes they lose their jobs over incidents like that. 

At this point I would like to stress that in addition to the question of
how many delay preferences (and thus, how many queues) are required in
large corporate networks, we also have the question of bandwidth
allocation.  Many of our customers run their SNA traffic encapsulated in IP
over an IP backbone, and more would like to do so.  However, since the SNA
traffic is often very critical to the business, they often run not only a
multi-level delay preference scheme, they also allocate bandwidth in the
routers to each traffic type, to make sure that a bandwidth allocation is
always available to a traffic type, regardless of delay preference (that
is, the leaky bucket for a particular queue must be serviced at least at
some particular rate, if there is traffic to be sent from that queue.)
This way, I always have some bandwidth available for even relatively low
priority traffic types, and I avoid starvation of the lower priority queues.  

Since now I not only have to allocate a delay priority to a traffic type,
but a bandwidth allocation as well, I need more granularity than just a
couple of bits for delay, since I may want to carve up the bandwidth
according to some association with not only the traffic type, but who is
paying for it, ala CBQ.

Since customers are already using and configuring their networks to run
this way by keying off the port numbers to identify the traffic type, and
since the port numbers will soon be encrypted for commercially significant
traffic running over public and private internets,  it seemed a no-brainer
to work towards a mechanism using the TOS Byte, to direct these traffic
types to specific queues in the router, that could be managed for both
delay and bandwidth allocation.  That's why we proposed the 4-bit Explicit
Service Class in our draft, at:  

http://diffserv.lcs.mit.edu/Drafts/draft-ellesson-tos-00.txt 

It's a very flexible approach that preserves what routers are already doing
in corporate networks, only using a different set of fields, since the set
they are using now will no longer do the job with IPSEC covering up the
port numbers. With the service class approach, a customer does not have to
do anything different functionally than he or she is doing today, and the
routers don't have to do anything different, other than look at a different
field to make their queuing decision.  This is not that hard!  

If someone wants to get fancier and implement the dropping schemes
discussed on this list, that can also be accomodated by our proposal.  So
you preserve what you have now, and you can go beyond it to prove the
theories that are being proposed on this list.  What's wrong with that?  

I'm afraid I don't understand the resistance to this proposal...what is the
downside?  

>In fact, you bring up an important issue here. Are we talking about
>_relative ordering_ of delay priorities or _definite numbers_ attached to
>the priorities? (i.e. it's about the semantics of the delay bits)
>I doubt you are talking about definite numbers, like 1ms for priority
>1 and 10ms for priority 2, etc, considering for instance varying propagation
>delays. If we are talking about the relative ordering, and we are selling 
>some delay profiles to customers, you think you can sell more than
>2 delay priorities in the Internet? How do the customers know you are
>not cheating when the "feels" between delay priorities are not set apart
>far enough? Relative ordering says nothing about the quantity of the gap,
>does it? And that's where the "feel" factor comes in, and I am not 
>as much convinced as you that you can create more than 2 "feel"s in the
>soft-guarantee Diff-Serv Internet. I wonder how you can convince a 
>subscriber of the worthiness of the 0.1ms difference his additional $5 
>has made (although the bigger problem is not the possibly small values of
>difference but it's the _unpredictability_ in the value from subscribers'
>point of view), when Joe Schmo in the neighbor who bought higher delay 
>profile just doesn't notice that huge difference in delay.
>
>>Regarding what Dave Clark said earlier on the 3 different "feels"
>>of delay, I am sorry, but I would prefer to see some numbers. Were
>>they simulation-based results? If yes, for what kind of network traffic-load
>>are we talking about? A dozen of TCP connections or several thousands of
>>flows in a backbone/NAP router? 
>
>I really hate to quote many times what other people said without consent,
but 
>with appropriate apologies, 
>
>:I don't think that we can construct more than about two levels of 
>:best effort service that have a significantly different
>:"feel". We went through the same exercise with real time. Initially, we
>:proposed to have three levels of real time service, that would be
>:distinguished by the "feel" of the delay. The delays had to be
>:substantially different, to make the distinction worth having, and after
>:we screwed around with a range of offered loads and queue service
>:disciplines, we finally realized that if we forced the three levels of 
>:delay far enough apart so that they really seemed different, the bottom 
>:two were worse than best effort. So we bagged the whole idea, and just 
>:went for the very simple, single level "controlled load". I suspect that 
>:the same will prove true for best effort service as well -- I doubt that 
>:you can pull the traffic apart into more than two queues, and still end up 
>:with queues that actually have much of a difference in the serice they
provide.
>
>My understanding was that he was summarizing the experience of the
>_Int-Serv group as a whole_. I was not following up Int-Serv activities
>until lately, so please correct me, Int-Serv gurus, if I am misinterpreting 
>something here. Actually, I will be grateful if some of you could
>answer Constantinos's questions since I also wonder what exactly
>baffled you.
>
>>Obviously, you will not get much delay "feel"
>>difference if there is no persistent queuing in the router. Also
>
>Exactly, plus varying propagation delays. Not many feels, definitely. 
>This was my point.
>
>>The fact that something has been
>>said by a person that everybody in this field  respects in the
>>maximum degree does not mean that it is necessary always true :-)
>
>You got me there, Constantinos, chalk one up for you. I guess the good
>old days of free-riding on giants' shoulders is over for me. ;-)
>
>
>>
>>Constantinos
>>
>>
>
>
>-- Hyogon
>---------------------------------------------------------------------------
---
>Hyogon Kim ([email-masked])	       		       |        (201) 829-4728
>Research Scientist				       |           MCC 1B-225B
>Internet Architecture Research Laboratory	       |      445 South Street 
>Bell Communications Research			       |  Morristown, NJ 07960
>---------------------------------------------------------------------------
---

Regards, 

Ed Ellesson
Sr. Engineer, TCP/IP Technology Management
IBM Networking Software Products
Research Triangle Park, NC 
[email-masked]
919-254-4115

---

From: "Alessio Casati" <[email-masked]>
Date: Wed, 3 Dec 1997 18:21:07 +0100

Ed,

Thanks for the enlightening details about IBM SNA networking experience.
I have some comments and questions I would ask you in order
to let me clear some doubts.

You say there are many thousands of SNA networks out there working fine using
queueing priority. I assume every single organisation has its own network operation center
deciding which kind of traffic to place in which class. Let's suppose every sender in the SNA
net decides autonomously in which class to place its own current traffic (much resembling what is 
likely to happen in the internet). What do you expect to happen?

I would also know if there exist statistics about the distribution of traffic over 
the different delay classes

-network control
-high priority
-medium priority
-low priority 
-routine

As far as I could understand from your message, there seems to be clustering in the
high priority and low priority classes.

 > guess, based on experience with customers' networks, but not a hard number).  

TIA

alessio