Subject: Traffic Distribution (was: Re: Don't let Delay eat up all TOS bits.)

From: Ed Ellesson <[email-masked]>
Date: Wed, 03 Dec 1997 15:06:15 -0500

Alessio, 

Sorry, I forgot to respond to the rest of your note.

At 06:21 PM 12/3/97 +0100, Alessio Casati wrote:
>
>I would also know if there exist statistics about the distribution of
traffic over 
>the different delay classes
>
>-network control
>-high priority
>-medium priority
>-low priority 
>-routine
>
>As far as I could understand from your message, there seems to be
clustering in the
>high priority and low priority classes.
>
> > As I mentioned in earlier postings, SNA uses four priority levels. By
> > default, the network control flows run at the highest priority.  Most SNA
> > networks use at least two additional priorities: "high", for interactive
> > traffic, and "low", for batch traffic, like print jobs, for example.  I am
> > told that the majority of users run their SNA networks with these three
> > priorities, but that maybe 25% use all four priorities (this is an
educated
> > guess, based on experience with customers' networks, but not a hard
number).  

Yes, for alot of our customers, there is a clustering of traffic into high
and low priority, however, the network control traffic can also be
significant for transient periods, like, for example, during times of link
or node failure and recovery.  The actual distribution of how much is high
and how much is low varies all over the place.  In some of our networks,
our customers size the network (that is, provision the link bandwidth) only
according to the peak high priority interactive traffic load.  This saves
them money.  Obviously, this would not work for the public Internet.  Other
customers size their links to accomodate both high and low, but when there
is congestion, the high priority goes first.  As I mentioned above the
medium priority gets used by some customers, but I do not have a feel for
how much traffic uses that priority value.

Forget about "routine".  That was a mistake in my first posting on this
subject, which I subsequently corrected in a later posting. 

We are proposing that the routers be configurable such that bandwidth
allocation by class of service can prevent starvation of the lower priority
queues.  We think this is key for the public internet, and also reflects
what routers are already capable of doing; its just that today they look at
the TCP/UDP header to figure out the service class, instead of the TOS byte.  

Regards, 

Ed Ellesson
Sr. Engineer, TCP/IP Technology Management
IBM Networking Software Products
Research Triangle Park, NC 
[email-masked]
919-254-4115

---

From: "Alessio Casati" <[email-masked]>
Date: Thu, 4 Dec 1997 11:04:03 +0100

Ed,

your useful and precise replies confirm what most urgent needs are and
what the most immediate goals should be.

we should basically

* let providers control the amount of traffic users are sending.

* concentrate on the definition of a two priority based
  service model, as Van proposes and SNA experience confirms, since
  medium priority (or any other additional level) doesn't seem to be much
  of use right now.

alessio

---

From: Steven L Blake <[email-masked]>
Date: Thu, 04 Dec 1997 11:39:03 EST

Alessio,

  I agree that the most expedient step for the WG is to standardize 
some reasonably simple SERVICE DEFINITION like the proposal in 
draft-nichols-diff-svc-arch-00.txt as a first cut for public Internet
deployment.  However, I think another goal of the WG should be to quickly
close on a SERVICE ENCODING (TOS/Class-byte layout) that can accommodate
new service definitions efficiently, and that can support experimentation
and network-operator customization in a multivendor-interoperable way.

  Looking at draft-nichols-diff-svc-arch-00.txt more closely, you will
observe that each packet can be associated with one of three states:

  P  A       State
  -  -       -----
  0  0       Assured - out-of-profile (normal best-effort)
  0  1       Assured - in-profile     (better-than best-effort)
  1  x       Premium  (in-profile)    (leased-line replacement)

A three-state system can't be encoded in less than two bits, so their
proposed encoding is maximally efficient.

  Now suppose that, after experimentation and deployment experience, we
conclude that there are a few more services we would like to standardize
which would allow the service provider to pack his network more 
efficiently and maximize revenue while satisfying customer requirements.
For example, suppose we choose to define the following services:

A - Premium 1 (peak-rate limited, conservative admission)
B - Premium 2 (conformance-marked, conservative admission)
C - Assured 1 (conform.-marked, high assurance - less conserv. admission)
D - Assured 2 (conform.-marked, medium assurance - much less conserv. adm.)
E - Network Control
  
We'll assume that the two premium classes are mapped into their own 
provisioned queues, and that the two assured classes are implemented via
three levels of drop-preference within a single queue.  Although this
should remain implementation-dependent, we have the requirement to mandate
that packets not get re-ordered within a service as a flow goes in-and-
out-of-profile.  We won't specify how network control traffic is queued;
only that it has a low loss probability and some minimum bandwidth
allocation and is always in-profile (e.g., it could be queued in the
Assured queue with low-drop preference or with pushout).

  Each packet received can be associated with one of the following
seven states:

S1 - Premium 1       - in-profile
S2 - Premium 2       - in-profile
S3 - Premium 2       - out-of-profile
S4 - Assured 1       - in-profile
S5 - Assured 2       - in-profile
S6 - Normal           (Assured 1,2 out-of-profile)
S7 - Network Control - in-profile

Seven states can be encoded with 3 bits.  One way that you don't want
to encode is to assign a bit A/B/C/D/E for each service as an indicator;
too inefficient.

  Now, how do you encode these services with drop-preference and delay-
priority bits (where we assume that each delay priority is mapped to a
separate queue)?  We'll look at two possible choices: 1-bit drop pref,
2-bit delay prio; 2-bit drop pref, 1-bit delay prio:

  drop pref(1)   delay prio(2)      State    drop pref(2)   delay prio(1)
  ---------      ----------         -----    ---------      ----------
      0              00              S1         00              0
      0              01              S2         00              1   *
      1              01              S3         01              1
      0              10              S4         10              1
!     0              11              S5         11              1
      1              11              S6         --              -             
%     1              10              S7         --              -

(*) we need to isolate Premium 1 and 2 into separate queues to provide the
    necessary level of service assurance for Premium 1

(!) we are mapping Assured 2 in/out-of-profile to a separate queue from
    Assured 1, although this could lead to potential re-ordering for
    Assured 1 in/out-of-profile

(%) we map Network Control to the Assured 1 queue, out-of-profile, even
    though this could violate the Network Control service requirement

  As you may see, the service definitions do not map at all to a mapping
with two drop-preference bits and one delay-priority bit, and don't map
well to a mapping with one drop-preference bit and two delay-priority bits
(there are other alternative mappings I haven't shown).

  Suppose instead that we make a minor change from the 1 drop/2 delay
mapping, and assume that the field we were calling delay priority is really
a service class field, which can take arbitrary semantics (i.e., as either
a queue identifier, drop-preference, or a delay-priority).  Then we
could use the following assignment:

   drop pref(1)    service class(2)   State
   --------        -------------      -----
      0              A - 00            S1
      0              B - 01            S2
      1              B - 01            S3
      0              C - 10            S4
      0              D - 11            S5
      1              C or D            S6
      ?                ?               S7

Oops!  We can't cleanly specify Network Control (S7).  We really need to
associate each state with a separate service class code point if we
only have three bits available.

  In draft-ellesson-tos-00.txt you'll notice that we proposed a four-bit
service class field and a one-bit drop-preference/conformance field.
Looking at the services we could envision for the future, we realized that
it was quite common that a service would benefit from a single-bit
conformance indicator, but that few would benefit from a multi-bit
conformance/drop-preference indicator (and in those cases, multiple
service class code points could be assigned).  We could support more
possible service definitions if we extended the service class field to
five bits, and eliminated the drop-preference bit, but we chose to
maintain some semblance of backwards compatibility with the current
TOS-field definition in RFC 1349.

  We really feel that the rationale presented in RFC 1349, Appendix B.2,
still holds, and that the service class concept in 
draft-ellesson-tos-00.txt is the most future-proof approach for
differentiated service encoding.

Regards,
/*=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
| Steven L. Blake                 [email-masked] |
| Internetworking Technology         (919)254-2030 (work) |
| IBM Networking Hardware Division   (919)254-5483 (fax)  |
| "You like the 'Net?  You ain't seen nothin' yet!"       |
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=*/

---

From: Juha Heinanen <[email-masked]>
Date: Fri, 5 Dec 1997 09:00:17 +0200

if i have understod vans paper correctly, there is no such thing as
out-of-profile premium traffic, since you drop the out-of-profile
premium packets.

i have the feeling that whole thing is getting far too complicated.  we
should start with a very simple scheme and experiment with it a year or
two and then make the final decisions.  so i propose that we start with
one dp bit and one delay bit.

i don't see why network control data could not be sent as low delay
in-profile packets.  if it turns out that one delay bit is not enough,
we can reserve another bit for that purpose. 

what comes to van's premium stuff,  we could tentatively reserve one bit
for that  too, and then see  how many vendors  actually implement it and
how many isps select that kind  of service.  it must  be the market, not
the ietf, that decides if premium service  needs to be implelemted  in
every router.

-- juha

---

From: Paul Ferguson <[email-masked]>
Date: Fri, 05 Dec 1997 07:19:37 -0500

>one dp bit and one delay bit.

Well, there is too complicated, and then there is too simplistic. ;-)

I'm of the opinion that one delay bit and one drop preference bit
is wholly inadequate. The suggestions in draft-ferguson-delay-drop-00.txt,
methinks, is not complicated, but goes beyond simplistic 1-bit
differentiation.

We'll talk more about this next week, I'm sure.

- paul

---

From: "Alessio Casati" <[email-masked]>
Date: Fri, 5 Dec 1997 12:02:12 +0100

On  5 Dec 97 at 9:00, Juha Heinanen wrote:

[...]
> so i propose that we start with one dp bit and one delay bit.

That's exactly what the joint Premium/Assured service model requires
What is questionable is whether the in/out concept may be extended to
the low delay queue or not. Van claims it makes no sense, and I admit
i find difficult to argument against such a claim, since low delay implies
short queues and I wonder how could we engineer proper values for
out low delay packets thresholds. 

> we can reserve another bit for that purpose. 

Network control  is certainly to be properly treated. I dont know if giving it low delay
is the best we can do if we don't want to impact negatively on low delay service (what
is the point of ingress shaping if great perturbations can stem from network
control?).  Giving it low dp might be another option.
 
> the ietf, that decides if premium service  needs to be implelemted  in

Perfectly agree with you here, including on the need to reserve right now a bit
for Van premium service (that is equivalent to the low delay bit, IMO).

As a final remark, support for existing diffservices (e.g. SNA priority levels) may be
regarded as adding value to the work being done. Not at the top of the actions list,
and only if it doesn't play against easy deployment of what's the main goal of
diffserv IMO.
Support for intserv aggregation and leaving some reseved bits for future use would be
a great plus.

alessio

---

From: Juha Heinanen <[email-masked]>
Date: Fri, 5 Dec 1997 17:19:29 +0200

no it doesn't since there is no out-of-profile traffic in the premium
service.

since low delay traffic needs short queues, the threshold needs to be
set to a low value.  consider for example an application where a
customer figures out that 90 % of the time he doesn't have more than 10
simultaneous voip calls from his site and subscribes to low delay
assured service accordingly.  occasionally when there is more than 10
calls those don't get blocked immediately at the ingress, but are
transmitted with high dp.  it would not make sense to prohibit this kind
of application.

   > the ietf, that decides if premium service  needs to be implelemted  in

   Perfectly agree with you here, including on the need to reserve right
   now a bit for Van premium service (that is equivalent to the low delay
   bit, IMO). 

as i argued above, van's premium bit must be separate from the two
assured bits. i don't udnersdtand how van can write his paper without
even mentioning this issue, i.e., what is the relationship of his bit to
the delay bit of assured service.  may be he is not reading this list?

-- juha

---

From: [email-masked] (Hyogon Kim)
Date: Fri, 5 Dec 1997 10:37:26 -0500 (EST)

Agreed. There are at least four different proposals for the use 
of the 8bits now (Priority, Drop Preference, Delay, and ECN). 
Even with minimalist approach, half of the space is gone. 
If each asks for double the number of bits (i.e. 2), voila, it's full. 
There is no one who doesn't know more bits bring finer granularity
in control, but we just can't give out the bits based on that
obvious claim since we do not know which among them will be more 
important, or even we don't know what other services we will need later.
(This is nothing against ECN but did we know when we started
this thread that it will come into our consideration? Maybe Sally but...)
I go with the gradual approach with minimal bit allocations. IMHO
it's not at all late to add more bits after we see how they work out 
in real diff-serv Internet.

>the ietf, that decides if premium service  needs to be implelemted  in

-- Hyogon
------------------------------------------------------------------------------
Hyogon Kim ([email-masked])	       		       |        (201) 829-4046
Research Scientist				       |           MCC 1G-231B
Internet Architecture Research Laboratory	       |      445 South Street 
Bell Communications Research			       |  Morristown, NJ 07960
------------------------------------------------------------------------------

---

From: [email-masked] (Hyogon Kim)
Date: Fri, 5 Dec 1997 10:49:18 -0500 (EST)

Paul,

>Well, there is too complicated, and then there is too simplistic. ;-)

There is nothing too simplistic here. Maybe 1 is a simplistic number,
but the effect of differentiation using even that one bit is never
that simple. Take RIO with one bit. If you expected that it would
show a simple and straightforward differentiation behavior, sorry,
it's complicated enough to write bunch of papers (I am writing a 
couple already :-) about it. I think what Juha was saying is that we
shouldn't throw in more complexity before we know sufficiently about 
"simplistic" differentiation.

>We'll talk more about this next week, I'm sure.

Yes.

-- Hyogon

---

From: Paul Ferguson <[email-masked]>
Date: Fri, 05 Dec 1997 11:12:59 -0500

I guess I must be missing something -- what the heck are you
wanting to save the bits for?

We have a problem to solve, and if there is consensus that
we need the whole byte (or most of it or whatever), then by
golly, we should use the whole byte, or whatever we agree
that is needed to get the job done. Let's not automatically
take a minimalist approach just because it seems like the
right thing to do -- if it doesn't adequately address the
problem, then it is all for naught.

As I have said on more than one occassion, I am willing to
be flexible on the number of bits for both drop preference
*and* delay indication, but 1 bit is certainly too small,
and I believe that 2 bits are inadequate as well, for both.

- paul

---

From: Paul Ferguson <[email-masked]>
Date: Fri, 05 Dec 1997 11:14:01 -0500

>"simplistic" differentiation.

Well, then -- I look forward to reading your paper so that I can
refute it.  ;-)

- paul

---

From: "Alessio Casati" <[email-masked]>
Date: Fri, 5 Dec 1997 17:17:12 +0100

> no it doesn't since there is no out-of-profile traffic in the premium

Juha, I know that, but as far as the required # of bits is concerned we have
no further need... You will agree with me we will not want a two delay classes assured
service to be deployed together with a Premium service...

> set to a low value. 

I'm afraid it would be so low a value to make hard to justify the need of the related computational
burden. And Anyway, since the two bits will be there, you can experiment and make the 
right choice!!!

> consider for example an application where a

I don't want to think to any particular application or service now. You would certainly be able to
find at least an application which is not favoured by a particular diffserv setting...
Anyway, as I told, two bits are there, both approaches can be encompassed.

> assured bits.

Here we disagree. We have to make a choice, IMO.

bye

alessio

---

From: Steven L Blake <[email-masked]>
Date: Fri, 05 Dec 1997 11:16:01 EST

> out-of-profile premium traffic, since you drop the out-of-profile

That's right; no one has said otherwise.  It's conceivable that a
lower grade of premium (less octane?) service could be defined which 
offers better delay that Assured, but whose non-conformant packets are
marked instead of dropped (and serviced in a queue with a low drop 
threshold).

> one dp bit and one delay bit.

It's a good sentiment to start with a simple service architecture as a
first cut.  But we need to close on the layout of the TOS byte RSN.  Fast
routing is moving into CMOS state machines, and it is going to become
increasingly difficult to deploy new TOS semantics in the future, unless we
adopt a mapping with general semantics today and encourage routing ASIC
designers to treat the TOS byte (or some sub-field) as a lookup key into a
configurable table which specifies the queue id, route selector, buffer
management policy, discard priority, FECN on/off, etc.

> we can reserve another bit for that purpose.

Because the amount of low delay traffic will presumably be regulated near
the source, and the amount of network control traffic will not.  Besides,
network control traffic will often be bursty TCP which could overrun a 
small queue.

Specifying another bit to indicate network control, in parallel with
a delay bit and a drop bit, is just a wasteful bit allocation policy (since
the other two bits are don't-cares when the network control bit is set).
Much better to use a code point within a field, ala RFC 1349.

> the ietf, that decides if premium service  needs to be implelemted  in

You've already used four bits to specify six states! (delay 0 in, delay 0 
out, delay 1 in, delay 1 out, network control, premium).  What a waste!

Regards,

---

From: "Alessio Casati" <[email-masked]>
Date: Fri, 5 Dec 1997 17:44:21 +0100

On  5 Dec 97 at 18:20, Juha Heinanen wrote:

> or let the market make the choise, i.e., does the delay bit belong to
> assured or premium service if we cannot afford to have a separate bit
> for van.

Here we agree :-). But the choice sooner or later will have to be
standardized.

alessio

---

From: Juha Heinanen <[email-masked]>
Date: Fri, 5 Dec 1997 18:20:47 +0200

-- juha

---

From: [email-masked] (Hyogon Kim)
Date: Fri, 5 Dec 1997 12:08:12 -0500 (EST)

>Well, then -- I look forward to reading your paper so that I can

Thank you, your healthy criticism will be more than welcome ;-).

-- Hyogon

---

From: [email-masked] (Kathleen Nichols)
Date: Fri, 05 Dec 1997 11:08:44 -0800

>marked instead of dropped (and serviced in a queue with a low drop 

Please note that such a service policy, applied to a flow, could lead
to egregious packet reorders. This is obvious for congestion, but can
also happen from unshaped best effort packets getting delivered before
shaped premium packets. The basic service types that were proposed by both
Jacobson and Clark&Wroclawski both avoid packet reordering. Is there
really support for such a service type?

---

From: Steven L Blake <[email-masked]>
Date: Fri, 05 Dec 1997 16:01:09 EST

> to egregious packet reorders. This is obvious for congestion, but can

Not if implemented in a separate queue from Premium High.  See my previous
note to Alessio Casati to see details on how this could be implemented.
A flow would either be marked Premium High, Premium Low-In, or Premium Low-
Out (or one of the other alternatives).

Of course, this would be impossible if Premium High and Premium Low were
encoded as separate TOS bits (another reason why a service class field is a
win).

> really support for such a service type?

We'll see.  There are two dimensions of distinction between the Premium
and Assured services as described in draft-nichols-diff-svc-arch-00.txt; 
conservative vs. less conservative service allocation policy (admission
control), and delay jitter.  It may or may not be worthwhile to provide a 
set of mechanisms and services which can cover all of these ranges,
someday. 

Regards,

---

From: Wu-chang Feng <[email-masked]>
Date: Fri, 05 Dec 1997 19:04:53 -0500

> Perfectly agree with you here, including on the need to reserve right now a bit for Van
> premium service (that is equivalent to the low delay bit, IMO).

It's seems more like a low-delay and low-loss bit combined (for traffic in envelope).  To
be maximally efficient with the bits (as Steve Blake's I-D addresses), maybe the premium
service should be encoded as (P=1 && A=1).  Then, the setting (P=1 && A=0) could be used to
select a different queue/service class which attempts to bound delay but makes no attempt
to traffic shape.  (Maybe the P=1 && A=0 setting could be the delay bit semantic of
Clark/Wroclawski?)

Wu

---

From: Tony Li <[email-masked]>
Date: 05 Dec 1997 23:58:53 -0800

> assured or premium service if we cannot afford to have a separate bit

If you let the market make a choice, and the market has to implement
something, we will end up picking one, more or less pseudorandomly, with a
great deal of weight being on what's trivial to implement.

If you don't like that outcome, you should decide.

Tony

---

From: Juha Heinanen <[email-masked]>
Date: Sat, 6 Dec 1997 11:06:52 +0200

well, the service you describe is low delay assured service, not lower
grade premium service.  there has been for a long time an understanding
on this mailing list that assured service needs the delay bit(s) and the
resulting low delay assured service has nothing to do with van's premium
service.

-- juha

---

From: Juha Heinanen <[email-masked]>
Date: Sat, 6 Dec 1997 11:13:47 +0200

there must be a misunderstanding in above.  assured service never
reorders packets if all packets that belong to the same flow are sent
with the same tos, ie., normal delay or low delay.

-- juha

---

From: Juha Heinanen <[email-masked]>
Date: Sat, 6 Dec 1997 11:21:49 +0200

wu,

we should clearly separate type of service from drop preference.  so far
at least the following types of service have been identified:

- assured normal delay
- assured low delay
- premium
- network control

out of these four types of service, high drop preference can only be
applied to the first two.  two bits is enough to encode the above four
types of service and one more bit is needed for the drop preference.
this means that we have used 3 bits out of 8 which sould be ok.

-- juha
---------------------------------
   Sender: [email-masked]
   Date: Fri, 05 Dec 1997 19:04:53 -0500
   From: Wu-chang Feng <[email-masked]>
   Organization: University of Michigan
   CC: Juha Heinanen <[email-masked]>, [email-masked], [email-masked]
   References: <[email-masked]>	([email-masked]) <[email-masked]>

   Wu

---

From: Wu-chang Feng <[email-masked]>
Date: Sat, 6 Dec 1997 10:22:18 -0500 (EST)

> at least the following types of service have been identified:

It seems to me that premium service is an implementation of an
assured low delay service (since the shaper insures that everything
is "In").  This is why I suggested the (D=1 && A=1) setting for premium 
traffic.  Something which gives you low delay by virtue of small queues
and reasonable scheduling (with no guarantees on packet loss and no traffic 
shaping) would then use (D=1 && A=0).  In contrast to the (D=0 && A=0/1) 
settings, you would not stick (D=1 && A=0) packets in the same queue as 
premium traffic nor would you allow a single connection to use both types 
of service (due to packet re-ordering problems) since each would maintain 
separate queues.  I think this is in-line with the bit semantics defined 
although the (D=1 && A=0) setting may not be very useful.....
 
Wu

---

From: Juha Heinanen <[email-masked]>
Date: Sat, 6 Dec 1997 19:16:21 +0200

is "In"). 

you could intepret it this way.

   This is why I suggested the (D=1 && A=1) setting for premium 

this kind of bit usage is possible, but is it necessary to go that far,
since by having a clean separation of a 2 bit tos field and a 1 bit drop
preference field would only consume 3 bits out of 8?  also, i consider
low delay/high drop preference combination as useful as normal
delay/high drop preference.

-- juha

---

From: "Alessio Casati" <[email-masked]>
Date: Wed, 17 Dec 1997 12:37:48 +0100

I appreciate very much your effort.
What I think about it is that I have no objection in using TOS bits
in order to identify the service class and 1 bit for conformance marking/DP. 
(its not so much different from ATM, where the VPI/VCI implicitly identify service
category and CLP the conformance, where the service category and network operator
choices make it useful; so the background experience is not lacking here).
And, yes, it is future proof (with the same extent with we can now think to use TOS bits for
diffs, since 4 bits for 16 codepoints are there). But the general feel while reading your proposal 
is that you want everything to be set out now forever. The feel is confirmed by the reservation of 
ECN bits. It's too early, since it's still a research issue. Let them be reserved bits now, you can 
always use them properly later. I would rather first identify reasonable services/mechanisms, then 
would work out some coding. 

alessio

------