Subject: Controlled-Load comments

From: "Roch Guerin ((914) 784-7038)" <[email-masked]>
Date: Fri, 23 Feb 96 10:55:19 EST

!!!!!!!!!!   WARNING  -- REASONABLY LONG NOTE    !!!!!!!!!!

John,

Some time ago I had sent you a number of comments on the CL doc but did
not hear from you.  Given that some of the recent postings to the mailing
list by Curtis and others as well as Juha's draft on a protected best
effort service raise a number of similar issues, let me try to recast
some of these earlier comments in that light.

I think we all agree that there is a need for a *minimum* service
specification, that is as simple as possible but nevertheless provides
a level of service above that of the current best effort.  The issue
though, as Curtis pointed in one of his notes, is to strike the right
balance between simplicity and making the service suitable to applications.
In particular, in order to be simple you have to limit the scope of the
requirements you are trying to satisfy.  I think that's where CL
needs some crispening, and possible narrowing of its goals.

Specifically, I'd like to categorize applications along the following axes:

 1. Adaptive vs non-adaptive
 2. Conformant vs non-conformant

where conformance is defined wrt to the traffic parameters that are
*declared* at the time the service is requested.  Clearly, not all
combinations along those two axes make sense, e.g., adaptive and
conformant is probably not very meaningful.  However, I think it is
useful to consider separately the requirements of applications in
each of the quadrants defined by these two axes.  In particular, this can
help determine what network elements need to do in order to satisfy the
requirements, i.e., the amount of resources (bandwidth and buffers)
to be *set aside* and what to do when running out or resources.
Let me, therefore, try to review the different possible combinations
and their requirements, starting with the simplest one, i.e.,
non-adaptive and conformant applications.

A. Non-Adaptive and Conformant
------------------------------

These are applications that declare traffic parameters, e.g., token
bucket depth b and token generation rate r, and will comply with this
declaration.  In exchange, such applications expect *all* their packets
to be delivered (in order) with a latency *comparable* to what they
would experience if they had a dedicated circuit of rate equal to
the service rate they request, i.e., typically the token generation r.

What this requires from the network elements is that they set aside
enough bandwidth, e.g., r, as well as enough buffers so that dropping
of packets is a rare event.  What that means, is that packets should be
delayed/buffered if they cannot be transmitted immediately, and the amount
of buffers set aside should be at least b plus whatever is needed to handle
the *perturbations* (increases in burstiness) caused by upstream network
elements.  Note that the Guaranteed Service specs specifies (in the Adspec)
the amount of buffers needed to never loose a packet, but if small losses
are tolerable you may not need the Adspec info to determine the amount of
buffers needed.  Typically, a network element could set aside something
like a*b where a is a term that accounts for the statistical
multiplexing gain at the network element.  It is up to each network element
to engineer the proper value.  Note also that while policing can be done
at the network boundary, it should NOT be done within the network.  In
other words dropping a packet or downgrading it to best effort because
it is not compliant with the specified TSpec, is a rule that should not
be applied at intermediate network elements for this kind of flows.

* The service specification for this case is relatively simple:

- The application specifies its TSpec (r, b) and the network can police it
at the boundary.  The application expects that *all* its packets will be
delivered in order and with a latency comparable to that provided by a
dedicated circuit at its requested rate.

- Concerning resources allocation, intermediate service elements must
allocate an amount of bandwidth at least equal to the requested service
rate, and set aside enough buffers so that packet loss is a rare event.
In addition, reshaping to the TSpec at intermediate service
elements is permitted and even advisable, at least every so often as it
reduces buffer requirements at downstream nodes.  On the other hand,
policing at intermediate service elements is prohibited.

B. Non-Adaptive and Non-conformant (or not necessarily Conformant)
------------------------------------------------------------------

These are applications which also declare traffic parameters, e.g., (r,b),
and expect the same kind of service as applications from category A
for traffic within this advertised traffic envelope.  The difference is
that the application may occasionally send at a higher rate, and is
willing to see degraded service for these excess packets.  For a service
element, the issues are

1). Allocate enough resources for the declared traffic parameters
2). Identify excess traffic
3). Determine what to do with excess traffic

As far as resources allocation is concerned, it should be essentially
similar to that for a type A application with the same traffic parameters.

The identification of excess traffic is a little less straightforward
as it needs to account for

- the own behavior of the network element, i.e., how good it is at serving
packets at the reserved rate,
- the impact of upstream network elements, i.e., avoid categorizing packets
as excess simply because they got lumped together by upstream nodes,
- the behavior of the source, i.e., whether it has indeed been sending
excess packets.

The identification of excess traffic is again something that needs to be
engineered at each network element so as to ensure a *reasonable* behavior.
As a general rule the token bucket that would be used at an intermediate
network element to identify excess packets should have the same rate r'=r
as the one declared by the application.  However, its token bucket size
b' should be bigger than the declared burst size b.  This is needed to
avoid categorizing too many conformant packets as excess.  A possible
rule of thumb is to take b' somewhere between b and 2*b.  Taking b'=b is
safer for the network, but will typically result in quite a few conformant
packets being classified as excess.  Taking b'=2b should be enought to
ensure that conformant packets are rarely classified as excess.  However,
taking b'=2*b represents a potential exposure for the network as in case
of congestion it may have to buffer up to b'=2b for each flow.  In general,
a possible engineering solution is to couple the conformance checking with
the buffer management, i.e., use b'=2*b as long as there are buffers
available and progressively decrease it to b'=b as the number of free
buffers goes down (any number of simple variations along those lines are
possible).

Once packets have been identified as excess, the options are then to:

a). Keep delaying them as long there is buffer space available,
b). Drop them,
c). Send them as best effort.

Option a) is probably not desirable for a non-adaptive application as it
will impact subsequent packets that may then see a much increased latency.
So, the choice is most likely between dropping and sending as best effort.
Which one of the two is appropriate is most likely application dependent
and, as Christian Huitema pointed it out in some earlier notes, it would
be useful if the service invocation would let you choose which of the two
is to be used. For example, if the application cannot really deal well with
re-ordering packets, then there is no point in wasting bandwidth sending
the excess packets as best effort.

* To summarize, the service specification for this case is as follows:

- The application specifies a TSpec (r, b) that represents the traffic
envelope for which it wants service guarantees.  However, it is free to
send more traffic if it wishes to.  In addition, the application specifies
the desired treatment for its excess traffic, i.e., drop or send as best
effort. The application expects that if it conforms to its TSpec *all* its
packets will be delivered in order and with a latency comparable to that
provided by a dedicated circuit at its requested rate.  The application
understands that the network does not provide any guarantee for packets
sent in excess of the specified TSpec.  However, it expects it will receive
a *fair* share of any additional bandwidth available to this service class.

- On the network side, service elements must set aside enough resources to
provide service guarantees to the traffic characterized by the TSpec. This
means allocating enough bandwidth to satisfy the specified rate r, and
setting aside enough buffers for a *conformant* flow, i.e., based on a call
admission rule similar to that used for case A.  In addition, the service
element is entitled to policing the traffic of the flow, but it is advisable
that they do so using a token bucket that has a *larger* token bucket depth
than the one specified in the TSpec.  It is recommended that the extent to
which the depth of the token bucket used for conformance testing exceeds the
value specified in the TSpec, be coupled to the amount of buffer available.
Packets that have been identified as excess will be either dropped or sent
as best effort if possible, according to what was specified by the
application.  Note that reshaping at intermediate service elements should
not be performed.

C. Adaptive and Non-conformant (or not necessarily Conformant)
--------------------------------------------------------------

These are applications which declare a rate parameter r and possibly (more
on this below) a token bucket size b.  The expectation is that the rate
parameter r is the minimum guaranteed rate but the application will try to
send more if needed and possible, and it expects to get a *fair* share of
any available bandwidth in excess of its guaranteed minimum rate r.
The application will rely on some higher level mechanism (flow control)
to throttle its rate in case of congestion.  The applications expects
that if it sends at its guaranteed rate r, it will experience minimum
packet loss and a latency *comparable* to that of a dedicated circuit
at rate r.

For service elements the requirements are as follows:

1). Ensure that the flow is always guaranteed its requested rate r,
2). Determine how to handle excess packets when the application sends
    at a rate that exceeds r and the available bandwidth is insufficient
    to sink this excess traffic,
3). Allocate *enough* buffers to this class of service to be able to
    deal with temporary overloads due to the latency in rate adjustments
    in the presence of congestion.

At a service element, the tough problem is not 1), but 2) and 3) which
are closely related.  First you have to decide on how much bandwidth is
available to this type of traffic at any given time.  In particular, it
may not be wise to let it starve totally lower priority traffic classes
such as best effort.  Once this has been set, then comes the issue of
what to do with excess packets when they start piling up.  This is
related to both buffer sizing and to what treatment might be most
suitable for a given application.

If the flow specified a b term, then this value can be used to decide when
a packet is an excess packet.  One option, if the flow is tolerant of
reordering and there is a lower priority best effort service class, is to
send the excess packets as best effort.  In this case, the specified
value for b is used to determine when packets start to overflow onto the
best effort queue.  Note that it is then important to
ensure that the best effort class has been allocated sufficient
buffer to handle (well) the excess packets.  The aspect of fairness
between best effort packets and these excess packets should also be
considered, e.g., do fair queueing between the two.

Another option, that is more suitable for application that are sensitive
to packet reordering, is to buffer excess packets as long as there is
buffer space available, and keep transmitting them at whatever rate is
available to the flow, i.e., at least r.  When buffer space is exhausted,
packets will be dropped. Note that the value of b should not be used as
the threshold value beyond which to start dropping packets.  This is
because you would then run into the problem pointed out by Curtis, where
you would need to specify a very large b in order to ensure decent
performance when you are blasting at a rate much higher than r.  Here,
the value of b should be used for buffer management to determine the
respective amount of excess traffic of different flows.  This could
then be used to determine which arriving packets to drop.  While this is
a possibility, it is not entirely clear how useful it would really be,
and it may very well be that specifying b is not needed for this
particular case, i.e., set b==0 as Curtis suggested.

Note that in general, the concept of burst size is really meaningful only
when associated with a particular rate, i.e., the amount of data that can
be dumped in excess of what your rate entitles you too.  Hence, it is a
quantity that is not very meaningful in the context of an adaptive
application, where the rate itself is fluctuating.  In that context, the
usefulness of a burst size term is mostly for the purpose of buffer
management and service differentiation, i.e., identify excess packets.
As was discussed, this is probably most useful for flows whose excess
packets can be sent at a lower service level, e.g., best effort, as the
*burst size* can then be used as the parameter to trigger this service
downgrading.  For flows whose packets need to be kept in order, such
a quantity may not be that useful as it cannot really be used for buffer
sizing.

* To summarize, the service specification for this case would be as follows:

- The application specifies a TSpec (r, b) that represents the traffic
envelope for which it wants service guarantees. The parameter b is
optional and not meant to specify a maximum burst size.  The application
is free to send traffic at a rate higher than r, but will not be provided
with any guarantees for its excess traffic. The application expects that
if it sends at rate r or less, *all* its packets will be delivered in order
and with a latency comparable to that provided by a dedicated rate r circuit.
It also expects that it will receive a *fair* share of any additional
bandwidth available to this service class.  In addition, the application
specifies the desired treatment for its excess traffic, i.e., maintain
packet ordering or not.  The application is expected to be responsive
to congestion in the network by adapting its rate (possibly down to r)
based on some higher layer flow control protocol that lets it react to
the onset of congestion, e.g., using a window flow control mechanism.

- On the network side, service elements set aside enough resources to
guarantee a minimum service rate of r to the flow.  Service elements
also guarantee that they will provide each flow with a fair share of
any additional bandwidth available to this service class.  In case of
congestion, services elements will handle excess packets according to
the treatment type specified by the application, i.e.,

a). Keep delaying them as long there is buffer space available, and then
    drop them,
b). Send them as best effort.

In case the application included a parameter b in its TSpec, the service
element will use it for the purpose of identifying excess traffic and/or
buffer management.  In particular, if the application specified that its
excess packets should be sent as best effort, the parameter b can be used
to determine which packet to downgrade to this service class.  Finally,
the service element is expected to allocate enough buffer space to deal
with temporary overload conditions due to the latency in rate decreases
by applications in case of congestion.

SUMMARY
-------

It is clear that there is a need for services with less stringent
guarantees than those specified by the Guaranteed Service.  The
deterministic delay and loss guarantees of the GS are appropriate for
some sensitive real-time applications, but their cost is too high for
applications that can tolerate less accurate guarantees.  On the other
hand, those other applications exhibit enough diversity in their service
requirements, that you cannot really satisfy them all with a single
service specification.  For example, I don't think that the same
service specification can successfully deal with both adaptive and
non-adaptive applications.  The CL spec should, therefore, probably
limit itself to either one of those two categories. Similarly, even if a
service specifies that it is intended for applications that are adaptive
or not, there is still a need to further specify the treatment
that excess packets (if any) should receive, delay, drop, send as best
effort, and various other combinations.  In general, I think that the
CL spec is trying to satisfy too many requirements at once and, therefore,
runs the risk on not really satisfying any of them well.  The broad
classification I would envision for those services that target lesser
service guarantees than GS, is as follows (please do not focus on
the names as they are just place holders based on my own views on the
stuff that has been discussed so far).

- Guaranteed Bandwidth (non-adaptive, conformant flows)

  User specifies TSpec (r, b) and agrees to comply with it, i.e., allows
  policing at the boundary and reshaping within the network.
  The service elements guarantee the availability of a bandwidth at least r
  to the flow, and dedicate enough buffer to it so that packet losses will
  be rare (to be engineered for each service element).

- Controlled Load (non-adaptive, not necessarily conformant flows)
  Two flavors CLa and CLb as a function of how excess traffic is treated
  (could be specified through a bit or a different service number)

  User specifies TSpec (r, b) but does not agree to comply with it, i.e.,
  may be sending more occasionally.  The user understand that no
  guarantees can be provided for its excess traffic.  However, it
  specifies either one of two handling for its *excess* traffic:

  a. Delay and drop, i.e., preserve ordering,
  b. Send as best effort, i.e., do not necessarily preserve ordering.

  The service elements guarantee the availability of a bandwidth at least r
  to the flow, a fair share of any additional bandwidth available, and
  dedicate enough buffer to it so that losses of non-excess packets are
  rare and few non-excess packets get classified as excess (to be engineered
  for each service element).

- Protected Best Effort (adaptive flows)
  Two flavors PBEa and PBEb as a function of how excess traffic is treated
  (could be specified through a bit or a different service number)

  This is based on Juha's service specification with additions to allow the
  optional downgrading of *excess* packets to best effort, i.e., do not
  maintain ordering, and the optional specification of the parameter b.

  User specifies TSpec (minimum guaranteed rate r, and optionally a *buffer
  sizing* parameter b).  It expects to be guaranteed a bandwidth of at
  least r, plus a fair share of any additional bandwidth that is available.
  In case of congestion, packets sent in excess of the guaranteed rate r
  will be handled in either one of two ways as specified by the user.

  a. Buffer as long as there is space available,
  b. Send as best effort when flow backlog exceeds given value, i.e., b.

  The service elements guarantee the availability of a bandwidth at least r
  to the flow, a fair share of any additional bandwidth available, and
  dedicate enough buffer to this service class to accommodate temporary
  overloads, i.e., there better be large buffers for this service class.

Well, that about concludes it.  I would very much like to see things moving
in a direction with narrower targets for the different service specifications,
i.e., distinguish between adaptive and non-adaptive flows and allow the
selection of how to treat excess traffic.  I've tried to articulate a possible
classification in the above, but there are certainly other ways of
doing it.  The main issue is that while simple is good, the same simple is
not good for everyone, so it is probably better to partition the service
space accordingly.

Roch

---

From: [email-masked] (John Krawczyk)
Date: Mon, 26 Feb 96 11:45:04 EST

Hi Roch,

A couple of comments:

First of all, the term "adaptive" has some history in the int-serv
literature as referring to delay-adaptive applications (those that can
move their playback point around).  I believe you are referring to
applications that can adapt their sending rate based on feedback from
the network.  Perhaps using the term "rate-adaptive" would keep me
from getting confused (is it only me?).

Secondly, I'd like to see the sections on policing modified to include
those cases of a shared reservation on a multicast session with
multiple senders.  For example, your "Guaranteed Bandwidth" service
does not allow dropping within the network, but if a reservation is
for less than the sum of the senders' rates, what do you do?

Thanks.

-jj