Subject: controlled delay

From: [email-masked] (Fred Baker)
Date: Mon, 20 Nov 1995 14:48:32 -0800

>    * addressing the issues of link overhead, fragmentation, and packet
>    rate raised in Stockholm.

Not all the comments raised in Stockholm were addressed. I will remind you
of them...

One issue that I have is that it is impossible for me to provide a trial
implementation of this service unless you tell me the bit pattern that you
want to see in the message. Theory is nice, but you really need to get down
to brass tacks if you want deployment experience.

I think you also need to think through the specification from the
application's perspective, as it is not at all clear that the information
is useful to the application.

>   The minimum policed unit, m,  is an integer measured in bytes.  All
>   IP datagrams less than size m will be counted against the token
>   bucket as being of size m.

What is the motivation for this? It seems that if the sending application
knows that it might periodically send bursts of messages that contain an IP
header and one byte of data, it will simply specify m=21. This is not
productive if you are trying to get the network to force the application to
send data of a reasonable size. It would be better to say that routers may
be configured with a minimum policing threshold that would apply to all
flows.

>   The TSpec can be represented by two floating point numbers in
>   single-precision IEEE floating point format followed by two 32-bit
>   integers in network byte order.  The first value is the rate (r), the
>   second value is the bucket size (b), the third is the minimum policed
>   unit (m), and the fourth is the maximum packet size (M).

The third and fourth of the two values? :^)

>   For each level of service, the network element exports three
>   measurements of delay (thus making nine quantities in total).  Each
>   of these characterization parameters is based on the maximal packet
>   transit delay experienced over some set of previous time intervals of
>   length T; these delays do not include discarded packets.  The three
>   time intervals T are 1 second, 60 seconds, and 3600 seconds.  The
>   exported parameters are averages over some set of these previous time
>   intervals.

This is the comment that I raised at Stockholm, and to which I recall there
being agreement to change this specification.

First off, export in what way? To SNMP, forcing each receiver to poll each
upstream router periodically to find out what's going on in order to make
the right decisions? Or is this an export to some unnamed object in an RSVP
message? How does the system that needs this information get it, and how
are the bits arranged?

Now I'm going to argue that, once you answer that question, this
information is itself of marginal real world utility.

Imagine just for fun that there are two or more variable rate streams, one
being the stream you are interested in and one being a video stream from
camera which is moving slowly, sending images at one time of the surface of
a choppy body of water - high rate and difficult to compress - and at
another time of a blank wall of a single uniform color - very low rate. The
camera started doing this five minutes ago. All streams are using the same
service level (or worse, the camera is using a higher service level than
you are). The camera's pan rate is such that it sees one picture for a
period of several seconds, switches over to the other over a period of few
seconds, and shows it for several seconds, and then pans back.

Heck, say it's the security camera in the boat dock at the beach, and its
field of view is misaligned to sometimes point at the wall, if you like :^)

OK, clearly, mean delay of the stream you are interested in is going to
vary significantly, but at a relatively low rate of change. You get the
information when you happen to look at it - and I don't suppose the typical
viewing application is going to send a poll to each upstream router to read
these things very often.

The only number in the lot that tells you much is the 60 second delay
average. The 3600 second number has had too little time to reflect much at
all, and the one second number is going to vary quite a bit depending on
which second you happened to read it. And frankly, the 60 second number
doesn't tell you much about what's happening *now*, which is what the
application is going to need to achieve the stated objective, which is
guidance of which service level to select *now*.

It seems to me that, to achieve that objective, what needs to be defined is
a moving average, perhaps an exponentially weighted one, that tells you
about delay characteristics over the past 30-60 seconds with an emphasis on
the most recent few, is carried in the TSPEC, and is updated at each hop so
that the receiver receives an estimate of the source-sink delay being
experienced without having to read each router on the path, and a similar
sum for each service level. Note that an individual stream may experience
the delays of different service levels at different points in the path - he
may be requesting service level 3, that may be merged into a stream of
level 2 somewhere upstream, and further up it may merge into a level 1
stream. So he really can't look at any of the "service level" numbers and
tell what he's experiencing other than to say that what is getting is no
worse than the service level average that he is requesting.

Given this type of definition, the specification should additionally
specify that each receiver should specify the lowest impact (highest
numbered) service level that achieves the delay it requires.

Another thought: what the service level really specifies, if you think
about it, is not the mean delay as much as it is the variation in delay
attributable to higher impact service levels. If you specify the highest
service level, you should get close to constant delay. If you specify the
lowest service level, you will get about the same delay characteristics
until the link approaches saturation. The principal differences between
service levels is that the lower service levels will experience higher
variability when the link is saturated.

So I submit that the issue here is not really a measurement of end to end
delay as much as it is a measure of the variation in end to end delay.
Specification of a higher service level doesn't directly affect end to end
delay except when a non-null subset of the links en route are saturated
with reserved traffic. Specification of a different service level directly
affects variation in delay.

>   The delays are measured in units of one microsecond.  An individual
>   element can advertise a delay value between 1 and 2**28 (somewhat
>   over two minutes) and the total delay added across all elements can
>   range as high as 2**32-1.  Should the sum of the different elements
>   delay exceed 2**32-1, the end-to-end advertised delay should be
>   2**32-1.

I have commented on this in the past. I think it's silly, as only a small
subset of deployed hardware has this precision of measurement. However, if
you are willing for me to average measurements that I *can* make and report
to a precision that I can't hope to measure, I'll shut up. :^)

>   Policing need only be done if the TSpec on the outgoing link is "less
>   than" (in the sense described in the Ordering section) the TSpec
>   reserved on the immediately upstream link.  A source merge point is
>   where the multicast distribution trees from two different sources
>   (sharing the same reservation) merge.  It is the responsibility of
>   the invoker of the service (a setup protocol, local configuration
>   tool, or similar mechanism) to identify points where policing is
>   required.  Policing is allowed at points other than those mentioned
>   above.

I disagree. Policing must be done at any place where the ingress rate may
exceed the egress rate; an obvious place is at the first hop router; the
sender may send at a rate higher than he advertised.

>   The token bucket parameters require that traffic must obey the rule
>   that over all time periods, the amount of data sent cannot exceed
>   rT+b, where r and b are the token bucket parameters and T is the
>   length of the time period.

Suppose the time period in question approximates the duration of a single
message? Won't the rate during the time period approximate the link rate
regardless of the value in the TSPEC?

I think you want to say something like "for all time periods in excess of
the amount of time required to transmit a single burst of size b at rate
r". I assume that the sender might send me the burst (a video frame) at the
rate of the connecting link.

>   TSpec's are ordered according to the following rule: TSpec A is a
>   substitute ("as good or better than") for TSpec B if (1) both the
>   token bucket depth and rate for TSpec A are greater than or equal to
>   those of TSpec B, (2) the minimum policed unit m is at least as small
>   for TSpec A as it is for TSpec B, and (3) the maximum packet size M
>   is at least as large for TSpec A as it is for TSpec B.

I'm a little confused by this. A TSPEC is a statement of what a single
sender is capable of (what his codec will do). Under what circumstances
would the characteristics of one sender be validly confused with that of
another sender? The receiver might request a reservation that merges the
characteristics of several senders, or several reservations might be
merged, but merging TSPECs seems anachronistic. And when is it valid for a
router to *change* a TSPEC? When is it valid for a router to advertise "he
said he could generate <mumble> but *I* have decided that he can generate
<mmmmph>"?

>   A possible implementation of controlled delay service would be to
>   have a queueing mechanism with three priority levels, with level 1
>   packets being highest priority and level 3 packets being lowest
>   priority.  Each controlled delay service level would be associated
>   with a target queue utilization level, say 20% for level 1, 50% for
>   the combination of levels 1 and 2, and 70% for the combination of all
>   three levels.  The utilization of the link, by each of the three
>   levels, would be measured over some relatively short time period
>   (say, 5 seconds, or 10000 MTU packet transmission times).  A new flow
>   would be admitted to level 1 if the measured usage of level 1, plus
>   the token bucket rate of the new flow, was below the target
>   utilization of level 1.  Similarly, a new flow would be admitted to
>   level 2 if the measured usage of levels 1 and 2, plus the token
>   bucket rate of the new flow, was below the target utilization of
>   levels 1 and 2.

Since a simple priority mechanism permits a higher priority stream to lock
out a lower priority stream as long as it maintains traffic in queue, it
seems to me that this is an invalid implementation. It does not in fact
control delay for any but the highest service level.

How about this:

    A possible implementation of a controlled delay service would be to have
    a queue for each service level, with an associated target maximum delay.
    When a message is enqueued, it is intended to be delayed no longer than
    the target maximum delay for the service level, so it is stamped at
    enqueue with a deadline for its start of transmission. This deadline is
    the target maximum delay minus the message's duration at the interface
    bit rate, which is to say the number of bits in the message divided by
    the interface bit rate. When considering which of the available queues
    to dequeue from, one inspects the message at the head of each queue and
    dequeues the message with the earliest deadline. Policing is minimally
    effected by discarding messages that are not dequeued until after their
    transmission deadline.

Having said that, you might also configure the percentages you suggested,
although I'm not sure that's necessary to the service.

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Some ideas are so crazy that only an intellectual could believe them...
        George Orwell

---

From: [email-masked] (John Krawczyk)
Date: Tue, 21 Nov 95 10:17:07 EST

On Mon, 20 November, Fred Baker ([email-masked]) wrote:

Hi Fred,

I think this is kinda mentioned in the document a couple of times, but
not exactly in your context, and it doesn't explicitly use the
"variation in delay" language:

   However, all three levels of controlled
   delay service will have average delays that are no worse than best
   effort service, and the maximal delays should be significantly better
   than best effort service when there is significant load on the
   network.

   All three levels of service should be given better service, i.e. more
   tightly controlled delay, than uncontrolled best effort traffic.  The
   average delays experienced by packets receiving different levels of
   controlled delay service and best-effort service may not differ
   significantly.  However, the tails of the delay distributions, i.e.,
   the maximum packet delays seen, for the levels of controlled delay
   service that are implemented and for best-effort service should be
   significantly different when the network has substantial load.

Perhaps some of the text from the latter paragraph (comparing CD to
BE) should be cut-and-pasted to discuss the delay distributions of the
different levels (ditto for the predictive document).

-jj