Subject: Concerns about exported numbers

From: "Roch Guerin ((914) 784-7038)" <[email-masked]>
Date: Fri, 22 Dec 95 14:33:11 EST

Reference:  Your note of Wed, 20 Dec 1995 12:14:16 -0800

Fred,

Let me try to take a crack at responding to some of the comments/questions
in your note, at least those concerning the guaranteed service. I'd like
to also take the opportunity to say that I consider the exported information
of that service specification to be key to providing coherent and consistent
end-to-end service characteristics.  I'll try to provide details clarifying
that statement in the rest of this note, but let me first respond to the
specific issues you raised.

>   No other exported data is required by this specification.
>
> No doubt, if I go read the references
>
>
>  [3] L. Zhang, "Virtual Clock: A New Traffic Control Algorithm for
>  Packet Switching Networks," in Proc. ACM SIGCOMM '90, pp. 19-29.
>
>  [4] D. Verma, H. Zhang, and D. Ferrari, "Guaranteeing Delay Jitter
>  Bounds in Packet Switching Networks," in Proc. Tricomm '91.
>
>  [5] L. Georgiadis, R. Guerin, V. Peris, and K. N. Sivaraja,
>  "Efficient Network QoS Provisioning Based on per Node Traffic
>  Shaping," IBM Research Report No. RC-20064.
>
>   [6] P. Goyal, S.S. Lam and H.M. Vin, "Determining End-to-End Delay
>   Bounds in Heterogeneous Networks," in Proc. 5th Intl. Workshop on
>   Network and Operating System Support for Digital Audio and Video,
>   April 1995.
>
>one of them will inform me of the formal meaning of the variables C and
>D;

Probably not directly as I don't think any of them uses this specific
notation.  However, all the "necessary" information is there!

>however, benighted person that I am, I don't have ready access to
>IBM Research Reports

All IBM Research reports that have not appeared in other publicly
available form, e.g., conf proceedings, journals, etc. are
available through the IBM Research Publication server whose URL is

               http://www.watson.ibm.com:8080

>or the proceedings of conferences I don't attend,
>and ACM SIGCOMM periodically stops sending me things, one of the issues
>they've missed mailing me being SIGCOMM '90 (I have had continuous
>registration for a long time, but periodically have to write to them
>and ask them to send recent issues, as they don't mail them to me. I
>had to ask recently for the entire year of 1994!)
>
>So it would be really really nice if some kind soul would put a
>paragraph into the specification that says "C means this" and "D means
>that". My english teacher in fifth grade taught me to do that...
>

This is a good idea (should not have given the credit to your English
teacher...).  Actually, the best way to do that is probably by means of
a few examples, that could be added to the "Guidelines for Implementors"
section, of what the C and D terms would be for some common schedulers
that router may use.  If there are no volunteer, I can probably take a
crack at putting together such a few paragraphs for both a WFQ scheduler
and a rate-controlled scheduler.  Others may want to contribute
similar inputs for other schedulers they use.  The current specs do
include a very brief mention along those lines in the section
"Guidelines for Implementors" towards the end of p.12, where it mentions
what C is in the context of WFQ.

>After you tell me what C and D are, I'll comment on whether they're
>implementable. It's not at all obvious that they are based on what I
>know.

C and D will typically be fixed quantities that depend on the type of
schedulers you use.  The harder part, implementation wise, is likely to
be the scheduler itself, but quite a few people have demonstrated that
schedulers that fit the "Guaranteed Service" model can indeed be
implemented, e.g., WFQ and rate-controlled.  We are ourselves working
on such an implementation, and I do not foresee any major show-stopper
that would invalidate the model.

>
>And I still don't really know what to do with the information. Yes, if
>I have a system that allocates memory to interfaces at flow negotiation
>time rather than on message receipt, I can plug the numbers into the
>formula.

I'm not sure I'm answering your question, but you don't need to allocate
memory at flow negotiation, you only need to check that you have enough
memory so that when you do memory allocation on message receipt, you
don't run out of it for flows to which you have made guarantees.
At least, that's how we are approaching the problem.

>But would the folks who do that mind raising their hands? At
>the IETF meeting, when we were discussing other values that might be
>useful for doing so, no-one (including those asking for the numbers)
>said that they were definitely going to do that. And if no-one is
>plugging the numbers into *that* formula, what has the number got to do
>with the price of eggs?
>
>-- btw, you might want to think about this one... --
>

As far as we are concerned, all the formulas and numbers we use serve
a useful purpose, in particular for flow admission purposes, i.e., to
decide that we need to deny a service request because we have run out
of buffer and/or bandwidth on a link. So, if you are asking whether
we'll make use of the formulas for the stuff we are implementing,
we'll certainly raise our hands on that one.

>All of the above comments are in relation to the standard Integrated
>Services model, which assumes that all interconnects are done via point to
>point lines and the system making the guarantees is in fact in control of
>the statistical behavior of the queues and media between itself and its
>next hop. There is some additional concern being raised about places where
>there might be a non-RSVP cloud between IS hops, but the assumption is
>being made that these are detectable using IP TTL - that those hops are
>routers.
>

You are perfectly correct in that crossing clouds can complicate things
a bit.  We did take a preliminary look at these issues in the specific
context of Guaranteed Service flows having to cross ATM clouds.  You
can find the results of this investigation in another IBM Research
Report, also available on our Web server.  The number is RC 20250.

>What the above reports display are queuing delays. What about propagation
>delays? Propagation delays include such media characteristics as:
>
>  - shared media such as broadcast LANs
>  - satellite hops (272 milliseconds delay per hop)
>  - ISDN circuit switch drop-and-add multiplexing points
>  - telco repeater delays
>  - layer 2 switching fabrics such as ATM or Frame Relay, which have
>    internal queuing, forward error correction, and speed of light delays
>
>If the accumulated delays are supposed to externalize the actual mean end
>to end delay, all of the above are non-trivial components and are neither
>measured nor reported, and it's not clear how they might be measured or
>reported. I suppose we could dedicate 43 K of RAM per VCI in an ATM switch
>to keep track of maximal end to end delays...
>

Let me try to answer this by first pointing that you need to differentiate
between fixed and variable delay components. For things like propagation
delay (satellite links included), fixed repeater or add-drop muxes, etc.
you'll have a single fixed value that needs to be advertised.  That's
essentially information to be exported through the D parameter. Now for
variable delay devices such as shared media LANs or ATM/Frame Relay
switches, you need to be able to upper bound them if you want to get
a Guaranteed Service when crossing them.  This may be hard to ensure
when you go over a shared medium LAN, unless its a dedicated LAN segment,
i.e., switched E/N, over which you can probably upper bound it.  However,
this is certainly feasible when crossing ATM clouds (again see the
above Research report for some details on this).  Bottom line is that
you want to know the fixed component of your end-to-end delay,
so that you can determine what to ask from the routers for the portion
that is under their control, i.e., through the scheduling.  Btw, there is
no mean end-to-end delay component in the Guaranteed Service, since
it only deals with bounds.

>-- the End to End Principle --
>
>Someone told me once about a principle that applies to services in general.
>It goes something like this:
>
>        "If something must be done at layer N for some applications but
>        not for others, posit the mechanism in that layer rather than
>        some lower layer."
>
>Now, clearly, not all applications or transports need a measurement of end
>to end delay. TCP, for example, plugs along quite happily without it. So
>the lowest layer at which one might expect to find a non-universal service
>that measures end to end delay is at the transport layer. I observe that
>RTP, frequently used by multimedia protocols with real time requirements,
>contains a time stamp that is expressly designed to use the network time
>protocol as a source of timing information. Father Time, in my hearing, has
>boasted that the quality of timing information that results from NTP is
>such that he can calibrate the thermostats in machine rooms around the
>world by the drift in their machine's reference clocks.
>
>Given that I have a way to determine what the end to end delay *is*, it
>seems that determining what it would be were I to ask for something
>better is determinable by experimentation - if you don't like the
>service you're getting, ask for a higher level and see if it is better;
>if you are gettingsignificantly better service than you need, try
>asking for a lower level and see what the effect is.

Not sure I follow you there.  The point is not to find out what you are
getting by experimentation.  Rather, what you want is to get enough info,
so that you can decide what to ask in order to get what you need.  Also,
asking for higher or lower levels of service does not help much unless
I know what these mean, and that all the routers on my path use the same
definition.  Imho, this is one of the strength of the Guaranteed Service
model, in that it ensures end-to-end consistency and performance.

>Less deterministic, yes, but implementable with no support from the network
>at all, and (unlike this model) reliably delivers as good a set of
>numbers as you care to calculate.
>

There, I don't understand how you can calculate anything and even less
deliver anything reliably, but maybe I missed something.

>
>-- My Recommendation --
>
>There is no non-RSVP Integrated Services implementation (ST-II has a
>different flow specification and set of things it does with it), and the
>RSVP implementations I am aware of (ISI and derivatives, Bay, Cisco) do not
>implement these exports at all. They are significantly problematic, and at
>the IETF meeting a few weeks ago no-one proposed that they be mandatory to
>implement.
>
>Can we please rely on the end to end services to make their own
>measurements, and forget these?

I strongly believe that relying only on end-to-end services to make
measurements will not allow for a coherent and efficient delivery of
inter-operable implementations, that can provide good end-to-end
performance guarantees.  I don't want to get into the debate of
whether we need reservation of resources to provide QoS guarantees,
but assuming we do then I don't believe we can get away with just
end-to-end measurements.

Roch

---

From: Hari Adiseshu <[email-masked]>
Date: Mon, 1 Jan 1996 22:33:27 -0600

Hi Fred

Having spent the last summer  at PARC working on the implementation of
OPWA and ADSPECs, I read with great interest your broadside against the IntServ
characterization parameters.

I think you've made 3 points in your message.

i) What are the C and D terms in guaranteed Service supposed to mean?
Can they be measured for all networks and fabrics, including ATM and 
Ethernet switches?

ii) Measuring delays to 1 microsecond precision , and tracking the
maximum delay for the last 3600 seconds is expensive in
terms of memory (43K per interface), and might involve redesign of
existing hardware switching engines.

iii) Past maximal delay values cannot be used to predict future delay values,
and therefore the exported delay parameters (in their current format)
are not of use.

There have been a flurry of mails on item i), so I would not like to
comment on that.

Regarding item ii), with a modification of the algorithm that
you've given, it is possible to obtain the peak delays using only
2, rather than 3600 variables. This involves measuring the peak
delay over the last 3600 to 7200 seconds, rather than strictly 
over the last 3600 seconds. 
(You simple keep two variables, say previous_hour_max, and 
current_hour_max.  Update current_hour_max with the current maximum, 
and at the end of every hour, copy current_hour_max into previous_hour_max.  
The maximum of previous_hour_max and current_hour_max measures the peak 
delay over the last 3600-7200 seconds.)
So the storage complexity isn't really all that bad.

Regarding redesign of hardware engines- well, all IntServ implementations
need to make admission decisions and ensure that flows get the
desired QoS, which means that they need to
track the  delays of service levels, as well as  delays of individual flows.
However, the precision of delay measurement might need to be relaxed.

The last point (regarding the utility of carrying delay information)
was pertinent, and one in which I'm inclined to agree with you.
I think you've pointed out earlier that it makes
little sense in carrying 1 s delays when PATH messages are not sent
immediately, but wait at each hop. In addition, providers might not
be willing to let sensitive data like delay violations be made public.

However, given a scenario in which a user wants to know the QoS 
(in terms of delay) associated with a given service level before committing to 
that level (especially since access to higher service levels is unlikely to 
be free), what are the alternatives ? 

A possible solution is to embed the  characterization
parameters in an IntServ MIB. So an end user application which wants
to know the current end to end delay for a given service level only
has to query the routers along the path, and compose the retrieved
parameters. This functionality  should be easy to combine with the disgnostic
messages now under development in RSVP. The drawback with this approach,
of course, is the possibility of message implosion, if all receivers
wanted to know the delays. However, if the majority of the receivers
were to base their service class selection on the observed application
behavior, as you suggest, then this method of accessing characterization
parameters would be more efficient that carting them in ADSPEC objects.

To summarize, while I disagree that adding characterization parameters
to ADSPEC objects would be unduly difficult, I agree that adding them
at this current stage, would add more complexity than value. I
suggest that the characterization parameters be accessed through
an IntServ MIB, and the idea of ADSPECs revisited only if the behavior
of applications warrants it.

Regards.

-hari adiseshu
+--------------------------------------------------------------------+
|Hari Adiseshu  Email: [email-masked]   Mail: CCRC           |
|Computer       Phone: 314-935-8563             Campus Box 1115      |
|Science Dept.  Fax:   314-935-7302             Washington University|
|http://dworkin.wustl.edu/~hari                 St. Louis  MO 63130  |
+--------------------------------------------------------------------+

---

From: [email-masked] (Fred Baker)
Date: Wed, 3 Jan 1996 13:07:59 -0800

At 10:33 PM 1/1/96, Hari Adiseshu wrote:
>I suggest that the characterization parameters be accessed through

From discussions I have had with John W, it sounds like C and D are
constants, not measurements. These would certainly be good things to put
into a MIB rather than repeat on the wire every few seconds. And I could
put these others into the int-serv MIB if that works for people.

I am still, like you, concerned with the meaningfulness (meaninglessness?)
of the numbers requested - if I know, as you suggest, what was the worst
delay experienced in the last 60-120 or 3600-7200 seconds, what is the real
utility of this? The 60-120 second number, by that approach, would never be
greater than the 3600 second number, as the two time domains overlap, and
if it is less, which is more meaningful? The fact that current conditions
allow for better delay characteristics, in which case I wonder why we need
the 3600 second number, or the fact that conditions have been worse, in
which case what's the point of the 60 second number?

Before we do either, I would love to get a full description of exactly how
the application is supposed to use those numbers. I'd like to know for sure
that we are measuring the right thing.

Maybe this is my "MIB designer" side coming out - folks like to take a lot
of nebulous measurements, throw in a bunch of counters and gauges that say
"I passed through this bit of code", without particularly thinkng about
what the numbers are good for. Then I get the application writers and the
network managers coming back to me asking for (this is literally a request
I received once) a statement of what they can deduce from each number
counting quickly or slowly, or not counting, and how that relates to the
price of tea in China. So before *I* get hammered with the question, I am
putting the question to the list, and submitting that absent a very clear
explanation that an application writer or network manager can use well, we
don't advertise the numbers in the first place.

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Some ideas are so crazy that only an intellectual could believe them...
        George Orwell

---

From: [email-masked] (Fred Baker)
Date: Fri, 19 Jan 1996 10:09:39 -0800

At 9:41 AM 1/19/96, Mark W Garrett wrote:
>Hmmm, not to beat this to death, but Jon seems to be saying
>drop *all* packets on a connection which exceeds its traffic description,
>and Fred seems to be saying drop (at worst) only the *excess*
>traffic of such a connection.  Is the disagreement one of intention
>or just semantics?

It is one of intention. I am very much unconvinced that instantly pitching
all excess traffic is a good thing, although I certainly will use no
heroics to preserve it.

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Suppose you were an idiot.  And suppose you were a [politician].
But I repeat myself. -- Mark Twain

---

From: [email-masked] (Christian Huitema)
Date: Mon, 22 Jan 1996 14:04:33 +0100

At 2:54 PM 19/1/96, John Krawczyk wrote:
>   On Fri, 19 January, Jon Crowcroft ([email-masked]) wrote:
>
>    >It seems that dropping excess traffic at the point of violation is
>    >reasonable. Any comments?
>
>   agree ...  though mandating order ing is not needed, however, for
>   network load adaptive applications, the loss at the receiver ought to
>   be as ordered as possible w.r.t the legal contracted delivery....
>   to enable the adaption to be timeley...
>
>Haven't we been here before?  The reason the specs were changed from
>the original "drop non-conformant traffic" position was because,
>otherwise, you can create a denial of service for best-effort
>receivers by making a too-small reservation.  (check the mail archive
>for details).
>
>-jj

I believe that all this debate exhibits a major confusion in the conception
of the "integrated" service.  Specifically, the group has long seemed to
equate the traffic class - real time or data - and the resource reservation
requirement.  The group assumption has been that real time is inelastic,
cannot adapt to network conditions, while data is elastic, can be treated
best effort.  I believe that we should reconsider this assumption.

There is ample evidence that we can build adaptive real time services.  IVS
video is one such service, and so is RAT for audio.  The mechanism for
supporting end to end adaptivity are specified in RTPv2, now a proposed
standard.  There is also ample evidence of a market demand for granting
level of services to non real time traffic, e.g. the pipe between two
locations of a corporation, the file transfer of some large and important
data, NFS access to a key server.  I propose that we build our model
accordingly:

* whether "excess" traffic should be dropped or queued should be a property
of the traffic itself.  There is a well delineated provision for this in
IPv6, where priority levels 0 to 7 apply to "data," whose excess should be
queued, and levels 8 to 15 apply to "real time," whose excess should be
dropped.  The feature could be retrofitted to IPv4 by specifying that
excess is dropped when the TOS is set to "best delay", queued otherwise.

* the amount of "reasonable" traffic should be determined from traffic
allocation, policies, etc.  Best-effort, here, should not mean queuing but
rather "fair share", i.e. the amount of service that a particular class
would get in the absence of reservation.

This model results in significant simplifications of the scheduling
machine.  Suppose a model that combines fair queuing (by default) and
weighted fair queuing (in the presence of reservation).  The reservation
would only be understoud as a "guaranteed bottom".  They would only take
effect if the amount of resource is more than what the class would get
anyhow, thus eliminating the "denial of service" attack mentioned in the
message.  The traffic handling would be entirely determined by packet
itself, thus eliminating the adverse effect on delay jitter noticed by Mark
Handley when some traffic is passed fast while other traffic is queued.  In
fact, we would be satisfied with exactly one reservation class, the
"guaranteed bottom", and with one very generic constraint, that "real time"
packets should never be kept in long queues, even in the absence of
reservations.  If an application requires a "hard guarantee", it will
simply ask for a "high" bottom, and stay within it.

Am I making it too simple ?

Christian Huitema