Subject: Re: interim meetings?

From: Curtis Villamizar <[email-masked]>
Date: Thu, 08 Feb 1996 22:30:33 -0500

In message <[email-masked]>, John Wroclawski writes:
> 
>    From: Curtis Villamizar <[email-masked]>
> 
>    I don't see the issue of real-time vs adaptive real time vs elastic
>    addressed.  A controlled load reservation for elastic traffic should
>    be supported.  There was no response to the idea of using zero, NAN or
>    positive infinity to indicate elastic traffic.  Adaptive real time is
>    a separate issue which I'm not offering any solution for.  It might be
>    that nothing further is needed for adaptive real time (it just gets
>    policed and adapts to the detection of loss).
> 
> Curtis,
> 
[ .. long reiteration of controlled load goals ... ]
> 
> - Other than the requirements on best-effort forwarding above, it
> doesn't say exactly what happens to a traffic stream when it exceeds
> the parameters. 
> 
> There are basically two choices. You can send some of the packets (a
> within-parameters subset) promptly and delay or drop the others as
> required by queueing. If you do this, some of the packets will arrive
> with minimal delay, but they will be reordered and the delay variance
> will be high.

Yes there are two choices (actually three have been mentioned).
That's exactly the point.  Once choice is terrible if you are delay
sensitive (just queue things up and let delay accumulate).  The other
is terrible for TCP (police and drop).  The third is horrible for
almost any type of traffic (take some portion of the traffic and delay
it, delivering it out of order).

> Or, you can treat all of the packets equally, delaying or dropping as
> needed. If you do this, there will be no reordering, but average delay
> and/or drop rate may be higher.

<g>  you've been paying attention.

> There are several points to consider here. The first is whether it
> matters, and if so how much. Intuitively, it seems that realtime
> streams would prefer the first behavior (sending some packets with no
> delay), because they could either set a long playback point and deal
> with reordered packets, or set a short playback point and use only the
> timely packets. TCP streams, though, might prefer the equal-treatment
> choice, because it minimizes reordering and consequent screw-up of
> retransmission mechanisms.

There's this assumption that if real time doesn't like traffic to be
delayed, the out of order packets might just as well not be delivered
since the added delay puts them beyond the playback point.  If the
playback point can be stretched that far, just add delay and keep the
ordering and be nice to TCP.

> Of course, things aren't really that simple. For example, in the
> delay-some case there is no guarantee that the same packets will be
> delayed if the flow hits another congestion point - since there is no
> packet marking the next congestion point might police, and hence
> delay, a completely different set. At the receiver, you would see both
> reordering -and- higher overall delay.

Sounds really awful.

> Implementation questions play a big part in this, too. At the Dallas
> IETF I talked about three possible implementations of controlled-load;
> there are certainly more. These have the property that they naturally
> do one or the other of the above over-load actions. For example, a
> simple per-flow WFQ, with no additional policing, implements
> controlled-load in a way which will treat all packets equally when the
> traffic exceeds the parameters. The other simple approach of using a
> token-bucket policer to put in-profile packets in a high-priority
> queue and out-of-profile packets in a lower priority queue naturally
> implements the other, delay-some approach. In my view, there will be
> cases where a particular hardware design can easily implement one but
> not the other of these behaviors.

So you provide a poor response to one of the two types of traffic and
no clue to the end-system that their traffic will be trashed by your
choice?  Doesn't sound to me like a good design.

> Considering all this - the fact that different applications might
> ideally like different over-load behaviors from the service, the fact
> that per-hop overload behavior is not necessarily related to
> end-to-end overload behavior, the related matter that there is
> actually no strong evidence at this time of either TCP or realtime
> apps strongly benefiting from one behavior over the other, and the
> problem that mandating just one behavior is going to throw
> -somebody's- implementation for a loop, I intentionally left the
> choice unspecified. I (continue to) believe that this is the right
> choice, but it is one area where more experimental evidence would be
> welcome.

OK.  So you didn't finish the job and left a big hole in the spec.
Lets plug the big hole and move forward!

> (Note that specifying that the controlled-load service provide -both-
> behaviors and allow the reserver to choose has the property that all
> implementations will have to provide two more or less independent
> internal mechanisms, of which at least one probably won't fit with the
> existing implementation. I wanted to avoid this unless it proves
> absolutely necessary).

Consider this.  Using WFQ:

	r > 0	b > 0
	  set Weight acording to r
	  set depth according to b
		when depth is exceeded drop
	r > 0  b == 0
	  set Weight according to r
		when depth get large do something reasonable (see below)

Something reasonable in order of simplicity would be:

  tail drop when sum of b==0 queueing exceeds some limit

  watch the sum of b==0 queues and use EPD or RED accordingly

  track history of each queue size and use EPD or RED, possible also
  considering the total b==0 queue usage.

Simple is not good enough if it is not useful.  As simple as possible
but no simpler please.

The idea behind controlled load is a simple but useful service.  It
should not be as simple as possible, regardless of whether it is
useful.

You can always allow an implementation to always reject reservations
with b==0 and keep the implementation as simple as the ones you
describe.  I'll ask and won't buy their product, but you can allow it.

> 				-john
> 
> PS: Hmmmm. The current draft of controlled-load is
> 
>    draft-ietf-intserv-ctrl-load-svc-01.txt
> 
> which was sent in and posted to this list in mid-December, and is
> available in the directory
> 
>    ftp://mercury.lcs.mit.edu/pub/intserv/current-drafts/
> 
> However, I learned today that the version at ds.internic.net, at
> least, is still the old one from before the December IETF. I'm at
> something of a loss as to why, but until it is fixed please get copies
> from mercury, or send a note and I'll email one back in reply.
> 
> -- 
> John Wroclawski
> [email-masked]

I was at Dallas.  I didn't see people agreeing with you.  I certainly
did not agree with you.  Fred apparently doesn't and is off writing
another draft to make up for the problems with what you have.  The -01
draft (I just picked it up) reflects what you said, for which there
was no concensus.

So please change it already!  Specifying b==0 behavior is easy.
Implementing it is quite easy too.  It will help a lot.  It may save
Fred some writing.

Curtis

---

From: John Wroclawski <[email-masked]>
Date: Fri, 9 Feb 1996 18:54:47 -0500

Strong words. Remember that we are talking about what happens when
both a) a flow is over its reserved capacity and b) the network is
loaded. The main goal at that point is to get adaptive traffic to back
off and to protect other network traffic from non-adaptive flows. For
adaptive flows (realtime or not) it's not expected to be a stable
operating point.

The spec does -not- force you to screw up TCP-like flows just because
they are over-limit. (it does potentially -allow- you to; a point
which could be changed, see below)

   The other is terrible for TCP (police and drop).

Strict police-and-drop is not an option if you want to avoid the
denial-of-service-to-best-effort-receivers problem. The current spec
disallows it. We talked about that in Dallas.  My 'two choices' were
your options one and three.

   The third is horrible for almost any type of traffic (take some
   portion of the traffic and delay it, delivering it out of order).

Well, no. Given an end-to-end delay curve like that, a realtime
application could easily choose a playback point which used only the
non-delayed packets, if it wanted to. A different listener in the same
session (perhaps the one which is making the archive recording,
instead of being interactive) might choose a point which used as many
pkts as possible, reordered or not.

Again, we are talking about what happens to traffic which is
over-reservation when the network is overloaded. It's -expected- to
get poor service, and if it's adaptive it's expected to back off. The
question for TCP is whether one variant of poorer service is
significantly preferable to the other in the circumstance that you are
trying to tell your TCP to slow down.

	   r > 0	b > 0
	     set Weight acording to r
	     set depth according to b

This is one of the implementations I talked about in Dallas. Why do
you think it does not work for TCP flows as well as realtime flows?
(aside from having the usual tail-drop problems)

It does depend on per-flow WFQ, which many people wish to avoid.
The document is written the way it is today in part to avoid making
this choice for you..

	   r > 0  b == 0

     [...]
     [...]

This is fairly similar to another of the implementations I mentioned
in Dallas, although I think you can avoid the separate FQ's in this
case.  I believe it, too, provides good service to both TCP and
realtime traffic when the network is not overloaded.

   describe.

As an aside, this notion that the b term is not useful for TCP traffic
is dubious. b is more or less buffer size. If I want a reservation
that just gives my TCP a "floor" rate r, I use that r and a big
(delay*bw) bucket b and let the tcp run as fast as it can. If I'm
happy with a (cheaper?) reservation for a fixed-rate transfer, I use
my r and a smaller b, and clamp the window size of my TCP down a bit.

   I was at Dallas.  [...] The -01 draft (I just picked it up)
   reflects what you said, for which there was no concensus.

What the document says now is the result of my thinking, what was said
in the second meeting, and hallway conversations with a number of
folks. Nonetheless, no concensus is correct, which is why it's a
draft, not an RFC. There were people in Dallas who thought the current
spec put too much restriction on over-limit behavior, people who agree
with what it says now, people who thought the behavior should be more
spelled out, and people who thought controlled-load should explicitly
and bluntly not address rate-adaptive or TCP traffic.

Some choices are:

1) leave the document alone.

2) leave the actual spec as is now, allowing both sorts of overload
   behavior.  Strengthen the evaluation criteria to strongly favor
   implementations which don't start reordering or delaying
   over-reservation flows until the overall link is overloaded.
   (rather than doing it just because the individual flow is over its
    token bucket parameters)

3) Continue to allow either sort of overload behavior, but
   make it part of the spec that implementations may not
   start reordering or delaying over-reservation flows until the
   overall link is overloaded.

4) Explicitly say that reordering of a flow's excess
   (over-reservation) packets is not acceptable.

5) Add a bit which chooses between delay-all and delay-some-and-
   reorder behavior when over-reservation. I think a bit is
   preferable to b==0 for the reason given above.

   5a) require a conformant implementation to support both.
   5b) let implementations reject requests of one sort or the other.

Personally, I'd prefer 2, 1, or 3, more or less in that order. I think
that requiring routers to support two different mechanisms when
there's no clearly established need for it is a bit of overkill.
Obviously, though, the final document will reflect whatever the group
agrees on. I'm not sure we're there yet, but if I'm wrong I hope
everyone will say so...

			-john

---

From: Curtis Villamizar <[email-masked]>
Date: Fri, 09 Feb 1996 22:34:16 -0500

> adaptive flows (realtime or not) it's not expected to be a stable

How do you think the Internet runs.  All over the place:

  a) people would like to get more bandwidth but would be willing to
     accept a fair share
  b) congestion exists

This is clearly the general case for trans-Atlantic links, most
smaller to medium providers, and is becoming more and more the case
even for larger providers.  Any provider that supports high bandwidth
users will occasionally have congestion when those users try to do
some work.

Translated.  The requirements of the Internet must be supportable.

> which could be changed, see below)

It does not provide any means whatsoever to provide quality support
for both real time and TCP.

> pkts as possible, reordered or not.

I did not hear consensus for this at the meeting.  I heard a lot of
discussion and a lot of disagreement.

> trying to tell your TCP to slow down.

No kidding.  Reservations in an underutilized network is not very
interesting.  Making things only work well for that case isn't very
interesting either.

> (aside from having the usual tail-drop problems)

This would only be done for non-adaptive service, moswt likely real
time.  TCP would not rate shape but go for the whole wire if it can
get it.  With this TCP performance would be horrible.

> realtime traffic when the network is not overloaded.

The point is this would provide good service for TCP if the network
was overloaded and non-adaptive real time using the other b>0 service
would also get lossles service if they rate shaped accurately.

> my r and a smaller b, and clamp the window size of my TCP down a bit.

This is really annoying since I sent a long message about this and you
didn't bother to reply.

As an example, imagine I sit at an SCC and I'd like to get 30 Mb/s
through a 44 Mb/s pipe, but would settle for 1 Mb/s if that was my
fair share.  I set r according to the minimum rate and b according to
the delay bandwidth if I actually got 30 Mb/s.  What I don't realize
is a dozen or so other people do the same thing at the same time.  The
router is required to give me that buffering before dropping.  On the
other hand if there were two such users who both set b according to
the 1 Mb/s d*bw, TCP performance would be quite poor (too little
buffering and TCP don't get along - proven empirically many times).

This is an argment by example.  If you want to reply to my previous
mail, you can dredge it from the archives.  A few people replied, some
in private.  No need to unless you want to.

>    5b) let implementations reject requests of one sort or the other.

You may want to support 3 behaviors:

	1.  the one you have (delay non-conformant)
	2.  police and drop
	3.  what I described as b==0

> agrees on. I'm not sure we're there yet, but if I'm wrong I hope

It is not really that hard for them to support the 3 behaviors I show
above and if your mechanism performs as poorly as I think it will,
we're stuck.

Curtis