Subject: The problem with tagging packets

From: Zheng Wang <[email-masked]>
Date: Mon, 03 Nov 1997 16:48:37 -0500

It seemsto me that one of the problems with tagging packets based on in/out 
profile is how to decide a profile for a user? In most cases, the profile
is destination-dependent. Suppose an ISP has several pipes to the regional
exchanges and to other countries. If a user is communicating with someone
in the regional backbone, it is very likely that the user can be allocated
fairly high rate. On the other hand, if the user is going through a thin
pipe to another country, the allocated rate would be much lower. Therefore,
the allocated rate can not be easily determined at the contract time, rather
it should depend on the size of the congested pipe and the average number 
of users on the pipe.So, weight-based sharing seems to be a much better
option.

Comments?

Cheers
Zheng

---

From: Paul Ferguson <[email-masked]>
Date: Mon, 03 Nov 1997 17:09:04 -0500

>of users on the pipe.So, weight-based sharing seems to be a much better

I thoroughly agree (perhaps not with your example, but at least with
your point). It makes much more sense, at least to me, to weight
the drop preference using more than 1 bit, and subsequetly using a
weighted RED mechanism to deterministically drop during congestion
situations.

A single 'in' or 'out-of-profile' bit seems naive.

- paul

---

From: Roch Guerin <[email-masked]>
Date: Mon, 03 Nov 97 18:13:18 -0500

>of users on the pipe.So, weight-based sharing seems to be a much better

Zheng,

I think that what you are pointing to is not a problem with tagging of
in/out of profiles, but with a specific definition for a profile,
i.e., one that is not destination specific.  If you take a profile
which is the TSpec for an RSVP flow, then you don't have that
problem, and knwoledge of in/out of profile can be useful.

Roch

---

From: [email-masked] (Hyogon Kim)
Date: Mon, 3 Nov 1997 20:49:13 -0500 (EST)

Paul, and Zheng,

>A single 'in' or 'out-of-profile' bit seems naive.

Naive of course if we thought that the profile would be effective all over 
the Internet :-) Well, my understanding was that the profiles an ISP sells
should be effective _at least_ in that ISP. Further guarantee could come
after ISPs make some sort of peering agreement, agreeing to "respect"
(whatever that means) each other's packet markings. Although I agree
that single bit 'in'/'out' marking is naive, I believe that for a "good"
ISP, under-provisioned links should be limited to special cases like 
trans-atlantic link. And I believe that they should be able to tell
customers "for any place in the U.S. you will get your rate 95% of the
time". (sorry for religious pitch)

I am really not "objecting" to the idea of using more bits - if really
necessary we will have to. But what I am saying is that if you have a 
hammer every problem begins to look like a nail, and I am wondering if 
we have a hammer called more bits. Do we all think that the thin pipe
problem is best solved with more bits? that the problem is a generic
problem worthwhile to solve with the bits? I think we should be able to 
say yes to the questions before we dispense this precious resource.

-- Hyogon
------------------------------------------------------------------------------
Hyogon Kim ([email-masked])	       		       |        (201) 829-4728
Research Scientist				       |           MCC 1B-225B
Internet Architecture Research Laboratory	       |      445 South Street 
Bell Communications Research			       |  Morristown, NJ 07960
------------------------------------------------------------------------------

---

From: Paul Ferguson <[email-masked]>
Date: Mon, 03 Nov 1997 21:19:40 -0500

>time". (sorry for religious pitch)

This is not an issue of whether a single profile would be used
pervasively across the entirety of the Internet, but rather, it
is an issue of how to effectively provide differential services.
I would submit that a one bit, binary selection of 'in' or
'out-of-profile' is insufficient to achieve this goal.

Consider the situation where a host (h) is sending data via
a local gateway router (r). The local gateway router (r) could,
for example, use token bucket thresholds to determine how to
mark traffic below and/or above an arbitrary threshold. At first
blush, this might appear to only require a single bit to indicate
whether the packet(s) are within the threshold parameter, or not.
However, consider that there may be several hosts (i.e. h1, h2,
etc.) behind (r), and also consider that an administrator may
wish for traffic below the threshold to be marked differently for
h1 than for h2. This requires a bit more granularity.

This is certainly not unrealistic, and in all likelihood, this
is exactly the type of granularity that is required. 

  h ----> r ----> R -----> R -----> [towards destination]

Of course, a preferential drop mechanism would be implemented on
all intermediate nodes (R).

Assumptions on the engineered capacity between the sender and
receiver can't realistically be made.

>say yes to the questions before we dispense this precious resource.

I would suggest that the 3 bit IP precedence field is wholly adequate
for this purpose. I would also suggest that this field is more appropriate
than the 4 bit TOS field, since the TOS field has been targeted for use
by at least one QoSR proposal which would use TOS values transposed into
OSPF LSA's. Whether or not this proposal is actually implemented in
practice is not germane to this particular thread, but I do believe
that use of the TOS field is more appropriate for QoSR applications
and the IP precedence field is more appropriate for drop preference
applications such as what we have been discussing.

Of course, this is just an opinion, so take it with a grain of salt.

- paul

---

From: Masataka Ohta <[email-masked]>
Date: Tue, 4 Nov 97 14:41:33 JST

Zheng;
> of users on the pipe.So, weight-based sharing seems to be a much better

True.

So, what we can meaningfully have is simple best effort or something
at least as complex as RSVP.

							Masataka Ohta

---

From: "Alessio Casati" <[email-masked]>
Date: Tue, 4 Nov 1997 12:03:18 +0100

The problem at hands is whether the single DP bit (or IPI) is sufficient in order to cope with 
underprovisioning along some routes.
The problem is twofold:

- is the profile as simple as a token bucket?

If so, then one can with some confidence state that along the path to the destination
you can have some chance of finding underprovisioned links. The point is, if you have 
multiple drop precedence levels, which kind of application can take advantage of them?
You would at least need to make the application responsible of marking the packet, the
network should not modify the level an application set on the packet, since it conveys
a degree of importance meaning... But by definition that would be an adaptive application.
So an early feedback mechanism can replace multiple drop levels.
Non adaptive application would not benefit of multiple levels at all.
TCP based applications, in addition, would naturally back off, and the profile (and the 
associated charge) will be suited to it. Multiple drop levels have no reason to be for
them.
I think the profile should take into account some further information, the destination
network(s) in the first place.

- is the committed profile decision merely local?

In my opinion it should be local. However it should be based on some information 
related to available resources in the provider network and at peering points.
Such an information should be conveyed by routing.

So, if the profile allows the sender to specify destinations, and if routing let the local
decision point to be aware of the available resources, you can make the likelyhood of having 
service degradation extremely low.

The solution to the problem is to set up proper interprovider agreements, provision resources 
according to experience and short term variations and have well defined profiles.

The semantic of marking should not be made so complex to take in more problems
then it can solve. Adding stuff and bits sometimes is no less naive than the 
lean soution is.  

Is anybody so kind to help me to make a reality check and change my opinion?

alessio

---

From: "Alessio Casati" <[email-masked]>
Date: Tue, 4 Nov 1997 12:37:16 +0100

> receiver can't realistically be made.

Paul, you are the "so kind anybody" I asked for ;-).
I understand your point, and this kind of marking makes me comfortable since shows an 
example of non application-based marking. Application-based marking (to me) doesn't seem to help 
much. It seems you are talking of a bit (or 2) set by administrators (and maybe providers) to add 
a hook used to identify most important packets (from the administrator anf the ISP viewpoint).
Just in order to check whether I understood your point, is this kind of marking related to the
profile the user selects (or is enabled to select)? That is, you have different profiles ranked on 
a per drop preference basis? Do you agree with me if i say the IPI bit would nontheless be useful
and have a different semantic wrt the drop precedence as it is defined here?

Bye

Alessio

---

From: Dimitrios Stiliadis <[email-masked]>
Date: Tue, 4 Nov 1997 08:07:07 -0500 (EST)

> all intermediate nodes (R).

I am afraid I don't understand your example or what the administrator
is trying to achieve. If a different marking is required 
between two hosts, you simply use different profiles. A 
different number of packets will be marked ... and all
probabilities will be derived from this. If we extend
this example the way it is presented, it implies that 
we must have as many bits as hosts .... ????

> applications such as what we have been discussing.

Actually it seems to me that using the available bits
for delay priorities is much more important. No matter
how many packets are marked, by changing the 
delay-priority/bandwidth one can control the drop 
probabilities inside the network. Not only that, but
it can give some firewall properties among different types
of applications.

CBQ and similar systems have the additional advantage,
that no matter what the traffic distribution is, they
can allocate bandwidth to different types of traffic.
If some weird traffic distribution appears, no matter
how many drop bits are there, it is not realy possible
to control how much bandwidth a particular type of 
application takes.

Dimitri

---

From: Paul Ferguson <[email-masked]>
Date: Tue, 04 Nov 1997 09:05:59 -0500

>profile the user selects (or is enabled to select)? That is, you have
different profiles ranked on 

Either the client host application can mark packets with the precedence,
and the ingress router can passively admit them, or the ingress router
can actively police & mark incoming packets.

>and have a different semantic wrt the drop precedence as it is defined here?

No, I don't agree -- I think it is much too simplistic.

The example I used earlier suffices here, I believe. For example, an
ingress route may use a token bucket threshold to mark all packets
from a particular host below 56 kbps as IP precedence '5' (binary 101),
and all packets from the same host in excess of 56 kbps as IP
precedence of '1' (binary 001). If there is no congestion in the
network, everyone's traffic gets delivered in a reasonable amount
of time and everyone is happy. However, when there is congestion,
all traffic in excess of 56 kbps from this host will be
probabilistically subject to drop.

Of course, there is a reliance on the architecture & design of
the network. If it is horribly designed, not differential CoS
or drop preference scheme will ever help you.

- paul

---

From: Zheng Wang <[email-masked]>
Date: Tue, 04 Nov 1997 10:29:55 -0500

As I see it, the tagging scheme is essentially an admission control 
mechanism. A user and its ISP agree on the profile for the user (for the 
simplicity, let's just use rate). The ISP will honor the traffic if below the
agreed rate and anything above the rate is tagged and subject to dropping
(within the ISP's own network and on its access link to the backbone).
So, an ISP can differentiate users by allocating different rates based on
the depth of the users' pocket.

An ISP can decide the rate easily if it has a simple topology (e.g., one
bottleneck access link) and stable traffic patterns (the average number of
users for the bottleneck link is stable). 

But if the topology is a little bit complicated or the traffic pattern
changes daily, the short-term rate (average over mins, for example) 
tends to change rapidly, and depends very much where you do.

Here we are talking about sender-based tagging. In the receiver-based tagging,
the the tagging scheme becomes a feedback control mechanism (like the DEC bit).
We are again relying on the source to reduce its traffic. But it is becoming
increasingly difficult to rely on the users' good behavors to manage the
Internet.

Cheers
Zheng

---

From: Paul Ferguson <[email-masked]>
Date: Tue, 04 Nov 1997 10:36:08 -0500

>we must have as many bits as hosts .... ????

Of course not, that would be ridiculous.

Using the IP precedence field allows 0-7 levels of
abstraction, if you will. Most service providers
that I have talked to have not expressed a desire
to implement more than 3, at least for the near term.
Using a traffic classifier at the ingress router is
a reasonable method of defining acceptable amounts
of granularity.

Perhaps we're also talking about the same thing.
A 'profile' could certainly be a 'template' or
'access list' or whatever which defines the
marking which is to take place, at what thresholds,
and on what traffic (source, destination, port, etc).

Of course, another variation is to use the extensible
intserv message objects (e.g TSpec), compared to the
threshold, and subsequently map IP precedence based
on threshold comparison within the same context of
variables.

Each of these methods as active admission schemes
at the ingress router -- the router could, of course,
passively admit traffic as it is marked by the clients.

>it can give some firewall properties among different types

I would submit that delay control/priorities brings us
full circle to a method to _determine_ a priori delay
characteristics and stateful maintenance of this criteria.

What I believe service providers are looking for is a
simpler method to provide differential services, one
which may not include negotiation of delay issues.

>can allocate bandwidth to different types of traffic.

No, they don't allocate bandwidth, they simply
deviate from the fundamental FIFO single queue
servicing model. I suppose you could look at it as
unfair queue servicing, which in turn could give a
certain class of traffic a preferential opportunity
to transmit packets on the available bandwidth.

>to control how much bandwidth a particular type of 

I'm not worried about bandwidth control, I'm more concerned
with preferential drop. If there is no bandwidth contention,
why worry about bandwidth control? There should be an opportunity
for best effort traffic to use more than it's 'share' if it
is available.

- paul

---

From: "Alessio Casati" <[email-masked]>
Date: Tue, 4 Nov 1997 17:27:19 +0100

On  4 Nov 97 at 9:05, Paul Ferguson wrote:

> It seems you are talking of a bit (or 2) set by administrators (and
> maybe providers) to add a hook used to identify most important packets (from the administrator and
> the ISP viewpoint).
P> Either the client host application can mark packets with the precedence,
P> and the ingress router can passively admit them, or the ingress router
P> can actively police & mark incoming packets.
> and have a different semantic wrt the drop precedence as it is defined here?
P> No, I don't agree -- I think it is much too simplistic.
P> 
P> The example I used earlier suffices here, I believe. For example, an
P> ingress route may use a token bucket threshold to mark all packets
P> from a particular host below 56 kbps as IP precedence '5' (binary 101),
P> and all packets from the same host in excess of 56 kbps as IP
P> precedence of '1' (binary 001). If there is no congestion in the
P> network, everyone's traffic gets delivered in a reasonable amount
P> of time and everyone is happy. However, when there is congestion,
P> all traffic in excess of 56 kbps from this host will be
P> probabilistically subject to drop.
P> 
P> Of course, there is a reliance on the architecture & design of
P> the network. If it is horribly designed, not differential CoS
P> or drop preference scheme will ever help you.
 
It's important now to stress why I have concerns about multiple drop preferences.
Maybe I'm wrong, but if you set different DP levels you'll have to negotiate 
not only the bandwidth, but on the buffer side too: if you allow fellow A to send more packets with 
lower DP than fellow B, then A is expected to receive better service than  B. So some negotiation 
with the ISP is needed in order to prevent badly biased network behaviour. I let you imagine the 
mess a double sided network provisioning and profile negotiation entails. 
That's why I still lean for a lean solution, (more Delay classes and IPI bit). Yet It's only an 
opinion, there's a lot of work to be done.

Bye

---

From: [email-masked] (Hyogon Kim)
Date: Tue, 4 Nov 1997 12:02:53 -0500 (EST)

Zheng,

>tends to change rapidly, and depends very much where you do.

Maybe we keep talking about different things here but I don't understand
why you should consider complex topologies. You don't have to look
at the entire topology to see whether a particular link is under-engineered 
or not. It is local to that particular link. I.e., the fraction of the
"out" (or "in") packets among admitted packets, R', is a good indication of, 

R = (sum_of_the_profiles_of_competing_flows_at_the_link / link_capacity).

As for the time scale, an over-engineering would have R' < 1 at any time
of the day, and 95% engineering would meet R' < 0.95 or something like
that. I'm sure you could be much more intelligent than this (I am aware that
provisioning the simple sum of the profiles is not a smartest thing to
do so don't flame me on this ;-) but my point is that whatever engineering 
model you use, basically you can always look at R' and its fluctuations.

>increasingly difficult to rely on the users' good behavors to manage the

Two questions I have on receiver-based tagging: 

1. should it be separate from the sender-based tag (I guess so)? 
If so, we should set aside one bit for rate and one more bit for delay...
so it's already 4 bits and we used up the precedence bit field plus
the reserved bit. Since Paul argues we should leave the TOS for QoS
routing, I guess we are done now about bit allocation ;-)

2. in case we use the receiver-based tag like ECN as suggested above,
would it mean that IP at the receiving end (i.e. the sender) should copy 
the bit somewhere in the TCP header to slow down the sender TCP?
Hmm...

-- Hyogon

---

From: Paul Ferguson <[email-masked]>
Date: Tue, 04 Nov 1997 11:25:12 -0500

>the depth of the users' pocket.

Yes, I agree with this in principle, but as I stated earlier,
I believe a single 'in' or 'out-of-profile' bit is simply too
simplistic. A variation of the theme you outlined above would
be that the ISP will honor the traffic if below the agreed
rate and invariably will have the opportunity to tag packets
which conform to this threshold. Anything above the rate is
also tagged, but with a different value, and is possibly subject
to dropping further along the path.

>increasingly difficult to rely on the users' good behavors to manage the

Of course. No one is obviating the need or desire to possibly
also rate-shape traffic. This is another valuable tool, and might
be done in a number of ways.

A variation would also be (using the earlier reference) the ISP
will honor the traffic if below the agreed rate and invariably
will have the opportunity to tag packets which conform to this
threshold. Anything above the rate could simply be dropped at
the ingress router, if this were the chosen policy.

- paul

---

From: Zheng Wang <[email-masked]>
Date: Tue, 04 Nov 1997 13:37:04 -0500

> R = (sum_of_the_profiles_of_competing_flows_at_the_link / link_capacity).

We are talking about the same thing but my starting point is that if I have 
an ISP, how can I start to provide differentiated services now (not what link
I need to expand before I can think about differentiated services).
Anything that depends on ISPs' bandwidth provision would not go very far.

What I am saying is that R will vary from one congested link to another one
(if you base your analysis on current ISPs' infrastructures), and R in fact
changes daily because the the number of competing flows change daily (or
hourly).

Cheers
Zheng

---

From: Zheng Wang <[email-masked]>
Date: Tue, 04 Nov 1997 13:52:37 -0500

> the ingress router, if this were the chosen policy.

But the ingress routers are far away and close to the traffic sources.

Suppose all users of an ISP are pulling traffic into the ISP (which
is more or less true for a dial-up ISP), and the bottleneck is the
link between the ISP and the backbone, what we really want do is to
honor the agreed rate on the incoming traffic at the bottleneck.

Unless the packets are tagged at the sources (Web servers) based
on the *receivers' profile*, the border router does not have the
info to discard packets.

Cheers
Zheng

---

From: Paul Ferguson <[email-masked]>
Date: Tue, 04 Nov 1997 13:45:34 -0500

>routing, I guess we are done now about bit allocation ;-)

I fail to see why would we need separate bits for sender
(originating host) and reciever (ingress router) tagging.
These two diffierent policy implementations -- sender
taging is passive admission; reciever marking is active
admission control and policing. Both implementations can
use the same bit space or bit values.

- paul

---

From: Jon Crowcroft <[email-masked]>
Date: Tue, 04 Nov 1997 22:39:20 +0000

>- is the profile as simple as a token bucket?

oh really - a profile is a LIST of
classifications (including wildcard matches for src or dst, net, port
and so on) AND for each, a list of flow specs....

which is why we need a new protcol (the Broker in Van's architecture)
and at least some non trivial extensions to RSVP to install it....

cheers
 >Is anybody so kind to help me to make a reality check and change my opinion?

 jon

---

From: Paul Ferguson <[email-masked]>
Date: Tue, 04 Nov 1997 17:52:25 -0500

>on the *receivers' profile*, the border router does not have the

The point in the topology where packets are marked is really
not the issue here, but rather the semantics of how they are
marked, how many bits will be used, and which bits they will
be. Let's tackle one issue at a time.

- paul

---

From: "Alessio Casati" <[email-masked]>
Date: Wed, 5 Nov 1997 10:29:05 +0100

> be. Let's tackle one issue at a time.

OK Paul,  can you suggest the semantic of more drop levels?

---

From: "Alessio Casati" <[email-masked]>
Date: Wed, 5 Nov 1997 11:02:36 +0100

On  4 Nov 97 at 22:39, Jon Crowcroft wrote:

> [...] we need a new protcol (the Broker in Van's architecture)

Perfectly agree.

> and at least some non trivial extensions to RSVP to install it....
 
Ideally I think the broker protocol should be much simpler than RSVP and not end to end,
and activated only when the broker, based on local information, detects insufficient
rersources (or an overflow on some threshold on resources usage) and wants to signal
to some network functionality the need for more, in order to prevent congestion.

alessio

---

From: "Eric S. Crawley" <[email-masked]>
Date: Wed, 05 Nov 1997 09:03:43 -0500

OK, I've been trying to keep quiet on this thread but this last exchange
has started things going in a direction that I don't think was intended.

Let's try to remember some of the reasons we got into this conversation to
begin with.

- Service providers have been hesitant to install per-flow, highly-dynamic,
QoS  protocols such as RSVP.
- Customers of the service providers are demanding more reliable and
consistent services.
- *SIMPLE* mechanisms are needed to allow providers to design service
offerings to meet the cusomter's needs.

I have to agree with Paul that we need to define very simple mechanisms to
achieve this goal.  It is not up to us to tell the providers how to
engineer their networks, just to give them the tools to separate and
classify the traffic.  They can then design their profiles, set up their
queuing, and engineer their networks to handle whatever profiles they come
up with.  When you start getting into things that "automatically" adjust
traffic engineering parameters in response to things like congestion, the
providers start getting very nervous.  This is where the "Bandwidth Broker"
ideas start getting us into a deja vu mode with RSVP.  I'm not saying that
we won't want a "Bandwidth Broker" but I don't want to tie up designing
simple mechanisms for separating traffic and allowing providers to design
service offerings with development of a YASP (Yet Another Signalling
Protocol).  If we have the separation and classification mechanisms, the
signalling can be added later.  Besides, if someone wants hard guarantees
on a particular flow, why don't they use RSVP anyway?

As far as Drop Preference goes, I think I have been converted to believe
that more bits is not necessarily a bad idea.  Where things get intersting
is when we combine multiple delay priorities with drop preference.  Here's
an example using just best effort traffic:

The provider offers three kinds of service:
- Traditional best effort on everything
- X kbits/sec of "high reliability" traffic with everything over X kbits
dropped if congestion is encountered.
- Y kbits/sec of "higher reliabiltiy" traffic with everything over X kbits
getting marked but not necessarily dropped.

At the customer's ingress point to the network, packets are marked
according to the profiles.  For the best effort service offering, packets
are marked with a DP of say 3.  For the second customer, all "in profile"
traffic is marked with DP 2 and the "out of profile" traffic is marked with
DP 7. [Note, I'm using higer DP values to mean greater priority for
dropping.] The third offering is marked with "in profile" traffic marked
with DP 1 or 2 and "out of profile" traffic marked with 3.  

In the ISP network, packets with a higher DP are dropped before packets
with a lower DP, based on whatever mechanism the vendor implements.  When
packets exit the provider's network, they are marked according to the
bi-lateral agreement the provider has with the neighbor.  The provider can
engineer their network to provide whatever statistical multiplexing gains
they want for each DP level.  If they want to overload a particular DP
level with too much traffic, then they can take that risk and the customers
can complain or not about the ability of the provider to meet their profile.

What I'm trying to separate here is the mechanism from the economics.  We
don't want to choose the economic models for the service providers.  We
want to provide a set of tools for them to create service offerings that
they can use to stay competitive.

I actually think that the problem of "receiver-based" marking can be
somewhat ignored.  The service contracts have to note that the customer is
paying for bandwith *into* the network and not necessarily out of the
network.  If a customer complains that they are not getting adequate
performance from a site that is outside the providers network, the provider
can use this as a tool to get the outside site connected to the provider's
network to assure that it can service the customers on the provider's network.

So, what needs to be standardized to make this happen?  In my opinion, the
only pieces that are needed for interoperability are the semantics of the
bits for drop and delay.  We need to decide how many bits are used for each
and what they generally mean (lower DP value means lower drop probability
or vice versa, etc.)  I that if we start assigning more than simple
ordering to these bits, we are asking for trouble.  Saying that delay
priority 1 means a particular type of service rather than just a priority
better than 0 (or 2 if you decide the ordering goes the other way) will
cause us to spin around endlessly trying to make service models that will
be useless the day they become an RFC.  The goal is to provide tools for
the providers and not to design full end-to-end services.  

[Sorry this rant went this long.]

	Eric

---

From: Paul Ferguson <[email-masked]>
Date: Wed, 05 Nov 1997 08:16:54 -0500

>OK Paul,  can you suggest the semantic of more drop levels?

Why not use the 3 precedence bits as drop preference
indicators?

000  High drop preference
001  --
010  --
010  --
011  --
100  --
101  --
110  --
111  Low drop preference

- paul

---

From: Zheng Wang <[email-masked]>
Date: Wed, 05 Nov 1997 10:46:06 -0500

Eric

I generally agree with much of what you said except the following

> network to assure that it can service the customers on the provider's network.
 
If a customer is paying for traffic into its ISP, tagging at the 
customer-ISP interface does not make sense since you can only tag 
packets going out of the ISP.

Let me illustrate with an example:

Suppose ISP X has 3 customers A, B, C respectively subscribed to 3 classes
(basic, silver and golden services, for example), and they all surf the
same Web server in the backbone. Suppose that the link between the ISP X and
its backbone is congested as a result of traffic coming from the Web server.

Customer A ---\			|
               \		| <----- traffic
Customer B ---------------------|------------------- Web server
               / congested link |
Customer C ---/ 		|
			    border router	

		ISP X			Backbone ISP

Tagging packets injected into ISP X by customer A, B, C does not help here
since the congestion is caused by incoming traffic from the web server.
How much bandwidth the Customer A, B, C get depends on the border router 
between
ISP X and its backbone provider. When entering ISP X, packets are not tagged
based on customer A/B/C's profile.

I believe that the receiver-based case is the most important one if
we believe that most of the traffic is web-based. 

Cheers
Zheng

---

From: Juha Heinanen <[email-masked]>
Date: Wed, 05 Nov 1997 18:47:28 +0200

> routing, I guess we are done now about bit allocation ;-)

why can't the same tos bits be used for both routing and scheduling?  in fact, i
don't understand how routing could be separated from scheduling.  if a packet is
supposed to get low delay/jitter, it means that it should get BOTH a low delay
route AND low delay treatment within the routers.  that is why i suggested in my
i-d that the first three bits of the tos octet are reserved for dp and the tos
field is used to describe delay characteristics.

-- juha

---

From: [email-masked] (Hyogon Kim)
Date: Wed, 5 Nov 1997 11:51:12 -0500 (EST)

Zheng, 

>Anything that depends on ISPs' bandwidth provision would not go very far.

I understand what you are saying. However, I still belive that
assumption(s) on provisioning is an essential part of differentiation,
if predictability is what we want to achieve. Well, in any case, it seems
to me that both of us don't have numbers to support our qualitative
arguments so I suggest we agree to disagree and leave it there.

>changes daily because the the number of competing flows change daily (or

Yes, R varies from link to link of course so I said "It is local to that
particular link" above. As for the feasibility of capturing R within
some range, again I disagree with you but I guess that's it.

Thanks,
-- Hyogon

---

From: [email-masked] (Hyogon Kim)
Date: Wed, 5 Nov 1997 12:01:30 -0500 (EST)

Juha,

gotcha. 

-- Hyogon

---

From: "Alessio Casati" <[email-masked]>
Date: Wed, 5 Nov 1997 17:59:05 +0100

> Why not use the 3 precedence bits as drop preference

Great. It's a backward compatible system. Do you think everybody should have to
support 8 drop levels in their boxes in order to be compliant, or, as Juha suggested,
would you let clustering some levels, thus letting vendors and providers making the choice they
think the best for their systems and networks?

---

From: [email-masked] (Hyogon Kim)
Date: Wed, 5 Nov 1997 12:31:23 -0500 (EST)

Paul,

first of all, a correction. Receiver-based tagging is not what
the ingress router from the sender side does, IMHO. It's done by 
whatever marking entity at the receiver side, which happens to
be responding to the traffic from the sender. E.g.,

A (web client) ---mA------------------> B (web server)
               <-------------------mB---

In the above picture, mA is the marker for A wherever it resides (A
or A's ISP) and similarly mB is the marker for B. So the 
receiver-based marking refers to the case where mB marks the
traffic B->A at the request of A. 

This is a complicated problem, which needs help from routing
protocols (BGP community I heard from someone - but I don't know
the details), OR, as I idly imagined, from some "escort" packets sent 
from A. It's a very stupid idea so I am kind of hesitant to say
it but here you go. The escort packet from A travels all the way
up to B, installing credits so that the unmarked (high drop
preference) packets from B can get a mark each hop. Since
the escort packets also goes through mA, they need the sender
mark bit (probably "marked" not to lose this pseud-signalling
packets), and to prevent routers from looking inside the packet to
realize the packet type, "escort". So the escort packet will
carry the credit in the receiver mark bit...... Cosidering
asymmetric routes, waste of router capacity usually limited
by pps, and the difficulty of specifying credit amount, it's
probably a crap. However, I was probing the possiblity.

In any case, the above web server/client scenario happens because
you don't want to pay a lot of money to open a public web
server which unfortunately(?) happens to be enormously popular. You don't 
want to pay money to make sure that your web pages are delivered
to casual browsers with high probability, except when you are
doing some web commerce or something like that.

In reality, the asymmetry is indeed a problem. I know an ISP
in a foreign country with has multiple T3 lines to U.S., and
the ratio of incoming traffic (from U.S) to outgoing traffic
is nearly 10:1. It's useless to do differentiation against
the outgoing traffic because it is the incoming link that
is congested. However, we should be very careful and ask the
question "is this a generic, systematic problem that we should
solve using more bits" again here, before we commit the 
bit resource.

-- Hyogon

---

From: Jon Crowcroft <[email-masked]>
Date: Wed, 05 Nov 1997 17:54:17 +0000

>and at least some non trivial extensions to RSVP to install it....

to extend this,

a user maybe a person, an appllication, a host, a subnet a net or an
AS

a profile may be for pairwise combinations of these, or group
destinations, or multicast groups as defined in the new BGMP spec (see
IDMR), with adequate provision for masking on unicast addresses, and
for source ASs or "root" ASs in BGMP as destinations

a profile is a list of these plus a qualectivity parameter - this is 
an indication of a type of flow e.g. :

best effort
guaranteed
controlled load
new TCP goodput (as per Dave Clark and John Wroclawski
and any new ones we come up with

togertehr with the flow spec leaky bucket if needed (not required for
best effort....)

the classical generalized port specifications are either wildcarded in 
places (e.g.  all traffic from src IP addr, or to dst IP addr, or between prefix
10/24 and 11/24) with wildcarded ports, or specific ports, or for UDP,
say its RTP, with qualiectivity parameters that have to be measured
(since we can't use ports to key quickly) to check

and get more general (shorter matches) as  you get away from the 
"edge of a net"< or approach the interface between backbone
providers....

degermark or other schemes allow small fast macthcing implementations
of these co,mbined with simple lookup or hash functions for ports and
other smaller well known fields.....

to put these in palce requires modest additions to RSVP (so it can
operate to routers from and to network management systems that run the
profile subscription database systems, ratehr than from src and dst
hosts) and can manage the addr masks etc...

ipsec can be used to key whether a src or dst is autneticable and
therefore allowd to use a profile that costs and that the user really
is the authed principal......

the rest is just normal routing....

the synergies from BGMP and mBGP+ are clear (many many profiels will
need none of the gubbins above).....

jon

---

From: [email-masked] (Hyogon Kim)
Date: Wed, 5 Nov 1997 13:09:43 -0500 (EST)

>and at least some non trivial extensions to RSVP to install it....

Before we make the profile that complicated, I would like to ask if
frame relay CIR does that, too. Although there is no rule against making
our profiles richer and more intelligent than CIR, at least the profile
idea came from CIR (IMO), and I want to see justifications or reasons 
why ours cannot but be richer than the original.

Hyogon

---

From: [email-masked] (Hyogon Kim)
Date: Wed, 5 Nov 1997 14:25:10 -0500 (EST)

My two cents: 

[First of all, I would be surprised to see if someone can show me
8 different drop preference levels indeed result in different 
throughputs disjoint enough from each other so that users can "feel" 
the difference. But in any case,]

one thing you should take into account when building a router that
supports 8 drop preferences based on RED is that the required buffer 
size gets larger and larger as the number of preference levels 
increases. Let me take a simple example  of 2 drop preference levels.
Suppose the aggregated profiles happens to be much smaller than the link 
capacity.  If source are TCP, the throughput is driven by "out"(i.e. level-2) 
packets because TCPs quickly go over their tiny profiles but still they 
find excess link capacity (e.g.  no drop). If you assign small thresholds 
to level-2 packets then you see funny things like low utilization, and that 
means each level must have sufficient buffer space of its own, large enough to
avoid such funny things (also see Floyd's original RED paper where she
mentions this briefly). Unless the thresholds are dynamically 
adjusted acoording to the incoming traffic pattern (which is also 
difficult when RED dynamics study itself is lacking), supporting 
many drop preference level increases queueing delay. I don't know
how this will intereact with low-delay queue. So, the picture looks
like:

  minth(1) < maxth(1) < minth(2) < maxth(2) < ... < maxth(8)

And this is one of the reasons why I do not like to jump on 3-bit
preference level field wagon. Hardly anything technical has been presented
(not to mention confirmed) as the supporting argument for 3-bit definition.

I admit that RED might not be the only queue manangement scheme we
can use, but what I am saying is that we do need more technical study
and that we must be sure of its benefits before we commit as many as 3 bits.

-- Hyogon

---

From: "Eric S. Crawley" <[email-masked]>
Date: Wed, 05 Nov 1997 14:53:36 -0500

Zheng,

I understand your example and making a receiver-based tagging system work
is very difficult (that is partially why I chose to ignore it ;)

There are some assumptions that you are making that I don't necessarily
agree with:

- I think that the most important customers for differential services are
those companies that are managing their own private networks now and want
to "outsource" this function to an ISP (I think the press refers to this
type of network now as an "ExtraNet").  With all the company locations
connected to the same ISP, sending and receiving is satisfied between the
locations.

- Rather than consider the lack of ability to provide differential services
to external sites as a bug, I consider it an opportunity for the ISP to
gain new customers.  When the customers complain to the ISP that they have
trouble getting to web site Foo, the ISP can contact web site Foo and
encourage them to get "Gold" service on their ISP.  This is identical to
the problem nowadays with link speeds to web server farms.  If customers
complain if popular web sites are behind 56Kb links, they are upgraded to
T1 links, etc.  I agree this isn't the best technical solution, but it is
one that fits the current economic models.

	Eric

---

From: Paul Ferguson <[email-masked]>
Date: Wed, 05 Nov 1997 14:59:08 -0500

Again, I'm not suggesting using 8 different drop levels
in the initial stage of deployment, just that using the
precedence bits allow for this many levels of indication.
The major hurdle here is developing a common interpretation
of these bit values which can be globally interpreted.

- paul

---

From: Wu-chang Feng <[email-masked]>
Date: Wed, 05 Nov 1997 15:36:37 -0500

>   minth(1) < maxth(1) < minth(2) < maxth(2) < ... < maxth(8)

I agree.  In order to ensure that the drop rates of high precedence packets
are strictly less than drop rates of low precedence packets over an arbitrary
packet arrival stream, you must set the threshold values here significantly
far apart.  I think we exchanged messages about this in the context of a
single bit ERED/RIO queue a couple weeks ago.  Extending this discussion
to 8 levels, certainly exacerbates the queue management problem of
achieving high utilizations when only low priority packets are in the network,
while protecting potential high priority packets arrivals from being subsequently
dropped.  Setting the threshold values far apart is a big problem since it means
that by the time service differentiation kicks in for the high precedence packets
a large queue at each router must be present.

Wu

P.S.  Before I get flamed for talking about implementation, let me just say
that I still don't understand the benefit that ignoring implementation issues
have when determining these bit semantics.

---

From: Steven L Blake <[email-masked]>
Date: Wed, 05 Nov 1997 17:16:37 EST

>   minth(1) < maxth(1) < minth(2) < maxth(2) < ... < maxth(8)

This isn't the only way that WRED can differentiate loss probabilities for different drop-preference levels.  Several DP levels could use the same
values for min_th(n) and max_th(n) with progressively larger values for
max_p(n) (the maximum loss probability for DP level n before avg > max_th(n)
where all packets of level n are discarded).  The thresholds could be
structured like so:

  min_th(1..m) < max_th(1..m) <= min_th(m..n) < max_th(m..n)

while

  max_p(1) < max_p(2) ... < max_p(m)

and

  max_p(m+1) < max_p(m+2) ... max_p(n).

Another way for WRED to differentiate loss probabilities is to use different
averaging functions for each DP level with progressively shorter time
constants [not that I think that this is a good idea].

> (not to mention confirmed) as the supporting argument for 3-bit definition.

I agree with Roch in that I would rather see one bit declared as "drop-
preference" and two or more bits declared as "service class".  What I mean
by that in implementation terms is that I would rather that the profile
meter only be able to dynamically toggle one bit to select in/out-of-profile
(it could select/authorize one of the service-class bit combinations for all
of the packets within a particular flow).  The nominal implementation would
be one queue per service class with two-level WRED within each queue.  This
wouldn't preclude single-queue multi-level WRED-style implementations when/if
link-speed required it, but it would establish rules in the bit semantics as
to which combination of bits if toggled dynamically could lead to packet
reordering problems.

Regards,
/*=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
| Steven L. Blake                 [email-masked] |
| Internetworking Technology         (919)254-2030 (work) |
| IBM Networking Hardware Division   (919)254-5483 (fax)  |
| "You like the 'Net?  You ain't seen nothin' yet!"       |
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=*/

---

From: Wu-chang Feng <[email-masked]>
Date: Wed, 05 Nov 1997 18:32:18 -0500

>> min_th(1..m) < max_th(1..m) <= min_th(m..n) < max_th(m..n)

If you really want DP to mean something regardless of the input traffic
applied, you need strict thresholds with the property Hyogon suggests.
Anything less will dilute the meaning of DP and make it hard to predict
any sort of end-to-end behavior.

I would much rather have the property that DP means an order of magnitude
less loss over non-DP regardless of the applied DP/non-DP traffic, than
have DP mean X% less loss over non-DP where X depends on how much
DP/non-DP traffic there is at a particular instant of time and how well behaved
each traffic stream is at that time.

I had some experiments where the I had the min_th and max_th of DP and
non-DP packets overlap, but which had max_p values which differed
by two orders of magnitude.  Unfortunately, this approach simply doesn't
protect the DP packets when non-DP traffic is either extremely bursty or
non-responsive.

Wu

---

From: "Kent W. England" <[email-masked]>
Date: Wed, 05 Nov 1997 15:14:49 -0800

>h1 than for h2. This requires a bit more granularity.
I think we have to decide what other header bits are included along with
the QoS bit. If the tuple includes src addr/port, dest addr/port and
QoS-bit then one bit is enough. If the QoS-bit is interpreted w/o reference
to other header bits, then we might need 16 or 32 bits (a flow ID).

>OSPF LSA's. 

Absolutely on target. But let's also always include what other bits we
assume the router is looking at along with the QoS-bit(s).
>Of course, this is just an opinion, so take it with a grain of salt.
Just a iota is needed, not an entire grain.

--Kent

---

From: Paul Ferguson <[email-masked]>
Date: Wed, 05 Nov 1997 19:15:38 -0500

[Eric: I like long, thoughtful rants. :-)]

I wholeheartedly agree with all of the points you made
(which I did not quote), especially with specific regard
to simplicity. Our job is to define semantics & mechanics,
let the masses determine how to apply the tools (we can
certainly make suggestions).

If both delay & drop preference indicator bits are desired,
why not use both the 4-bit TOS field (for delay indication)
and the 3-bit precedence field (for drop preference)?

The TOS field could used in a fashion similar to discussed
in RFC1349. I'm not necessarily suggesting that we use these
values, but at least it's something to think about.

[snip]

4.  Specification of the TOS Field

   As was stated just above, this memo redefines the TOS field as a four
   bit field.  Also contrary to RFC-791, this memo defines the TOS field
   as a single enumerated value rather than as a set of bits (where each
   bit has its own meaning).  This memo defines the semantics of the
   following TOS field values (expressed as binary numbers):

                    1000   --   minimize delay
                    0100   --   maximize throughput
                    0010   --   maximize reliability
                    0001   --   minimize monetary cost
                    0000   --   normal service

[snip]

The values expressed in the precedence field could be used
as the drop preference indication -- lower precedence values
indicate higher probability of discard in times of congestion,
higher values indicate lower probability.

Also, simply because there is 'more than one' bit available
in each field, as well as various combinations of each, does
not necessarily mean that actually using each and every
combination is a good thing in practice.

- paul

ps. I'm also willing to document this in a draft, either
individually or as a collective effort.

---

From: Paul Ferguson <[email-masked]>
Date: Wed, 05 Nov 1997 20:12:54 -0500

>to other header bits, then we might need 16 or 32 bits (a flow ID).

I'm not sure this is such a great idea. Existing mechanisms
to identify src addr/port, dst addr/port directives (as well
as other information) are already available for classification
purposes -- no need to complicate a diff-serv traffic classification
mechanism with this (it is really implementation specific).

Even if you have flow-based information available, 1 bit is arguably
not enough with which to do anything useful.

>assume the router is looking at along with the QoS-bit(s).

On the flip side of this :-) one could always use the 4-bit TOS
field for delay indication, should there be a public outcry that
this is needed, assuming that (a) the semantics in RFC1349
were honored or (b) the semantics in RFC1349 were changed.

- paul

ps. As an aside, I also understand that the ospf wg is
deprecating the support for TOS due to lack of
implementation.

---

From: Paul Ferguson <[email-masked]>
Date: Wed, 05 Nov 1997 20:30:26 -0500

>think the best for their systems and networks?

I think we need to define semantics, and let the masses determine how
many they want to use.

- paul

---

From: David Black <[email-masked]>
Date: Wed, 05 Nov 1997 20:32:26 -0500

Eric,

I agree with almost everything in your original message, except for the
desire to ignore receiver-based tagging.  I have another scenario to
add to Zheng's, but first let me comment on your reply to him.

> connected to the same ISP, sending and receiving is satisfied between the

That's among the most valuable paragraphs posted to this list recently. Recall
Dave Clark's comment in Munich that the major difficulty in this area is
defining the problem that we're trying to solve.  I disagree with you in that I
think businesses looking to outsource networks are among, but not exclusively
the most important customers ... but I commend you for opening up this *very*
important area of discussion.

> one that fits the current economic models.

The risk in this model is that it discourages ISP to ISP agreements among ISPs
with overlapping territories.  ISPs are already using enhanced service within
their own networks as a competitive weapon, and the problem with this scenario
is that it can easily lead to the situation FedEx described in Memphis where
one has to do business with a bunch of ISPs because none of them will enter
into QoS contracts with any major competitor.  I don't believe that the ISP
business (or even the backbone portion of it) is going to turn into a
winner-take-all scenario where we get a dominant player (and that's what has to
happen to avoid the FedEx-like result under these conditions).

Now let me add another scenario to the discussion by turning Zheng's example on
its head.  Suppose the web site is buying enhanced service, the customers
aren't, and the congestion is in the other direction, causing the customers
difficulty in accessing the web site.  If the web site could buy enhanced
service for customers accessing the web site, that would be a competitive
advantage -- the customers notice that this web site works well, that one
doesn't and know which one to try first next time.  Ignoring the technical
problems for the moment, what's wanted is that the customers' ISPs mark packets
going to this web site and treat them preferentially in exchange for some cut
of the extra money the web site is paying for enhanced service.  This
receiver-pays model is a very popular and powerful way of doing business -- 800
(toll-free, WATS, or Freefone) numbers are so popular that the US just had to
create a second area code to accomodate them.  If there's any widespread move
to traffic metering (which I don't see yet), this analogy is dead on, and
even without metering, the added value of bypassing congestion for John Doe
to buy stuff from a business's web site makes it powerful.

So, if we could build it (receiver-based marking), I think they almost
certainly would come (and with a vengeance) ... but I have severe concerns
about the scalability of the resulting mechanism due to the difficulty of doing
aggregation, leading me to agree that making such a system work is "very
difficult".  I assert that the rewards from doing so would be commensurately
large.

--David

============================================================================
David L. Black            THE OPEN GROUP     Voice: +1 (617) 621-7347
The Open Group                                 Fax: +1 (617) 621-8696
Eleven Cambridge Center      RESEARCH       E-Mail: [email-masked]
Cambridge, MA  02142        INSTITUTE         http://www.opengroup.org/~dlb/
============================================================================

---

From: Paul Ferguson <[email-masked]>
Date: Wed, 05 Nov 1997 20:53:11 -0500

>difficult".  I assert that the rewards from doing so would be commensurately

Reciever-orientation sounds remarkable like RSVP.  ;-)

Let's get the easy stuff out of the way first
(sender-based marking).

- paul

---

From: Jon Crowcroft <[email-masked]>
Date: Thu, 06 Nov 1997 07:25:45 +0000

>individually or as a collective effort.

that'd be very useful - i think the achitecture for deploying profiles
is the other key piece of the picture - who's working on that?

can anyone from the traffic theory world
 tell me what happens if one deployed 1 lelvel of DP in some
regions, and n in other regions, and differentiated trafficc was mixed
across mixed region types?

thanks

 jon

---

From: "Alessio Casati" <[email-masked]>
Date: Thu, 6 Nov 1997 10:57:22 +0100

> QoS  protocols such as RSVP.

Yes, I agree, RSVP is not a piece of cake for providers.

> - Customers of the service providers are demanding more reliable and

Yes, Van was very clear on this point, it is the only hope for internet self-sustainability
(these customers are going to pay for these services)

> offerings to meet the cusomter's needs.

Yes, we everybody hope we are able to provide providers with that, that's why we are having 
message eschange these days on this mailing list.
 
> achieve this goal.

I even lean for a simpler solution than Paul's (although it maybe naive, but don't know yet!!!!)

>  It is not up to us to tell the providers how to
> traffic engineering parameters in response to things like congestion,

You are misunderstanding my take: I didn't mean I want a machanism reacting to 
congestion, I want to give providers a tool to have some alarm on thresholds and signals letting
them to rearrange their provisioning or deny service requests (if they want) in order to prevent 
congestion. If a provider out there wouldn't be happy with that, please submit your complaints to 
this list!!!!! 

The point is whether the protocol should be standard or not... It can be either.

Let's analyse what I wrote.

   > >> [...] we need a new protcol (the Broker in Van's architecture)
   
   Here I expressed the need for a new protocol ( not a standard protocol, might be proprietaty
   so don't care about it, Eric). And not a signaling protocol, NOT  YASP!!!!
 
   > >> and at least some non trivial extensions to RSVP to install it....

   Please note, what follows isn't a plea for this Jon 's statement, rather it's a
   radically different position...

   > >Ideally I think the broker protocol should be much simpler than RSVP and
   
   Thus it can be intraprovider, at the access or exchange points ( and might be proprietary)

   > >to some network functionality the need for more, in order to prevent congestion.
                                                       *******************************

   I didn't write REACT to congestion!!!!

> the providers start getting very nervous.  This is where the "Bandwidth Broker"

Yes RSVP services are like executive planes, first class services and best Effort are
like Business class and coach class on planes. The bulk will use BE, people needing some 
good and reliable services will choose first class, VIP will use RSVP (if they can get the
executive plane, that is if  RSVP will be sufficiently deployed).

So, don't worry Eric, we are not going to kill beautiful ideas (look below!!!)

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
  "A beautiful idea has a much greater chance of being a correct idea
   than an ugly one."
                     -- Roger Penrose, "The Emperor's New Mind", 1989

---

From: "Alessio Casati" <[email-masked]>
Date: Thu, 6 Nov 1997 11:37:40 +0100

> of these bit values which can be globally interpreted.

That's why I asked the question here below.... I think you can have multiple drop levels
but what's hard is then to use them. That's why, cautiosly, I still lean for the lean solution.
The lowest the number of the DP bits (and drop classes) the more comfortable I feel, right now.

[...]
>>would you let clustering some levels, thus letting vendors and providers making the 
>>choice they think the best for their systems and networks?

---

From: "Alessio Casati" <[email-masked]>
Date: Thu, 6 Nov 1997 11:49:52 +0100

[...]

> have when determining these bit semantics.

I think you now know I share your take. I share your P.S. too!

Alessio

---

From: "Alessio Casati" <[email-masked]>
Date: Thu, 6 Nov 1997 12:32:23 +0100

> Let's get the easy stuff out of the way first

Good point.

---

From: "Alessio Casati" <[email-masked]>
Date: Thu, 6 Nov 1997 14:18:51 +0100

Steven's reply suffices from the theoretical point of view.
>From the architectural point of view I lean for as few levels
as possible (2 is my favourite figure, with the IPI info 
associated to the bit).

---

From: "Alessio Casati" <[email-masked]>
Date: Thu, 6 Nov 1997 15:11:20 +0100

> any sort of end-to-end behavior.

I predicted in early times that solutions with more stuff and bits may be no less
naive than lean solutions are. Now can everybody see the mess we meet
and how hard is to assign a crispend to end semantic to multiple drop levels?
"In Profile" is a crisp semantic, "High drop precedence / Low drop precedence"
can be used only in a fuzzy way!

alessio

---

From: Dimitrios Stiliadis <[email-masked]>
Date: Thu, 6 Nov 1997 08:15:40 -0500 (EST)

Paul, 

I have a simple question : Is there any study (formal analysis please)
of a scheme that would have multiple drop bits and will keep
the utilization high ? If yes, please point me to that ?

If no, I have to agree that the only thing that we know how
to use is Delay priorities ... or better bandwidth 
(because, bandwidth + buffers => delay , or vice-versa)

A Drop Bit is definately required to tag packets out of 
profile or excessive use of resources. What are the
rest going to do ? and how ?  

Dimitri

PS I am not even talking about implementation ... I simply 
ask for a formal argument.

---

From: Paul Ferguson <[email-masked]>
Date: Thu, 06 Nov 1997 09:44:05 -0500

>can be used only in a fuzzy way!

Not enough granularity.

- paul

---

From: Jim Boyle <[email-masked]>
Date: Thu, 06 Nov 1997 10:18:03 -0800

Zheng is right on here (along with Hong Kim's note an hour later).  In the
international networks, these customers are a more likely focus than all
the web servers around.  I'm not suggesting there are different bits needed
or anything, just that receiver based tagging is something that shouldn't
be discounted.

regards,

Jim

---

From: David Black <[email-masked]>
Date: Thu, 06 Nov 1997 11:18:32 -0500

> Let's get the easy stuff out of the way first

Upon further thought, I may have written too quickly, so
	(1) No it doesn't have to look anything like RSVP, AND
	(2) It could be easy via reuse of a common marking mechanism
What's required is that sufficient care be taken to avoid overspecification
of the sender-based marking mechanism to unnecessarily prohibit receiver-

> they can use to stay competitive.

If we can build a marking mechanism primarily targeted at sender-based
marking, here's a plausible way of turning it into a receiver based marking
mechanism that will address the enhanced web service problem, has reasonable
scaling properties, and doesn't require major new inventions of technology.
The major pieces are:
	- Mark traffic to entire CIDR-blocks of destinations, not individual
		IP address destinations.  This avoids scalability problems
		-- it won't scale quite as well as CIDR routing, but the point
		is to do marking at the edges of the network (where there's
		a bit more leeway in scalability), not the core and hence
		this ought to be ok within limits (e.g., minimum size
		of a CIDR block to be marked).
	- Allocate a second IP address out of such a marked CIDR block
		for each web server that purchases enhanced service, and
		use a NAT to remap the marked second IP address onto the
		server's real IP address to avoid renumbering headaches.
		The NAT could do other things such as only translate traffic
		for the http port.  Of course, if it's a brand new server,
		directly using an address from the marked block may work
		(depends on the routing topology).
	- Any ISP that marks a given CIDR block also arranges their DNS so
		that names of web sites buying the enhanced service resolve
		to the IP address in the marked block.  For ISPs other than
		the one that hosts the web site, this is part of the contract
		that divides the revenue from the enhanced service.
		Alternatively, just change the root DNS records to point to
		the new address to avoid DNS complications at the risk of
		increased dislocation if the web server should give up
		enhanced service.
	- Note that the agreements on what CIDR blocks get marked need to be
		done out of band.  People talking to people is good enough
		to get started.  I'm definitely NOT advocating use of routing
		protocols to piggyback information on what blocks to mark to
		avoid breaking CIDR aggregation.

I assert that as long as the work we do focusses on what bits are used to mark
a packet and how a packet marked in such-and-such a fashion should be treated
differently from unmarked ones (or ones marked in other ways), a structure like
the above can be layered on top of this mechanism to realize receiver-based
marking.  This is not the only way to do receiver-based marking (nor even
necessarily the best way), but it's a proof by example that it is possible
without major additions of complexity or the need for an RSVP-like protocol.

--David

---

From: [email-masked] (Hyogon Kim)
Date: Thu, 6 Nov 1997 13:39:06 -0500 (EST)

David,

>> they can use to stay competitive.
                  [....]
>--David

I agree. And this is why I oppose to allocating (possibly unnecessarily)
large number of bits to drop preference and delay levels before even having
a chance to consider other problem dimensions like receiver-based marking (who
knows what else to come), and before we have some technical conviction
that so many bits are really necessary for particular fields.

-- Hyogon

---

From: Zheng Wang <[email-masked]>
Date: Thu, 06 Nov 1997 14:30:49 -0500

Folks,

I'd like to present here an alternative scheme for differentiated service
called "User-Share Based Differentiation (USD)" which I believe will solve
some of the problems that tagging has (fixed profile, receiver-based tagging
and unfriendly sources).

The USD has two major components:

1) The basic flow granularity is a user (a dial-up customer or a corporate
network) who pays for the service. All traffic originated from or 
destined to a user is regarded as an aggregated flow. An ISP deals with 
traffic at the user level.

2) The service contract is based on "Shares". A user and its ISP agree on the
number of "Shares" that the user has. The ISP guarantees that under congestion
the bandwidth is allocated according to the "Shares".  

I believe that this simple model has a number of interesting features:

- It is a very simple service contract. If one user pays for 3 Shares,
it will get 3 times the bandwidth that a user who has 1 Share has.

- It will immediately have benefits both for ISPs and users (traffic
isolation according to what users pay)

- The "profile" here is not a fixed one but a relative one. The bandwidth 
a user gets is proportional to its "Shares". It works over pipes of any size
and with any number of competing flows.

- It works both for sender-based traffic or receiver-based traffic.

- It can be implemented with weighted fair queuing (WFQ) which is extensively
studied and well understood.

- Under congestion, the scheme provides hard flow isolation (WFQ has been
shown to provide near-optimal bandwidth isolation between flows). It does
not rely on end systems' cooperation but the end systems do traffic control
for their own benefit (misbehaving users can only hurt themselves).

- it allows many levels of aggregation. A retail ISP can be a single user of 
its
backbone ISP, hence it's own users do not need to be visible inside backbone.
In general, when packets are close to the source ISP, the sender's
shares have more influence and when the packets move close to the
destination, the receiver's shares becomes more important.

- it allows incremental improvement, for example installing
source-destination pair granularity reservation for VPN.

In some degree, one can view USD something between end2end reservation,
and tagging or a sort of long-term reservation on the granularity of
per-user flow.

Your comments are welcome.

Cheers
Zheng

---

From: [email-masked] (Hyogon Kim)
Date: Thu, 6 Nov 1997 14:44:47 -0500 (EST)

Steven,

>constants [not that I think that this is a good idea].
            [ ..... ]
>Regards,

Of course it's not the only way :-). Given 4 configurable parameters in RED,
we could play with one or more of them in order to create differentiation
effects. Considering this mailing list's dislike of implementation
details at this stage of the discussion, I will just mention one thing
about what you suggested: max_p() is not a good handle. It can't protect
profile users under the attack of the non-adaptive flows. On the
other hand, max_th() can protect customer profiles even without the
help from policing. By the way, fairly(whatever that means) distributing 
excess capacity to "out" packets outside profiles, is another problem 
which may be solved either by policing, or as Paul suggests, by using more 
drop preference bits.

-- Hyogon

---

From: Paul Ferguson <[email-masked]>
Date: Thu, 06 Nov 1997 16:13:47 -0500

Okay, so please provide an overview of the details.

- paul

---

From: Wu-chang Feng <[email-masked]>
Date: Thu, 06 Nov 1997 16:18:39 -0500

> which may be solved either by policing, or as Paul suggests, by using more drop preference bits.

This can also be done by extending TCP congestion control.  Since current distribution of "excess" capacity is done using TCP congestion control,
another way which we have studied is to integrate the marking in with the congestion control mechanisms at the source.  Without doing this, you run
into solutions which increasingly add more and more complexity into the network routers.

---

From: Zheng Wang <[email-masked]>
Date: Thu, 06 Nov 1997 16:42:00 -0500

> Okay, so please provide an overview of the details.

I though I have described the details in that message:-)

1) A user subscribes to its ISP for a number of "shares" (or an ISP
can have a number of packages - each associated with a number of
shares). 

2) The ISP installa the user's IP address or prefix and its associated
shares in its routers (not necessarily all routers but the most congested
ones such as access/border routers)

3) When a link is congested, WFQ starts to provide isolation for each
user on the link using its "shares" as the weight in WFQ.

If you take RSVP, change it from 5-tuple granularity to 1 (src or destination),
change its end2end nature to a single ISP, change its per-session timescale
to much longer timescale (weekly/monthly), you start to get something looks
like USD.

Cheers
Zheng

---

From: Paul Ferguson <[email-masked]>
Date: Thu, 06 Nov 1997 16:58:26 -0500

>can have a number of packages - each associated with a number of

What exactly are 'shares'? Is this a percentage of fixed bandwidth,
expressed as bits per second?

>ones such as access/border routers)

What is the mechanism used to 'install shares'? Is this a
token bucket spec threshold? Also referred to as a profile?

>user on the link using its "shares" as the weight in WFQ.

This is reasonable.

>to much longer timescale (weekly/monthly), you start to get something looks

Not sure I quite understand the breadth/depth of tour last comments here.

- paul

---

From: Zheng Wang <[email-masked]>
Date: Thu, 06 Nov 1997 17:22:28 -0500

> expressed as bits per second?

No - it's just a number - it's the "weight" that is used in WFQ.

A user's percentage share of a bottleneck bandwidth 
= (user's share)/(total shares of active flows on the link)

> token bucket spec threshold? Also referred to as a profile?

We need a simple protocol for a management system to talk to the routers
to install the user/share mapping. This is the part that needs some 
standardization.

Essentially User-Share Based Differentiation applies WFQ on a per-user basis.

Cheers
Zheng

---

From: Paul Ferguson <[email-masked]>
Date: Thu, 06 Nov 1997 17:45:24 -0500

>to install the user/share mapping. This is the part that needs some 

And this is where I believe your proposal breaks down.

Not to intentionally sound facetious, but I am adamantly
opposed to YASP (Yet Another Signaling Protocol) to
provide a *simple* set of mechanisms for diff-serv.

I am in the process of writing my thoughts & ideas
down into a draft, but I would certainly welcome
comments from other folks (other than the same 4 or
5 people who have contributed 100% of the volume of
traffic on this list in the past few days) on their
thoughts on the matter.

In the meantime, I still like the idea of using the
TOS bits field for delay indication, and use of the
IP precedence field for drop preference.

We can begin to discuss semantics, if you like,
on how these bit values might be interpreted --
this would be somewhat productive.

I'll certainly provide suggestion in the draft.

- paul

---

From: Zheng Wang <[email-masked]>
Date: Thu, 06 Nov 1997 18:14:41 -0500

> provide a *simple* set of mechanisms for diff-serv.

First, if you regard this as a signalling, anything can be viewed as
signalling. We are talking about a simple protocol that delivers the
user/share mappings to the routers on a very long time scale (weekly
or monthly). What is your concern, overheads, complexity, scalability?
In fact, we can use an existing file transfer protocol like FTP 
to do the job and all we need to agree on the semantics of the mappings.

Second, if you use tagging. The ingress routers need to tag or check the
tagging. To do that, they need to know the users' profiles and you need
to send the profiles to the routers anyway. So what protocol do you use
to send the profiles to the routers? I will use what whatever you use or
call it whatever you call it.

Cheers
Zheng

---

From: Paul Ferguson <[email-masked]>
Date: Thu, 06 Nov 1997 18:24:49 -0500

>call it whatever you call it.

I do not believe that we need to develop a protocol which
instructs the router (or whatever) how to construct profiles.

At least not immediately.

I am first and foremost concerned with defining the semantics
of the indictors themselves, so that we can proceed with the
mechanics of the network plumbing. This is quite important.

- paul

---

From: Fred Baker <[email-masked]>
Date: Thu, 6 Nov 1997 15:41:33 -0800

>expressed as bits per second?

I'm sure it's very much what we do with WFQ now. Imagine the case where you
have N flows that are constantly keeping data in queue, and they all have
the same weight. Each will average 1/N of the link. Each gets 1 share, and
there are N shares.

Now imagine a case where you want to give different flows different
percentages. For example, if IP Precedence is used as an index into a table
of weights, and a flow with IP Precedence K gets K+1 times the throughput
of a flow with IP Precedence of 0, Zheng would say that it got K+1
"shares", out of a possible N+K.

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
	NT 5: the power of Windows, the simplicity of Unix.

---

From: Paul Ferguson <[email-masked]>
Date: Thu, 06 Nov 1997 19:27:14 -0500

>"shares", out of a possible N+K.

That makes much more sense. ;-)

- paul

---

From: Geoff Huston <[email-masked]>
Date: Fri, 07 Nov 1997 13:49:41 +1100

>Essentially User-Share Based Differentiation applies WFQ on a per-user basis.

So instead of looking at weighted fair queueing on a per connection 
basis, you are advocating to increase the coarseness to applying
relative weights to aggregated conenctions which share a common
source address.

What exactly is the win here? I would've thought this would defeat
much of the attributes of weighted fair queueing given that within
each pseudo conenction you are back to the vagries of round
robin packet queueing presumably.

Geoff

---

From: Geoff Huston <[email-masked]>
Date: Fri, 07 Nov 1997 13:54:01 +1100

>mechanics of the network plumbing. This is quite important.

What you are therefore saying Paul is that you equip routers
with a fixed set of functionality levers and use indications in the
packet to trigger one of more of these levers during the 
router's scheduling function.

I agree that defining the semantics of the indicators
is the primary initial task

  Geoff

---

From: Geoff Huston <[email-masked]>
Date: Fri, 07 Nov 1997 13:58:48 +1100

>call it whatever you call it.

I believe the fundamental point here is the 'atom' of fair queueing.

If the aim of the exercise is to provide more predictable performance
levels to conversation sessions then you need to operate
the WFQ algorithm  on the full 4-tuple of a connection.

If you want a more abstract notion of allocation of relative network
resource quantities to end systems, then your concept of
using only a 1-tuple of source address may do this, BUT
at the same time you will lose any concept of attempting
to bound performance parameters in so doing, as you
have yet to define the scheduling algorithm used 
WITHIN each common source address flow aggregate.

  Geoff

---

From: Paul Ferguson <[email-masked]>
Date: Thu, 06 Nov 1997 22:16:32 -0500

>router's scheduling function.

Yes, exactly.

Of course, the 'delay indicator' bit semantics would 'trigger'
a classifier to map into a non-FIFO queue servicing discipline.
Drop preference may not be immediately used.

- paul

---

From: Jon Crowcroft <[email-masked]>
Date: Fri, 07 Nov 1997 08:41:47 +0000

this fits VERY well with our work on multiplying the cosntants in TCP
so you get more than 1 TCP-worth of capacity....more on that
later...but

basically the difference between int-serv and diff-serv lies in
reducing 
1/the complexity of the admission test
to a subscribe time rather than call time
2/ reducing the complexity of the requirements on the queueing and
policing functions (e.g. timescales over which they operate)

btw
i am opposed to any of these complex multi-level precedence bits (and
any drop preference differentiation at all  since this happes natually
y with precedence querueing+RED :-)

since the primary goal is to get something
a) deployable
b) almost signaling free (subsceribe time signalling can still use
RSVP of a kind....but doesn't need it everywhere)
c) that addresses user requirements now (e.g. get my web page faster,
or make this mbone audio app work "properly") rather than last year or
in the next millenium

anything that has implications for 

 jon

---

From: Greg Waters <[email-masked]>
Date: Fri, 07 Nov 1997 05:05:07 -0500

At 04:42 PM 11/6/97 -0500, you wrote:
>If you take RSVP, change it from 5-tuple granularity to 1 (src or
>destination), change its end2end nature to a single ISP, change its
>per-session timescale to much longer timescale (weekly/monthly), you start
>to get something looks like USD.

You described a simple diff-serv model, which is good.  But I may have missed
part of why your scheme is useful, or how it is scalable.  Some questions:

1. I understand that prefixes of all 1- or N-share users will be installed
   at all interfaces likely to be used in either direction by those users,
   if the interfaces have recurring congestion.  So multi-share users A and
   B cause destination prefixes to be installed at their access router, for
   traffic from the ISP to the users.  And users A and B cause source
   prefixes to be installed at access routers of all congested sites that
   they access, for traffic from the ISP to those sites.

   ("Access router" here means at the ISP's end of the link.  The number of
   prefixes includes single-share users because even those packets must be
   lumped into WFQ flows.  A weight-1 flow is still a flow.)

   That is the proposal, right?  Or did server-to-user packets get defined
   service only if the server's "user" subscribed to a "shares" package?

2. When a packet's two addresses match two "shares" prefixes, may the
   router use either prefix and share, instead of the larger share?  Of
   course, if _every_ user buys a package (such as single-share), then
   most packets from a multi-share user will also match a single-share
   prefix, which slows down the router's decision.

3. I understand that the number of flows participating in WFQ on each link
   with recurring congestion is as many as the total number of users
   (multi-share or not) who use that link during congestion.

   When you say "WFQ", it implies one of the published packet-level
   approximations to true fair queueing.  Any of those scheduling algorithms
   is quite expensive for a fast link with many thousands of WFQ flows.
   Does your proposal mean to use multi-thousand-flow WFQ on each link,
   or do really mean stochastic fair queueing, or did you envision that
   fast links don't follow your proposal exactly.

Thanks  --Greg Waters, [email-masked]

---

From: Jon Crowcroft <[email-masked]>
Date: Fri, 07 Nov 1997 14:40:17 +0000

>   fast links don't follow your proposal exactly.

you can apply WFQ to individual flows at the ingress router, and to
the aggregate flows  near the center - you actually hget higher
utilisation (or your "admission test" (subscribe time for the profile)
is more relaxed) if you do this...

but i don't see why you can't JUST do rate limited policing at the
edge and precedence queueing in the center for now...

 jon

---

From: Zheng Wang <[email-masked]>
Date: Fri, 07 Nov 1997 10:00:14 -0500

> the WFQ algorithm  on the full 4-tuple of a connection.

If you want to do that, just use RSVP and guaranteed/controlled load.
But for many reasons, it is difficult to deploy such end2end reservation
at the present (e.g., lacking of accounting suuport, inter-ISP
settlement, adminstrative support at the user organizations, etc)

> WITHIN each common source address flow aggregate. 

In the diff-serv, we are looking at longer term resource allocation
rather than per-session reservation.

Cheers
Zheng

---

From: [email-masked] (Hyogon Kim)
Date: Fri, 7 Nov 1997 11:07:28 -0500 (EST)

Zheng,

>  [   ...   ]

I don't know what gremlins are lurking ahead  but it seems to me your
idea is neat (at least conceptually). It indeed solves receiver-based
tagging headache rather nicely if we let the routers from the server
side look at not only the shares of the server but also those of the
client (one more memory lookup), unfriendly sources end up punishing 
themselves by using WFQ, obviates the need to fight over bit 
allocations (allowing ISPs to quickly implement their own share market(!) 
not waiting for IETF to standardize the bits), and yes it has more 
flexibility than fixed profile schemes. 

A minor implementation concern would be, as someone suggested, the 
scalability of WFQ. And for share installation, I thought we could use
existing network management protocol, maybe? And.... I am sure 
some venture company will pop up and publish monthly report on
the prices of shares and the corresponding average bandwidth if
this gets implemented: MCI share is 64Kb/s at $18.99 and Sprint's
is .... ;-)

All in all, I think USD is worthwhile to push further.

-- Hyogon

---

From: Zheng Wang <[email-masked]>
Date: Fri, 07 Nov 1997 11:35:01 -0500

>    they access, for traffic from the ISP to those sites.

First a general comment. if we want to differentiate users, the network has
to know about the users' profile or share or whatever agreement with the ISP
in order to enforce the agreement. We cannot avoid that.

Within an ISP, its users' share/profile determine the sharing over a congested
link. So generally you check dest for traffic coming into the ISP and
src for traffic going out of the ISP. 

>    prefix, which slows down the router's decision.

In this case, both the src and dest are inside the same ISP.
Either of the shares should do and the ISP can decide which to use.
If we believe that the majority of the traffic is from a web server to the
users, the users' share is important.

>    (multi-share or not) who use that link during congestion.

The total number of flows/users on a single link is limited by its
bandwidth/buffer size. The number of queues you support is really 
a parameter of your pipe size and buffer size.  

>    fast links don't follow your proposal exactly. 

There are over 10 different implementation of WFQ and many of them
are designed for hardware implementation. In fact, since we just care 
about bandwidth isolation not delay bound, we can pick up a much
simpler WFQ scheme. Put it another way, if you 
can support RSVP, it's much easier to support USD.

Cheers
Zheng