Subject: RE: The problem with tagging packets

From: Paul Ferguson <[email-masked]>
Date: Mon, 10 Nov 1997 06:54:59 -0500

At 11:37 AM 11/10/97 +0200, Kilkki Kalevi NRC/Hki wrote:

>
>I hope that this differentiated service effort will not try to re-invent   
>ABR, RSVP or VBR with tagging. They are already invented and they are not   
>too simple.
>

Ditto.

>If someone can present a simple one-bit approach (without any RSVP or ABR   
>type of mechanism) that works smoothly with current TCP/IP applications   
>under highly dynamic conditions, I will be very interested to analyze it.
>

I am having a hard time understanding what the infatuation is with
doing diff-serv with 1 bit? Why is this such a sticking point?

- paul

---

From: Jon Crowcroft <[email-masked]>
Date: Mon, 10 Nov 1997 12:18:31 +0000

>doing diff-serv with 1 bit? Why is this such a sticking point?
 
because its a question of 
0/ explaining why network subscribers or network providers
need more than 1, and then explaining to each one agfain and again how
to use them and how its reflected in their bill and how to re-write
all their performance monitoring tools to complrehend and verify
this.

1/ perceived (rightly or wrongly) implementation complexity 
at ingress routers, in the core routers and in brokers...

now is it easier to understand why 1 bit is easier to understand? :-)

 jon

---

From: Paul Ferguson <[email-masked]>
Date: Mon, 10 Nov 1997 07:27:06 -0500

>now is it easier to understand why 1 bit is easier to understand? :-)

No, not really. This is akin to stating that Decisions are Bad (tm).
I don't agree. Again, simply because it might be 'easier' to understand
a simple 1 bit differentiation does not preclude us from defining semantics
which use more than 1 bit. I believe we have the possibility here to 
define semantics which make selection of the differentiation granularity
an implementation issue that users decide for themselves.

- paul

---

From: "Alessio Casati" <[email-masked]>
Date: Mon, 10 Nov 1997 15:34:50 +0100

> doing diff-serv with 1 bit? Why is this such a sticking point?

I would be more fair than that. I think the discussion is (although it's getting
a little boring) about whether we really need more drop preference bits, looking at real
applications and services. If some reason exists, OK, let us use more bits. Otherwise
let's save them for other future need.
I think people claiming the need fore more bits shoud speak-up now and clearly 
tell to 1-bit-people why they'd like using them, showing which scenario they have in 
their mind, the applications benefiting of their use and in which way an ISP would
use them.

If those needs are only a customisation to some provider's need, then let me say freedom exists to
call the favourite vendor and ask him an ad hoc solution.

Standardisation should cover only general needs and not custom needs.

That's always my naive opinion.

Alessio

---

From: Paul Ferguson <[email-masked]>
Date: Mon, 10 Nov 1997 10:05:23 -0500

>their mind, the applications benefiting of their use and in which way an ISP would

The first scenario that leaps to mind where multiple drop preferences
would be needed is when there are several 'downstream' nodes (or prefixes)
which an administrator may wish to define multiple thresholds for 'priority',
or rather, drop preference, one for each node (or prefix, or application).
In the simplest case, let's assume that:

 h1----+
       |
 h2----+---R---->
       |
 h3----+

h1, h2, and h3 can be either individual hosts on a common link-layer
media, or for the sake of this discussion, can even be downstream
networks connected to an aggregation router (R) by a wide area. Also,
for the sake of simplicity, let's also assume that h1, h2, and h3
all connect to R at the same speed, Y.

In a single bit-oriented marking scenario, R uses a single token-bucket
to mark packets in excess of a single threshold X, allowing each downstream
node (or nodes with prefix) to burst to Y. Packets in excess of X have
the single bit 'set' when the threshold is exceeded. If more granularity
is needed here for various thresholds, multiple token-buckets can be used
to define multiple thresholds (one for each 'profile') Xsub1, Xsub2, etc.;
packets in excess of each profile threshold have their 'bit' set indicating
a preference for drop eligibility in the face of congestion.

However, when all of these packets are sent along their way beyond this
marking router, there is no granularity beyond 'in' or 'out' in which to
base a preference on which packets to be drop eligible when congestion
occurs, assuming that there is a preferential drop mechanism at each
intermediate node router (a la WRED). So, although I have taken pains
to segregate traffic with some amount of granularity, insofar as defining
multiple marking thresholds (e.g. Xsub1, Xsub2, etc.) at the ingress
marking router, the granularity at which they are dropped at
intermediate nodes is binary (drop eligible or not), which is not
exactly what I want to accomplish -- I want to drop with more
granularity than a simple binary 'in' or 'out'.

In a scenario where multiple drop preference bits can be used at the
ingress marking router, I can now indicate more than one level of
drop preference.

Even in a link-layer realm, multiple drop preferences can be achieved
with ATM CLP and Frame Relay DE, since each VC can have unique thresholds.
This is one of the reasons why commercial Frame Relay and ATM services
have been popular.

>Standardisation should cover only general needs and not custom needs.

I agree. That's why we should leave the decision on whether to simply
use an 'in' or 'out-of-profile' decision (or whether to use multiple
levels of 'in' or 'out'), drop preference, and/or delay indication
to the user.

- paul

---

From: Sanjay Mithal <[email-masked]>
Date: Mon, 10 Nov 1997 12:07:20 -0500

All,

Following the recent discussion on differentiated services it appears
that there are only two issues for standardization that this group should
be concerned about, viz.,

1. The number of bits in the TOS octet used to indicate the ``class of
   service''.

2. The number of bits used to indicate the drop priority of the
   packet.

All other issues are implementations specific.

Having made these statements I will justify them individually:

1. The number of bits used to signal the class of service defines the
   granularity, i.e., the total number of supported services in the
   service model. The specific semantics of the use of these bits
   should be upto the concerned service provider.

   The underlying assumptions behind the proposal for the
   differentiated service model are :
	a) Signaling protocols such as RSVP are complex, inefficient
	and unreliable.

	b) Many application flows are short lived (hence cannot incur
	flow-set-up latency delays) or dont care about strict
	performance guarantees.

	c) In the era of standardization, it is difficult to
	distinguish one service provider's offerings from another,
	unless customizable, easy to deploy service provisioning tools
	are provided.

   Addressing (c) above is THE reason not to specify the semantics of
   the TOS bits. 

   The obvious question is what happens in the case of multi-service
   provider networks. The simple answer is that service providers are
   going to arrive at bilateral carrier agreements so that respective
   carriers will shape (multi-service) traffic into the ``pipes'' they
   buy from their counterparts. 

   Note :
   -----
	It is imperative to observe that a differentiated services
	model as proposed, i.e., using only the TOS bits to do packet
	classification IMPLIES usage based pricing on a per packet
	(most likely per byte leve) basis. In addition, the number of
	packets per class, per flow (based on any definition of a flow
	the service provider chooses to implement) will have to be
	kept at each egress router in the network.

2. The issue of the number of drop priority bits required depends on
   the specific implementation of the router, i.e.,

	a) Only ONE drop priority bit is needed if the router is
	capable of per flow (again based on the level of flow
	granularity that the packet classification process in the
	router is capable of) queueing and scheduling. The reason is
	that the packet classification process (determined by various
	packet filters) provides a finer granularity than that
	provided by the TOS bits, hence when each flow has a separate
	queue, we are only interested in whether the drop bit on each
	packet is set or not.

	b) In the event that the router is not capable of per flow
	queueing, then at the very least we need two drop priority
	bits corresponding to the two bits in the TOS field used for
	packet classification. Additional bits will be needed
	depending on the number of drop priorities (as opposed to the
	number of service classes) the service provider wishes to
	implement.

	However, it is clear that the next generation of routers on
	which this service model is most likely to be deployed in a
	ubiquitious manner will be per flow enabled, and equipped with
	sophisticated schedulers as well. Thus, it would appear that
	only one bit be used for drop priority.

   Once again, the semantics of the drop priority bits DO NOT need to
   standardized. We need to establish consenus only on the number of
   bits, leaving the specific actions to the service provider in
   question.

   All this flexibility will contribute to the ability of a service
   provider to customize its service offerings to some classification
   of subscribers that is determined not by some external
   ``standards'' body, but by actual customer profile data that is
   collected.

   Further, we must keep in mind a simple, flexible, yet robust
   charging model for whatever we propose, for without this charging
   model, this service model is not going to be implemented.

Sanjay Mithal

-- 
Sanjay Mithal
Member of Technical Staff
High Speed Networks Research Dept.
Room 4C-528, Bell Laboratories,
Holmdel, NJ 07733.

Tel: (732) 332-5163
Fax: (732) 834-5379

[email-masked]

---

From: Sanjay Mithal <[email-masked]>
Date: Mon, 10 Nov 1997 22:34:46 -0500

Paul,

You gave the following example as an existence proof for the necessity of 
multiple drop bits. However,an underlying assumption that you make is that the 
router is not capable of per flow queueing, where a flow could be defined by
say (SA,DA). 

This issues you describe in fact disappear the moment you have per flow 
queueing and one bit marking for drop precedence.

> a preference for drop eligibility in the face of congestion.

Finer granularity can be obtained with single bit drop priority + per flow 
queueing, achieving the same effect as multiple thresholds. 

> ingress marking router, I can now indicate more than one level of

Again, the issue of multiple thresholds only arises when an implementation does
not do per flow queueing.

As per latest information realeases from most major router vendors, service 
providers have succeeded in getting in their requirements for per flow 
queueing inside the network, and not just at the core.
Further, just about all the gigabit routere vendors are also advertising this 
feature, hence it is reasonable to assume that the routers on which this 
service will ubiquitiously deployed will have this capability.

> >Standardisation should cover only general needs and not custom needs.

I agree completely, so why are we standardizing the service description? I was 
hoping for some explanations as to why we need to standardize something beyond 
the number of bits for indicating the service class and the number (and 
position) of bits for packet drop precedence.
Could you enlighten me on this please.

Thanks,

Sanjay

Tel: (732) 332-5163
Fax: (732) 834-5379

---

From: "Alessio Casati" <[email-masked]>
Date: Tue, 11 Nov 1997 12:05:44 +0100

> Could you enlighten me on this please.

I thank you for this question, and I hope to be able to give an appropriate answer.
I think service providers indeed must be free to implement whatever service they want,
but, since the service must work end to end, providers must know which kind of functionality the
network is by default going to offer. Yes, one can say interprovider agreements and the pricing
structure are going to get everything settled. Let me say, this is somehow risky and can lead 
backbone providers to set the rules instead of the IETF, since everybody will have to rely on their
basic services.
We must agree, I think, on some very basic semantic. The syntax is a minor problem (though we must 
keep in mind the constraints it is going to impose). "In profile" semantic is  very simple
and it seems very acceptable by everybody (from my standpoint). Furthermore it is of great help
in bad guys identification and it does not impose any hard implementation constraint. And it is
independent from the definition of ISP services. Profiles (sevice defition) and profile meters
(playing major role in service actuation) will be the way ISP will differentiate (in addition to 
the deployment of a well engineered network and well engineered access to backbone services).
TCP profiles must be based on standard TCP behaviour. Aside from, it providers are totally free.

I think, in addition, that if some delay based differentiation is going to be implemented (and I 
think it should), then we must agree what it exactly means. (only scheduling priority, some
average, statistical, queueing delay bound...)

hope it helps

alessio

---

From: Roch Guerin <[email-masked]>
Date: Tue, 11 Nov 97 08:41:33 -0500

>queueing and one bit marking for drop precedence.

Sanjay, it actually seems to me that if you have per flow queueing
(buffer management) you don't need any bit.  A bit is needed to
identify out-of-profile (at least in the context I'm assuming) packets
so that the router can recognize them and avoid that they impact
in-profile packets in case of congestion, i.e., ensure some level of
flow isolation.  If I have per flow buffer accounting, I can ensure
that no flow grabs more than what it is entitled to since I monitor
exactly how much resources each use.  So you can do away with the
marking requirement.

In general, whether we talk about drop precedence (DP) or delay
indication (DI) to use Kalevi's notation, the goal is to avoid having
to maintain per flow state and instead rely on the aggregated
information that these bits provide.  We can argue about the
complexity of maitaining per flow state (I also think it is not that
difficult, although there certainly are limits), but my view is that
if we are willing to maintain that kind of information we don't need
any DP or DI bits.  The information they provide is redundant with the
per flow information we maitain anyhow.  

So, I would limit the discussion on the semantics and use of DI and DP
bits to environments where we don't want to maintain per flow state,
and are looking for aggregate solutions.

Roch

---

From: Paul Ferguson <[email-masked]>
Date: Tue, 11 Nov 1997 09:00:43 -0500

>and are looking for aggregate solutions.

Exactly (you just saved me the trouble of responding to several
earlier messages). We should not be making any assumptions on
the ability to interact with any complementary flow-based
mechanisms.

- paul

---

From: "Alessio Casati" <[email-masked]>
Date: Tue, 11 Nov 1997 15:32:19 +0100

On 11 Nov 97 at 8:41, Roch Guerin wrote:

[...] snip
> and are looking for aggregate solutions.

I totally agree with Roch (also with the snipped part).

---

From: Sanjay Mithal <[email-masked]>
Date: Tue, 11 Nov 1997 08:34:50 -0500

Alessio,

Thank you for your reply. I still have some questions:
> network is by default going to offer.

Isnt it true that the service provider usually "works"with the vendor for 
features and functionality. Hence, the default service is in that sense known.
What I am referring to is the following : suppose the router is capable of flow
isolation capacity larger than the number provided in the TOS bit range. Then
an intranet comprising of only these routers will be able to provide finer 
granularity of service. Now suppose another carrier can only support say 4 
such classes. Then the edge routers in the finer granularity providers network 
will
shape its traffic into these four "pipes"that it connects with into the other 
carriers network. Thus the problem reduces to clever traffic engineering.

> the deployment of a well engineered network and well engineered access to backbone services).

I agree. Do the content of these profiles have to be standardized is my 
question ?

> think it should), 
Do you think that the routing protocol can incorporate this information,e.g. a 
QoS based routing scheme will typically account for this information ? 
Further, in the gigabit world, it appears that most of the contribution to the 
delay will come from propagation delays and not queueing delays. Hence the 
route computation can be modified to include the "static" propagation delays 
into the calculation.

thanks

Sanjay

Tel: (732) 332-5163
Fax: (732) 834-5379

---

From: Sanjay Mithal <[email-masked]>
Date: Tue, 11 Nov 1997 08:42:26 -0500

Roch,
I agree that you do not need the DP bit in the event we have rate based 
congestion control. However, what do we do with an application like TCP
that responds only to explicit packet discard ?

Sanjay

Tel: (732) 332-5163
Fax: (732) 834-5379

---

From: Roch Guerin <[email-masked]>
Date: Tue, 11 Nov 97 09:50:13 -0500

>However, what do we do with an application like TCP

The discard will still be explicit (you've got to tell me what you
mean by an implicit discard ;-), it's just that which packet gets to
be discarded is not based on a particular bit, but on the fact that
that flow is exceeding its buffer allocation.  Note that the latter
can have some dymanicity, i.e., be a function of how many other flows
are currently active.
Roch

---

From: "Alessio Casati" <[email-masked]>
Date: Tue, 11 Nov 1997 16:36:17 +0100

I owe you a reply, sorry for I'm late, Paul.

 h1----+
       |
 h2----+---R1(ISP1)---->|
       |                |
 h3----+                | 
                        |
 h1----+                |
       |                |
 h2----+---R2(ISP2)---->|
       |                |  BACKBONE Provider
 h3----+                |
                        |
                        |
 h1----+                |
       |                |
 h2----+---R3(ISP3)---->|
       |                |
 h3----+

If  ISPx defines its own partial ordering (in terms of DP) then, when each is going to inject 
packets into the backbone, some likelyhood exists that  something unwanted may  happen unless a
crisp definition of each level semantic is given (the service is otherwise impredictable and
hard to define). Since a crisp definition is, in my opinion, rather hard, and since end to end 
global service requires the definition of a total ordering, then one of the following will hold:

- this WG gives up defining each level's end to end semantic ( In profile/ Out profile suffices)
- somebody is so clever to give an agreeable definition of a total ordering.
- somebody is able to define an implementation idependent way of using more bits as DP
  not relying on an agreed upon semantic.

> levels of 'in' or 'out'), drop preference, and/or delay indication

So why are we wasting our time on this list? Our work would already be done by
telling which bits are available for use and let people do whatever they want. Am I wrong?

Alessio

---

From: Paul Ferguson <[email-masked]>
Date: Tue, 11 Nov 1997 10:44:09 -0500

>telling which bits are available for use and let people do whatever they want. Am I wrong?

Not intentionally trying to be facetious, but I'm getting tired
of repeating myself. We need to define the semantics, or else
(as Geoff Huston pointed out) we'll be inundated with several
non-interoperable implementations.

- paul

---

From: "Alessio Casati" <[email-masked]>
Date: Tue, 11 Nov 1997 17:47:10 +0100

On 11 Nov 97 at 8:34, Sanjay Mithal wrote:
 
> will shape its traffic into these four "pipes"that it connects with into the other 

In my opinion per packet treatment by backbone providers must be known in advance
and if the expected behaviour is not end to end, then unpredictable events may
happen and service definition may be very hard.

> I agree. Do the content of these profiles have to be standardized is my 

Profiles are service definitions. Any provider (whether in the access or backbone) is going to 
meter users based on that. It's in the provider's own best interest to both avoid misbehaviours
and to provide the contracted services. My feeling is that given these preconditions, we can rely
on the In/Out router semantic and let everybody be free to define profiles and the appropriate
meters.

> route computation can be modified to include the "static" propagation delays 

I think routing should be aware of the provisioned resources for each existing class.
That would greatly enhance the performance of diff services.

alessio

---

From: Masataka Ohta <[email-masked]>
Date: Thu, 13 Nov 97 17:48:49 JST

Roch;

> and are looking for aggregate solutions.

But, no one, not even a leaf provider, is motivated to reduce
packet priority, unless one is annoyed by the congestion, at which
location tracing of flows does not scale.

For example, if a leaf provider have only a single flow (or multiple
flows of equally important customers) to an outside congestion point,
all the packets from the provider will be sent with the highest priority.

> difficult, although there certainly are limits),

The limitation is in scalability and is, thus, fundamental.

						Masataka Ohta

---

From: "Alessio Casati" <[email-masked]>
Date: Thu, 13 Nov 1997 14:11:16 +0100

Whatever the will of a provider is, it will have to face some limitation on the amount 
of Class* traffic it is enabled to inject into another provider's (transit) domain, due to  
contracted profiles reasons. Marking and likely discard occurs for excess traffic. Service 
degradation would result.  This forces everybody to be wellbehaved and nice.

alessio

P.S. Highest priority is not going to come for free.

-----
On 13 Nov 97 at 17:48, Masataka Ohta wrote:

[...]
> all the packets from the provider will be sent with the highest priority.
[...]

---

From: Masataka Ohta <[email-masked]>
Date: Fri, 14 Nov 97 13:39:30 JST

Alessio;

> of Class* traffic it is enabled to inject into another provider's (transit) domain, due to  

No. The next hop provider does not mind the injection from the leaf
provider, even if the injected flows cause congestion in the third
next hop provider or beyond, as long as the total traffic from the
leaf provider is not so much.

So, leaf providers are motivated to give highest possible priority
to such traffic.

							Masataka Ohta