Subject: Traffic Shaping for Multicast flows

From: Ajay T Joseph <[email-masked]>
Date: Mon, 17 Feb 1997 14:34:59 -0500

Question about Traffic Shaping:

Upon reading the guaranteed QOS draft - Traffic shaping is required
within the network to accomodate the heterogenous TSpec's of multicast
links. From an implementation point-of-view isn't it very expensive to
perform shaping on a per flow basis, especially for high-speed links
within the network. Is there any way around this problem?

--Ajay Joseph--
GTE Labs
Waltham, MA

---

From: Craig Partridge <[email-masked]>
Date: Mon, 17 Feb 97 16:11:58 -0800

within the network.

It depends on your implementation.  Once you decide to shape (even for
aggregates) you're usually OK to shape for smaller flows -- the only
question is how big the memory for flow state is going to be (and thus
how many total flows you can support).

Craig

---

From: Roch Guerin <[email-masked]>
Date: Tue, 18 Feb 97 06:42:00 -0500

Ajay>    Upon reading the guaranteed QOS draft - Traffic shaping is required
Ajay>    within the network to accomodate the heterogenous TSpec's of multicast
Ajay>    links. From an implementation point-of-view isn't it very expensive to
Ajay>    perform shaping on a per flow basis, especially for high-speed links
Ajay>    within the network.

Craig>It depends on your implementation.  Once you decide to shape (even for
Craig>aggregates) you're usually OK to shape for smaller flows -- the only
Craig>question is how big the memory for flow state is going to be (and thus
Craig>how many total flows you can support).

Exactly.  The main scalability issue when doing reshaping, or in
general providing any kind of per-flow guarantee, is the need to store
state (service and/or traffic) parameters for individual flows and
update them as packet get transmitted/received.  In that respect,
shaping is no different from scheduling, e.g., need to maintain per
flow state parameters when doing wfq or any of your favorite
scheduler.  In the context of our implementation, the same component
actually implements both the shaper and scheduler, and the number of
flows you can support is a function of how much you are willing to pay
for control memory (can go up to 64k flows).

One other issue you need to worry about when doing reshaping, that
actually arises whenever you need to deal with possibly non-conforming
traffic and at the same time provide service guarantees, is that you
have to be careful on how you do buffer management, i.e., when do you
start dropping.

Roch

---

From: Ashish Mehra <[email-masked]>
Date: Tue, 18 Feb 1997 10:43:35 -0500 (EST)

> have to be careful on how you do buffer management, i.e., when do you

I concur with all the above points. However, I think another important
issue affecting scalability (but not necessarily for multicast flows) 
is the mechanism employed to do traffic shaping (i.e., timer management, 
etc.). The following points are relevant in the context of traffic 
shaping at hosts (say servers), but may also apply to routers.

Traffic shaping (policing + delay-till-compliance) of non-compliant 
packets at hosts involves computing the time to compliance (done 
via variables stored in the flow state and application of the policing
function), and delaying (i.e., buffering) the non-compliant traffic
in two ways: blocking the application's thread doing the send/write call
(lets refer to this as session-level shaping) or buffering packets
(lets refer to this as packet-level shaping). Now, at the host one
must rely on OS timers to delay threads or packets. So one issue
is the number of timers outstanding and the overheads associated with
timer operations (setting/handling/canceling) - aggregation (or 
aggregated timer handling) can surely help here! 

Packet-level shaping only incurs the overhead of OS timers, but then
proper buffer management becomes crucial, as Roch pointed out.
For example, non-compliant traffic may continue obtaining best-effort
buffers and affecting traditional best-effort traffic, especially
under controlled load service. This "interference" also depends on 
the OS thread scheduling and preemption policy. 
The other issue is the cost of blocking/unblocking threads if doing
session-level shaping. The big advantage of session-level shaping
is that it permits immediate feedback to an application violating
the traffic specification. Further, it may actual ease the realization
of flow aggregation. However, blocking/unblocking threads is relatively
expensive and includes a context switch in the path. Further, while
packet-level shaping triggers packet transmissions off timer interrupts,
session-level shaping may incur an additional "OS scheduling delay" 
over and above the desired shaping delay. One may therefore have to
change the buffer management policy and policing function to 
ensure that traffic spec. is enforced such that a flow does not
lose credits "on average". 

All the above issues affect scalability, primarily from the host's
perspective, and some form of flow aggregation or aggregated 
traffic shaping would greatly help.
These and some additional concerns are explored in 
"On the Performance Impact of Supporting QoS in TCP/IP Protocl Stacks"
that I have coauthored with Dilip Kandlur and Debanjan Saha.
Please contact me if you want the postscript.

One final comment: the fundamental issue here is the accuracy of the
traffic specification and how often the application violates it.
I understand that at routers (re)shaping may be required even if
the application does not violate the traffic spec.
Please don't flame me for saying this, but I think that we need to
spend more time figuring out how applications will select a 
traffic specification, especially legacy apps that may benefit
from QoS or even new apps that may have very complex communication
structures.
 
Please correct me if I am wrong or misguided about the above.

Ashish Mehra