Subject: Re: A Generic Socket Interface for QoS

From: Anindo Banerjea <[email-masked]>
Date: Mon, 22 Aug 1994 01:38:42 -0700

I have followed with interest the recent discussion on the e2e list
about an API for QoS. Some of you may be aware that we did an
implementation of realtime network services at UC Berkeley a couple of
years back. And that it has been running on the Sequoia 2000 network
for a while. Our API was not designed to integrate into the UNIX
architecture. The main motivation was ease of implementation, so we
tried to keep as much as possible out of the kernel. The lessons we
learned while implementing and using it might be of interest to you. It
might be of passing interest to note that after working with our
implementation for a while we made a wish-list of things to change if
we ever reimplemented and one of the items on the list was an API
pretty much like Craig's suggestion (at our end the socket API idea was
tossed out by Steve Mccanne while complaining about our API :-)
trying to make his teleconference application work (VIC) with our realtime
system).

Our communication model for realtime communication is connection
oriented, and maps well onto the ATM model. There is a connection
establishment phase which maps onto the ATM VC setup. The signaling
software also performs resource reservation and admission control, in
much the same way as the ATM model. The traffic specification and QoS
specs are different from your model (Token Bucket), but we can leave it
out of this discussion since enough people are picking in that
already.

In our model the signaling software is a user level process and the
application sets up a connection (ATM VC) by making a procedure call
to a library. This makes it look like an RPC to the client, under the
hoods this becomes a message which is sent to the signaling process.
The procedure call (establish_request) specifies the clients traffic
description (peak rate, average rate, averaging interval) and the QoS
requirements (delay, jitter, loss rates). This causes the signaling
process to initiate an establishment (VC setup) and if successful the
signaling software returns a VC identifier.

The application process then creates a socket and then associates
the already established VC id with it using a setsockopt. The calls
look like this.

    /* theParmBlock contains the traffic specs and the QoS requirements */
    if ((status = RcapEstablishRequest(&theParmblock, &lcid, &ipAddr)) < 0) {  
        perror("RcapEstablishRequest");
        fprintf(stderr, "%s\n", RcapStrError(rc));
        exit(1);
    }

    if ((sock = socket(AF_INET, SOCK_DGRAM, IPPROTO_RMTP)) < 0)
    {
        perror("socket");
        exit(1);
    }

    if (setsockopt(sock, IPPROTO_RTIP, RTIP_ASSOC, &lcid, sizeof(u_short)) < 0)
    {
        perror("setsockopt");
        exit(1);
    }

On the destination side things are a bit more complicated, since the
destination essentially has to be ready and waiting for the connection.
Else the signaling protocol, after trying to find the application,
perhaps waiting for a while, must return a NACK to the source. To do this
we have a register, receive, accept protocol which maps loosely onto
the UNIX listen accept model. A third call is needed because the destination
application has the option of refusing the connection (e.g. because
sufficient resources are not available at the destination to support
the QoS requirement. The virtual circuit created is then bound to a socket
as before.

    if ((rc = RcapRegister(port, N_buffer_requests)) < 0 ) {
        fprintf(stderr, "%s\n", RcapStrError(rc));
        exit(1);
    }
    if (( rc = RcapReceiveRequest(port, &theParmblock, &source, &lcid)) < 0) {
        fprintf(stderr, "%s\n", RcapStrError(rc));
        exit(1);
    }
    if ((rc = RcapEstablishReturn(lcid, SUCCESS, 0, &control)) < 0) {
        fprintf(stderr, "%s\n", RcapStrError(rc));
        exit(1);
    }
    if ((RtipSocket = socket(AF_INET, SOCK_DGRAM, IPPROTO_RMTP)) < 0) {
        perror("socket");
        exit(1);
    }
    if (setsockopt(RtipSocket, IPPROTO_RTIP, RTIP_DESTINATION_ASSOC, &lcid,
                   sizeof(unsigned short)) < 0) {
        perror("setsockopt");
        exit(1);
    }

To remove a channel:
        if ((rc = RcapCloseRequest(lcid, 0))<0)
           fprintf(stderr, "RcapCloseRequest: %s\n", RcapStrError(rc));

To unregister a port: 
        if ((rc = RcapUnregister(port))<0)
           fprintf(stderr, "RcapUnregister: %s\n", RcapStrError(rc));

The good part of this design is that now the signalling protocol can be
built in user space (or on another machine if you feel like it). This
makes it easier to implement and maintain, as well as restart if things
get hung during experimentation (as noted by Keshav).

The bad part is that the only association between the VC and the
application is the lcid (local channel id), and so in the
implementation we have any process can modify the QoS specs or remove a
VC created by some other process as long it has the lcid. That could be
fixed by introducing some idea of the ownership of the lcid, or by
introducing capabilities as in Keshav's description. But making the
socket the only interface to the network is, I think, cleaner.  It is
also easier to program for someone used to TCP/IP networking and that
is probably the paramount consideration.

Another problem we faced especially in the initial part of our experiments
was that often the application would die or get killed and the VC's would
not be removed. This left zombie resources remaining in the network and
we had not paid enough attention to garbage collection in our design.
The solution we came up with is based on the fact that the communication
from the application to the signalling process was based on a connection
and when the application died, this connection is killed. The signalling
process kept track of the association between the lcid and the control
connections, and killed the VC when the control connection went away.
I wonder if Keshav has faced the same problem, since I don't know
if his RPC's have an underlying connection.

This problem would also be more cleanly solved by having the socket
the only interface, since when the data socket is closed the VC can
be torn down. Since the socket will be closed in most cases even if
the process is killed, this takes care of most of the garbage collection
problems.

The only down side with the socket based API is that part or all of the
signalling code will be in the kernel. This is a performance win, but
performance is less of an issue for call set up, since this is done
once before data is transferred, and at least in my mind the duration
of the call is long compared to the time to do a setup, so the cost is
pretty much amortized. However this is more of a pain to implement and
maintain, especially in a research environment where you want to keep
making small changes to the code.

As I mentioned before, after working with our implementation for a
while and setting up applications to run on it, we reached the
conclusion that the realtime channel paradigm must be integrated more
seamlessly with the underlying communication abstraction of the OS, if
making the system usable to the application programmer is of concern.
Since we are using BSD, our proposed solution looked almost exactly
like what Craig has come up with. However, since we were more
interested in exploring the networking issues, and were prepared to put
up with the less than perfect application interface, we left the API as
it was.

Now after promising to not talk about the traffic specs and QoS specs,
let me make a few comments about what I believe to be the right way to
go.  I think that the QoS specification should be in terms of the
performance provided by the network and the traffic injected into the
network.  But it should not tie us into one particular traffic
description/ policing/ traffic shaping mechanism in the network. In
other words the QoS descriptor should be independent of the mechanism
used in the network to make sure that the QoS is met.

I tend to agree with Mark Garrett that while the token bucket describes some
forms of traffic well, and provides a fairly good way to reserve resources
for it, it is not a panacea to all realtime problems. And Craig's argument
that stat muxing does not work and that you have enough best effort traffic
to fill up the background leads to the corollary that peak rate allocation
is all we need to do. Clearly no one agrees to that. So the issue open
to research is how to share the bandwidth between sources as efficiently
as possible to support as many realtime streams as possible.

Now the network can make more intelligent decisions about sharing if it
has more information about the traffic. The (B,R) characterization is good
as far as two parameter specifications go, but multiple (B,R) pairs,
or multiple (rate, interval) pairs would provide more information and
might make for better utilization of the network. Work by Zhang
and Knightly shows considerable improvement in network utilization by
using more complex traffic descriptors. The question of whether the
increase in complexity is justified by the improvement in network
usage depends a lot on the nature of the traffic being produced by
the set of applications running in your neck of the woods. This
means that even if we pick one QoS right now, there is no way of
saying if that will continue to be the right choice tomorrow, or
in some one else's network with a different set of applications running
on it. It would be good to hide this from the application layer as much
as possible by letting the network describe the QoS parameters in the
was that it knows best, and translating to it from a application based
QOS which remains unchanged.

The point is that the frontier is moving and standardizing on something
that might not be the right way to go would hurt us. Obviously at some
underlying portion of the hierarchy the more generic QoS descriptor
would be translated to whatever the underlying network is using. And
if the network uses token bucket descriptors, then that gets used. However
if someone figures out a better way to do things, it should be possible
to change the network description without changing the application code.
Our API does not do that. Perhaps Craig's suggestion involving multiple
protocol families will solve that problem. I'll wait with anticipation
for the promised final summary of the discussion.

---

From: "Bruce A. Mah" <[email-masked]>
Date: Mon, 22 Aug 1994 09:29:40 -0700

A few follow-up comments about the "UC Berkeley experience"...  (So
you know where we're coming from, Anindo and I are the co-implementors
of RCAP, the Real-Time Channel Administration Protcool, which does
signalling for the Tenet Real-Time Protocol Suite.  Comments from our
co-designers are welcome.)

Anindo Banerjea writes:
> tried to keep as much as possible out of the kernel. 

Also a factor for us was that at the time, none of the implementors
had much (any?) experience doing kernel hacking.  After subsequently
having done a little kernel work in both UNIX and Sprite, I personally
would feel a lot more confident about doing a more socketized version
of RCAP (our signalling protocol), were we to do this again.

> The lessons we

In fairness to Steve:  We all spent quite a bit of time talking about
what the next API would look like and (I believe) came to the
conclusion that we couldn't really take the time to do it right ("not
enough hours in the day" syndrome).  I don't think he intrinsically
hates the idea.
 
> hoods this becomes a message which is sent to the signaling process.

Just a real minor nitpick: It's sort of more like a "local RPC"
(i.e. cross-domain procedure call).  Through the library, the
application "calls" the RCAP signalling entity on the same machine by
sending a local message, which then sends a message along the path of
the channel to set things up.

> the already established VC id with it using a setsockopt. The calls
[code snippit deleted]

Obviously if we're using the socket as a handle to the real-time
channel, there's no need for the application to even know what the
local VCI is.  If there are any exceptions, they can be handled via
socket options.

At one time, we were considering having some analogue to the BSD
socket layer that would use the VCIs directly.  This, I think, was the
main reason for giving the VCI to the user process, then letting it
handle the association with the socket.  If we were to do it again,
I'd definitely remove the distinction, at least from the applications'
point of view.

> the QoS requirement. The virtual circuit created is then bound to a socket

[code snippit deleted]

To be more precise:

RcapRegister					<=> listen
RcapReceiveRequest + RcapEstablishReturn	<=> accept

One question here is whether the destination process should have the
right to refuse a connection after having seen the connection
establishment request.  We took the viewpoint that the destination
process can make the final accept/reject decision, because there may
be higher-layer admission control decisions that can only be made by
the application.

> get hung during experimentation (as noted by Keshav).

Portability was a big plus.  It took only a few hours to get the
original Ultrix implementation of RCAP to build on SunOS, HP/UX and
IRIX.

> making small changes to the code.

In 1991 (when we were designing the Tenet Suite) we had an "Ohmygod,
let's not touch the kernel" sort of an attitude, because we didn't
have too much experience doing this kind of thing.  Speaking just for
myself, I don't think it's such a big deal now, and I don't think that
putting some parts of the signalling in kernel is inherently evil (in
particular, the part the glues together the socket interface and the
local signalling entity).

For more info on what we did and why we did it, you-all can take a
look at my Masters report, which describes our signalling protocol in
a moderate amount of detail.  It can be found at:

ftp://tenet.berkeley.edu/pub/tenet/Papers/Mah93.ps

Bruce.