Subject: Re: Generic QoS Socket Interface

From: Craig Partridge <[email-masked]>
Date: Wed, 17 Aug 94 17:41:32 -0700

Hi Keshav:

    Thanks very much for the comments.  I want to think some of them over
some more but I have a two immediate comments.

    If I understand Craig correctly, he wants to work under the constraint
    that existing applications ought to be easily ported to the QoS enhanced
    sockets.

Not quite my goal.  My goal is that all QoS needy applications use the same
basic interface, regardless of protocol stack (IP with integrated services,
ATM, IPX with integrated services, whatever).

    The first problem with setsockopt is that it is a one-way street - an
    application can specify what quality it wants, but cannot receive
    information back about what information the network can actually provide.
    One way out is to have a poll loop on getsockopt(), waiting for 
    something to change.  This is not elegant. Another, dirtier, solution
    is the one proposed earlier in this thread, where an application would paus
   e 
    awaiting a signal, which would be delivered on call completion.

I don't think so.  When using connect() the semantics of connect() are that
the call doesn't return until the connection is made.  For UDP without
QoS this is trivial and connect() returns immediately but for UDP w/QoS
(and for TCP now) connect() would wait until the net OK came back and then
could examine the QoS.  And UDP already supports multiple calls to connect.

I'll note that I think the interface *should not* enforce a given negotiation
model.  So one way (only get to say QoS and not dicker) should work as should
two way.

Craig

PS: What are the error scenarios in the user interface?  The code for the
usual case looks OK but what cleanup has to be done (say if socket fails
after a vci is created)?

---

From: [email-masked]
Date: Wed, 17 Aug 94 19:44 EDT

Craig, Gopal, Geoff and others,

	Here are some thoughts on how application ought to specify a QoS 
to the network (and how this QoS should be provided by the network). 
Much of this arises from work that we have been doing at Bell Labs 
since June 92 in the Xunet testbed.

What we have in place now is a 'native-mode' ATM stack that allows
applications to specify per connection QoS, and signaling and OS
hooks as well as the protocol stack to provide this QoS. 
Some of this work will appear in a paper in the coming Sigcomm.

If I understand Craig correctly, he wants to work under the constraint
that existing applications ought to be easily ported to the QoS enhanced
sockets. This is achieved by keeping the socket interface the same,
but putting hooks into the bind/connect calls to set up a
QoS provisioned ATM VCI. The QoS would be specified by the setsockopt call.

I believe that we should view the QoS in the following way. The client
sends a specification via signalling to the server. The server has
a chance to accept the connection, and _modify_ this QoS, depending on
its current load. The modified string is then used to relax the 
network's reservation state, and passed back to the client. The client
sees the modified QoS, and if it doesnt like it, tears down the call.
This way, both the client and the server have a chance to negotiate
the QoS before tha call completes.

is the one proposed earlier in this thread, where an application would pause 

In our solution, the client makes a call to a library routine, which in
turn makes an RPC to the signalling entity (running in user space - 
more on this later).  The library routine also passes in what is 
effectively a handle into an RPC stub where the signaling entity can
call back back when signaling completes. When this stub is called, the client 
can see what QoS value was returned, and if this is not satisfactory, 
the socket() call need not be made.

The second problem with Craig's solution is that by using setsockopt 
to pass in QoS info, he is forcing the implementation of signaling 
to be in the kernel (unless the bind/connect calls 
make an upcall from the kernel to a signaling entity in user space). 
Having worked with two signaling implementations, 
it is very clear, at least to me, that putting
signaling in the kernel is a bad idea. There are far too many things
that can go wrong, and it is a big win to be able to restart signaling
in user space, instead of rebooting the kernel each time.

Now, if you accept that signaling is in user space, then separating
call establishment (and QoS specification) from the socket abstraction
is a small next step. This is what we have done.
The signaling entity is a user level process, which is accessed by
a call-estblishment library using RPC-like calls.
This approach has three advantages:

1) signaling can run in user space 
2) client see the modified QoS string.
3) if the qos string changes (because today's flavor is different
   from yesterday's), we recompile and run, instead of rebooting
   every system we have.

Such libraries are easily provided in all operating systems,
and it is trivial to add it to Unix. So, this solution satisfies Craig's
constraints.

A code fragment from our implementation:
client:
        vci = open_connection(dest, service, portno, out_comment, out_res, in_commen
t, &in_res);

	/* dest is the dest ATM address, service is like a /etc/services port,
	   portno is the local RPC stub, out_res is the outgoing qos string,	
	   the comment strings are for experimanting with */

        sd = socket(AF_XUNET, SOCK_DGRAM, 0);
        addr.family = AF_XUNET;
        addr.chan    = vci;
        connect(sd, &addr, sizeof(addr) 

On the server side, the corresponding calls are 

	export_service(service, portno);
	cookie = await_service_request(listen_sock, comment, &res);
	vci = accept_connection(cookie, comment, res);
	sd = socket(AF_XUNET, SOCK_DGRAM, 0);
        addr.family = AF_XUNET;
        addr.chan    = vci;
	bind(sd, &(addr), sizeof(addr))

Note the use of a cookie (capability) which makes sure that no one
can hijack an open connection. More details about this are in the
Sigcomm paper.

------------------------

The Chorus work has carried out the QoS concept much farther, but
without Craig's constraints. Their use of 'direct connections'
is very nice (and something that one Xunet student wants to do in IRIX/Plan 9).
But, the use of a dummy upcall to an application sounds pretty 
complicated as an API. One has to be fairly sure of the 'average'
case, and further, the avarage and worst cases should be close
to each other. Otherwise, the scheduler doesnt have much to go on.

-------------------------

Gopal proposes that protocol stack code be linked into an app
to allow per-process QoS from the scheduler. This seems to be 
a good idea. We are working on similar lines using Plan 9.
In any case, since we have a non-multiplexing stack, we can
easily associate a process with one or more tasks (each layer
of the stack is an independently schedulable task). This would
allow scheduling of the protocol tasks and the user process
in a reasonable manner, even if the protocol lies inside the
kernel. At the moment, we only have hooks for task scheduling,
and are fishing around for policies.
-----------------------------

Any comments on our design are welcome, either on this list,
or in person during Sigcomm.

keshav