Subject: Re: A Generic Socket Interface for QoS

From: [email-masked] (Larry Backman)
Date: Mon, 8 Aug 94 17:20:09 EDT

>>     
 >>     Several folks have asked me to put my views on how to enhance the BSD
 >> socket (and presumably WINSOCK) interfaces to support quality of service.
 >> 
 >>     The ideas here stem from a basic premise.  If at all possible, we want
 >> a single, generic interface for QoS support.  Just as sockets are a generic
 >> interface for networking protocols so far, any extensions for QoS should
 >> also be generic possibly with bells and whistles for various protocol
 >> families.   
 
 Much of this thinking is (hopefully) getting tracked in Winsock-2,
 the successor which is going to cure all the oversights in Winsock.

 >>     My view is that, if one agrees with this analysis, the right thing
 >> is to find some way to set the QoS between the time socket() is called and
 >> the time connect() or listen() is called.  And there already exists a very
 >> generic system call for customizing sockets -- setsockopt().  So my proposal
 >> is that we standardize one or more options for setsockopt to set a QoS.  (And
 >> use getsockopt() to get any information about QoS, such as after doing an
 >> accept()).
 set/getsockopt() fits well with the Winsock paradigm but..
 >> 
 >>     Internally in socket implementations there's some logic to this approach
 >> too.  A call to connect() is the first time that any packets are sent.
 >> So one can enhance the internal kernel code to pass the QoS structure down
 >> with the first packet (e.g., the TCP syn) through the transport layer to
 >> IP (for IP QoS) and the link level driver (e.g., ATM, Fast Ethernet, FDDI)
 >> so that lower layers can do what they have to (e.g., IP call RSVP, ATM do a
 >> setup, FDDI request for bandwidth) to implement the QoS.
 what happens if the setsockopt() is done after the connect()?  In Winsock
 we have these clever little asynch handlers which introduce truly amazing
 race conditions in poorly coded mass market applications.

 A setsockopt(QOS) after a connect was issued but before the connect
 asynch notification could be a real joy for both Winsock stack provider
 as well as winsock app developer.
 >> 
 >> 
 >>     Second,  how to integrate scheduling?  Remember that in most cases where
 >> we're asking for a QoS guarantee, we also need to make sure that the BSD
 >> scheduler knows to give the real-time application some guarantees about
 >> CPU time.  Unfortunately, there is no defined way for BSD to provide guarantees
 >> about CPU time yet.  One option is to integrate this into the QoS request,
 >> but I think that's overloading the setsockopt() call.  An alternative is
 >> to assume the application is smart enough to request the CPU resources it
 >> needs at the same time it sets up the socket.  E.g., there's some bit of code
 >> like:
 >> 
 >>     sp = getschedulebyname("mpeg-2");
 >>     scheduler(sp);
 >> 
There is no equivalent in shipping Windows 3.1/3.11.  Chicago has
some concept of scheduling but is pie in the sky.  In the meantime
depending on the architecture of a Winsock vendors stack they can
have some limited control over scheduling or none at all.  Not
a pretty picture for realtime app's w/ millisecond needs.

I'm not sure how viable QOS can really be on a platform where 110 
millisecond granularity is the best guarentee available.  Of course
all the vendors can promise Mbytes/sec. of data thruput in controlled
performance tests; but QOS is about guarenteed delivery under all
circumstances, right?

Larry Backman
FTP Software

---

From: [email-masked]
Date: Thu, 11 Aug 1994 12:35:07 +0100

At  1:41 pm 6/8/94 -0700, Craig Partridge wrote:
   
>Here's an attempt to present my ideas in about three pages of text (short
>enough that one can easily read it).  I'm giving this wide circulation
>to solicit opinions as I'm about to start implementing this as part of
>a larger project to integrate ATM and QoS into a BSD socket interface.

Craig's suggestion for a QoS API in BSD seems well thought out and
plausible. It was of considerable interest to me as we have also designed a
QoS API. However, we had different design goals: we didn't attempt to be
UNIX compatible at all. In fact, our API is an extension of the Chorus API.
The only constraints we imposed on ourselves were that the API should have
a Chorus 'look and feel'.

I agree with Craig's assertion that you need to integrate scheduling into
the API. The way we did it was through the use of 'rthandlers' attached to
sockets (rtports in our jargon). Rthandlers are C functions (containing
application code) whose address is passed to the system in an
'attachhandler(rtport, * function)' call. Rthandlers get upcalled by the
system when data arrives. So the scheduler knows how much time an rthandler
will take, rthandlers take a boolean flag argument that chooses one of two
paths to be taken through the function. At admission control time an
infrastructure thread calls the rthandler with the flag set 'true' (say)
which makes the upcalling thread take a 'dummy run' through the code. This
is intended to take as long as a 'typical' call of the rthandler (the
application programmer had better make sure this is the case!). This way,
the infrastructure can take an initial estimate (which can be refined by
dynamic monitoring) of the number of CPU cycles the application needs for
each packet on the connection. This can then be used to configure the
scheduler.

In fact, the source as well as the sink end of connections use rthandlers.
To make this work, we have the idea of the *connection itself* being the
active entity with the application being passive. The application provides
code but supplies no threads to run it. Instead, the threads are
transparently allocated at connect time and upcall the application code at
times determined by the QoS of the connection. This reverses the usual API
model of an active application and a passive system (it's more like a
window manager model of callbacks in fact). The advantage is that
programming is considerably simpler for the application programmer because
the infrastructure can do so much.

Of course how much of this (if any) it would be desirable to take into the
BSD world is another question. You could use UNIX signals to implement our
rthandler notion with the convention that the application call pause()
straight after the connection is set up (like standard SIGIO except that no
additional send() or recv() system call is needed). I may just be dreaming
though :-).

Last, I thought I'd add a wish list for additional features in a QoS API.
These are taken from our current design. I'm sure that many possibilities
for inclusion were considered and rejected by Craig for very good reasons
to do with the limitations of UNIX(!) - both API conventions and underlying
functionality. But maybe its worth airing a few possibilities to see what
people think...

1. The QoS on connections seems to be geared to 'stream' data. But what
about 'real-time' messages - e.g. RPCs with bounded latency? These are
arguably equally important for distributed multimedia applications in
general.

2. We saw a need for structuring continuous media applications as pipelines
of processes (largely local but also distributed). And you don't want to
have QoS over the network only to lose it when you pass your data over
local pipes (or whatever). We thus have a uniform set of abstractions that
can connect processes both across the network and locally. Most of the QoS
management of the latter reduces to scheduling and memory mapping of
course. 

3. Our API/ infrastructure allows users to set up direct connections where
one (or both) ends can be a kernel-managed device (e.g. an audio card).
These direct connections avoid bringing data into user space when all you
want to do is shove it straight back to the kernel. It is relatively easy
to fit this idea into our API because of our notion of 'active' connections
and 'passive' applications (see above).

4. As well as supplying the threads (see above) the infrastructure also
supplies the buffer in our API. The address of the buffer (and its size) is
passed to the application code as an argument of the rthandler. The
application then fills/ consumes the data and, if it's a source rthandler,
returns the amount of valid data in a return argument. Application code can
expect to lose its buffer memory when it lets the rthandler return, so the
infrastructure can do cleverer buffer management - e.g. use wired pages
which can be safely reclaimed when necessary.

5. Our API does some 'typechecking' - i.e. when you create a socket (rtport
in our jargon) you pass a set of QoS parameters as arguments - i.e. I guess
this boils down to saying you combine the socket() and setsockopt() calls
in BSD terms. This causes no resource allocation until connect time but the
connect stage checks that the actual QoS requested in the connection is not
more stringent than that with which the rtport was originally created with.

6. We have 'commitment' as a QoS parameter which will (or is intended to)
work in the end-system as well as the network. We thus (attempt to) provide
'hard' guarantees for connections that want them. This is hard to do in an
environment where you get arbitrary interrupts hitting you and must also
support a UNIX personality(!) We do not yet have any empirical evidence as
to how well this will work... 

All comments welcome!

Geoff

---

From: [email-masked] (Bob Braden)
Date: Mon, 22 Aug 1994 10:04:00 -0700

OK, I will be able to do it.

Bob Braden

---

From: [email-masked] (Bob Braden)
Date: Mon, 22 Aug 1994 12:24:45 -0700

*> From [email-masked] Mon Aug 22 10:22:52 1994
  *> Date: Mon, 22 Aug 1994 10:04:00 -0700
  *> From: [email-masked] (Bob Braden)
  *> To: [email-masked], [email-masked], [email-masked]
  *> Subject: Re: A Generic Socket Interface for QoS
  *> Content-Length: 44
  *> X-Lines: 6
  *> 
  *> 
  *> 
  *> 
  *> OK, I will be able to do it.
  *> 
  *> Bob Braden
  *> 

Sorry, my mailer did something bizarre.  Please ignore this message
(a gratuitous request, surely!)

Bob Braden