Subject: A Generic Socket Interface for QoS

From: Craig Partridge <[email-masked]>
Date: Sat, 06 Aug 94 13:41:58 -0700

Hi folks:
    
    Several folks have asked me to put my views on how to enhance the BSD
socket (and presumably WINSOCK) interfaces to support quality of service.
Here's an attempt to present my ideas in about three pages of text (short
enough that one can easily read it).  I'm giving this wide circulation
to solicit opinions as I'm about to start implementing this as part of
a larger project to integrate ATM and QoS into a BSD socket interface.

The Basic Premise

    The ideas here stem from a basic premise.  If at all possible, we want
a single, generic interface for QoS support.  Just as sockets are a generic
interface for networking protocols so far, any extensions for QoS should
also be generic possibly with bells and whistles for various protocol
families.   [This work also motivates the ATM socket work I'm doing with
Trevor Mendez and Shawn Smith -- we've got code using a PF_ATM protocol
entry table and mapped calls like bind() and connect() into ATM actions]
The point is that regardless of whether one is using IPv4, IP6, or ATM,
the interface should be the same.  The job of operating systems is to
provide such generic interfaces in a way that makes programming them easy
and comfortable for applications programmers.

The Interface

    If one looks carefully at the BSD socket interface, its logic is similar
to that of UNIX process creation.  First, you create a socket using socket().
Then you customize it, using bind(), setsockopt(), fcntl() and ioctl().
Finally you make the socket operational (i.e. ready to send data or to
accept connections) using connect(), or listen()/accept().
[The similarity to process creation is that one calls fork() to create the
process, then customize the new process's privileges, open files, etc,
and then call exec() to get the new image loaded].

    From the perspective of QoS, we obviously want the QoS in place once
the socket becomes operational.  I.e., after connect() or listen()/
accept() returns, we expect our QoS requirements to be in place and thus
to be able to start sending or receiving data.  Note, parenthetically, that
we'd obviously like to be able to check on the QoS at any time during the
life of the active socket.

    My view is that, if one agrees with this analysis, the right thing
is to find some way to set the QoS between the time socket() is called and
the time connect() or listen() is called.  And there already exists a very
generic system call for customizing sockets -- setsockopt().  So my proposal
is that we standardize one or more options for setsockopt to set a QoS.  (And
use getsockopt() to get any information about QoS, such as after doing an
accept()).

    Here's a brief outline.  Let's suppose that there's one UNIX QoS structure
for all protocol families (a topic discussed further later).  We can define
an option SO_QOS.  An video application that wanted to set a QoS could do the
following:

    if ((s = socket(AF_FAMILY, SOCK_STREAM, 0)) < 0)
	{ perror("socket"); exit(1); }

    if ((qos = getqosbyname("mpeg-2")) == (struct qos *)0)
	{ fprintf(stderr,"unknown service\n"); exit(1); }

    if (setsockopt(s,SOL_SOCKET,SO_QOS,(char *)qos,sizeof(*qos)) != 0)
	{ perror("setsockopt"); exit(1); }

    if (connect(s,(struct sockaddr *)dst,sizeof(*dst)) != 0)
	{ perror("connect"); exit(1); }

to get a socket opened with an established QoS.  If no QoS is set, the
socket QoS defaults to best effort (i.e. what it is today).

    Internally in socket implementations there's some logic to this approach
too.  A call to connect() is the first time that any packets are sent.
So one can enhance the internal kernel code to pass the QoS structure down
with the first packet (e.g., the TCP syn) through the transport layer to
IP (for IP QoS) and the link level driver (e.g., ATM, Fast Ethernet, FDDI)
so that lower layers can do what they have to (e.g., IP call RSVP, ATM do a
setup, FDDI request for bandwidth) to implement the QoS.

    I envision that we'd implement routines like getqosbyname() so application
writers would not have to know the innards of the QoS structure unless they
needed to.  They'd just need to know what type of application they were running.

A Few Open Questions

    I'm aware of two open issues with this interface.
    
    First, is the QoS structure generic (e.g., SOL SOCKET) or per-protocol
family specific?  This isn't a big deal.  A generic structure would be nice but
if we have protocol specific formats it just means that getqosbyname has
two args (protocol family and name of service), and setsockopt has QOS
option values for each protocol family, and we have to define a new QoS
structure for each protocol family.  A bit of a nuisance for application
writers but that's it.

    Second,  how to integrate scheduling?  Remember that in most cases where
we're asking for a QoS guarantee, we also need to make sure that the BSD
scheduler knows to give the real-time application some guarantees about
CPU time.  Unfortunately, there is no defined way for BSD to provide guarantees
about CPU time yet.  One option is to integrate this into the QoS request,
but I think that's overloading the setsockopt() call.  An alternative is
to assume the application is smart enough to request the CPU resources it
needs at the same time it sets up the socket.  E.g., there's some bit of code
like:

    sp = getschedulebyname("mpeg-2");
    scheduler(sp);

in the code that gets the right scheduling parameters and calls the scheduler
(via a new system call) to get that scheduling.

I welcome any comments folks have.  However, I do ask that if you believe I've
got this design all wrong, you provide an alternative theory of BSD works now
and show how your proposal fits your model.  I think we're much more likely
to have a successful approach if it fits with what has gone before.

Thanks!

Craig

---

From: Jon Crowcroft <[email-masked]>
Date: Sun, 07 Aug 94 14:08:19 +0100

>CPU time.  

Craig

why do you need to tell the scheduler anything new - the QoS guarantee
for packets is done by sceduling packets in routers - surely  the
scheduler in the end system can make the same calcuation (or derived
one) insterad of burdening the application with an extra system
call...?

 jon

---

From: Craig Partridge <[email-masked]>
Date: Sun, 07 Aug 94 08:18:02 -0700

Hi Jon:
    
    You have to talk to the scheduler because the workload on the end
systems is not simply the cost of moving packets through queues.  You also
have to allocate time for the application to process the data.  And different
applications will have different processing requirements (there's a big
difference between, say, deciding where to place some bits on a screen and
doing some compression or decompression of video).  In other words, the
amount of time required per packet varies widely on a per application basis
and only the application knows how much time it needs per packet if it is
to keep up with the data stream.

Craig

---

From: [email-masked]
Date: Fri, 12 Aug 94 09:46:06 -0400

Craig,

----------------------------------------------
FROM Jon Crowcroft

---------------------------------------------

>to keep up with the data stream.

I have to agree with Jon.  The 1003.4 Realtime spec will set realtime
switches per implementations.   I do not believe IP6 should add
scheduling to the API for networking.   These should be distinct. What
we don't want to do with sockets is overload them to solve the world
hunger problem on end systems.  

At the IP6 implementors meeting in Toronto we also heard a request to
add management interfaces to sockets.  I also believe this leaving the
paradigm for sockets.  I think right now its more important to define
and efficient and expedient sockets interface that will:

  1.  Be backwards compatible (and binary) with IPv4.
  2.  Possibly let the connect() determine the IP6 or IPv4 stack.
  3.  Update gethostbyxxx() calls that need to understand IP6 | IPv4.

Lets solve 80% of the engineering problem and get an API out for folks
to work with on IP6.   

I am working with Bob Gilligan to write the new IP6 API spec and input
is needed and I am not hard lined on the above but I would like to see
good justifications to alter the present paradigm of the BSD sockets
model.

I do agree QOS should be a set sockopt.

????

/jim

---

From: [email-masked]
Date: Fri, 12 Aug 94 16:11:49 -0400

Craig,

> setup, FDDI request for bandwidth) to implement the QoS.

I think you would also need to add similar logic between "listen()" and
"accept()" on the server side since QoS relates to the capacity/load
of (1) network interface (including IP and the link level driver) on 
the sender end system, (2) the network components along the way,
and (3) network interface on the receiver end system.

> (via a new system call) to get that scheduling.

Can we separate the QoS guarantee for communication (i.e., the socket
and protocol stack) from the QoS for the rest of the application here
at first?

In general, an application will, e.g., 'do some processing on the packet"
(e.g., decode the image), then send or receive packets from the socket
in a loop. Thus with the qos in place for sockets, the user can
expect QoS guarantee up to/from the point of send or receive, but
not beyond. The QoS for 'do some processing on the packet" before sending
or after receiving should be a separate CPU scheduling issue.
However, I do understand that the QoS guarantee for just the 'communication'
part still need the CPU to some extend, but I think it is easier
to deal with using your logic on the internal socket implementations.

--Chia

---------------------------------------------------------
Chia Shen                          Email: [email-masked]
Research Scientist                 Work : (617) 621-7528
Mitsubishi Electric Research Labs. Fax  : (617) 621-7550
Cambridge Research Center        
201 Broadway,
Cambridge, Mass. 02139
---------------------------------------------------------

---

From: Craig Partridge <[email-masked]>
Date: Fri, 12 Aug 94 21:41:09 -0700

Hi Jim:
    
    I think we're having an agreement of sorts.

    All my note was trying to say is that a setsockopt() won't solve all
the problems related to doing real-time.  This isn't to say that I want
us to put the scheduling stuff into the API -- that's not our problem --
the socket part is.  But if we standardize the socket part and don't
point out that scheduling is also needed, we're likely to surprise
somebody.

Craig

---

From: Gopal <[email-masked]>
Date: Sat, 13 Aug 1994 12:53:20 -0500

> From: Craig Partridge <[email-masked]>
> Date: Sat, 06 Aug 94 13:41:58 -0700
> Subject: A Generic Socket Interface for QoS
> 
> Hi folks:
>
At Washington University in St.Louis, we have been looking at a general solution to
the problem of providing QoS guarantees in endsystems specially with a view to
support multimedia applications. In fact my PhD topic deals exactly with this
problem and the work that I have done till date is available in the form of a
technical report over mosaic. The URL for my home page

http://dworkin.wustl.edu/~gopal/hmpg.html.
Click on the anchor labeled "How to provide end-to-end QoS guarantees".

You are all welcome to go through it and comment freely on its contents.
I must confess that I'm just now beginning to grapple with implementaion
issues such as the ones raised by Craig, but I shall attempt to answer some
of his questions on the basis of what I have proposed in the design.

> structure for each protocol family.  A bit of a nuisance for application

The QoS structure in my design is not generic but depends on the "application
class". I have 4 QoS classes -- One is the 'isochronous' class for compressed and
uncompressed CM data, the other is the 'burst' class for bulk transfer (such as
for files in a mosaic type of application that need certain rate of transfer), 
the third is the 'delay bounded message' class for RPC type applications,
and the fourth is the 'rate bounded message' class for DIS like applications
that could require guarantees on the rate at which messages are delivered.
The QoS parameters for each class, as well as the justification for the
choice of classes and the parameters is in the report. 

> about CPU time yet.  One option is to integrate this into the QoS request,

My design provides a solution to the scheduling problem but cannot be used in BSD
as is. This is because it is based on a user level protocol implementation model
and this choice is motivated by the fact that in most OSs the process (or thread) is
the basic unit of resource management. Therefore if we map QoS parameters to
resource attributes then the existing resource management will ensure that the QoS
guarantees are provided. To be able to do this, we have to integrate protocol code
that implements the data transfer path into the process theat owns the network
connection so that the process can then be provided with appropriate resources to
carry out the protocol processing. These ideas are definitely not new and people at
CMU have been exploring this route and have reported performance figures too. The
way our solution differs is that we plan to automate the mapping of QoS parameters
to resource attributes in a protocol independent manner (i.e the protocol code does
not have need to know about resource management). We are considering as resources
the CPU, NIA and the network connection.
We propose the use of real-time threads to provide guaranteed processing. I don't
know if BSD will provide us this facility in the near future and so I'm looking to
put this into netBSD.

Of course we are keenly alive to the fact that user level protocols are lagging
behind in performance compared to their kernel resident cousins (BSD). Work done
by the University of Washington, Seattle folks does solve some of the problems.
However, we have proposed several performance improvement implementation techniques
for user level protocols that we will implement and evaluate.

> Craig

I shall follow with considerable interest the discussions on the BSD API and try to
incorporate it into our implementation as much as possible.
-gopal

---

From: [email-masked]
Date: Mon, 15 Aug 94 08:40:17 -0400

Craig,

ACK OK.  

/jim