Subject: TCP and/or sockets vs. the last message in full-duplex applications

From: [email-masked]
Date: Mon, 1 Mar 1999 11:45:19 PST

The OMG seems to be in the process of adopting a messy fix to a bug in either TCP or the sockets interface.  I was wondering if you could give me a reality check as to whether it really is a bug, whether the bug is in TCP or certain APIs and their interfaces, what's the prognosis for getting it fixed, and whether the work-around outlined below is the appropriate one.  Apologies if this is not the right forum (and what is the right one?).

The issue below concerns the OMG's standard RPC protocol, GIOP, and its mapping to TCP, IIOP.  The problem occurs when a server initiates a connection closure at roughly the same time the client starts writing a new request message.  Clean shutdown messages, sent from server to client just before closure, were recently introduced to GIOP, but the bug seems to prevent them from being seen when the timing is close.  The proposed fix is to make the server: send the clean shutdown, close the sending half of the TCP connection, consume data until EOF or timeout, then close the receiving half.  This seems like an inappropriate amount of bother just to reliably send a message and close.

---------------- Forwarded  Message ----------------------

Date: Sun, 28 Feb 1999 14:49:41 PST
From: Jonathan Biggar <[email-masked]>
Subject: Re: Draft Final Proposal for CloseConnection Issue for Review
To: Bill Janssen <[email-masked]>
cc: [email-masked], [email-masked]

Bill Janssen wrote:
> 
> I notice a problem with another section of this.
> 
> >    * After reliably issuing a CloseConnection message, the issuing orb
> >      may close the  connection.  Some transport protocols (not
> >      including TCP) do not provide an ?orderly disconnect? capability,
> >      guaranteeing reliable delivery of the last message sent. When
> >      GIOP is used with such protocols, an additional handshake needs
> >      to be provided as part of  the mapping to that protocol's
> >      connection mechanisms,  to guarantee that both ends of the
> >      connection understand the disposition of any outstanding GIOP
> >      requests.
> 
> Actually, TCP does not seem to provide reliable delivery of the last
> message sent.  Take the case where the server sends a CloseConnection,
> then closes the TCP connection.  If the client is writing to the socket
> during this, it will send data to a closed connection on the server.
> The server will then respond with a TCP reset.  This reset ``jumps the
> queue'', and instead of seeing the CloseConnection message on its next
> read on the socket, the client will see an ECONNRESET error.  This means
> that one side may still see connection failures while the other side
> presumes to have sent an orderly shutdown.

This is actually dependent on the TCP implementation.  Some throw away
any data pending to read when the ECONNRESET occurs, others keep it.  It
would be best to document this using the TCP standard rather than the
sockets implementation of it anyway.  In the real TCP standard, each
direction of the connection can be shutdown (i.e. send a FIN)
independently.  In practice, for sockets, use the shutdown() call
instead, with the argument 1, which means close the sending side only.

I would rewrite this paragraph like this:

   * After reliably issuing a CloseConnection message, the issuing orb
     may close the  connection.  Some transport protocols (not
     including TCP) do not provide an "orderly disconnect" capability,
     guaranteeing reliable delivery of the last message sent. When
     GIOP is used with such protocols, an additional handshake needs
     to be provided as part of  the mapping to that protocol's
     connection mechanisms,  to guarantee that both ends of the
     connection understand the disposition of any outstanding GIOP
     requests.  For TCP, the orb should only shutdown the sending side
     of the connection, and then read and discard any incoming data
until
     it receives an indication that the other side has also shutdown. 
At this
     point, the TCP connection can be closed completely.

-- 
Jon Biggar
Floorboard Software
[email-masked]
[email-masked]

---------------- End of Forwarded Message ---------------

---

From: Craig Partridge <[email-masked]>
Date: Tue, 02 Mar 1999 09:49:24 -0800

Well there are two issues here:

1. What to do with a FIN if you have data still to read

    Jumping the queue sounds busted to me.

2. Whether an reliable close is possible

    It is not -- it is an instance of the two armies problem.

Craig

In message <99Mar1.114532pst."105927"@augustus.parc.xerox.com>, [email-masked]
erox.com writes:

>
>The OMG seems to be in the process of adopting a messy fix to a bug in either 
 >TCP or the sockets interface.  I was wondering if you could give me a reality
 > check as to whether it really is a bug, whether the bug is in TCP or certain
 > APIs and their interfaces, what's the prognosis for getting it fixed, and wh
 >ether the work-around outlined below is the appropriate one.  Apologies if th
 >is is not the right forum (and what is the right one?).
>
>The issue below concerns the OMG's standard RPC protocol, GIOP, and its mappin
 >g to TCP, IIOP.  The problem occurs when a server initiates a connection clos
 >ure at roughly the same time the client starts writing a new request message.
 >  Clean shutdown messages, sent from server to client just before closure, we
 >re recently introduced to GIOP, but the bug seems to prevent them from being 
 >seen when the timing is close.  The proposed fix is to make the server: send 
 >the clean shutdown, close the sending half of the TCP connection, consume dat
 >a until EOF or timeout, then close the receiving half.  This seems like an in
 >appropriate amount of bother just to reliably send a message and close.
>

---

From: [email-masked] (Alan Cox)
Date: Wed, 3 Mar 1999 01:43:08 +0000 (GMT)

>     Jumping the queue sounds busted to me.

Jumping the queue destroys everything. Been there, done that while I was
still a tcp writing newbie. You break ftp, everything and people bitch. I
thus find it hard to believe anyone got away with the mistake in a release
product without receiving hate mail by the truckload

Alan

---

From: "Mike Spreitzer" <[email-masked]>
Date: Tue, 2 Mar 1999 23:58:32 PST

>     Jumping the queue sounds busted to me.

No, the FIN isn't the problem here.  The RST is.

>     It is not -- it is an instance of the two armies problem.

The problem at hand is more one of reliable data delivery before the close.
If only the RST were handled differently upon receipt at the client side
(i.e., didn't cause discarding of the segments received from the net but not
yet delivered to the app), we'd have this reliable data delivery.  I think
one possible answer would be to refine the RST bit into two bits: a more and
a less severe one.  But I'm also not sure it's right for the server to
respond with RST in the case at hand.

Mike

---

From: Henrik Frystyk Nielsen <[email-masked]>
Date: Wed, 03 Mar 1999 10:13:33 -0500

At 11:45 3/1/99 PST, [email-masked] wrote:

>The proposed fix is to make the server: send the clean shutdown,
>close the sending half of the TCP connection, consume data until
>EOF or timeout, then close the receiving half.  This seems like an
>inappropriate amount of bother just to reliably send a message and
>close.

This is old news in HTTP world - in fact (although I can't find any mails
in the HTTP archives to show this), I believe to have first found this
problem in early 1995 when experimenting with cross-atlantic HTTP/1.0 PUT
requests between MIT and CERN. The problem that I saw for HTTP/1.0 PUT is
the following:

* Client A sends a PUT request with a large body to the server
* The server B sends back a short 401 Access Denied response
  and closes the connection in both directions.
* A receives the response but has already sent a large part of
  the body because of the RTT across the Atlantic
* A ACK's the response but the ACK is pending behind the data
  already sent to B
* B sees that data is still coming and sends a RST to A
* A gets the RST and passes it up immediately to the application
  dropping the HTTP response because the seq counter hasn't
  been updated.
* A gets a RST but no HTTP response - it doesn't know what happened.

At the time I talked with Dave Clark about it and he pointed out that it
was not a bug but normal behavior.

The problem can also occur in HTTP/1.1 pipelining which we found when
implementing this in the libwww HTTP code. This is described in an IETF
draft by Jim Gettys and Alan Freier that was never finished [1], section 8:
	
   In simple request/response protocols (e.g. HTTP/1.0), a server can go
   ahead and close both receive and transmit sides of its connection
   simultaneously whenever it needs to. A pipelined or streaming
   protocol (e.g. HTTP/1.1) connection, is more complex [Frystyk et.
   al.], and an implementation which does so can create major problems.

   The scenario is as follows: an HTTP/1.1 client talking to a HTTP/1.1
   server starts pipelining a batch of requests, for example 15 on an
   open TCP connection.  The server decides that it will not serve more
   than 5 requests per connection and closes the TCP connection in both
   directions after it successfully has served the first five requests.
   The remaining 10 requests that are already sent from the client will
   along with client generated TCP ACK packets arrive on a closed port
   on the server. This "extra" data causes the server's TCP to issue a
   reset which makes the client TCP stack pass the last ACK'ed packet to
   the client application and discard all other packets. This means that
   HTTP responses that are either being received or already have been
   received successfully but haven't been ACK'ed will be dropped by the
   client TCP. In this situation the client does not have any means of
   finding out which HTTP messages were successful or even why the
   server closed the connection. The server may have generated a
   "Connection: Close" header in the 5th response but the header may
   have been lost due to the TCP reset. Servers must therefore close
   each half of the connection independently.

This has in HTTP/1.1 rev 6 [2] been moved to section 10.4

	If the client is sending data, a server implementation
	using TCP SHOULD be careful to ensure that the client
	acknowledges receipt of the packet(s) containing the
	response, before the server closes the input connection.
	If the client continues sending data to the server after
	the close, the server's TCP stack will send a reset packet
	to the client, which may erase the client's unacknowledged
	input buffers before they can be read and interpreted by
	the HTTP application.

I believe that all modern HTTP servers in fact do the half-close. However,
this is not without problems as the server also has to be able to protect
itself. It therefore has to have some feeling of a "reasonable lingering
time" for the connection.

Henrik

[1] http://www.ics.uci.edu/pub/ietf/http/draft-ietf-http-connection-00.txt
[2] http://www.ics.uci.edu/pub/ietf/http/draft-ietf-http-v11-spec-rev-06.txt
--
Henrik Frystyk Nielsen,
World Wide Web Consortium
http://www.w3.org/People/Frystyk

---

From: "Mike Spreitzer" <[email-masked]>
Date: Wed, 3 Mar 1999 09:03:04 PST

Actually, it looks to me like things would be great if we could just do the
"CLOSE" operation defined in the TCP RFC (739).  The "Event Processing"
section there outlines a series of transitions in which the TCB is not
discarded immediately at close time but rather is put through a series of
state transitions that do not call for RST to be sent (even if user data
arrives).  So one analysis is that sockets close() isn't TCP's "CLOSE".  Is
shutdown() with (how==2)?

Cheers,
Mike

---

From: "Eric A. Hall" <[email-masked]>
Date: Wed, 03 Mar 1999 20:10:14 -0800

> implementing this in the libwww HTTP code. This is described in an
> IETF draft by Jim Gettys and Alan Freier that was never finished [1],
> section 8:

>    The remaining 10 requests that are already sent from the client
>    will along with client generated TCP ACK packets arrive on a
>    closed port on the server.

So when would the server have received the ACKs for the data it sent?
This just never would have worked in anything resembling reliability.

-- 
Eric A. Hall                                            [email-masked]
+1-650-685-0557                                    http://www.ehsco.com

---

From: Henrik Frystyk Nielsen <[email-masked]>
Date: Fri, 05 Mar 1999 11:13:51 -0500

>This just never would have worked in anything resembling reliability.

It was less an issue knowing this wouldn't work than making sure it would
actually be fixed in HTTP servers moving from HTTP/1.0 (where it wasn't an
issue) to HTTP/1.1.

Henrik