Subject: Re: [tcpm] On the implementation of TCP urgent data (IETF Internet Draft)

From: Bill Squier <[email-masked]>
Date: Fri, 27 Feb 2009 18:13:11 -0500

On Feb 27, 2009, at 5:29 PM, Christos Zoulas wrote:

> In article <[email-masked]> you write:
>> Hello, folks,
>>
>> We have published a revision of our IETF Internet-Draft entitled  
>> "On the
>> implementation of TCP urgent data". The document is available at:
>> http://tools.ietf.org/id/draft-gont-tcpm-urgent-data-01.txt (you can
>> also get the document in other fancy formats, such as PDF, at
>> http://www.gont.com.ar/drafts).
>>
>> This document describes current issues relevant to the implementation
>> and use of TCP urgent data, aims to change the IETF specifications so
>> that they accommodate what virtually all implementations have been  
>> doing
>> wrt urgent data.
>>
>> The TCPM working group of the IETF is currently deciding whether to
>> adopt this document as a working group item, so that your input  
>> will be
>> very much appreciated.
>>
>> To voice your opinion, please send it to [email-masked], and CC me
>> ([email-masked]), so that I make sure that your post makes it  
>> to
>> the mailing-list, even if you are not subscribed to it.  
>> (Alternatively,
>> you can send me your input, and I could forward it to the [email-masked]
>> mailing-list).
>
> The behavior on Windows is more complex than this. I think that it  
> maintains
> much more data than a single byte and this can lead to resource  
> exhaustion.
> This behavior is used and required to be present by SQL Server, so  
> Microsoft
> is unwilling to change it. Bill Squier <[email-masked]> discovered  
> this and
> can provide you with details.

I haven't had time to read the article completely, but I did skim the  
Windows section, and Christos is correct.  Windows exhausts a non- 
paged fixed size pool of kernel memory (leading to a crash) whenever  
the receiver does not drain OOB data.  Thus, any Windows TCP receiver  
which does not expect OOB data is an avenue of attack.  For example,  
the NetBIOS listener.  I refer here to Windows versions <= XP.  I have  
not performed any testing on Vista and later.  Microsoft has been  
aware of this issue for years.  Jerry and James can speak to whether  
they have ever addressed it.

Further, I noticed that some of your analysis discusses OOB data  
bleeding in-line.  This is almost certainly caused by an interaction  
(on the _sender_) of Nagle and the fact that TCP defines only a single  
OOB pointer.  The receiver is not returning bytes which it knows to be  
OOB bytes inline, the _sender_ is accidentally placing more than a  
single byte of OOB in each packet that it sends.

The receiver has no way to know that, as the only means of  
communication about OOB data between sender and receiver is a single  
pointer.

I have added James and Jerry to this thread as well.

[note to [email-masked]; we are not subscribers, please preserve the  
cc: if you wish to solicit our feedback]

-wps

---

From: Jerry Leichter <[email-masked]>
Date: Sat, 28 Feb 2009 16:51:55 -0500

I haven't seen a copy of the paper, so I have to respond indirectly.

Microsoft confirmed the problem.  It was present in all their stacks  
prior to Vista.  Based on our repeated complaints, they eventually  
produced a patch for XP.  For some time, it was one of their "if you  
know to ask we'll give it to you" patches.  For all I know, they  
eventually decided to roll it in to some regular patch release.

The "OOB data leaking in line" has many causes.  I know of at least  
two completely different mechanisms:

	- On the sender, the problem can occur because you can only send
		one OOB byte per segment, but there is no correlation
		between segments and OOB write requests.  Consider the
		situation where the segmentation code wants to send a
		segment, and within the range of bytes it wants to send,
		there are two that were marked OOB.  The only way to handle
		this correctly is to send two segments, even if one of them
		is artificially short - perhaps even a single byte.  I
		haven't checked implementations in detail, but I've
		never seen this mentioned and I would be surprised if
		implementations get this right.

	- On the receiver, there is a messy case, the details of which
		I forget, in which the implementation doesn't keep enough
		state to properly handle two "close" OOB bytes.  If you
		look in Stevens, he actually reproduces a comment from
		the original BSD code which describes the problem, but
		basically says "you lose".  It's not clear anyone ever
		bothered to fix this.

The fundamental issue is this:  OOB bytes exist at the socket API  
level, but there is no such concept at the TCP level.  The urgent  
pointer exists at the TCP level, but it's basically invisible at the  
socket API level.  The theory is that you can use the urgent pointer  
to implement OOB bytes - and, in fact, that's true.  I've convinced  
myself of this by writing out a semi-formal definition of the  
theoretical semantics at the two levels and sketching an  
implementation that actually presents the right semantics at the  
socket API level.  I'm pretty sure that no one actually implements  
this right other than Microsoft (ironically) because it requires the  
ability to allocate space for OOB bytes at the receiver dynamically,  
and I know of no implementations that do so other than Microsoft's.   
By counting the allocated bytes against the window size you are  
willing you offer, you can avoid the problem that MS ran into, where  
there's no way to bound the amount of space allocated.  The  
implementation is potentially expensive, but it's only expensive for  
applications that actually send many OOB bytes - which is a reasonable  
tradeoff.

What actually happens is that the TCP stack implementers seem never to  
have believed in OOB on general principles, and in any case it seems  
that the TCP stack implementors (the network guys) and the socket API  
implementers (the OS guys) don't seem to talk to each other much.  So  
this delicate implementation issue, which lives exactly at the  
interface between the two, gets lost in the shuffle.  (Note that the  
multiple-OOB-bytes-in-a-segment problem probably cannot be solved  
without changing the TCP stack/socket API interface, since typically  
there is no way for the segmentation layer even to *see* that there  
are multiple outstanding OOB bytes:  The only available interface is a  
single urgent pointer value that gets passed across.)

There's an even more fundamental problem that it's worth pointing  
out:  The actual urgent pointer protocol has a flaw.  The UP is a 16- 
bit offset into the stream.  As long as no segment can be more than  
2^16 bytes - as was true when TCP was designed, because the windows  
size is represented in 16 bits - this works.  But if you use scaled  
windows, it's possible to be in a situation where you need to send a  
UP value that cannot be represented, because it's too far into a large  
segment.  There's an obvious work-around - send a short segment when  
this happens - but I doubt anyone does this.  In fact, I would guess  
that the offset gets calculated ignoring the 16-bit limitation and the  
actual UP sent is the lower 16 bits of the true UP.  Obviously, this  
can cause severe problems to any use of TCP that actually relies on  
the UP being correct!

Finally, there's yet another level of problem:  Some router-like  
devices (Cisco PIX firewalls - in their default configuration! - are a  
known example) simply turn off the Urgent bit!  This is to block a  
very old (10+ years?) Windows bug.  However, this is a disaster for  
any program that actually relies on OOB data/the UP.
                                                         -- Jerry

On Feb 28, 2009, at 3:36 PM, Fernando Gont wrote:

> Bill Squier wrote:
>
> (Added Andrew Yourtchenko (draft co-author) to the recipient's list)
>
> Comments inline...
>
>
>> Windows section, and Christos is correct.
>
> IIRC, we did the windows tests with cygwin. Maybe this is what lead to
> different results?
>
> That said, the next version of our Internet Draft will include text
> describing the buggy implementations you are referring to.
>
>
>> bleeding in-line.  This is almost certainly caused by an  
>> interaction (on
>> the _sender_) of Nagle and the fact that TCP defines only a single  
>> OOB
>> pointer.  The receiver is not returning bytes which it knows to be  
>> OOB
>> bytes inline, the _sender_ is accidentally placing more than a single
>> byte of OOB in each packet that it sends.
>
> There is no problem with that. TCP just provides a mechanism for  
> marking
> the end of urgent data. Just a mark.
>
>
>
>> communication
>> about OOB data between sender and receiver is a single pointer.
>
> Exactly. Any data that's before the pointer should be considered  
> "urgent".
>
> Thanks!
>
> Kind regards,
> -- 
> Fernando Gont
> e-mail: [email-masked] || [email-masked]
> PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1
>
>
>
>
>

---

From: Jakob Heitz <[email-masked]>
Date: Sun, 01 Mar 2009 20:27:02 -0800

inline...

Jerry Leichter wrote:
> have believed in OOB on general principles,

and neither does RFC 793.
Urgent data does not represent OOB data.
As soon as the urgent pointer is set, it means all data up to that point 
has become urgent.
It wasn't urgent before.
It basically means "The user has become bored with the output and has 
pressed Ctrl-C: all urgent data is to be flushed".
Fine, this is just one interpretation, but if you fantasize any more 
into it than that,
you run the risk of inventing really tiresome bugs.

> and in any case it seems that the TCP stack implementors (the network 
> guys) and the socket API implementers (the OS guys) don't seem to talk 
> to each other much.  So this delicate implementation issue, which 
> lives exactly at the interface between the two, gets lost in the 
> shuffle.  (Note that the multiple-OOB-bytes-in-a-segment problem 
> probably cannot be solved without changing the TCP stack/socket API 
> interface, since typically there is no way for the segmentation layer 
> even to *see* that there are multiple outstanding OOB bytes:  The only 
> available interface is a single urgent pointer value that gets passed 
> across.)

There is never more than one OOB byte. Actually, there is never more 
than zero OOB btyes.
If the urgent pointer changes, it means the user just pressed Ctrl-C again.
Flush some more.

> out:  The actual urgent pointer protocol has a flaw.  The UP is a 
> 16-bit offset into the stream.  As long as no segment can be more than 
> segment.

The urgent pointer is from the start of a segment, not the start of the 
window.
How can you send a segment longer than 64k? The IP length field is only 
16 bits.

>   There's an obvious work-around - send a short segment when this 
> happens - but I doubt anyone does this.  In fact, I would guess that 
> the offset gets calculated ignoring the 16-bit limitation and the 
>>> (on
>>> the _sender_) of Nagle and the fact that TCP defines only a single OOB
>>> pointer.  The receiver is not returning bytes which it knows to be OOB
>> There is no problem with that. TCP just provides a mechanism for marking

---

From: James Chacon <[email-masked]>
Date: Mon, 2 Mar 2009 22:19:13 -0600

On Feb 27, 2009, at 5:13 PM, Bill Squier wrote:

>>> ([email-masked]), so that I make sure that your post makes  
>>> it to
> I haven't had time to read the article completely, but I did skim  
> the Windows section, and Christos is correct.  Windows exhausts a  
> non-paged fixed size pool of kernel memory (leading to a crash)  
> whenever the receiver does not drain OOB data.  Thus, any Windows  
> TCP receiver which does not expect OOB data is an avenue of attack.   
> For example, the NetBIOS listener.  I refer here to Windows versions  
> <= XP.  I have not performed any testing on Vista and later.   
> Microsoft has been aware of this issue for years.  Jerry and James  
> can speak to whether they have ever addressed it.
>
> (on the _sender_) of Nagle and the fact that TCP defines only a  
> single OOB pointer.  The receiver is not returning bytes which it  
> knows to be OOB bytes inline, the _sender_ is accidentally placing  
> more than a single byte of OOB in each packet that it sends.
>

Actually we have seen leaks in-band when simply sending one byte at a  
time of OOB. It appears if the distance between 2 bytes exceeds the  
available OOB marker (which from memory was 16 bits) and nothing reads  
on the receiving end then on some impls the bytes leak in-band.

We've seen that on linux and on Solaris to my recollection.

James

---

From: Jerry Leichter <[email-masked]>
Date: Tue, 3 Mar 2009 08:24:36 -0500

On Mar 2, 2009, at 11:19 PM, James Chacon wrote:
> Actually we have seen leaks in-band when simply sending one byte at  
> a time of OOB. It appears if the distance between 2 bytes exceeds  
> the available OOB marker (which from memory was 16 bits) and nothing  
> reads on the receiving end then on some impls the bytes leak in-band.
>
I don't dispute the observations, but the explanation makes no sense  
to me.  Yes, the Urgent Pointer is a 16-bit offset from the beginning  
of the segment in which it appears (or something very much like  
that).  It would make no sense to store the UP position as just the 16- 
bit offset inside the implementation, since that's impossible to  
interpret without the segment starting point.  The received segment  
itself already has the (32 bit) position in the stream of the segment,  
and the pair is unambiguous.  An implementation could store the pair,  
but why?  The 32-bit sum of the UP and the segment starting position  
is 2 bytes smaller and contains exactly the same information.

With bugs anything is possible, of course, but it's hard to see why  
having the distance between successive OOB bytes being > 2^16 should  
be significant in and of itself.  How was this tested?  Are you sure  
it's really "gap > 2^16" or could it be "gap large enough" or "enough  
data buffered"?  I can see this leading to the failure mode that I  
described earlier:  User receiver hasn't read data for a long time;  
receiving TCP lets its receive window close as it's running out of  
buffer space; data piles up in sender's TCP stack buffers; eventually  
we have two OOB bytes within the data to be sent; sender doesn't keep  
track of this, or tries to send a segment with two OOB bytes and can  
only encode one.
                                                         -- Jerry

---

From: James Chacon <[email-masked]>
Date: Tue, 3 Mar 2009 09:24:41 -0600

On Mar 3, 2009, at 7:24 AM, Jerry Leichter wrote:

>> the available OOB marker (which from memory was 16 bits) and  
>> nothing reads on the receiving end then on some impls the bytes  
>> leak in-band.
>>
> to me.  Yes, the Urgent Pointer is a 16-bit offset from the  
> beginning of the segment in which it appears (or something very much  
> like that).  It would make no sense to store the UP position as just  
> the 16-bit offset inside the implementation, since that's impossible  
> to interpret without the segment starting point.  The received  
> segment itself already has the (32 bit) position in the stream of  
> the segment, and the pair is unambiguous.  An implementation could  
> store the pair, but why?  The 32-bit sum of the UP and the segment  
> starting position is 2 bytes smaller and contains exactly the same  
> information.
>
> it's really "gap > 2^16" or could it be "gap large enough" or  
> "enough data buffered"?  I can see this leading to the failure mode  
> that I described earlier:  User receiver hasn't read data for a long  
> time; receiving TCP lets its receive window close as it's running  
> out of buffer space; data piles up in sender's TCP stack buffers;  
> eventually we have two OOB bytes within the data to be sent; sender  
> doesn't keep track of this, or tries to send a segment with two OOB  
> bytes and can only encode one.
>                                                        -- Jerry
>

It's entirely possible your reasoning is also the cause. To be honest  
I never felt like diving into linux networking code to figure out the  
reason :)

I just know we have a reasonably reliable test which can replicate  
this on linux and the likelihood of seeing that fixed was low based on  
experiences with the select API and it's bugs.

James

---

From: Fernando Gont <[email-masked]>
Date: Tue, 03 Mar 2009 14:50:44 -0200

Jerry Leichter wrote:

>     You refer to the data pointed to by the urgent pointer as the urgent
> data.  This is incorrect.  TCP doesn't directly have a concept of urgent
> *data*; what it has is an urgent *mode*.  
[....]

This was the intent of the text. I'll go through the I-D again and will
try to remove any instances of what you describe.

> Note that this is a fine concept, but *it's completely unsupported by
> the socket API*.  While the socket API can tell you that you've reached
> the end of the urgent data (a read() stops at the urgent byte, and
> there's an ioctl() that tells you that you've just read it), there's no
> way to know when you've *entered* urgent mode.  At best, you can get a
> signal, but there's no way to figure out how that relates to the
> position in the TCP stream.  

Not sure what you mean.

> Given all that, I think your suggestion that there is no OOB data, only
> Urgent Mode, is exactly backwards.  

Isn't this the converse of what you stated at the very beginning of your
post??

> Unless I'm inside of the TCP stack,
> Urgent Mode is a meaningless concept.  I can't control when it starts, I
> can't detect when it starts, I can't even ask if I'm *in* urgent mode. 

huh? What about SIOCATMARK?

>     You comment that based on tests under Cygwin, OOB data is always
> delivered in-line.  If so, this is a Cygwin issue - I can tell you quite
> definitely that if you use the Windows socket implementation, by default
> OOB data does *not* get delivered in-line.

Will try this.

>     You recommend that applications that use OOB data request that it be
> delivered in line.  This is pointless - it fixes nothing.

This is the intented semantincs for TCP urgent mode. And nevertheless,
the code should be prepared to handle "urgent data" in-band, because
some middle-boxes while clear the URG bit, thus effectively putting
"urgent data" back "in-line".

>     The RFC's today *do not allow* middle boxes to turn of the UP bit.

And what? These are widely-deployed middleboxed. If your app depends on
tcp urgent mode, it will break.

> Either we change the spec to completely eliminate the whole concept of
> urgent data, or we assert that the spec is right, and these middle boxes
> don't conform.  It's absurd to specify something and then tell
> applications "well, it's in the spec, but you can't rely on it".  

It's absurd to neglect what's a fact.

> And,
> frankly, if NDIS's can't get this right - too bad.  Fix them.

This is impossible. The issue here is that there's the semantics of the
UP as specified by the IETF specs, and there are the semantics that real
implementations are using (which can be overridden by a system-wide toggle).

>     BTW, Microsoft actually *relies* on OOB data.  I don't know the
> details - something to do with aborting database transactions.  I expect
> you'd see significant opposition from them to an attempt to remove
> Urgent Mode from the specs.

Microsoft's implementation of TCP urgent mode is completely broken.
Shame on them.

>     When you come right down to it, Urgent Mode as specified by TCP is
> unambiguous (ignoring the age-old RFC793/RFC961 which is irrelevant in
> today's world:  Everyone has agreed on the same interpretation).  The
> actual problems are:
> 
>     - Middle-box implementations that *illegally* modify data in transit;

huh? You claim that RFC 793 is irrelevant, and then claim that clearing
the UP is illegal????

> None of these can be fixed at the TCP spec level, since that's not where
> the problem is.  (Well, the first is a matter of politics:  Getting
> implementers to actually implement the specs as they exist!)  

This is completely wrong. If you get implementers to implement the specs
as they exist, you'd be e.g. changing the semantics of the urgent
pointer (UP), which would basically break all those applications that
have not yet been broken by middleboxes clearing the URG bit.

Our draft aims at:

* Clarifying the concept of TCP urgent mode (no "out of band" data
channel. Simply a mark that signals whether you are in urgent mode or not)

* Changing the specs so that the semantics of the UP accommodate all
real implementations (i.e., "poinst to the last byte of urgent data" vs.
"points to the byte following the last byte of urgent data".

* Raising awareness about "urgent mode" not being reliable in the
current Internet (e.g., Cisco PIX clears the URG bit).

> So while I
> think it's important to raise these issues with IETF, this document is
> aimed at the wrong target.

I'm not sure what's the target you think our I-D should aim at.

Thanks!

Kind regards,

---

From: David Borman <[email-masked]>
Date: Thu, 12 Mar 2009 15:45:30 -0500

> try to remove any instances of what you describe.

RFC 793 specifies all three: the urgent pointer, urgent mode, and  
urgent data.  Urgent data is *all* the data before the urgent  
pointer.  Urgent mode is when you are reading the urgent data.  What  
TCP does not have is out of band (OOB) data.  Writing just one byte in  
urgent mode makes all the previously written and unread data urgent  
data.

>> position in the TCP stream.

select() should tell you when you have entered urgent mode, through  
the errorfds descriptor set (which is exceptional conditions, not just  
errors).

> huh? What about SIOCATMARK?

The sender controls when the connection leaves urgent mode, and by  
doing so, implicitly makes all the previously unread data urgent  
data.  On the receiver, select() will tell you when you've gone into  
urgent mode, and SIOCATMARK will tell you when you are at the mark.

>> OOB data does *not* get delivered in-line.

There is no OOB data in TCP.  BSD was wrong when it implemented it as  
such, but there were existing applications (telnet) that were using  
the OOB interface, so in order to not break existing binaries, the  
SO_OOBINLINE socket option was created so that you could turn off the  
broken OOB semantics.  That option has been around since at least  
4.4BSD.  Any application that uses the TCP urgent pointer and does  
*not* turn on SO_OOBINLINE has to deal with that decision, but it is  
out of scope for the IETF.

> "urgent data" back "in-line".

There is no OOB data in TCP.  Applications should be setting  
SO_OOBINLINE and using select to find out when they have transitioned  
into urgent mode, and using SIOCATMARK to determine when they are at  
the end of urgent mode.

> It's absurd to neglect what's a fact.

Middle boxes shouldn't be turning off the URG bit.

Applications shouldn't be using or relying the OOB socket interface on  
TCP connections.

TCP provides the urgent pointer as mechanism that the applications can  
use to indicate an "interesting" point in the TCP data stream.  It is  
not a record marker.  It is not one-to-one (if you don't read up to  
the mark before a new urgent pointer arrives, you lose the knowledge  
of the earlier urgent pointer).  It is up to the application to define  
what "interesting" means, what it should do with the data it reads  
while in urgent mode, and what to do once it gets to the mark.  An  
application can even be written so that it tolerates whether the  
implementation implemented the urgent pointer as a pointer to the last  
byte of urgent data, or the first byte of non-urgent data.

> Microsoft's implementation of TCP urgent mode is completely broken.

OOB != Urgent mode.  TCP does not have OOB data.  You can't get rid of  
something that was never there.

> current Internet (e.g., Cisco PIX clears the URG bit).

Yep, those are the issues.

> I'm not sure what's the target you think our I-D should aim at.

TCPM is the right place to discuss whether the urgent pointer is the  
last byte of urgent data, or the first byte of non-urgent data.

			-David Borman
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: David Borman <[email-masked]>
Date: Fri, 13 Mar 2009 09:55:31 -0500

Jerry,

On Mar 12, 2009, at 5:32 PM, Jerry Leichter wrote:

> On Mar 12, 2009, at 4:45 PM, David Borman wrote:
>> pointer.  Urgent mode is when you are reading the urgent data.   
>> What TCP does not have is out of band (OOB) data.  Writing just one  
>> byte in urgent mode makes all the previously written and unread  
>> data urgent data....
> A couple of comments:
>
> 	- *TCP* has no notion of "unread data" - as it has no notion of
> 	  *read* data either.  TCP can't possibly be altered to introduce
> 	  any such concept without first adding in a whole model of
> 	  the interaction between the TCP stack and its buffering and
> 	  user mode and its interaction with that buffering.  I can't
> 	  imagine why you would want to do that - or how you would do it
> 	  in a way that accurately reflects all existing implementations.

TCP buffers in sequence received data until the application reads it,  
that is what I mean by unread data.  TCP can even buffer data and  
defer making it available to the application until more data arrives;  
that is why there is a PUSH bit, so that the sender can indicate that  
no further data is coming, and any buffered data up to that point  
should be made available to the application for reading.  When you  
have unread data, and a new packet arrives with the URG bit, all that  
unread data becomes urgent data.

> 	- Because of this, "urgent data" is not a very interesting concept.
> 	  Urgent mode is to some degree, but even that's a mess.  Actual
> 	  TCP implementations involve two weakly-synchronized state
> 	  machines, one implementing TCP, one implementing the user code.
> 	  Urgent mode is a property of the former.  You mention later that
> 	  you can use the exception flags from select() to tell if you
> 	  entered urgent mode - one of the synchronization points between
> 	  the two machines.  However, it's a very limited synchronization.
> 	  For one thing, entering urgent mode is not the only thing that
> 	  causes the exception flag to be set - and you have no effective
> 	  way to check what the bit really means.  For another, if you're
> 	  executing a read() at user level when a segment with URG bit
> 	  set arrives, you're stuck.  I suppose you could alternate
> 	  select()'s with one-byte read()'s, but let's get real here.

You can register for the SIGURG signal to get out of band notification  
when urgent data has arrived.  If you have more than one file  
descriptor, you can use select() to determine which descriptor has the  
urgent condition.  You then use read() and SIOCATMARK to read the  
urgent data.  You can do reasonable read() calls, because read() will  
not return data that spans the mark in a single read() call.  The  
mechanisms for dealing with TCP urgent pointer through the socket  
interface have been around for at least 20 years, since the days of  
4.3/4.4 BSD networking, none of this is new.

> 	- I completely agree with you that TCP doesn't have OOB data.  I
> 	  said as much.  So?  TCP doesn't "have" ASCII or Unicode or HTTP
> 	  either.  All of these can be *built on top of* TCP.
>
> 	- Then again, *TCP* doesn't have a user API either.  That, too, is
> 	  built over a TCP implementation.

>
>
> 	- *The socket API* has OOB data.  The implementer's problem is how
> 	  to build a correct implementation of the socket API's OOB data
> 	  on top of the TCP primitives.  In fact, as I showed in a separate
> 	  message to Mr. Gont (which I'd be happy for forward if you're
> 	  interested; it's a bit long, and I don't want to just dump it
> 	  on a list of unknown size), it's actually not hard to implement
> 	  the socket OOB semantics of TCP, using urgent mode, as it is
> 	  specified today.  The implementation can even be equivalent in
> 	  performance to today's implementations, for streams that never
> 	  use OOB data.  It's true that one can improve the implementation
> 	  by making minor changes to TCP - but realistically the costs
> 	  would far outweigh the benefits.
>
> In summary:  This is not, and has never been, a TCP problem.  The  
> issues are at the boundary between the TCP stack and the socket API,  
> affecting code on both sides of that boundary.

The only TCP issue is the mismatch between the definition of the  
Urgent pointer as being the last byte of the Urgent data, and the  
typical implementation of it being the first byte of non-urgent data.

> Finally, on the question of whether the socket API should be changed  
> to eliminate OOB data:  That's perhaps an arguable position, exactly  
> because the implementations are so bad; but the implicit message in  
> the way you state it - that TCP is what matters, and the socket API  
> is just some minor thing - is exactly backwards.  There are at most,  
> what, a couple of hundred engineers in the entire world, working on  
> at most a few tens of distinct code bases, who actually deal with  
> *TCP*.  There are hundreds of thousands of engineers dealing with  
> millions of applications that depend on the socket API.

>
>
> Changing TCP is a big job, but one that would be doable if there  
> were sufficient need:  Look at IPv6.  Changing the socket API is  
> impossible - if you think IPv6 is taking a long time to roll out,  
> that's nothing compared to how long it would take a new,  
> incompatible socket API replacement.  Chances are it would never  
> happen.

I never said the socket API should be changed.  The mechanisms for  
dealing with TCP urgent data have existed for a long time.  The  
MSG_OOB flag provides a mechanism for an application to set the Urgent  
pointer when writing to a TCP socket.  On the receiver side, setting  
SO_OOBINLINE leaves all the TCP data inline (as it should).  Using  
SIGURG, SIOCATMARK and read() (and select() when needed), the  
application is notified when it should go into Urgent mode, and is  
able to identify when it has read up to the mark.  This all works  
today.  There is nothing to change at either the socket or TCP level.

			-David Borman

---

From: Jerry Leichter <[email-masked]>
Date: Thu, 12 Mar 2009 18:32:13 -0400

On Mar 12, 2009, at 4:45 PM, David Borman wrote:
> TCP does not have is out of band (OOB) data.  Writing just one byte  
> in urgent mode makes all the previously written and unread data  
> urgent data....
A couple of comments:

Finally, on the question of whether the socket API should be changed  
to eliminate OOB data:  That's perhaps an arguable position, exactly  
because the implementations are so bad; but the implicit message in  
the way you state it - that TCP is what matters, and the socket API is  
just some minor thing - is exactly backwards.  There are at most,  
what, a couple of hundred engineers in the entire world, working on at  
most a few tens of distinct code bases, who actually deal with *TCP*.   
There are hundreds of thousands of engineers dealing with millions of  
applications that depend on the socket API.

Changing TCP is a big job, but one that would be doable if there were  
sufficient need:  Look at IPv6.  Changing the socket API is impossible  
- if you think IPv6 is taking a long time to roll out, that's nothing  
compared to how long it would take a new, incompatible socket API  
replacement.  Chances are it would never happen.

                                                         -- Jerry

---

From: Andrew Yourtchenko <[email-masked]>
Date: Fri, 13 Mar 2009 21:32:50 +0100 (CET)

On Thu, 12 Mar 2009, Jerry Leichter wrote:

>
> In summary:  This is not, and has never been, a TCP problem.  The issues are 
> at the boundary between the TCP stack and the socket API, affecting code on 
> both sides of that boundary.

That's somewhat the problem is that it's "noone's problem". In my opinion, 
the best thing is a strongest possible message to the application 
developers "if you were planning to use this, don't". Whether the 
definition of is one byte left or right does not matter - the ambiguity is 
*already* there. Granted, it can be mitigated within the specific 
administrative domain, but if I talk to you across the internet, there's 
absolutely no guarantee whatsoever (try to interrupt an FTP transfer from 
a randomly selected server and see how many actually do not hang - 
amongst those that keep the tcp/21 connection of course)

>
> Finally, on the question of whether the socket API should be changed to 
> eliminate OOB data:  That's perhaps an arguable position, exactly because the 
> implementations are so bad; but the implicit message in the way you state it 
> - that TCP is what matters, and the socket API is just some minor thing - is 
> exactly backwards.  There are at most, what, a couple of hundred engineers in 
> the entire world, working on at most a few tens of distinct code bases, who 
> actually deal with *TCP*.  There are hundreds of thousands of engineers 
> dealing with millions of applications that depend on the socket API.

The initial discussion we had with Fernando that gave a rise to this draft 
was of the flavour "TCP Urgent considered harmful for application 
developers".

> sufficient need:  Look at IPv6.  Changing the socket API is impossible - if 
> you think IPv6 is taking a long time to roll out, that's nothing compared to 
> how long it would take a new, incompatible socket API replacement.  Chances 
> are it would never happen.

All depends on the business needs, but the TCP Urgent is indeed not 
among the most widely used ones. A clear and well-structured semantics of Out-Of-Band 
Notifications might be of some use though, IMHO - but then, one could as 
well use the separate UDP channel along - though it is not as "convenient".

thanks,
andrew

---

From: David Borman <[email-masked]>
Date: Thu, 19 Mar 2009 15:17:30 -0500

Hello TCPM members,

At the TCPM session on Monday, I intend to take a hum on whether or  
not the WG wants to take on the issue of the definition of the TCP  
Urgent pointer (the last byte of urgent data), and its contradiction  
with most TCP implementations (the first byte after the urgent data),  
as documented in draft-gont-tcpm-urgent-data-01.txt.  It would be nice  
to get a feel from the mailing list prior to the meeting.

Please reply whether you are in favor of, against or have no opinion  
on adopting this as a WG item.

		-David Borman, TCPM co-chair