Subject: [tcpm] Extending the TCP option space - yet another approach

From: Andrew Yourtchenko <[email-masked]>
Date: Tue, 12 Apr 2011 23:26:52 +0200

Folks,

my observation is - there's an ongoing "musical chairs" game with
respect to TCP options space. As a WG I think we should take a step to
tackle it upfront and see what (if) can be done about it.

During the IETF in Prague, I discussed an idea offline with several
people, and the rough consensus  from what I heard was that I should
package it up for further beating up into a draft, to hear more from
the WG at large.

So here it is: http://tools.ietf.org/html/draft-yourtchenko-tcp-loic

NB: the draft deliberately omits the discussion of the hard problem of
the two parallel 3WHS operations and their timing to get a robust
solution. The draft is there to get a sense from the community on the
general approach - whether it may be used as a building block going
forward or not at all.

flame on, please! :-) (I know, the name does implicate :-)

cheers,
andrew

---

From: "Anantha Ramaiah (ananth)" <[email-masked]>
Date: Tue, 12 Apr 2011 15:18:56 -0700

To add more...

    History : TCP option space issue, when I raised it 5 years back, was
deemed as a "solution in search of a problem".  There was no convergence
reached on any of the proposals that were being discussed. Adam Langley
also tried to revive the TCP options discussion sometime back.

Time passed by and currently with many proposals seeking a new TCP
option etc., (includes MPTCP) the TCP option space is more and more
becoming a real issue. I chatted with the chairs during this IETF and
they seem to agree that this is an area where TCP WG can do something. 

>From my part I had volunteered to write up an ID ( a small document)
describing the problem and reasons for doing TCP option space extension.
I am working on it.

-Anantha

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
Of
> Andrew Yourtchenko
> Sent: Tuesday, April 12, 2011 2:27 PM
> To: [email-masked]
> Subject: [tcpm] Extending the TCP option space - yet another approach
> 
> Folks,
> 
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: Joe Touch <[email-masked]>
Date: Tue, 12 Apr 2011 15:48:03 -0700

Hi, all,

On 4/12/2011 3:18 PM, Anantha Ramaiah (ananth) wrote:
> deemed as a "solution in search of a problem".

I would describe it as a "unsolvable problem" with many proposed 
solutions, none of which was viable.

Here's the nutshell, as I recall it:

- extra space after the 3WHS is easy; create an option, and once it's 
ACK'd you can use it (in each direction)

- extra space is useful only if it exists in the SYN, however; that's 
where space is at a real premium

- extra space in the SYN is not possible
there's no way to provide the space before you negotiate support in a 
backward compatible way

There were several mechanisms, some of which attempted to use data in 
the SYN (which corrupts the data path if not supported), and even one 
that suggested just doubling all the TCP header fields (Mark Allman).

Regardless of need or utility, it is simply not possible in the SYN.

So basically:
- we can trivially add space after a SYN exchange; it that's useful, 
let's do it

- if the real need is for space inside the SYN, a real solution needs to 
be presented that is backward compatible

Non-backward compatible solutions to increased SYN space are trivial too 
- use a new protocol number. ;-)

Regarding Andrew's proposal, IMO it isn't useful:

- outboard boxes and some intermediate systemsdump packets with 
erroneous checksums -- or worse, they update the checksums ;-( the 
latter would be really bad

- even if they didn't, if you support the new method, how long do you 
hold the first SYN you get in hopes of seeing a SYN-LO? that's the 
trouble with dual-open tricks; you get stuck waiting for something that 
might never happen, and end up delaying everything

If I missed anything, please post.

Joe

---

From: Jakob Heitz <[email-masked]>
Date: Tue, 12 Apr 2011 18:59:50 -0400

> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of Joe Touch

Hold it until the next packet comes. If that packet isn't
a SYN-LO, then it's not coming. Nothing is delayed.

--
Jakob Heitz.

---

From: Joe Touch <[email-masked]>
Date: Tue, 12 Apr 2011 16:03:24 -0700

> a SYN-LO, then it's not coming. Nothing is delayed.

What next packet? If it's a legacy TCP, you're waiting for a SYN 
retransmission, and you've now killed their performance.

Joe

---

From: Joe Touch <[email-masked]>
Date: Tue, 12 Apr 2011 16:06:14 -0700

FWIW, I'd be glad to write up the problem and why a solution cannot 
exist, so we can stop going around this block ;-)

Joe

---

From: Wesley Eddy <[email-masked]>
Date: Tue, 12 Apr 2011 20:09:45 -0400

> ACK'd you can use it (in each direction)

I agree.

> where space is at a real premium

I disagree; there's definitely utility beyond just having this 
capability in the SYN, some of this was defined in the LO/SLO draft 
linked to below.

> there's no way to provide the space before you negotiate support in a

I agree.

> that suggested just doubling all the TCP header fields (Mark Allman).

The ones I recall are:
- LO/SLO : http://tools.ietf.org/html/draft-eddy-tcp-loo-04
- DO redefinition : http://tools.ietf.org/html/draft-kohler-tcpm-extopt-00
- tcpx2 : https://tools.ietf.org/html/draft-allman-tcpx2-hack-00

Andrew's LOIC is interesting, and significantly different than these. 
If there's evidence that it can survive some significant fraction of 
middleboxes, that would be good; this was a concern with all 3 of the 
proposals listed above.

> Regardless of need or utility, it is simply not possible in the SYN.

The SLO option was an attempt to handle that by sending long options 
that you would have liked to be on the SYN later (i.e. on the ACK 
completing the handshake); in implementation, this meant calling the 
function in the kernel for processing SYN options on a received packet 
even though the packet wasn't a SYN, which was slightly awkward, but 
fairly simple.

> - we can trivially add space after a SYN exchange; it that's useful,

The LO option is one way to handle that:
http://tools.ietf.org/html/draft-eddy-tcp-loo-04

-- 
Wes Eddy
MTI Systems

---

From: Joe Touch <[email-masked]>
Date: Tue, 12 Apr 2011 17:39:25 -0700

Hi, Wes,

On 4/12/2011 5:09 PM, Wesley Eddy wrote:
...
> capability in the SYN, some of this was defined in the LO/SLO draft

If there is, it's a lot easier to do as a single mod.

...
> Andrew's LOIC is interesting, and significantly different than these.

It's a bit of revisiting tcpx2, and just assumes that the endpoint tries 
to start two connections at the same time. That's been suggested before, 
but has problems:

	- how long do you wait to figure out which connection to accept?
	- what happens when either attempt's SYN is lost?

AFAICT, all variants here end up costing either the legacy or the new 
variant excessive delays, and that's not going to fly.

> even though the packet wasn't a SYN, which was slightly awkward, but

This was discussed on this list back in 2004. The key interfering issue 
raised then, AFAICT, was intervening middleboxes that transparently 
rewrite segments.

I'm not sure whether that's still an issue, or whether other issues were 
raised, but let's recall that thread before exploring this as if we 
haven't discussed it before ;-)

> http://tools.ietf.org/html/draft-eddy-tcp-loo-04

Sure - but that would need to be factored out as a separate thing if 
deemed useful.

Joe

---

From: Mark Allman <[email-masked]>
Date: Tue, 12 Apr 2011 22:17:22 -0400

> and even
> one that suggested just doubling all the TCP header fields (Mark
> Allman).
>
> [...]
>
> Non-backward compatible solutions to increased SYN space are trivial
> too - use a new protocol number. ;-)

Note, my proposal did both of these.

allman

---

From: <[email-masked]>
Date: Wed, 13 Apr 2011 10:32:52 +0100

> FWIW, I'd be glad to write up the problem and why a solution 
> cannot exist, so we can stop going around this block ;-)

Please do. It will be interesting to compare that to all the
different proposals for doing increased option space.

(Increasing available option space for SACK use can increase
TCP's performance in unusual environments - paths with
high bw*delay products.

But since performance can be increased further in those
environments by simply using something else other than TCP,
that's arguably a moot point.)

L.

http://sat-net.com/L.Wood

---

From: Mark Allman <[email-masked]>
Date: Wed, 13 Apr 2011 08:38:53 -0400

> It's a bit of revisiting tcpx2, and just assumes that the endpoint
> tries to start two connections at the same time. That's been
> suggested before, but has problems:
> 
> AFAICT, all variants here end up costing either the legacy or the
> new variant excessive delays, and that's not going to fly.

I don't think this is entirely true if one keeps a little history.
Appended is a table and some text I sent several weeks ago in the
context of the fast open proposal.  It shows that the lions share of
connections over the course of a day were between IPs that had a
previous connection.

Further, if you fire two SYNs and act on the first SYN+ACK you add no
delay.  

  o If that SYN+ACK is for the new thing, great.  We win and we move
    along.

  o If that SYN+ACK is old-style then we perhaps progress without some
    new fangled TCP item.  But, we can still observe a new style SYN+ACK
    if it comes along and record that it did show up for future usage.

That doesn't add delay and the cost is something you have to be willing
to do anyway: operate without the new wrinkle.

Of course, one of the key aspects the above discussion and the data
below doesn't touch on is multiple points of attachment.  I.e., just
because I know that sitting here in my office I can get some TCP-new
through to host X doesn't mean I will be able to do so when I go to
campus.  So, certainly there will need to be some re-learning and
depending on mobility patterns that may or may not be a pain.  (Note:
the information that I can get TCP-new to host X from my office is not
for nothing on campus.  That at least says host X is cool with it.)
Further, paths change, middleboxes change, etc. and so you'd need some
robust fallback mechanism (e.g., because even though you "know"
something from previous history it doesn't work anymore) and that may
well add delay.  My point is that with a bit of work we can likely make
added delay (a) the exception and (b) not all that big.

allman

Well, I grabbed a connection log from yesterday for a monitoring point
that watches a fiber-to-the-home setup that serves 100 houses in
Cleveland, OH to test this out.  (Nothing magic here ... it just
happened to be handy data.)  For each outgoing TCP connection I looked
at whether there was a previous connection from the current source host
to the current destination host within the last [threshold] seconds.
The results for various thresholds:

            30sec      60sec      120sec      180sec      240sec      Inf
-----------------------------------------------------------------------------
Total       15.3M      15.3M      15.3M       15.3M       15.3M       15.3M
Prev         5.1M       6.1M       7.2M        7.8M        8.2M       15.1M
NoPrev      10.1M       9.2M       8.1M        7.4M        7.1M        145K

(modulo rounding errors; you get the point ...)

Where "Prev" means there was a previous connection in the time interval
and "NoPrev" means there was no previous connection.  The "Inf" column
removes the time threshold and just tests whether there was a previous
connection or not.  The analysis is not perfect.  E.g., the same source
could be used for a few machines behind a NAT in the given house.  But,
it gives a general idea.

---

From: Joe Touch <[email-masked]>
Date: Wed, 13 Apr 2011 08:21:16 -0700

Hi, Mark,

> connections over the course of a day were between IPs that had a

Although that may be true, increased address sharing means that IP 
addresses aren't the whole story. History cannot be used to affect 
future connections without verification, and that verification needs to 
happen during the SYN, so you can't *use* extended options in the SYN 
regardless.

>      if it comes along and record that it did show up for future usage.

The problem is that TCP is designed to respond to options it doesn't 
know by ignoring them.

If the SYN that arrives first is the one with the long options, you need 
to ensure that the long options aren't misinterpreted. That's always 
been the challenge.

The current proposal gets around this by using phantom data (e.g., "2") 
to change the checksum. However, there are devices that repack things 
that might just recalculate the checksum rather than checking it first.

Further, let's say we send SYN with incompatible long options with 
phantom data (+2 checksum), and there's an error that makes the 
resulting checksum OK. That SYN could then be accepted and acted on by 
an endpoint that doesn't support the option.

I'm concerned about weakening TCP's robustness this way.

Joe

---

From: Andrew Yourtchenko <[email-masked]>
Date: Wed, 13 Apr 2011 18:17:20 +0200

> to do anyway: operate without the new wrinkle.

Precisely. However, I thought we need to also account potential path
asymmetry. This makes for the competing threeway handshakes (and needs
further thinking, that's why I excluded it in my writeup). Maybe I'm
overthinking it though.

> added delay (a) the exception and (b) not all that big.

I think memoizing the behavior should happen together with some kind
of ephemeral "host identification" - due to NATs, as Joe mentions.  In
that light fast open + extending the TCP option space (by whatever
method) would make a pretty happy couple. Long options by themselves
are just plumbing. But if they have the fast open use case glued to
them, that's a killer app to push it forward.

cheers,
andrew