Subject: Re: [tcpm] I-D Action: draft-borman-tcpm-tcp4way-00.txt

From: Bob Briscoe <[email-masked]>
Date: Mon, 3 Nov 2014 23:21:55 +0000

David,

Thx for updating the draft & filename, and 
particularly for the adding the roster of 
existing TCP options. However, to be honest, I 
think tcp4way has got caught up in following a 
false trail. Let me explain, and I'll add other critique afterwards too.

1) Following a false trail?

Tcp4way aims to give the client a second chance 
to negotiate some more TCP options on a 
subsequent SYN/ACK. The original reason you 
wanted this (from emails back in the summer) was 
because it would be really complex to allow a 
second segment from the client to negotiate more 
options if the hosts had already made an interim 
decision after the first round. They might need 
to revoke some of their earlier decisions, which 
would be extremely complex and often impossible.

However, all that tcp4way has achieved is to send 
more segments before making a decision. That 
means more segments on which TCP options cannot be /used/ yet.

I've calculated how many segments in both 
directions before a 4WHS server enters the established state:
* Without TFO: 3 segments, including 2 that could carry data
* A resumed connection with TFO & IW=3: 7 segments, all potentially with data.
* A resumed connection with TFO & IW=10: 21 
segments, all potentially with data.

So all tcp4way achieves is to prevent TCP options 
being included on more segments.

I believe it was wrong to conclude (back in the 
summer) that the problem is all about holding 
back a move to the established state. Most 
existing TCP options apply to segments of any 
kind, irrespective of whether the host is in the 
established state when it sends them, and 
irrespective of whether they even carry data.

In all the following examples, if the client 
wants to support option X and the server can 
support X, the server would ideally like to start 
using X on the first SYN/ACK, whether or not the 
client is offering to delay moving into the established state for a round:
* TCP-AO
* The following sub-options of the tcpcrypt CRYPT 
option: DECLINE, PKCONF*, NEXTK1*, NEXTK2*, 
INIT1, INIT2, UNKNOWN, SYNCOOOKIE, ACKCOOKIE, REKEY, REKEYSTREAM.
* the tcpcrypt MAC option
* MPTCP_JOIN, ADD_ADDR(2), DSS, MP_PRIO, MP_FAIL
* SACK permitted
* MSS
* TFO
* TS (if used for RTT calculation)
* UTO

The only ones that are meaningless on the first SYN/ACK are:
* SACK (because the SACK block for zero previous 
ACKs would be pretty uninteresting. However SACK 
would be useful on ACKs from the client while still in SYN-RCVD)
* TS (if used for PAWS, 'cos there's no risk of wrap on the first few segments)
* WS (The client can't use a large RWND until it 
has opened up cwnd thru a few rounds of 
slow-start. However, this is arguable, because 
there might be occasions where a connection can safely inherit a large cwnd).

Some option are only meaningful once data has 
started (e.g. UTO, MPTCP-DSS, WS), but even those 
can be used prior to connection establishment.

We can't know which of these two lists 
yet-to-be-defined new options will be in. If the 
past is a good predictor of the future, the first 
list is more likely (ie. the ones tcp4way doesn't help much).

2) Latency

Tcp4way always introduces latency. How much depends on which scenario:
* TCP client sends request first (by far the majority of connections): 1 RTT
* TCP server sends data first (e.g. resuming a push connection): 0.5 RTT

As JohnL said, "any added latency is nearly 
fatal". I strongly agree with every word, except "nearly".

3) Firewalls & Intrusion Detection Systems

Outgoing SYN/ACK:
To a firewall protecting a private network, an 
outgoing SYN/ACK is as abnormal as an incoming 
SYN. I guess most firewalls will not have a rule 
blocking an outgoing SYN/ACK. But given it's 
abnormal, I'm sure some IDSs will block it.

4W Reserved flag:
I suspect some firewalls / IDSs might drop 
packets with a Reserved flag set. I suspect we'll 
see a fair amount of 'correction' of Reserved flags back to zero too.

I appreciate that the 4W flag isn't crucial to your scheme.
But the second SYN/ACK is fundamental.

Even if you don't use a Reserved flag, you've 
still got to find something 'abnormal' to signal support for the 4WHS.

I don't have data for Reserved flag discard or 
normalisation. If tcp4way used a TCP option, 
[Honda] reported %stripped options to the following port numbers:
  port 80: 14%
  port 443: 6%
  port 34343: 4%.
I suspect a lot of the option stripping is due to 
connection splitters, which won't set options 
that they don't understand for the ongoing 
connection. I'm sure they won't set unrecognised 
Reserved flags either. So I suspect traversal 
will be just as bad for Reserved flags as for options.

It is particularly important for TCP options that 
provide security (e.g. TCP-AO, tcpcrypt) to aim 
for 100% middlebox traversal. If attempts to use 
security fail so often they become normal, it is 
too easy for an attacker to target any connection 
it wants to snoop or alter by stripping options 
or flags, which just looks like yet another middlebox problem.

I.e. when middlebox interference looks just like 
a downgrade attack, it's really easy to make a 
downgrade attack look like middlebox interference.

__________________
They are my 3 main criticisms. The rest are nits...

4) Nit: Reflection of Reserved Flags

Simple reflection of Reserved flags cannot be 
used to confirm that the server agrees with the 
client. There are TCP implementations that just 
reflect whatever Reserved flags the client sends, 
whether or not they support them.

This is fairly easy to work-round. Similar to ECN 
negotiation [RFC3168, Sec 6.1.1.2] you just 
define the confirmation as a different pattern of 
flags (but this burns more flags).

Richard Scheffenegger was testing the Alexa top 
1M Web servers in Apr with flags on SYN as follows:
         > CWR ECE SYN
The correct response of ECN-capable servers was:
         <     ECE SYN ACK
I know he found a fair number (sorry I don't have 
the %) that just dumbly reflected:
         < CWR ECE SYN ACK
And when he set the nonce flag on a SYN sent to 
these same broken Web servers, all of them just 
dumbly reflected back the new pattern:
         > NS CWR ECE SYN
         < NS CWR ECE SYN ACK

5) Nit: Remembering EDO support across connections

S.7.3.2.1 says that server EDO support can be 
saved alongside a TFO cookie. I don't believe 
this is safe. During an upgrade, there might be 
some server replicas that support EDO and others 
that don't. The legacy servers would then ignore 
the EDO option, but still pass TCP options as a corrupt payload to the app.

[BTW, it is safe for a client to hold solely the 
TFO capability of a server in a cookie, because 
(unlike EDO) TFO doesn't put data that is not 
intended for the app in the payload. So, if a 
replica server that picks up the resumed 
connection does not support TFO, it will simply 
ignore the cookie and just hold the data back until the 3WHS completes.]

6) Nit: Unknown Option

Altho it would be nice to think that this might 
help, there doesn't seem to be a case where it 
will. Can you give an example where the Unknown Options is useful?

7) Nit: T/TCP is obsolete.

T/TCP had vulnerabilities and was obsoleted by RFC6247.

Bob

At 16:21 24/10/2014, David Borman wrote:
>I’ve just posted an update to the TCP Four-Way handshake document.
>
>1) I changed the name of the document to include 
>tcpm: draft-dborman-tcpm-tcp4way-00.txt
>
>2) I’ve added text that any implementation 
>that indicates support for the Four-Way 
>handshake is also indicating that it properly 
>handles unknown TCP options, as per RFC 
>1122.  We keep dancing around that issue, and 
>this gives us a way to stop worrying about it anymore, and
>a way for an implementation to know for sure 
>that it isn’t talking to a legacy TCP implementation.
>
>3) I’ve added a preliminary section to discuss specific TCP options
>
>4) I’ve added text about the TS option to 
>clarify RFC 7323, that it is the presence of 
>TSopt in both a sent and received packet with 
>the SYN bit that enables the option, i.e. it is 
>not limited to the specific instance of the 
><SYN> and <SYN,ACK> packets of the three-way handshake.
>
>
>And something entirely new:
>
>5) I’ve added a new TCP option, the Unknown 
>Option option.  Currently, the primary way to 
>determine that the other side of the connection 
>doesn’t support a specific option is by lack 
>of response to that option, because it silently 
>ignores it.  The Unknown Option option provides 
>a mechanism where a TCP can inform the other 
>side that it does not understand or support any 
>given option for a connection.  This can be very 
>useful for sending TCP options in non-SYN 
>packets, especially if the option is not 
>explicitly enabled through the SYN exchange, or 
>it is a one-way option.  UTO comes to mind.
>
>The document needs more work, but I wanted to 
>get an update out today, before Mondays submission cutoff.
>
>                         -David Borman
>
> > On Oct 24, 2014, at 11:01 AM, [email-masked] wrote:
> >
> >
> > A New Internet-Draft is available from the 
> on-line Internet-Drafts directories.
> >
> >
> >        Title           : TCP Four-Way Handshake
> >        Author          : David Borman
> >       Filename        : draft-borman-tcpm-tcp4way-00.txt
> >       Pages           : 21
> >       Date            : 2014-10-24
> >
> > Abstract:
> >   One of the limitations of TCP is that it has limited space for TCP
> >   options, only 40 bytes.  Many mechanisms have been proposed for for
> >   extending the TCP option space, but the biggest challenge has been to
> >   get additional option space in the initial SYN packet.
> >
> >   This memo presents a optional four-way TCP handshake to allow
> >   extended option space to be used in SYN packets in both directions.
> >
> >
> > The IETF datatracker status page for this draft is:
> > https://datatracker.ietf.org/doc/draft-borman-tcpm-tcp4way/
> >
> > There's also a htmlized version available at:
> > http://tools.ietf.org/html/draft-borman-tcpm-tcp4way-00
> >
> >
> > Please note that it may take a couple of 
> minutes from the time of submission
> > until the htmlized version and diff are available at tools.ietf.org.
> >
> > Internet-Drafts are also available by anonymous FTP at:
> > ftp://ftp.ietf.org/internet-drafts/
> >
> > _______________________________________________
> > I-D-Announce mailing list
> > [email-masked]
> > https://www.ietf.org/mailman/listinfo/i-d-announce
> > Internet-Draft directories: http://www.ietf.org/shadow.html
> > or ftp://ftp.ietf.org/ietf/1shadow-sites.txt
>
>_______________________________________________
>tcpm mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/tcpm

________________________________________________________________
Bob Briscoe,                                                  BT

---

From: David Borman <[email-masked]>
Date: Tue, 4 Nov 2014 15:01:37 -0600

> On Nov 3, 2014, at 5:21 PM, Bob Briscoe <[email-masked]> wrote:
> 
> David, 
> 
> Thx for updating the draft & filename, and particularly for the adding the roster of existing TCP options. However, to be honest, I think tcp4way has got caught up in following a false trail. Let me explain, and I'll add other critique afterwards too.
> 
> Tcp4way aims to give the client a second chance to negotiate some more TCP options on a subsequent SYN/ACK. The original reason you wanted this (from emails back in the summer) was because it would be really complex to allow a second segment from the client to negotiate more options if the hosts had already made an interim decision after the first round. They might need to revoke some of their earlier decisions, which would be extremely complex and often impossible.

No.  My original point is that negotiating TCP options in packets without the SYN bit requires additional complexity because ACK-only packets are not reliably transmitted.  The 4WHS adds a client SYN/ACK packet that is reliably transmitted, and is distinct from the original client SYN packet.

For a non-directional option, it doesn’t matter which side sends it first, hence the server SYN/ACK can propose using additional non-directional options, which can then be confirmed in the client SYN/ACK of the 4WHS.

Because both sides know that the other side supports extended option space, both of the SYN/ACK packets can take advantage of it.

> However, all that tcp4way has achieved is to send more segments before making a decision. That means more segments on which TCP options cannot be /used/ yet.

Again, no.  Options in the initial SYN and SYN/ACK are not delayed.  Nothing from the client side is delayed, because it decides the full set of what is being used when it gets the servers SYN/ACK, because it knows which options from the SYN were accepted by the server, and which options proposed by the server it is willing to acknowledge when it sends the SYN/ACK.  It’s the same point as with the 3WHS.

The server won’t know if the client supports the additional options that the server proposed in the SYN/ACK until it gets the client’s SYN/ACK, so from the server standpoint, there’s one more RTT for those additional options.

> 
> I've calculated how many segments in both directions before a 4WHS server enters the established state:
> * A resumed connection with TFO & IW=10: 21 segments, all potentially with data.
> 
> So all tcp4way achieves is to prevent TCP options being included on more segments.

I don’t follow this logic at all.  With the 4WHS the server enters established state at the exact same time as it does with the 3WHS, when its SYN is acknowledged.  Whether that is an ACK-only packet with a 3WHS, or a SYN/ACK with a 4WHS, from the server side it still happens at the same point in the exchange.  From the servers perspective, the only difference is that for the 4WHS it sends a final ACK and does final option enablement, and for the 3WHS it doesn’t. 

> 
> I believe it was wrong to conclude (back in the summer) that the problem is all about holding back a move to the established state. Most existing TCP options apply to segments of any kind, irrespective of whether the host is in the established state when it sends them, and irrespective of whether they even carry data.

I never said the problem was about holding back established state.

I said that if the servers SYN/ACK contained additional options, with the 4WHS the client can respond with a SYN/ACK to complete negotiation of those additional options, whereas the 3WHS doesn’t have that possibility.  And the only reason those options wouldn’t have been in the SYN is because there wasn’t enough space, but with EDO there is now space in the SYN/ACK.

> 
> In all the following examples, if the client wants to support option X and the server can support X, the server would ideally like to start using X on the first SYN/ACK, whether or not the client is offering to delay moving into the established state for a round:

If X was in the SYN, it can.  If X wasn’t in the SYN, it can’t, because it doesn’t know that the client supports X, it’ll have to wait for the SYN/ACK to know if the client supports X.

> * TCP-AO
TCP-AO is not negotiated.

> * The following sub-options of the tcpcrypt CRYPT option: DECLINE, PKCONF*, NEXTK1*, NEXTK2*, INIT1, INIT2, UNKNOWN, SYNCOOOKIE, ACKCOOKIE, REKEY, REKEYSTREAM.
> * SACK (because the SACK block for zero previous ACKs would be pretty uninteresting. However SACK would be useful on ACKs from the client while still in SYN-RCVD) 
> * WS (The client can't use a large RWND until it has opened up cwnd thru a few rounds of slow-start. However, this is arguable, because there might be occasions where a connection can safely inherit a large cwnd).
> 
> Some option are only meaningful once data has started (e.g. UTO, MPTCP-DSS, WS), but even those can be used prior to connection establishment.

> 
> We can't know which of these two lists yet-to-be-defined new options will be in. If the past is a good predictor of the future, the first list is more likely (ie. the ones tcp4way doesn't help much).

I’m not quite sure what your point is with just an enumerated list.  For each of them, the question is, if the option is not included in the initial client SYN, can it make sense in the servers SYN/ACK?  For example, if a server doesn’t receive TFO in the SYN, it could still send TFO in the SYN/ACK, which would just be ignored by non-TFO implementations.  Directional options, where the servers option is dependent on first knowing the contents of the clients option, are not candidates.

The 4WHS does not give the client a second chance to initiate option negotiation, that would require a 5WHS.

> As JohnL said, "any added latency is nearly fatal". I strongly agree with every word, except "nearly”.

If you are talking about latency of delivering user data, there isn’t any.  See my message entitled "We already support multiple versions of the TCP header”, and look for the section entitled “Delivering User Data”.  Those changes will be in the next revision of the 4WHS draft.

> 3) Firewalls & Intrusion Detection Systems
...

I don’t subscribe to the notion that today’s deployed middle boxes get to dictate all future development of the TCP protocol.  Yes, ignoring them would be silly, but kowtowing to them is equally silly.

We need to first decide on the destination, and then find a path to that destination.  If we just look at the paths through the middle boxes, we’ll still get somewhere, but will it be where we want to go?

...

Did you miss sections 4.3.2 and 4.3.3 where I discussed why this is not an issue?

> S.7.3.2.1 says that server EDO support can be saved alongside a TFO cookie. I don't believe this is safe. During an upgrade, there might be some server replicas that support EDO and others that don't. The legacy servers would then ignore the EDO option, but still pass TCP options as a corrupt payload to the app.
> 
> [BTW, it is safe for a client to hold solely the TFO capability of a server in a cookie, because (unlike EDO) TFO doesn't put data that is not intended for the app in the payload. So, if a replica server that picks up the resumed connection does not support TFO, it will simply ignore the cookie and just hold the data back until the 3WHS completes.]

Good points, I’ll take that out.

> 
> 6) Nit: Unknown Option
> 
> Altho it would be nice to think that this might help, there doesn't seem to be a case where it will. Can you give an example where the Unknown Options is useful?

I did, UTO.  It would also be useful for late enablement of TS.  Basically, it is useful for using any option which does not require that use of that option be negotiated in the initial SYN exchange.

The main purpose of the Unknown Option option is to provide a generic way to inform the other side that you don’t understand a particular option, when that option first appears mid-stream, rather than support for that option being negotiated in the initial SYN exchange.  Otherwise, if you send an option for the first time mid-stream, the only way to know that the other side doesn’t support it is by lack of any response to that option.  That requires tracking what sequence number you first sent it with, and waiting for the ACK for that sequence number to decide on a non-response.  But sequence number tracking doesn’t work if you don’t have any new data to send, so you need another mechanism.  If you know the other side support the Unknown Option option, then you know it’ll tell you if you send any other options that it doesn’t understand. 

> T/TCP had vulnerabilities and was obsoleted by RFC6247.

Yes, that’s why the document says:

      Though now obsolete [RFC6247], this is a good
      example of a directional TCP option.

> Bob

		-David

---

From: David Borman <[email-masked]>
Date: Tue, 4 Nov 2014 19:15:41 -0600

> On Nov 4, 2014, at 3:01 PM, David Borman <[email-masked]> wrote:
...
> We need to first decide on the destination, and then find a path to that destination.  If we just look at the paths through the middle boxes, we’ll still get somewhere, but will it be where we want…
...
I re-read that, and I could be clearer.

We need to first decide what we want TCP to look like, and then find a way to transition from current implementations to the new design, which might include middle boxes having to change, rather than letting current middle box implementations dictate how we can or can’t change TCP.

	-David

---

From: Joe Touch <[email-masked]>
Date: Tue, 4 Nov 2014 17:27:18 -0800

+1

> On Nov 4, 2014, at 5:15 PM, David Borman <[email-masked]> wrote:
> 
>

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 7 Nov 2014 21:41:05 +0000

David,

At 21:01 04/11/2014, David Borman wrote:

> > Thx for updating the draft & filename, and particularly for the 
> adding the roster of existing TCP options. However, to be honest, I 
> think tcp4way has got caught up in following a false trail. Let me 
> explain, and I'll add other critique afterwards too.
> >
> > Tcp4way aims to give the client a second chance to negotiate some 
> more TCP options on a subsequent SYN/ACK. The original reason you 
> wanted this (from emails back in the summer) was because it would 
> be really complex to allow a second segment from the client to 
> negotiate more options if the hosts had already made an interim 
> decision after the first round. They might need to revoke some of 
> their earlier decisions, which would be extremely complex and often impossible.
>
>No.  My original point is that negotiating TCP options in packets 
>without the SYN bit requires additional complexity because ACK-only 
>packets are not reliably transmitted.  The 4WHS adds a client 
>SYN/ACK packet that is reliably transmitted, and is distinct from 
>the original client SYN packet.

OK, understood (altho RFC793 does not require the ACK of the 3rd leg 
to be pure. It can ACK the SYN/ACK and send data - I can dig out some 
recent literature searches I did on why there is usually a 3rd ACK, 
if anyone's interested). But, granted, in the general case your point 
is valid that there might not be any data at this point.

>For a non-directional option, it doesn't matter which side sends it 
>first, hence the server SYN/ACK can propose using additional 
>non-directional options, which can then be confirmed in the client 
>SYN/ACK of the 4WHS.
>
>Because both sides know that the other side supports extended option 
>space, both of the SYN/ACK packets can take advantage of it.
>
> > However, all that tcp4way has achieved is to send more segments 
> before making a decision. That means more segments on which TCP 
> options cannot be /used/ yet.
>
>Again, no.  Options in the initial SYN and SYN/ACK are not 
>delayed.  Nothing from the client side is delayed, because it 
>decides the full set of what is being used when it gets the servers 
>SYN/ACK, because it knows which options from the SYN were accepted 
>by the server, and which options proposed by the server it is 
>willing to acknowledge when it sends the SYN/ACK.  It's the same 
>point as with the 3WHS.
>
>The server won't know if the client supports the additional options 
>that the server proposed in the SYN/ACK until it gets the client's 
>SYN/ACK, so from the server standpoint, there's one more RTT for 
>those additional options.

Only those options that fit on the first SYN can start being used 
early. The rest still have to wait. Given the point of this exercise 
is to make more space for options, you are saying that you can only 
provide more space for those options that are prepared to wait.

> >
> > I've calculated how many segments in both directions before a 
> 4WHS server enters the established state:
>I don't follow this logic at all.

That's possibly my point. I don't think your logic is relevant to the 
problem at hand.

>With the 4WHS the server enters established state at the exact same 
>time as it does with the 3WHS, when its SYN is 
>acknowledged.  Whether that is an ACK-only packet with a 3WHS, or a 
>SYN/ACK with a 4WHS, from the server side it still happens at the 
>same point in the exchange.  From the servers perspective, the only 
>difference is that for the 4WHS it sends a final ACK and does final 
>option enablement, and for the 3WHS it doesn't.

My point is that most TCP options are useful on /segments/. The 
problem at hand has nothing to do with the state that the TCP 
state-machine has reached. It has to do with:
a) minimising the round trip times before all TCP options can start to be used;
b) minimising how many segments might be exchanged before all TCP 
options can start to be used.

> >
> > I believe it was wrong to conclude (back in the summer) that the 
> problem is all about holding back a move to the established state. 
> Most existing TCP options apply to segments of any kind, 
> irrespective of whether the host is in the established state when 
> it sends them, and irrespective of whether they even carry data.
>
>I said that if the servers SYN/ACK contained additional options, 
>with the 4WHS the client can respond with a SYN/ACK to complete 
>negotiation of those additional options, whereas the 3WHS doesn't 
>have that possibility.  And the only reason those options wouldn't 
>have been in the SYN is because there wasn't enough space, but with 
>EDO there is now space in the SYN/ACK.

OK I accept that your goal was not about holding back established state.

But I'm not impressed. You just allowed up to 21 segments to go past 
without being able to use some TCP options that you are still in the 
process of negotiating.

> >
> > In all the following examples, if the client wants to support 
> option X and the server can support X, the server would ideally 
> like to start using X on the first SYN/ACK, whether or not the 
>If X was in the SYN, it can.  If X wasn't in the SYN, it can't, 
>because it doesn't know that the client supports X, it'll have to 
>wait for the SYN/ACK to know if the client supports X.

Yes. And that's why I'm not particularly impressed.

> > * The following sub-options of the tcpcrypt CRYPT option: 
> DECLINE, PKCONF*, NEXTK1*, NEXTK2*, INIT1, INIT2, UNKNOWN, 
> SYNCOOOKIE, ACKCOOKIE, REKEY, REKEYSTREAM.
> > * SACK (because the SACK block for zero previous ACKs would be 
> pretty uninteresting. However SACK would be useful on ACKs from the 
> client while still in SYN-RCVD)
> > * WS (The client can't use a large RWND until it has opened up 
> cwnd thru a few rounds of slow-start. However, this is arguable, 
> because there might be occasions where a connection can safely 
> inherit a large cwnd).
> >
> > Some option are only meaningful once data has started (e.g. UTO, 
> MPTCP-DSS, WS), but even those can be used prior to connection establishment.
>
> >
> > We can't know which of these two lists yet-to-be-defined new 
> options will be in. If the past is a good predictor of the future, 
> the first list is more likely (ie. the ones tcp4way doesn't help much).
>
>I'm not quite sure what your point is with just an enumerated list.

(Other than mistakenly including TCP-AO) I've divided up the options 
into those that make sense as early as the first SYN/ACK and those 
that could probably wait. So,...

* A 4WHS client has to consider all the options it thinks might be 
needed and decide which can wait. I'm not convinced that it makes 
sense to wait for any of the options in the first list. For some of 
them, it certainly gets messy to start without them, then to add them.
* All the particularly large options are in the first list.
* The very large majority of connections only consists of a few 
segments. 4WHS provides extra space after most connections will have 
finished. So, in the large majority of connections, 4WHS doesn't 
solve any problem at all.

>For each of them, the question is, if the option is not included in 
>the initial client SYN, can it make sense in the servers 
>SYN/ACK?  For example, if a server doesn't receive TFO in the SYN, 
>it could still send TFO in the SYN/ACK, which would just be ignored 
>by non-TFO implementations.  Directional options, where the servers 
>option is dependent on first knowing the contents of the clients 
>option, are not candidates.

The example you have chosen illustrates my concern in a number of ways.

TFO isn't a problem on the first connection in which it is used. It 
consumes 2B on the SYN, and 6-18B on the SYN/ACK.

TFO hits the space problem when it is used on a subsequent 
connection. The client sends the cookie it got from the previous 
connection /in the SYN/. There is no TFO option in the SYN/ACK on the 
resumed connection. The whole point of TFO is for the client to prove 
to the server that it had a previous valid connection.

Secondly, there is no point deferring TFO to the second SYN/ACK, 
because the 'F' in TFO means 'Fast'.

Similarly for tcpcrypt. There's no point having sent up to 21 
segments of data in the clear before any keys have been agreed. OK, 
you might argue that if you're using tcpcrypt you send no data until 
the keys have been agreed. However, the point of tcpcrypt is meant to 
be opportunistic encryption. I.e. it can be turned on for /all/ 
connections with zero config and minimal performance hit. If turning 
on tcpcrypt causes every connection to have to wait an extra round 
before it can send data, it will not be ubiquitous. It will be nobiquitous.

> > As JohnL said, "any added latency is nearly fatal". I strongly 
> agree with every word, except "nearly".
>
>If you are talking about latency of delivering user data, there 
>isn't any.  See my message entitled "We already support multiple 
>versions of the TCP header", and look for the section entitled 
>"Delivering User Data".  Those changes will be in the next revision 
>of the 4WHS draft.

There is latency added to any user-data (or empty segments) that want 
the benefit of the TCP options we're trying to make space for.

In 4WHS, it's the extra space that suffers the latency. It doesn't 
appear soon enough to be useful.

>I don't subscribe to the notion that today's deployed middle boxes 
>get to dictate all future development of the TCP protocol.  Yes, 
>ignoring them would be silly, but kowtowing to them is equally silly.

I'll substitute your corrected text for your next para...

>We need to first decide what we want TCP to look like, and then find 
>a way to transition from current implementations to the new design, 
>which might include middle boxes having to change, rather than 
>letting current middle box implementations dictate how we can or 
>can't change TCP.

With inner space, I also started from where we should want TCP to end 
up, and found a way to transition to it. Here's my starting principles:
1) Options should be in the TCP header. Extensions should be within 
the TCP Data.
    [I just submitted a paper to the IAB SEMI workshop that gives the 
rationale for that:
    <http://bobbriscoe.net/projects/2020comms/tcp/briscoe-inspace_semi.pdf>.
    ]
2) Minimal delay before options can be used
3) Choice between fire-and-forget options and reliable-in-order options
4) Host control over which options and base header fields are 
available for interaction with middleboxes, and which are strictly e2e.
5) Enforcement of the above choice through authentication and/or encryption.
6) Incremental deployment (against legacy servers and legacy middleboxes)

You say "transition ... might include middleboxes having to change". 
That earns you a patronising smile. Nonetheless, starting from 
principles is a sentiment I can wholeheartedly agree with.

>Did you miss sections 4.3.2 and 4.3.3 where I discussed why this is 

I read these sections and they have misunderstood the problem.

There is an easy way to solve the problem tho. (which I suggested in 
the text that you've removed from the thread).

> > S.7.3.2.1 says that server EDO support can be saved alongside a 
> TFO cookie. I don't believe this is safe. During an upgrade, there 
> might be some server replicas that support EDO and others that 
> don't. The legacy servers would then ignore the EDO option, but 
> still pass TCP options as a corrupt payload to the app.
> >
> > [BTW, it is safe for a client to hold solely the TFO capability 
> of a server in a cookie, because (unlike EDO) TFO doesn't put data 
> that is not intended for the app in the payload. So, if a replica 
> server that picks up the resumed connection does not support TFO, 
> it will simply ignore the cookie and just hold the data back until 
> the 3WHS completes.]
>
>Good points, I'll take that out.
>
> >
> > 6) Nit: Unknown Option
> >
> > Altho it would be nice to think that this might help, there 
> doesn't seem to be a case where it will. Can you give an example 
> where the Unknown Options is useful?
>
>I did, UTO.  It would also be useful for late enablement of 
>TS.  Basically, it is useful for using any option which does not 
>require that use of that option be negotiated in the initial SYN exchange.
>
>The main purpose of the Unknown Option option is to provide a 
>generic way to inform the other side that you don't understand a 
>particular option, when that option first appears mid-stream, rather 
>than support for that option being negotiated in the initial SYN 
>exchange.  Otherwise, if you send an option for the first time 
>mid-stream, the only way to know that the other side doesn't support 
>it is by lack of any response to that option.  That requires 
>tracking what sequence number you first sent it with, and waiting 
>for the ACK for that sequence number to decide on a 
>non-response.  But sequence number tracking doesn't work if you 
>don't have any new data to send, so you need another mechanism.  If 
>you know the other side support the Unknown Option option, then you 
>know it'll tell you if you send any other options that it doesn't understand.

You might want to say all that in the draft. It's impossible to infer 
all this without you having said it.

BTW, this is why one of my starting principles was:
   "Choice between fire-and-forget options and reliable-in-order options"

Inner-space-01 gives you the latter. Inner-space-02 (sitting on my 
machine waiting to send) gives you both.

The Unknown Option option only solves the negative part of this 
problem, i.e. inferring /lack/ of support. Inner space solves the 
positive side as well, i.e. inferring both hosts support option X, 
and co-ordinating the point in each half-connection when X starts to 
become enabled.

This makes it much easier to design and implement options that 
require co-ordination between both ends for when a change occurs 
(e.g. a re-key, or switching something on or off), precisely in the 
case you point out, when there is no new data.

This relates to a point I was making to Yoshifumi: simplicity isn't 
just in the scheme itself, it also concerns improving how simple it 
is to design and implement new TCP options.

>Yes, that's why the document says:
>

OK. My mistake.

Cheers

Bob

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Mon, 10 Nov 2014 16:38:30 +0000

Hi,

Just to share some (one of the many test run results) of the actual data collected for port 80:

I tested by sending a <SYN,ECE,CWR,NS> packet; the expected response for non-ECN would be a simple <SYN,ACK>, and for ECN-enabled hosts, a <SYN,ACK,ECE>.

Live:  15210 (89.0255)      [targets which actually established a connection, regular + ECN]

not-ECN: 8465 {55.6542}

SA:        8214 {54.0039} [a regular SYN,ACK, RFC793]

Bad_SAEC:  21 {0.138067}  [the Nonce bit was not reflected, the other ECN bits were]

Bad_SAECN: 210 {1.38067}  [all unknown flags are reflected ]

Bad_SAC:   0 {0}          [never observed a host with this combination ]

Bad_SACN:  0 {0}          [never observed a host with this combination ]

Bad_SAEN:  0 {0}          [never observed a host with this combination (RFC 3540 Nonce) ]

Bad_SAN:   20 {0.131492}  [ECN flags cleared, but NS bit reflected]

So, there are middleboxes that "understand" the ECN flags ECE and CWR (and properly clear them), but still simply reflect the unknown NS flag (SAN).
Other middleboxes (or paths) clear the "unknown" NS flag, but reflect both ECE and CWR (SAEC); the majority of broken implementations simply reflect all unknown flags (SAECN); Finally, SAEN is not strictly a forbidden combination under ECN Nonce, but we have never seen ECN Nonce support in the wild anyway.

Another note to make: On port 80, more broken behavior was seen than on port 21, probably due to a the higher number of paths featuring some kind of web-enhancing device (load balancer, proxy, ...);

Overall, the use of new flags will have to deal with a population of up to 2.0% of paths/hosts which won't deal with unknown flags properly.

Richard Scheffenegger

Simple reflection of Reserved flags cannot be used to confirm that the server agrees with the client. There are TCP implementations that just reflect whatever Reserved flags the client sends, whether or not they support them.

This is fairly easy to work-round. Similar to ECN negotiation [RFC3168, Sec 6.1.1.2] you just define the confirmation as a different pattern of flags (but this burns more flags).

The correct response of ECN-capable servers was:
         <     ECE SYN ACK
I know he found a fair number (sorry I don't have the %) that just dumbly reflected:
        < CWR ECE SYN ACK
        < NS CWR ECE SYN ACK

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 10 Nov 2014 18:12:54 +0000

Richard,

Thank you - so useful to have actual data!

However, I'm having trouble understanding a couple of points about the table.

* I assume the first number in each row is the number of responses observed.

* However, I'm not sure what the second numbers 
are - the ones in () or {}. They seem to add up 
to a little over 200, so they can't be 
percentages. The ones in {} are 2.11 times the 
percentage. The one in () is 1.88 times the percentage}.

* Also, I'm not sure what the difference between 
the two rows tagged not-ECN and SA is meant to be. Surely SA /is/ not-ECN?

Bob

At 16:38 10/11/2014, Scheffenegger, Richard wrote:
>Hi,
>
>Just to share some (one of the many test run 
>results) of the actual data collected for port 80:
>
>I tested by sending a <SYN,ECE,CWR,NS> packet; 
>the expected response for non-ECN would be a 
>simple <SYN,ACK>, and for ECN-enabled hosts, a <SYN,ACK,ECE>.
>
>
>
>Live:  15210 (89.0255)      [targets which 
>actually established a connection, regular + ECN]
>not-ECN: 8465 {55.6542}
>So, there are middleboxes that “understand” the 
>ECN flags ECE and CWR (and properly clear them), 
>but still simply reflect the unknown NS flag (SAN).
>Other middleboxes (or paths) clear the “unknown” 
>NS flag, but reflect both ECE and CWR (SAEC); 
>the majority of broken implementations simply 
>reflect all unknown flags (SAECN); Finally, SAEN 
>is not strictly a forbidden combination under 
>ECN Nonce, but we have never seen ECN Nonce support in the wild anyway.
>
>Another note to make: On port 80, more broken 
>behavior was seen than on port 21, probably due 
>to a the higher number of paths featuring some 
>kind of web-enhancing device (load balancer, proxy, …);
>
>Overall, the use of new flags will have to deal 
>with a population of up to 2.0% of paths/hosts 
>which won’t deal with unknown flags properly.
>
>
>Richard Scheffenegger
>
>
>reflect whatever Reserved flags the client 
>sends, whether or not they support them.
>
>This is fairly easy to work-round. Similar to 
>ECN negotiation [RFC3168, Sec 6.1.1.2] you just 
>define the confirmation as a different pattern 
>of flags (but this burns more flags).
>

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Mon, 10 Nov 2014 18:32:11 +0000

Hi  Bob,

yes, that list excludes the ECN-enabled sessions.

The ()-bracket behind "Live" is the percentage of responsive IP servers (20k IPs were sampled in this instance)
The {}-bracket are the relative percentages of all live hosts, that behaved in a particular way, ie.  54% would be non-ECN, regular SA (not mirroring unknown flags),  0.14, 1.38 and 0.13 % would not properly handle unknown TCP header flags in some way, and the remainder were ECN-capable TCP stacks. Which, by the way is not the same as stating that these sessions (44,35%) would actually constitute actually working ECN paths - only a small fraction of internet paths which have ECN-capable TCP stacks on both ends deal properly with the IP-ECN part at this time... I believe something like 2% of paths (all live hosts) would be supportive of AQM/ECN at an congestion point at this time overall - ie. a ECN mark would get through on the IP layer and dealt with properly on the TCP layer.

That was omitted here, as the ECN-sessions were broken down further during these tests, ie. if the path would properly handle IP-ECN flags, mirror or normalize IP-ECN flags, keep the "TOS" flag of the SYN for the session,  if ECT(1) and CE would elicit proper responses actually etc;

As Brian stated early on during this campaign, there are many more ways of brokenness than one would think of...

Richard Scheffenegger

From: Bob Briscoe [mailto:[email-masked]]
Sent: Montag, 10. November 2014 08:13
To: Scheffenegger, Richard
Cc: David Borman; [email-masked] ([email-masked])
Subject: RE: [tcpm] I-D Action: draft-borman-tcpm-tcp4way-00.txt

Richard,

* However, I'm not sure what the second numbers are - the ones in () or {}. They seem to add up to a little over 200, so they can't be percentages. The ones in {} are 2.11 times the percentage. The one in () is 1.88 times the percentage}.

* Also, I'm not sure what the difference between the two rows tagged not-ECN and SA is meant to be. Surely SA /is/ not-ECN?

Bob

Hi,

Richard Scheffenegger

        < NS CWR ECE SYN ACK