Subject: Re: [tcpm] New Version Notification for draft-touch-tcpm-tcp-edo-01.txt

From: Pasi Sarolahti <[email-masked]>
Date: Fri, 2 May 2014 14:18:29 +0300

On 26 Apr 2014, at 01:19, Joe Touch <[email-masked]> wrote:

> I'd like to ask the chairs if we can consider a call to make this a WG doc?
> 
> (given the work of the group is supposed to happen on the list - and I don't attend meetings in person these days -this doesn't seem like it needs to wait for a meeting to proceed)

We should discuss the possible WG adoption on the list before the meeting, and some people have already voiced their support. But there is value in discussing this in the f2f meeting, too, before confirming WG adoption (hum). Perhaps Wes can present this, or as the last resort the presentation could be proxied by chairs. I don't see any reason to hurry this so that we couldn't wait until the Toronto meeting.

- Pasi

---

From: Wesley Eddy <[email-masked]>
Date: Fri, 02 May 2014 10:45:24 -0400

On 5/2/2014 7:18 AM, Pasi Sarolahti wrote:
>> (given the work of the group is supposed to happen on the list -
>> and I don't attend meetings in person these days -this doesn't seem
>> like it needs to wait for a meeting to proceed)
> 
> We should discuss the possible WG adoption on the list before the
> meeting, and some people have already voiced their support. But there
> is value in discussing this in the f2f meeting, too, before
> confirming WG adoption (hum). Perhaps Wes can present this, or as the
> last resort the presentation could be proxied by chairs. I don't see
> any reason to hurry this so that we couldn't wait until the Toronto
> meeting.
> 

That would be fine with me.  IMO, it would be nice to know what
the perceived mailing list consensus is, and use the meeting to
confirm that.  It's only a couple months away.

I think Joe would agree that once adopted, this should be able to
proceed relatively rapidly.

-- 
Wes Eddy
MTI Systems

---

From: Joe Touch <[email-masked]>
Date: Fri, 02 May 2014 08:02:47 -0700

> confirm that.  It's only a couple months away.

I'm not in a rush, but as a point of order:

	- work happens on the mailing list, NOT at the meetings

	- decisions at the meeting are confirmed on the list,
	not the other way around

That's why I don't see why we need to wait for the meeting. Meetings are 
places where we typically hash out issues that arise on the list, e.g.:
	a. post to the list
	b. need for interactive clarification arises
	c. meeting supports interactive clarification
	d. decision happens back on the list

Somewhere along the line this all got reversed. I would like to 
encourage the chairs to use case as an opportunity to try the intended 
process.

Joe

---

From: John Leslie <[email-masked]>
Date: Fri, 2 May 2014 11:27:44 -0400

Joe Touch <[email-masked]> wrote:
> 
> Meetings are places where we typically hash out issues that arise
> on the list, e.g.:
> 	a. post to the list
> encourage the chairs to use case as an opportunity to try the
> intended process.

   +1

--
John Leslie <[email-masked]>

---

From: Yoshifumi Nishida <[email-masked]>
Date: Fri, 2 May 2014 13:27:14 -0700

Hi Joe,

>         d. decision happens back on the list

I personally would like to put a line between c. and d. like:
           c1. gauge consensus from both the list and the meetings
I think having two ways to gauge consensus is good thing.
--
Yoshi

---

From: Joe Touch <[email-masked]>
Date: Fri, 2 May 2014 13:33:46 -0700

> On May 2, 2014, at 1:27 PM, Yoshifumi Nishida <[email-masked]> wrote:
> 
> Hi Joe,
> 
>> That's why I don't see why we need to wait for the meeting. Meetings are places where we typically hash out issues that arise on the list, e.g.:
>>         a. post to the list

It's good but should not be required.  Meeting attendance isn't required and officially all IETF business is supposed to happen on the list. So holding up work to involve meetings should not ever happen IMO. 

Joe

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Sat, 3 May 2014 08:54:50 +0000

Hi Joe,

> It's good but should not be required.  Meeting attendance isn't required and officially all IETF business is supposed to happen on the list. So holding up work to involve meetings should not ever h…

Meeting attendance is not required. Still, I don't understand why not running an adoption call about two weeks after posting -00 really holds up work.

TCPM is not the only working group that has a high bar for adopting new work, and I think being conservative regarding std track TCP changes has always been a community consensus (e.g., our previous charter even told us that every new TCPM milestone requires IESG approval - that was certainly too conservative). Also, in TCPM review cycles are unfortunately a rather scarce resource, and the chairs have to consider this as well when adopting new work. All this has nothing to do with this draft specifically.
 
Regarding draft-touch-tcpm-tcp-edo, I am particularly interested in more feedback from implementers and users before starting the formal adoption call. We usually get very valuable feedback from implementers during meetings, and that would be one reason not to rush right now - but it is the feedback I am interested in, not the meeting. If corresponding aspects are discussed on the list before the next meeting, I'll be more than happy as well.

Thus, instead of further arguing about process aspects, I am more interested in thoughts on:

* From designers of TCP extensions requiring more option space: Is draft-touch-tcpm-tcp-edo a good solution for your needs and thus the right way to move forward?

* From implementers: Are there deployment issues beyond what is described in the draft already? And, would this mechanism be implemented in the Internet? 

Michael

---

From: John Leslie <[email-masked]>
Date: Sat, 3 May 2014 08:29:50 -0400

Scharf, Michael (Michael) <[email-masked]> wrote:
> 
> Meeting attendance is not required. Still, I don't understand why not
> running an adoption call about two weeks after posting -00 really
> holds up work.

   (I don't understand how you're measuring "two weeks".)

   To me, it's more a question of how putting this question off until
the rush of IETF week can possibly help.

   We need to determine whether this work can get traction it TCPM; and
we need to get IETF support for doing this work in TCPM -- whose charter,
lest we forget, is quite clear about us doing "small" TCP changes.
  
> TCPM is not the only working group that has a high bar for adopting
> new work,

   A 30-second hum after a ten-minute presentation isn't my idea of
what "a high bar" should mean.

> and I think being conservative regarding std track TCP changes has
> always been a community consensus (e.g., our previous charter even
> told us that every new TCPM milestone requires IESG approval -
> that was certainly too conservative).

   The IESG today doesn't require "milestone" changes to come before
a formal telechat; but I can imagine some concern about whether this
particular change (invalidating the "Data offset" field) fits within
our charter or requires a change to the charter wording. I'd much
prefer that such a question not be squeezed into the IETF week rush.

> Also, in TCPM review cycles are unfortunately a rather scarce resource,
> and the chairs have to consider this as well when adopting new work.

   Absolutely: new work should not be started without commitments for
enough review cycles!

> Regarding draft-touch-tcpm-tcp-edo, I am particularly interested in
> more feedback from implementers and users before starting the formal
> adoption call.

   Ah, but _what_ feedback?

   Commitment to review, certainly.

   Do you need data on how crowded the option space is?

   Do you need security considerations? Does that include what happens
as the IP-layer path changes to include more or less middleboxes?

   Do you need analysis of how this sort of change interacts with
MPTCP?

   Do you need analysis of which middleboxes can't be modified to
recognize this proposed option?

   Please be specific.

> We usually get very valuable feedback from implementers during
> meetings, and that would be one reason not to rush right now -

   I really don't follow why adoption as a Working Group Draft would
get in the way of considering such feedback.

> but it is the feedback I am interested in, not the meeting. If
> corresponding aspects are discussed on the list before the next
> meeting, I'll be more than happy as well.

   Fair enough!

> * From designers of TCP extensions requiring more option space: Is
>   draft-touch-tcpm-tcp-edo a good solution for your needs and thus
>   the right way to move forward?

   We're desperate! We'll get behind anything!

> * From implementers: Are there deployment issues beyond what is
>   described in the draft already?

   Good lord, yes! (That deserves a separate thread -- but again,
surely that can be discussed for a WG Draft as easily as for an
individual draft, can't it?)

> And, would this mechanism be implemented in the Internet? 

   I'd be astounded if it wouldn't be implemented in Linux!

   What beyond that do you need?

---

From: Olivier Bonaventure <[email-masked]>
Date: Sat, 03 May 2014 21:51:51 +0200

Michael,

> * From implementers: Are there deployment issues beyond what is described in the draft already? And, would this mechanism be implemented in the Internet?

My main concern with this draft are the possible interactions with 
segment splitting/coalescing middleboxes.

The draft considers that these middleboxes are rare and should not 
constitute a problem. Unfortunately, I have to disagree with this 
statement. An important concern are the recent NICs that support TSO or 
GRO. These NICs are very widely used today and there are environments 
where it is very difficult for a TCP stack to determine whether it runs 
with such a NIC a not. A typical case is a virtual machine running on a 
host that exposes a driver that does not correspond to the physical NIC 
that is actually used.

The EDO option proposed in this draft does not unfortunately work 
correctly through a middlebox that splits segments. Concerning the 
following example.

A host that has negotiated EDO during the three way handshake sends a 
segment that contains :
- TCP header indicates 4 bytes of option, followed by EDO option 
indicating an option of 100 bytes, 100 bytes of option (e.g. a very long 
SACK), 1000 bytes of data

The middlebox does not understand the EDO option but needs to split the 
segment. Some middleboxes will copy all options in the two segments. 
This results in  :

segment 1 : TCP header indicates 4 bytes of option, followed by EDO 
option indicating an option of 100 bytes, 100 bytes of option (e.g. a 
very long SACK), 500 bytes of data

segment 2 : TCP header indicates 4 bytes of option, followed by EDO 
option indicating an option of 100 bytes, 500 bytes of data

The receiver will try to parse 100 bytes of option in the second segment...

In their IMC paper entitled "Is it Still Possible to Extend TCP?", 
Michio Honda et al. provide the following recommendation
http://conferences.sigcomm.org/imc/2011/docs/p181.pdf

"All the NICs we tested correctly copied the options to all the split 
segments. TSO is now sufficiently commonplace so that designers of 
extensions to TCP should assume it. The implication is that TCP options 
must be designed so that when they are duplicated on consecutive 
segments, this does not adversely affect correctness or performance."

EDO should be modified to take this recommendation into account to be 
deployable in today's Internet.

One possibility could be to include a checksum inside the EDO option.

   +--------+--------+--------+--------+----------+----------+
   |  Kind  | Length |  Header_length  |       Checksum      |
   +--------+--------+--------+--------+----------+----------+

The checksum should be computed over the entire extended option by the 
sender. When a receiver receives a segment with the EDO option, it MUST 
verify the EDO checksum. If the checksum is correct, then the extended 
option can be parsed. Otherwise, the entire segment should be silently 
dropped.

The checksum could solve the problem mentionned above with the NIC that 
segment data, but additional work is required to check whether such 
modification would solve all possible problems with middleboxes.

With Multipath TCP, we managed to extend TCP while preserving 
connectivity. Multipath TCP can cope with various types of middleboxes 
but not all possible behaviours. For some of these behaviours, Multipath 
TCP fallsback to regular TCP to ensure that the application continues to 
operate without any loss of connectivity. The EDO option should also 
have this goal in mind.

Best regards,

Olivier

-- 
INL, ICTEAM, UCLouvain, Belgium, http://inl.info.ucl.ac.be

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Sun, 4 May 2014 09:54:23 +0000

John,

I am convinced that TCPM is the right place to discuss draft-touch-tcpm-tcp-edo. If we have strong support and commitments regarding the technical approach, we'll sort out the question whether we need a charter update or not.

Regarding implementer and user feedback, I'd really be interested in a study like the IMC 2011 paper (or a verification that those insights still apply). I might be wrong, but my assumption is that due to deployments of MPTCP, fast open, etc. it may perhaps become simpler to deploy new TCP extensions in parts of the Internet. That would be important to know.

But this kind of work may require help from the TCPM community, and it will require time to complete. For instance, just as a thought from an individual community member, the outcome of such a study could become an informational companion document like RFC 2416, RFC 2884, etc.

Michael

________________________________________
Von: John Leslie [[email-masked]]
Gesendet: Samstag, 3. Mai 2014 14:29
An: Scharf, Michael (Michael)
Cc: Martin Stiemerling; [email-masked]
Betreff: Re: [tcpm] New Version Notification for draft-touch-tcpm-tcp-edo-01.txt

   Ah, but _what_ feedback?

   Please be specific.

   Fair enough!

---

From: Costin Raiciu <[email-masked]>
Date: Mon, 5 May 2014 10:49:05 +0300

I'd like to add to what Olivier has said.

One question I had about this draft is whether the options included in the EDO are supposed to be carrying sequence numbers or not. Two options:
1. they carry sequence numbers: in this case we are changing the semantics of options in TCP, sending them reliably, orderly and subject to flow control. Which may be just fine for most types of options, but its not conforming to the spec and I suspect it opens a can of worms when we consider retransmissions of old options etc. For instance, if one would send MPTCP's Data ACKs in EDO reliably and subject to flow control we can end up with deadlocks.

2. they don't carry sequence numbers. In this case, a box that does not know EDO and looks at packets it will see more data sent sent the combination of sequence number / length allows. Many boxes will just drop the extra bytes, others may reset the connection. Based on our IMC 11 experience, I expect that at least 1/3 of access networks (e.g. DSL / Cellular / Hotspot  / etc) will break EDO. So this option is a no go, in my opinion.

Another thing: the draft says that NAT will work with EDO. However, most NATs have a passive FTP mode - which means they will look through the TCP payload for the host's IP address and rewrite with their own. 
With EDO, the NAT could be messing up options placed in EDO space in the payload. To avoid this you need a checksum, as Olivier says. 

I suspect that the solution has to be something in the direction of (1), where the EDO space is quoted explicitly  (have an SACK like options that says bytes XX-YY from the payload carry in fact  options). On top of that, we need a review of what options can be sent over EDO and what options cannot be sent over it. It's not going to be easy, in my opinion.

Cheers.
Costin

On 3 May 2014, at 22:51, Olivier Bonaventure wrote:

> My main concern with this draft are the possible interactions with segment splitting/coalescing middleboxes.
> 
> The draft considers that these middleboxes are rare and should not constitute a problem. Unfortunately, I have to disagree with this statement. An important concern are the recent NICs that support TSO or GRO. These NICs are very widely used today and there are environments where it is very difficult for a TCP stack to determine whether it runs with such a NIC a not. A typical case is a virtual machine running on a host that exposes a driver that does not correspond to the physical NIC that is actually used.
> 
> The EDO option proposed in this draft does not unfortunately work correctly through a middlebox that splits segments. Concerning the following example.
> 
> A host that has negotiated EDO during the three way handshake sends a segment that contains :
> - TCP header indicates 4 bytes of option, followed by EDO option indicating an option of 100 bytes, 100 bytes of option (e.g. a very long SACK), 1000 bytes of data
> 
> The middlebox does not understand the EDO option but needs to split the segment. Some middleboxes will copy all options in the two segments. This results in  :
> 
> segment 1 : TCP header indicates 4 bytes of option, followed by EDO option indicating an option of 100 bytes, 100 bytes of option (e.g. a very long SACK), 500 bytes of data
> 
> segment 2 : TCP header indicates 4 bytes of option, followed by EDO option indicating an option of 100 bytes, 500 bytes of data
> 
> In their IMC paper entitled "Is it Still Possible to Extend TCP?", Michio Honda et al. provide the following recommendation
> "All the NICs we tested correctly copied the options to all the split segments. TSO is now sufficiently commonplace so that designers of extensions to TCP should assume it. The implication is that TCP options must be designed so that when they are duplicated on consecutive segments, this does not adversely affect correctness or performance."
> 
> EDO should be modified to take this recommendation into account to be deployable in today's Internet.
> 
> The checksum should be computed over the entire extended option by the sender. When a receiver receives a segment with the EDO option, it MUST verify the EDO checksum. If the checksum is correct, then the extended option can be parsed. Otherwise, the entire segment should be silently dropped.
> 
> The checksum could solve the problem mentionned above with the NIC that segment data, but additional work is required to check whether such modification would solve all possible problems with middleboxes.
> 
> With Multipath TCP, we managed to extend TCP while preserving connectivity. Multipath TCP can cope with various types of middleboxes but not all possible behaviours. For some of these behaviours, Multipath TCP fallsback to regular TCP to ensure that the application continues to operate without any loss of connectivity. The EDO option should also have this goal in mind.
> 
> Best regards,
> 
> 
> Olivier
> 
> 
> -- 
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: John Leslie <[email-masked]>
Date: Mon, 5 May 2014 10:15:55 -0400

Scharf, Michael (Michael) <[email-masked]> wrote:
> 
> I am convinced that TCPM is the right place to discuss
> draft-touch-tcpm-tcp-edo. If we have strong support and commitments
> regarding the technical approach, we'll sort out the question whether
> we need a charter update or not.

   :^)

> Regarding implementer and user feedback, I'd really be interested
> in a study like the IMC 2011 paper

   I assume you mean "Is it still possible to extend TCP?"

> (or a verification that those insights still apply).

   This paper reported research on 135 paths using 3 port numbers:
HTTP, HTTPS, and "middle-of-nowhere", and showed at best 96% passage
of unknown options. It discussed the operation of middleboxes (including
HTTP proxies), and concluded that MPTCP could detect damage and fallback,
TcpCrypt is fragile (and can't fallback after being enabled), and TCP
Extended Options (draft-eddy-tcp-loo) was unsafe.

   I take it on faith that those insights are still valid.

> I might be wrong, but my assumption is that due to deployments of
> MPTCP, fast open, etc. it may perhaps become simpler to deploy new
> TCP extensions in parts of the Internet. That would be important
> to know.

   I think it is a given that deployment will depend upon successful
negotiation end-to-end.

   Given the (still-growing!) extent of middlebox damage, I suggest
it will also be necessary to detect middlebox damage throughout the
life of the connection.

   Nonetheless, I see large portions of the Internet where expanding
the TCP Options field could function today. I predict that offending
middleboxes will be replaced over time (perhaps five years). So long
as on offending middlebox can be identified and folks understand the
damage it causes, replacing them need not be expensive.

> But this kind of work may require help from the TCPM community,
> and it will require time to complete. For instance, just as a thought
> from an individual community member, the outcome of such a study
> could become an informational companion document like RFC 2416,
> RFC 2884, etc.

   Repeating the IMC-2011/p181 study sounds more like an IRTF task;
but I certainly agree it would be worthwhile.

====

   I do expect a standardized EDO to be more complex than tcp-loo;
but there are a number of approaches which are (IMHO) sufficiently
safe. I believe touch-tcpm-tcp-edo to be a better starting point
than eddy-tcp-loo was; but it will need significant work before
submitting to the IESG.

---

From: John Leslie <[email-masked]>
Date: Mon, 5 May 2014 10:41:09 -0400

Costin Raiciu <[email-masked]> wrote:
> 
> One question I had about this draft is whether the options included
> in the EDO are supposed to be carrying sequence numbers...

   That's probably a bad idea: while we can depend upon the endpoints
to negotiate a common understanding or visibly fail, the same is not
true of middleboxes. A middlebox _can_ cause negotiation to fail, but
it has no way to signal exactly how it _does_ interpret an option.

> 1. they carry sequence numbers: in this case we are changing the
>    semantics of options in TCP... I suspect it opens a can of worms...

   Agreed.

> 2. they don't carry sequence numbers. In this case, a box that does not
>    know EDO and looks at packets it will see more data sent sent...
>    Many boxes will just drop the extra bytes...

   Which would be a disaster.

>    So this option is a no go, in my opinion.

   I don't agree that adding sequence numbers can be made safe; so I
don't agree with Costin's division into two parts.

> Another thing: the draft says that NAT will work with EDO.

   I'd prefer not to argue that yet...

> However, most NATs have a passive FTP mode - which means they will
> look through the TCP payload for the host's IP address and rewrite
> with their own. 

   (That's one of the reasons I prefer to have "Data Offset" be a clearly
invalid value whenever it is overridden by EDO: any middlebox will then
have fair warning that it is clueless.)

> With EDO, the NAT could be messing up options placed in EDO space in
> the payload. To avoid this you need a checksum, as Olivier says. 

   Checksum isn't the _only_ way; but some sort of checksum is probably
a good idea.

> I suspect that the solution has to be something in the direction of
> (1), where the EDO space is quoted explicitly (have an SACK like
> options that says bytes XX-YY from the payload carry in fact  options).

   I don't quite follow; but that may be a workable approach... (I think
of somehow overlaying the "Urgent pointer".)

> On top of that, we need a review of what options can be sent over EDO
> and what options cannot be sent over it.

   I don't think _anything_ should be "cannot" there; but we'll have
some work in how to ration what options go in the SYN (which, alas,
we're going to have to give up on expanding).

> It's not going to be easy, in my opinion.

   It never was easy. :^(

---

From: Joe Touch <[email-masked]>
Date: Mon, 05 May 2014 08:40:33 -0700

On 5/3/2014 12:51 PM, Olivier Bonaventure wrote:
>> draft-touch-tcpm-tcp-edo a good solution for your needs and thus the
>> right way to move forward?
>>
>> described in the draft already? And, would this mechanism be
>> implemented in the Internet?
>

These already affect a few things - splitting replicates timestamps and 
repeats ACKs, and coalescing might need to drop SACK blocks and drops 
some of the timestamp info.

A key question for the group is the extent to which we want to support 
these or even detect them in our core mechanisms, or whether we should 
suggest that they be detected (and replaced) by existing mechanisms 
(e.g., TCP-AO or IPsec).

> GRO.

These are not merely middleboxes that resegment. They ought to be 
integrated with the endsystem sufficient to support TCP.

So we should consider the impact of network rewriting boxes vs. TSO/GRO 
NICs as separate issues, IMO.

 > These NICs are very widely used today and there are environments
> correctly through a middlebox that splits segments.

Again, we're back to the middlebox example:

> Concerning the

A middlebox that examines the contents of TCP traffic - or worse, 
modifies it - needs to follow all of RFC1122 host requirements. It's 
interpreting content - if it sees a SYN with an unknown option, it ought 
to have removed it or dropped the SYN.

If it doesn't, you have a device that is broken and needs to be replaced.

We should design TCP to be liberal in what it accepts, but no Internet 
protocol should be designed to overcome deliberate implementation errors.

Agreed.

> segments, this does not adversely affect correctness or performance."

This tells me your TCP interface was implemented incorrectly.

The correct interface passes options (including the extended option *and 
its associated contents*) ***SEPARATELY*** from the user data.

If that is done all the way down to the TSO, the TSO will correctly copy 
the option into individual segments correctly.

If, however, you implement this badly - and pass TCP segments as a whole 
to the offload engine, then the offload engine *has no business* 
splitting or coalescing segments.

> One possibility could be to include a checksum inside the EDO option.

That logic would apply too EVERY option, and now we have a TCP that 
wastes space primarily to locate and debug implementation errors.

I don't agree that TCP should be designed to overcome deliberate 
implementation errors.

Joe

---

From: Wesley Eddy <[email-masked]>
Date: Mon, 05 May 2014 11:50:04 -0400

> have fair warning that it is clueless.)

I think I agree.  IMHO, the tradeoff comes down to whether it's more
desirable for the middlebox to end up doing something violent like:

- dropping the segment
- resetting the TCP connection
- generating an ICMP
- crashing

versus just doing something to the segments which the receiver can
detect and possibly work-around with greater size and complexity in
the mechanism itself.

It ends up being a matter of transition and whether we really
expect offending nodes to be found and fixed over time, and can
stand some deployment hassles and complaints in the meantime.
If not, then the design needs to be a bit more complicated forever
in order to accommodate today's legacy.

If there are significant advantages to being able to use long
options that relate to real business problems, then I think the
legacy won't be as much of a problem, but don't really know.

-- 
Wes Eddy
MTI Systems

---

From: Joe Touch <[email-masked]>
Date: Mon, 05 May 2014 08:56:16 -0700

On 5/5/2014 12:49 AM, Costin Raiciu wrote:
> in the EDO are supposed to be carrying sequence numbers or not. Two
> options:
>
> semantics of options in TCP, sending them reliably, orderly and
> subject to flow control.

We're not doing that any more than the options in existing segments 
within the existing Data Offset are so covered.

The option isn't buried in the data stream - this was considered for 
MPTCP, but rejected, for some of the reasons you mention.

So EDO option space ought to be used the same way - with the same 
limitations (or not) of existing option space.

...
> Another thing: the draft says that NAT will work with EDO. However,
> most NATs have a passive FTP mode - which means they will look
> through the TCP payload for the host's IP address and rewrite with
> their own. With EDO, the NAT could be messing up options placed in
> EDO space in the payload. To avoid this you need a checksum, as
> Olivier says.

See my response to Oliver's issue on this. Such a NAT ought to have 
either dropped the EDO in the initial SYN or dropped the SYN completely.

Joe

---

From: Joe Touch <[email-masked]>
Date: Mon, 05 May 2014 08:57:46 -0700

> have fair warning that it is clueless.)

It had that warning when it saw the EDO option and decided that it could 
ignore it quietly.

Joe

---

From: Joe Touch <[email-masked]>
Date: Mon, 05 May 2014 09:00:21 -0700

PS - one other alternative would be to allow the header checksum to be 
optional - users can turn it on or off as needed.

It could be part of EDO (determined by the length of the EDO option 
itself) or as a separate option (which would then be useful for all 
other options as a "bad NAT traversal detection option".)

Joe

---

From: Costin Raiciu <[email-masked]>
Date: Mon, 5 May 2014 19:04:01 +0300

> A middlebox that examines the contents of TCP traffic - or worse, modifies it - needs to follow all of RFC1122 host requirements. It's interpreting content - if it sees a SYN with an unknown option, it ought to have removed it or dropped the SYN.
> 
> We should design TCP to be liberal in what it accepts, but no Internet protocol should be designed to overcome deliberate implementation errors.

As much as I agree with this purist view of the Internet, I also believe it is unworkable if we want stuff deployed. 

Say we actually stuck to the plan and designed EDO for a world with reasonable middleboxes. Then we implement it in Linux, then we turn it on by default. Then, all of a sudden the Internet stops working, and you have disgruntled users cursing Linux right and left....

Trouble is, for middleboxes to change you need to have a working protocol (that is maybe suboptimal) and a strong need for the change, and push from OS/app vendors etc. 

Cheers.
Costin

> The correct interface passes options (including the extended option *and its associated contents*) ***SEPARATELY*** from the user data.
> 
> If that is done all the way down to the TSO, the TSO will correctly copy the option into individual segments correctly.
> 
> If, however, you implement this badly - and pass TCP segments as a whole to the offload engine, then the offload engine *has no business* splitting or coalescing segments.
> 
> That logic would apply too EVERY option, and now we have a TCP that wastes space primarily to locate and debug implementation errors.
> 
> I don't agree that TCP should be designed to overcome deliberate implementation errors.
> 
> Joe
>

---

From: Joe Touch <[email-masked]>
Date: Mon, 05 May 2014 09:14:09 -0700

On 5/5/2014 9:04 AM, Costin Raiciu wrote:
>> We should design TCP to be liberal in what it accepts, but no
>> Internet protocol should be designed to overcome deliberate
>> implementation errors.
>>
> As much as I agree with this purist view of the Internet, I also
> believe it is unworkable if we want stuff deployed.
>
> Say we actually stuck to the plan and designed EDO for a world with
> reasonable middleboxes. Then we implement it in Linux, then we turn it
> on by default. Then, all of a sudden the Internet stops working, and you
> have disgruntled users cursing Linux right and left....

Linux already runs lots of things we didn't approve, and never seems to 
take any blame. I don't see avoiding Linux user ire as a viable approach.

> Trouble is, for middleboxes to change you need to have a working
> protocol (that is maybe suboptimal) and a strong need for the change,
> and push from OS/app vendors etc.

For middleboxes to change, they have to start breaking things. As long 
as we adapt to the errors they make, we're helping create their market.

I agree with your overall point that we do need to consider how/whether 
this can avoid being held hostage by middleboxes, and a *temporary* 
header checksum might be a way.

But ultimately, that still means the new option will not work through 
middleboxes, and lots of people will not consider that the fault of the 
middlebox.

Joe

---

From: John Leslie <[email-masked]>
Date: Mon, 5 May 2014 12:30:25 -0400

> desirable for the middlebox to end up doing something violent like:

   This should always be "legal" for a middlebox. (I certainly prefer
it to damaging some unrelated part of the packet.) It's easy enough
to detect.

> - resetting the TCP connection

   This isn't quite "legal", but is certainly easy to detect.

> - generating an ICMP

   Would anyone actually do this? (Hopefully, it would be in combination
with dropping the packet.)

> - crashing

   That's my most-preferred one. There's no excuse to blame anyone but
the middlebox vendor. ;^)

> detect and possibly work-around with greater size and complexity in

   I'd _much_ rather not start down the road of guessing what
unrelated part of the packet has been damaged!

> stand some deployment hassles and complaints in the meantime.

   IMHO, that's always the case -- it just is, sometimes the transition
takes decades. :^(

> in order to accommodate today's legacy.

   Now we're arguing over price: how much is "a bit"?

> legacy won't be as much of a problem, but don't really know.

   We've already over-packed the Options field: there are already
_multiple_ business cases for supporting an expanded field.

   How much of a problem the legacy proves to be depends a lot on
how we design EDO...

---

From: John Leslie <[email-masked]>
Date: Mon, 5 May 2014 12:36:15 -0400

> It had that warning when it saw the EDO option and decided that it could 

   This points out another failure of the TCP-Options design: for such
a basic building block, Options should have carried an indicator of
what to do with an unrecognized option -- e.g. ignore, drop the packet,
etc.

   Joe is right that "conservative in what you send" sort-of implies
that you don't pass on an option when you have no idea what it might
actually mean...

   (But that horse left the barn many years ago.)

---

From: Joe Touch <[email-masked]>
Date: Mon, 05 May 2014 09:51:28 -0700

> what to do with an unrecognized option -- e.g. ignore, drop the packet,

Right - we built that into IPv6, but TCP missed that boat.

However, it still requires that middleboxes would do the right thing; 
their incentive to "keep working" suggests they would treat this like 
all other Internet standards - something to violate if it makes them money.

So even if the flag were there (it sort of is already - but "ignore" on 
pass-through should have been interpreted as "drop"), it's not clear 
middlebox vendors would do as we proscribe.

Joe

---

From: Joe Touch <[email-masked]>
Date: Mon, 05 May 2014 14:58:20 -0700

Some comments on the approaches for handling the connection:

...

Agreed.

Agreed.

> with dropping the packet.)

This ought to be legal only when the midbox sends a router-like ICMP 
error. The only one that makes sense to me is "protocol unreachable".

However, this could be interpreted by the source as the fact that TCP 
isn't reachable at all, not just TCP-with-EDO.

So I don't think this is viable at all.

> the middlebox vendor. ;^)

I like that one too.

Joe

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Tue, 6 May 2014 10:13:03 +0000

> > extensions to TCP should assume it. The implication is that TCP
> > options must be designed so that when they are duplicated on
> > consecutive segments, this does not adversely affect correctness or
> performance."
> 
> to the offload engine, then the offload engine *has no business* splitting
> or coalescing segments.

I can agree with the assessment, that TSO is typically done wrong, and that the API was not properly designed.

Most TSO NICs only deal with "typical" header flags - from my ECN research I know that non-ECT (CE; ECT(0); ECT(1)) IP flags are not necessarily delivered correctly by LRO (only the IP bits of the first segment that was received are usually handed up to the software TCP stack, even when they change; thus ECN information in up to 200 segments may never reach the TCP layer for handling (and recent measurements also indicate that for ECN, the gating issue is not TCP ECN, but the IP ECN flags getting forwarded end-to-end properly...

On the LRO TCP side, any "unexpected" TCP header bits (EDO wouldn't account for this; setting DO to an invalid value might) would typically stop the fastpath processing, and get individual segments delivered - thus the RFC3168 ECN TCP signaling usually works (as ECE and/or CWR bits are set in those segments).

As for a TSO sender: if the NIC driver is broken, the sending TCP can ensure that the hardware doesn't perform invalid splits on segments containing EDO - it has to be provided as a NIC driver workaround, sure; 

Richard Scheffenegger

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 22 May 2014 18:10:33 +0100

Joe,

Returning to the question of adoption, we have to 
address the question of why previous attempts to 
do this have failed. I don't believe it is as 
simple as that they tried to include options on 
SYNs, so all we have to do is avoid the SYN problem.

1) There is obviously the re-segmentation 
problem, which Olivier/Costin have usefully 
highlighted, and I agree an optional checksum would at least detect this.

2) However, I think the main problem is that many 
important cases will need as large or larger TCP 
option space on the SYN as on non-SYNs.

The option space pressure for all the following 
(except SACK) is at least as critical for the SYN as for non-SYN segments:
* SACK          (SYN << non-SYN)
* MPTCP (SYN > non-SYN - typically)
* Timestamp     (SYN = non-SYN)
* Window scale  (SYN > non-SYN)
* TCP-AO        (SYN = non-SYN)
* TFO init      (SYN << non-SYN - but no use without TFO resume as well)
* TFO resume    (SYN >> non-SYN)

Given the above list, if bigger TCP options are 
not available for SYNs, is a critical mass really 
going to be persuaded that it is worth the effort 
of implementing, deploying, debugging, 
supporting, etc? And we need a critical mass, 
because until EDO is deployed at both ends it 
does nothing, so early implementers have to work on faith.

Admittedly, EDO is partly trying to make space 
for future options and partly trying to solve a 
problem we already have with existing options. 
So, I admit that the relative size of existing 
options is not the whole story. However, even new 
options have to fit with the existing ones.

3) The EDO draft implies that it is provably 
impossible to increase the option space on a SYN. 
A couple of ways have been proposed to solve this problem:
* LOIC <draft-yourtchenko-tcp-loic-00> that sends 
two parallel SYNs; a regular one and one with a 
longer TCP option space AND a newly defined 
checksum calculation, so that it will be discarded by legacy TCPs.
* An out-of-band control channel, e.g. <draft-paasch-mptcp-control-stream>

Much earlier in this thread, you dismissed the latter, wrongly I believe:

> >At 16:00 25/04/2014, Zimmermann, Alexander wrote:
> >> * Sec 4:
> >>    - Should we mention „draft-paasch-mptcp-control-stream“ here?
> >
> > I don't think so, any more than mentioning 
> FTP's control channel. In both cases, a 
> separate data channel is used for control info. 
> The MPTCP approach isn't applicable to 
> individual TCP connections - it only works in 
> MPTCP because the group of connections is co-associated.

This seems to miss the point that there could be 
a whole class of solutions where we create an 
associated connection, precisely in order to add 
a control channel of unlimited size to one (or 
more) data channels. This brings its own 
problems, not least it loses the intrinsic 
security binding when control and data are in the 
same segment. So, I wouldn't separate off a 
control channel if we were starting from scratch. 
But it's probably the most promising approach, 
given we have to add a carbuncle to a wart.

In fact there are some similarities between 
parallel SYNs and parallel channels.

4) Finally, the EDO draft cites 
<draft-ananth-tcpm-tcpoptext-00> as if it is just 
another solution. It's not. It's actually a very 
useful survey of all the previous attempts to 
solve this problem, including a useful 
enumeration of the problems that have to be surmounted.

The arguments on this thread show that we don't 
agree on the problem space. So, I suggest we 
adopt Anatha's draft, and as we develop it, we 
agree on the problem we are trying to solve 
first. Boring, but apparently necessary.

You could precis this whole email as "Necessity 
and impossibility are the mothers of invention".

Regards

Bob

________________________________________________________________
Bob Briscoe,                                                  BT

---

From: John Leslie <[email-masked]>
Date: Thu, 22 May 2014 13:53:51 -0400

> SYNs, so all we have to do is avoid the SYN problem.

   Of course, it's not that simple; but we've so far failed to convince
folks that the schemes for expanding option-space for SYNs are safe.

> highlighted, and I agree an optional checksum would at least detect this.

   Yes.

> option space on the SYN as on non-SYNs.

   No.

   We have not yet passed the point where the needed initial-negotiate
can't fit in an un-expanded SYN.

   We will, of course, if we continue to do nothing. :^(

> * TFO resume    (SYN >> non-SYN)

   Of those, certainly MPTCP can adapt to a really-small SYN.

   The Timestamp/Window-Scale is scary, but not actually fatal.

   (We may, of course, need to eventually design a way to push the
initial-negotiate issue beyond the _initial_ SYN into the SYN-ACK, but
that's not something we need to solve in tcpm-tcp-edo.)

> does nothing, so early implementers have to work on faith.

   The critical-mass will be slow to develop anyway. What will drive
the speed of adoption, IMHO, is whether it is perceived as risk-free.

> options have to fit with the existing ones.

   They can't, already. :^(

> impossible to increase the option space on a SYN. 

   I don't read it that way.

> checksum calculation, so that it will be discarded by legacy TCPs.

   That's too big of a change to ask folks to believe it safe.

> * An out-of-band control channel, e.g. <draft-paasch-mptcp-control-stream>

   That's workable, but really doesn't belong as part of an option-space
expansion update to TCP.

> Much earlier in this thread, you dismissed the latter, wrongly I believe:

   I don't agree; but that question is not germane.

> given we have to add a carbuncle to a wart.

   Yes.

> enumeration of the problems that have to be surmounted.

   How would _you_ like it cited?

> first. Boring, but apparently necessary.

   I don't believe we'll be taken seriously trying to resurrect a
two-year-old draft.

   IMHO, we need to concentrate on a smaller piece of the problem.

   YMMV, of course...

---

From: Joe Touch <[email-masked]>
Date: Thu, 22 May 2014 10:58:37 -0700

Hi, Bob,

On 5/22/2014 10:10 AM, Bob Briscoe wrote:
> Joe,
>
> Returning to the question of adoption, we have to address the question
> of why previous attempts to do this have failed. I don't believe it is
> as simple as that they tried to include options on SYNs, so all we have
> to do is avoid the SYN problem.

There haven't been any I can recall that didn't either focus on SYN 
space extension or include it as a key component.

> 1) There is obviously the re-segmentation problem, which Olivier/Costin
> have usefully highlighted, and I agree an optional checksum would at
> least detect this.

An optional checksum would help protect TCP from such "attacks" - 
re-segmentation isn't supported, and already breaks a number of existing 
TCP features (TCP-AO, TCP MD5, ACK clocking requirements, etc.).

> 2) However, I think the main problem is that many important cases will
> need as large or larger TCP option space on the SYN as on non-SYNs.

Oh, I certainly agree with this. The point of this proposal is twofold:

	a) (primary) to put to bed the notion that 'there is a way'
	to extend SYN option space without contaminating connections to
	legacy hosts

	b) (secondary) to do the only extension possible - post-SYN.

> The option space pressure for all the following (except SACK) is at
> least as critical for the SYN as for non-SYN segments:
> * SACK          (SYN << non-SYN)
> Given the above list, if bigger TCP options are not available for SYNs,
> is a critical mass really going to be persuaded that it is worth the
> effort of implementing, deploying, debugging, supporting, etc?

Maybe never; I don't know. This isn't a "build it and they will come" 
proposal; it's intended to document the 'negative' about SYN extension 
primarily. The extension of non-SYN is intended primarily to show how it 
can trivially be done, not to argue that it's a great thing to do it.

> And we
> need a critical mass, because until EDO is deployed at both ends it does
> nothing, so early implementers have to work on faith.
>
> Admittedly, EDO is partly trying to make space for future options and
> partly trying to solve a problem we already have with existing options.
> So, I admit that the relative size of existing options is not the whole
> story. However, even new options have to fit with the existing ones.
>
> 3) The EDO draft implies that it is provably impossible to increase the
> option space on a SYN.

It states it, with the condition of not contaminating connections to 
legacy TCPs.

> * LOIC <draft-yourtchenko-tcp-loic-00> that sends two parallel SYNs; a
> regular one and one with a longer TCP option space AND a newly defined

The problem happens when the non-legacy endpoint gets the legacy SYN 
first - or only. There's no good way to do support dual-stack without 
incurring huge delay penalties waiting for a SYN with the new feature.

> * An out-of-band control channel, e.g. <draft-paasch-mptcp-control-stream>

That won't work for the first SYN to a new endpoint, which is always 
where the problem exists. Subsequent SYNs can always be affected by 
persistent state, if desired.

>> > I don't think so, any more than mentioning FTP's control channel. In
>> both cases, a separate data channel is used for control info. The
>> MPTCP approach isn't applicable to individual TCP connections - it
>> only works in MPTCP because the group of connections is co-associated.
>
> This seems to miss the point that there could be a whole class of
> solutions where we create an associated connection, precisely in order
> to add a control channel of unlimited size to one (or more) data
> channels. This brings its own problems, not least it loses the intrinsic
> security binding when control and data are in the same segment. So, I
> wouldn't separate off a control channel if we were starting from
> scratch. But it's probably the most promising approach, given we have to
> add a carbuncle to a wart.
>
> In fact there are some similarities between parallel SYNs and parallel
> channels.

The problematic case is first-contact. All other contacts - either 
within a single connection or with subsequent connections can be handled 
in any number of ways - a control channel, state that's left-behind, etc.

> 4) Finally, the EDO draft cites <draft-ananth-tcpm-tcpoptext-00> as if
> it is just another solution. It's not. It's actually a very useful
> survey of all the previous attempts to solve this problem, including a
> useful enumeration of the problems that have to be surmounted.

I'll correct that.

> The arguments on this thread show that we don't agree on the problem
> space. So, I suggest we adopt Anatha's draft, and as we develop it, we
> agree on the problem we are trying to solve first. Boring, but
> apparently necessary.

I don't agree with much of the content of that doc regarding the 
enumeration of requirements. I don't agree that this is a useful place 
to start, nor did the WG when it was first proposed AFAIR.

Joe

---

From: Wesley Eddy <[email-masked]>
Date: Thu, 22 May 2014 14:58:22 -0400

>     b) (secondary) to do the only extension possible - post-SYN.

I agree with this.  In my view, we actually have had consensus
for a long time that extending the SYN is "really hard" (requiring
some creativity), whereas extending the options space on other
packets can be done quite a bit more trivially.

So, the problem is decomposed into 2 parts; the harder part and
the easier part.  EDO is leaving the harder part alone for now;
although it may ultimately be the more useful one to solve.  I
don't see this as a problem with EDO or reason not to do it ...
any future solution for SYNs may differ substantially, but would
seem like it should not impact how options are extended on other
segments.

-- 
Wes Eddy
MTI Systems

---

From: Joe Touch <[email-masked]>
Date: Thu, 22 May 2014 14:05:43 -0700

On 5/22/2014 11:58 AM, Wesley Eddy wrote:
> some creativity),

Well, AFAICT we've also known for a long time that it's basically 
impossible when needed - i.e., to new endpoints, without impacting 
legacy endpoints (i.e., making their connections to new systems slow or 
misinterpreting option info as user data).

Yes, there are a few other potential solutions that need to be discussed 
as non-viable, but IMO that's an important part of this doc.

At the very least, even if it's not provably impossible, the obvious 
solutions don't work and it's harder than it looks.

Joe

---

From: David Borman <[email-masked]>
Date: Thu, 22 May 2014 16:12:47 -0500

> 	to extend SYN option space without contaminating connections to

Add to that "without adding any additional packet exchanges."  That’s really the issue.  This can be done, within the existing TCP processing, but at the cost of an additional RTT, which everyone tries to avoid.  The receiver could respond to the initial SYN with another SYN, which *can* take advantage of an extended option space because it now knows that the other side supports EDO.  The originator incurs an additional RTT before it can send data (2 vs 1 RTT), the receiver has no delay for when it can send data (1.5 RTT).

Besides the additional RTT for the originator, the biggest problem with this would be all those blasted firewalls that would drop the returning SYN-only responses.  The other way to do this is that when the SYN/ACK comes back indicating that the other side supports EDO, then the originator sends another SYN with the expanded option space.  That has a cost of a full RTT in both directions.

But the issue still remains the same: for that initial SYN packet, with no a priori knowledge, there is no way to extend the option space and maintain 100% backwards compatibility.

			-David Borman

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 23 May 2014 11:03:25 +0100

Joe, and everyone else who wants to work on this,

Just because it's easier to make a chocolate teapot than a cast-iron 
one, doesn't imply that there is any need for chocolate teapots.

IOW, we will be asking the IESG to spend reviewer time on EDO, so we 
need to give some plausible indication that someone might find it 
useful and it's not just an academic exercise. The current draft 
solely gives SACK + MPTCP + TCP-AO as an example, but is that really 
something that can't be done today?

Adding more complexity to the TCP stack (with the potential for more 
vulnerabilities) is only worthwhile if there's an identifiable 
benefit, otherwise few production stacks are going to implement it anyway.

Bob

At 22:05 22/05/2014, Joe Touch wrote:

>legacy endpoints (i.e., making their connections to new systems slow 
>or misinterpreting option info as user data).
>
>Yes, there are a few other potential solutions that need to be 
>discussed as non-viable, but IMO that's an important part of this doc.
>

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 23 May 2014 13:13:36 +0100

David,

There is certainly no need for an additional RTT.

Let me propose a couple of concrete strawmen (concrete men?) to show 
this. I'm not claiming these are novel. I know Joe and others think 
that we've seen enough of attempts to extend the TCP options on the 
SYN, but Joe should know that when he claims something is impossible, 
he wakes sleeping dragons in people like me.

1) Parallel control channel
___________________________
Client A sends two SYNs back-to-back to an existing well-known port (e.g. 80).
* SYN D, establishes a regular data connection, with sufficient TCP 
options to be workable but they still fit within the existing 40B option limit.
* SYN C establishes another parallel connection to the same 
well-known port that looks like regular data from the outside (it 
could even be an extension to HTTP to ensure middleboxes will let it 
pass), but it talks a new app-layer 'TCP control' protocol inside.

If there is no support for the new app-layer protocol on port 80 the 
control channel just shuts down with a suitable HTTP error, while SYN 
D has opened a data connection with sufficient TCP options to be workable.
If the new app-layer TCP control protocol is supported on port 80, 
the parallel control channel (C) adds unlimited additional control 
flexibility to the data channel (D) hardly any added latency.

Establishing a similar control channel in the opposite direction 
would be fairly trivial.

There are few, if any, middlebox problems with the above approach. 
However, there are certainly other problems, but no more 
insurmountable than all the problems that have already been discussed 
with taking the 'easy' route of EDO:
* A secure binding would have to be added to bind channel C to a 
secret known only to the originator of channel D, otherwise it would 
open up data channels to spoof control channel attacks. This binding 
could be built on a TCP-AO option in channel D.
* Channel C would need some way to refer to the segments of channel D 
that was robust against re-segmentation.
* The main problem is that the two channels don't share fate; a 
control packet can be delayed relative to the point in the data 
stream at which it is attempting to exert control, possibly for a RTT 
if it is lost and has to be retransmitted. However, this is not 
insurmountable. The control protocol could include a mode to 
"synthesise shared fate", by making the data channel buffer data 
until an associated control segment had arrived. This would duplicate 
the latency impact of a loss or delay on either channel, but one can 
imagine mitigations that would consign this latency impact to corner cases.
* It's a bit of a mess, but that comes with the territory when trying 
to fix legacy protocol problems.
* The internal stack architecture seems to require a trombone back 
down into the kernel from user-space, but that is not insurmountable 
- a shim within the kernel on port 80 (for example) could redirect 
control channel data across to the "TCP control channel module" in 
the kernel, while passing non-control channel connections to user-space.

2) Build on LOIC
______________________
Long option with invalid checksum <draft-yourtchenko-tcp-loic-00>

>    That's too big of a change to ask folks to believe it safe.

When I read an idea, I don't take it as set in stone and just find a 
hole and dismiss it. I see it as a potential stepping stone to a 
solution and think about how it could be done better. In fact, Andrew 
Yourtchenko said that was the intention of his write-up of LOIC.

I believe that an approach worth further thought would be a mixture 
of the control channel idea and the invalid checksum idea. I'm thinking of:
* a pure control SYN (C) sent first, then a base SYN (D) sent 
back-to-back, both to the same port.
* SYN C would contain something invalid to cause a legacy TCP stack 
or legacy app to discard it (and hopefully less probability that a 
middlebox would), e.g. a payload that is invalid for the application 
protocol on the port.
* there would be additional TCP options in the payload of SYN C to be 
added to the TCP options that arrived separately on the base SYN
* The control SYN could be bound crytographically to the base SYN (as 
already described).
* It could use the shim-like control stack arangement described earlier.

By focusing solely on extending the SYN, this would avoid the ongoing 
shared fate problems that a separate control channel suffers 
throughout the connection. There would still be shared fate problems 
with 2 SYNs (e.g. the two SYNs get re-ordered), but the protocol 
would have to be designed to be robust to that (naively, SYN D could 
include a new TCP option that told a new stack to wait a few ticks 
for a SYN C, but that would be vulnerable to meddleboxes). Not insurmountable.

Bob

At 22:12 22/05/2014, David Borman wrote:

>Add to that "without adding any additional packet 
>exchanges."  That's really the issue.  This can be done, within the 
>existing TCP processing, but at the cost of an additional RTT, which 
>everyone tries to avoid.  The receiver could respond to the initial 
>SYN with another SYN, which *can* take advantage of an extended 
>option space because it now knows that the other side supports 
>EDO.  The originator incurs an additional RTT before it can send 
>data (2 vs 1 RTT), the receiver has no delay for when it can send 
>data (1.5 RTT).
>
>Besides the additional RTT for the originator, the biggest problem 
>with this would be all those blasted firewalls that would drop the 
>returning SYN-only responses.  The other way to do this is that when 
>the SYN/ACK comes back indicating that the other side supports EDO, 
>then the originator sends another SYN with the expanded option 
>space.  That has a cost of a full RTT in both directions.
>
>
>But the issue still remains the same: for that initial SYN packet, 
>with no a priori knowledge, there is no way to extend the option 
>space and maintain 100% backwards compatibility.
>
>                         -David Borman

---

From: David Borman <[email-masked]>
Date: Fri, 23 May 2014 11:33:34 -0500

Hi Bob,

> There is certainly no need for an additional RTT.

Oh, there will always be proposals for ways to avoid the additional RTT, but at what cost?  A parallel control channel is a lot of complexity if its only purpose is to get more option space for a single, initial SYN packet.

> Let me propose a couple of concrete strawmen (concrete men?) to show this. I'm not claiming these are novel. I know Joe and others think that we've seen enough of attempts to extend the TCP options on the SYN, but Joe should know that when he claims something is impossible, he wakes sleeping dragons in people like me.

What is impossible is to expand the option space in the initial SYN packet and maintain 100% backwards compatibility with the following restrictions:
   1) No additional packets are sent, just the initial SYN
   2) There is no a priori knowledge as to whether or not the remote host understands expanded option space.

Every proposal that maintains 100% backwards compatibility has to send additional data in other packets in some form.  A parallel data channel, a second SYN, whatever, they all require additional data in separate packets.

			-David

---

From: Joe Touch <[email-masked]>
Date: Fri, 23 May 2014 09:55:45 -0700

Hi, Bob,

> one, doesn't imply that there is any need for chocolate teapots.

You don't get a cast iron teapot just because you want one either ;-)

> need to give some plausible indication that someone might find it useful
> and it's not just an academic exercise.

Sometimes the answer "you can't have A, but at least here's B" is more 
than an exercise; it educates the community. By not providing either 
answer, we have continued to drag this issue around the block for far 
too long -- and spent far too many cycles in this and other WGs seeking 
solutions.

 > The current draft solely gives
> SACK + MPTCP + TCP-AO as an example, but is that really something that
> can't be done today?

Current total for SYN options in widespread concurrent use (as already 
described in sec 6.4):

	2	SACK permitted
	10	timestamp
	3	window scale
	4	MSS
	------------------
	11 bytes

The current DO field is 4 bits, with a max value of 15 = 60 bytes for 
the total header, less 20 for the base TCP header which leaves 40 for 
options.

So let's see what happens when we add:

	11	widespread basic options
	16	TCP-AO
	20	MPTCP
	--------------------
	47

That's more than 40.

> vulnerabilities) is only worthwhile if there's an identifiable benefit,
> otherwise few production stacks are going to implement it anyway.

There are two identifiable benefits:

	1) explain the ways we already know we can't extend the SYN
	so we stop wasting time trying them repeatedly (i.e., education)

	2) provide a solution for the other segments, so that can be
	used - e.g., for large SACK responses

	3) educate the community

Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 23 May 2014 10:01:24 -0700

>     2) There is no a priori knowledge as to whether or not the remote host understands expanded option space.

I would pose the conditions as follows:

	1) is robust to packet loss, reordering, and alternate paths
		any assumption of "back to back" packets isn't

	2) does not negatively impact legacy hosts
		e.g., dual-stack/multiple SYN approaches fail here when
		a legacy system is trying to connect to an updated one
		(which has to wait to see if the updated SYN will show)

	3) succeeds through basic NAT/NAPTs
		anything that uses a nonstandard checksum will fail
		to support the extended space

	4) addresses new connections to hosts not previously contacted
		it's trivial to retain state across sequences of
		connections, as per RFC 2140.

> Every proposal that maintains 100% backwards compatibility has to
> sendadditional data in other packets in some form. A parallel data channel,
 > a second SYN, whatever, they all require additional data in separate
 > packets.

+1

---

From: Joe Touch <[email-masked]>
Date: Fri, 23 May 2014 10:14:42 -0700

On 5/23/2014 5:13 AM, Bob Briscoe wrote:
> this. I'm not claiming these are novel. I know Joe and others think that
> we've seen enough of attempts to extend the TCP options on the SYN, but
> Joe should know that when he claims something is impossible, he wakes
> sleeping dragons in people like me.

I slay dragons all the time ;-)

("Defender of the cruciform packet"* is one of my mottos)

> 1) Parallel control channel

You can send in whatever order you want; packets will be reordered, 
lost, and sent along alternate paths.

FWIW, do these use the same source port and ISN?
	- if they do, it'll reset the connection
	- if they don't, you're now limiting the number of
	concurrent connections to roughly half:
	http://www.isi.edu/touch/pubs/infocomm99/infocomm99-web/

> * SYN C establishes another parallel connection to the same well-known
> port that looks like regular data from the outside (it could even be an
> extension to HTTP to ensure middleboxes will let it pass), but it talks
> a new app-layer 'TCP control' protocol inside.

What happens when they arrive out of order? What happens when you get D 
but not yet C? How long do you wait for C?

This is the problem with dual-stack approaches - new endpoints penalize 
legacy endpoints if there's a stall, and undermine new endpoints if they 
don't.

> control channel just shuts down with a suitable HTTP error, while SYN D
> has opened a data connection with sufficient TCP options to be workable.
> If the new app-layer TCP control protocol is supported on port 80, the
> parallel control channel (C) adds unlimited additional control
> Establishing a similar control channel in the opposite direction would
> be fairly trivial.
>
> However, there are certainly other problems, but no more insurmountable
> than all the problems that have already been discussed with taking the
> 'easy' route of EDO:
> * A secure binding would have to be added to bind channel C to a secret
> known only to the originator of channel D, otherwise it would open up
> data channels to spoof control channel attacks. This binding could be
> built on a TCP-AO option in channel D.

Yes, that's another problem.

> that was robust against re-segmentation.

Which means it won't work in the current Internet, because 
resegmentation is also widespread (though evil, IMO).

> * The main problem is that the two channels don't share fate;a control
> packet can be delayed relative to the point in the data stream at which
> it is attempting to exert control, possibly for a RTT if it is lost and
> has to be retransmitted. However, this is not insurmountable. The
> control protocol could include a mode to "synthesise shared fate", by
> making the data channel buffer data until an associated control segment
> had arrived. This would duplicate the latency impact of a loss or delay
> on either channel, but one can imagine mitigations that would consign
> this latency impact to corner cases.
> * It's a bit of a mess, but that comes with the territory when trying to
> fix legacy protocol problems.
> * The internal stack architecture seems to require a trombone back down
> into the kernel from user-space, but that is not insurmountable - a shim
> within the kernel on port 80 (for example) could redirect control
> channel data across to the "TCP control channel module" in the kernel,
> while passing non-control channel connections to user-space.
>
> 2) Build on LOIC
> ______________________

Won't work through current NATs, which won't recalculate the checksum 
properly.

> I believe that an approach worth further thought would be a mixture of
> the control channel idea and the invalid checksum idea. I'm thinking of:

Again, please don't assume back-to-back means anything.

> * SYN C would contain something invalid to cause a legacy TCP stack or
> legacy app to discard it (and hopefully less probability that a

But so will a NAT, etc.

> shared fate problems that a separate control channel suffers throughout
> the connection. There would still be shared fate problems with 2 SYNs
> (e.g. the two SYNs get re-ordered), but the protocol would have to be
> designed to be robust to that (naively, SYN D could include a new TCP
> option that told a new stack to wait a few ticks for a SYN C, but that
> would be vulnerable to meddleboxes). Not insurmountable.

AFAICT, it is.

*with a nod to Raiders 3.

---

From: "Eggert, Lars" <[email-masked]>
Date: Mon, 26 May 2014 09:37:43 +0000

Hi,

On 2014-5-23, at 18:55, Joe Touch <[email-masked]> wrote:
> Current total for SYN options in widespread concurrent use (as already described in sec 6.4):
> 
> 	2	SACK permitted
> 	10	timestamp
> 	3	window scale
> 	4	MSS
> 	------------------
> 	11 bytes

I think it was Mark Allman who questioned a while ago whether the benefits of the timestamp option are worth spending 10 bytes in every packet. (But I can't find the email now.)

> 	11	widespread basic options

If we could shave those off, we'd be at 37 even with AO. It wouldn't make the option space any bigger for the future, but it would get us out of the current situation, and would make a combination of some crypto and MPTCP be deployable.

Lars

---

From: Brian Trammell <[email-masked]>
Date: Mon, 26 May 2014 12:00:33 +0200

> If we could shave those off, we'd be at 37 even with AO. It wouldn't make the option space any bigger for the future, but it would get us out of the current situation, and would make a combination…

SACK and WScale are in use by the majority of sources, but (depending on where you're looking from), only about a third of senders ever use the timestamp options (lots of sources; the most recent is CCR April '14, Honda et al "Rekindling Network Protocol Innovation with User-Level Stacks", introduction), largely an effect of OS-level defaults. So an either-or approach for TS or TCP-AO would probably work.

(Doing this would make in-path passive measurement of RTT on mostly-unidirectional flows basically impossible, but in a post-BCP188 world we should probably be working on instrumenting endpoints for debugging and monitoring, anyway...)

Cheers,

Brian

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Mon, 26 May 2014 10:06:22 +0000

> -----Original Message-----
> From: tcpm [mailto:[email-masked]] On Behalf Of Eggert, Lars
> Sent: Monday, May 26, 2014 11:38 AM
> To: Joe Touch
> Cc: [email-masked]
> Subject: [tcpm] timestamp options (was Re: New Version Notification for
> draft-touch-tcpm-tcp-edo-01.txt)
> 
> Hi,
> 
> I think it was Mark Allman who questioned a while ago whether the
> benefits of the timestamp option are worth spending 10 bytes in every
> packet. (But I can't find the email now.)

And we have already had a long discussion on late enablement of timestamps. Given that we submitted 1323bis in the meantime, I actually think we could spend some cycles on saving those 10B from SYNs (if there was agreement that it is doable). 

Michael (chair hat off)

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Tue, 27 May 2014 11:36:50 +0000

Well,

(my arithmetic for the usual options comes to a total of 19 bytes, not 11).

For late TS, we would still need at least 2 Bytes during <SYN>/<SYN,ACK> to allow that capability.

However, as the useful values of Window Scale only cover 0..14 [1], one approach to save SYN option space would be to negotiate up to 4 binary (on/off) plus WS in a new single 3-byte option.

The following TCP options would lend themselves for that:

SACK permitted 
Late TS (new)
EDO OK (new)

Some recent scans did still reveal that quite a number of paths (port 80) only negotiate MSS, but refrain from doing any other option.

	 2	SACK permitted
	10	timestamp
	 3	window scale
	 4	MSS
	------------------
	19 bytes

Could become
	
	 4	MSS
	 3	WScale (new) + [SAckOk | LateTSOk | EDO ok]
	------------------
	 7 bytes

But

       7    compacted common options
	16	TCP-AO
	20	MPTCP
	--------------------
	43 bytes (still in need of EDO on SYN)

Regards,

Richard Scheffenegger

[1] http://tools.ietf.org/html/draft-ietf-tcpm-1323bis-21#section-2.1

> From: tcpm [mailto:[email-masked]] On Behalf Of Scharf, Michael
> (Michael)
> Sent: Montag, 26. Mai 2014 12:06
> To: Eggert, Lars; Joe Touch
> Cc: [email-masked]
> Subject: Re: [tcpm] timestamp options (was Re: New Version Notification
> for draft-touch-tcpm-tcp-edo-01.txt)
> 
> And we have already had a long discussion on late enablement of
> timestamps. Given that we submitted 1323bis in the meantime, I actually
> think we could spend some cycles on saving those 10B from SYNs (if there
> was agreement that it is doable).
> 
> Michael (chair hat off)
>

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Tue, 27 May 2014 11:48:35 +0000

> For late TS, we would still need at least 2 Bytes during
> <SYN>/<SYN,ACK> to allow that capability.
> 
> However, as the useful values of Window Scale only cover 0..14 [1], one
> approach to save SYN option space would be to negotiate up to 4 binary
> (on/off) plus WS in a new single 3-byte option.

I might be wrong, but I've thought so far that "replacing" the window scale option would be pretty difficult. We already had reports of middleboxes that parse this option because they try to understand rwnd (or they do not parse the option even if they should, or they have entirely broken logic to mess up TCP...). I think one example are firewalls that try to detect whether segments are in-window. Another example could be PEPs that try to tune rwnd (if they still exist in the wild).

My understanding so far was that a solution only for timestamps would be the most low-hanging fruit.

Michael

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Tue, 27 May 2014 11:56:02 +0000

Hi Michael,

Well, if it's just for Timestamps, just use an "invalid" length of 2 as a "Late TS permitted". 

Maintaining the same option value and this would be rather straight forward. 

Personally, I'd like to combine this with a change in semantics, if SACK is negotiated on the same session as well, to allow more efficient loss recovery (lost retransmission detection during a loss recovery episode). I should have a unpublished draft sitting around here about that aspect.

The TS option handling seems to be rather generous by middleboxes, in general. 

Richard Scheffenegger

> Sent: Dienstag, 27. Mai 2014 13:49
> To: Scheffenegger, Richard; Eggert, Lars; Joe Touch
> Cc: [email-masked]
> > However, as the useful values of Window Scale only cover 0..14 [1],
> > one approach to save SYN option space would be to negotiate up to 4
> > binary
> I might be wrong, but I've thought so far that "replacing" the window
> scale option would be pretty difficult. We already had reports of
> middleboxes that parse this option because they try to understand rwnd (or
> they do not parse the option even if they should, or they have entirely
> broken logic to mess up TCP...). I think one example are firewalls that
> try to detect whether segments are in-window. Another example could be
> PEPs that try to tune rwnd (if they still exist in the wild).
>

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Tue, 27 May 2014 12:12:17 +0000

> Maintaining the same option value and this would be rather straight

Interesting proposal. But actually we are not really short of option numbers. If we get e.g. 8 bytes in a significant number of SYNs (which is our real bottleneck), I think we could also spend a new option codepoint.

> Personally, I'd like to combine this with a change in semantics, if
> SACK is negotiated on the same session as well, to allow more efficient
> loss recovery (lost retransmission detection during a loss recovery
> episode). I should have a unpublished draft sitting around here about
> that aspect.

A change in semantics would also be a reason for a new codepoint, imho.

Michael

> > My understanding so far was that a solution only for timestamps would
> be
> > the most low-hanging fruit.
> >
> > Michael
> >

---

From: Yoshifumi Nishida <[email-masked]>
Date: Tue, 27 May 2014 11:36:48 -0700

Hi folks,

I've submitted a proposal for this a while ago:
http://tools.ietf.org/html/draft-nishida-tcpm-apaws-00
Although it's expired now, I will submit a new version within a couple of
weeks or so.
The late negotiation is also discussed in the draft.
As it will change the semantics of TS, it presumes to use new code point
(or use timestamp field in SYN). But, I'm happy to discuss on this point.

One tricky thing for RFC1323 is that it uses TS for two purposes:
timestamping and PAWS and one purpose of the draft is to separate them.
I appreciate any feedback on the draft.

Thanks,
--
Yoshi

On Tue, May 27, 2014 at 5:12 AM, Scharf, Michael (Michael) <
[email-masked]> wrote:

> Interesting proposal. But actually we are not really short of option
> numbers. If we get e.g. 8 bytes in a significant number of SYNs (which is
> our real bottleneck), I think we could also spend a new option codepoint.
>

---

From: Joe Touch <[email-masked]>
Date: Tue, 27 May 2014 13:04:19 -0700

Hi, all,

Late negotiation has too many problems, as have been repeatedly raised 
on this list:

	- prevents use of the timestamp to help address PAWS for
	the initial SYN

	- a new connection that either avoids TS in SYN or
	uses just "use TS" (two-byte) flag can't know when
	to engage the timestamp and what initial value to use

Joe

On 5/27/2014 11:36 AM, Yoshifumi Nishida wrote:
> Although it's expired now, I will submit a new version within a couple
> of weeks or so.
> On Tue, May 27, 2014 at 5:12 AM, Scharf, Michael (Michael)
> <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>      > Well, if it's just for Timestamps, just use an "invalid" length
>     of 2 as
>      > a "Late TS permitted".
>      >
>     numbers. If we get e.g. 8 bytes in a significant number of SYNs
>     (which is our real bottleneck), I think we could also spend a new
>     option codepoint.
>
>      > > From: tcpm [mailto:[email-masked]
>     <mailto:[email-masked]>] On Behalf Of Scharf,
>      > Michael
>      > > (Michael)
>      > > Cc: [email-masked] <mailto:[email-masked]>
>      > > [email-masked] <mailto:[email-masked]>
>     [email-masked] <mailto:[email-masked]>

---

From: Yoshifumi Nishida <[email-masked]>
Date: Tue, 27 May 2014 23:06:35 -0700

Hi Joe,

I'm wondering if you presume that using the existing code point for late
negotiation.
Suppose If we use new code point, do we still have problems you mentioned
below?

Thanks,
--
Yoshi

On Tue, May 27, 2014 at 1:04 PM, Joe Touch <[email-masked]> wrote:

> Hi, all,
>
> Late negotiation has too many problems, as have been repeatedly raised on
> this list:
>
>>      > The TS option handling seems to be rather generous by middleboxes,
>> in
>>      > general.
>>      >
>>      > Richard Scheffenegger
>>      >
>>      > > broken logic to mess up TCP...). I think one example are
>> firewalls
>>      > that
>>      > > try to detect whether segments are in-window. Another example
>> could
>>      > be

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Wed, 28 May 2014 08:17:29 +0000

Hi Joe,

> 	- prevents use of the timestamp to help address PAWS for

This one is genuine. However, given the prevalence of non-TS TCP sesssions and no wide-spread havoc caused by NOT having PAWS on the SYN, it appears to me that the impact is manageable. (Of course it would be nicer to have PAWS on SYN, but as long as SYN option space is limited...)

> 	to engage the timestamp and what initial value to use

Well, as this would be a new mechanism, there should be some broad guidance given; I would assume that the latest onset - if late TS is negotiated for - would be when either direction has sent at most 2^31 - (2^16 << Window.scale) bytes; and a receiver that sees the full TS must then start to reflect TS as usual, independent of its own relative offset into the session. And never relinquish from sending TS even when some packets - ie. retransmissions, or due to middleboxes on a changed path - not contain a TS from the other side.

As for the semantic change, I propose that when the session has SACK enabled as well as late TS, then the TS should reflect the TS of the segment triggering the ACK (we can discuss what should happen during in-sequence delayed ACK processing though; the relevant aspect for me is during loss - SACK recovery - to not mask the timestamps of retransmitted packets as per 1323bis; For legacy RTTM processing, the presence of a SACK option is indicative enough to NOT do normal RTTM processing - although a sender could choose to do so particularly for the loss recovery episodes. Having the (unique) Timestamp/SACK block combinations during loss recovery would allow more efficient, faster and also packet conserving recovery of lost retransmissions - unlike the current method of last resort, RTO.

Best regards,

Richard Scheffenegger

> From: Joe Touch [mailto:[email-masked]]
> Sent: Dienstag, 27. Mai 2014 22:04
> To: Yoshifumi Nishida; Scharf, Michael (Michael)
> Cc: Scheffenegger, Richard; Eggert, Lars; [email-masked]
> > As it will change the semantics of TS, it presumes to use new code
> > point (or use timestamp field in SYN). But, I'm happy to discuss on this
> point.
> >
> >      > Maintaining the same option value and this would be rather
> straight
> >      > forward.
> >
> >      > The TS option handling seems to be rather generous by
> middleboxes, in
> >      > general.
> >      >
> >      > Richard Scheffenegger
> >      >
> >      > > middleboxes that parse this option because they try to
> understand
> >      > rwnd (or

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 28 May 2014 18:13:24 +0100

Joe,

>lost, and sent along alternate paths.

Of course.

But as I suggested initially, we can standardise the protocol so that 
an upgraded host synthesises shared fate. Eg. a 2-octet TCP option on 
the D-type SYN that says "Hold for x [ms] to wait for the 
supplementary C-type SYN, where x is a lot less than the usual time 
you hold SYN connection state (e.g. x=2). If SYN C hasn't arrived by 
then, continue without it, and discard it if it arrives."

And if the C-SYN arrives first, hold it for y [ms] waiting for the 
corresponding D-SYN. Where for example y=2 as well.

>         http://www.isi.edu/touch/pubs/infocomm99/infocomm99-web/

Unless we can think of a way for the C-SYN to be discarded by a 
legacy TCP stack (but not a middlebox), I'm assuming we need to get 
the C-SYN up to the legacy app-layer before discard... So, I was 
assuming they use different source ports (otherwise, as you say, a 
legacy TCP stack could reset the D SYN connection if it arrived second)

The ISN on the C-SYN is redundant - we might be able to think of 
another use for that field.

On an upgraded host, max concurrent connections would only be 
slightly impacted, because of the v small timeout of C-SYNs.

The semantics of the option-space-extension option would be to only 
hold the C-type SYN for a timeout and only the D-type SYN creates 
full connection state (I'm deferring SYN cookie behaviour to tomorrow 
for now - this is a straw man).

Admittedly, the number of concurrent connections a /legacy/ host can 
support could reduce by up to half (if all remote TCP clients are 
sending the new option). But they have the choice of upgrading to the 
new stack to stop wasting their memory.

>What happens when they arrive out of order? What happens when you 
>get D but not yet C? How long do you wait for C?

See above.
The timeouts might be standardised, or they might be declared in the 
option as a hint (for the host to ignore if it is under stress).

>This is the problem with dual-stack approaches - new endpoints 
>penalize legacy endpoints if there's a stall, and undermine new 
>endpoints if they don't.

Have I satisfied you that this can be solved sufficiently? Bearing in 
mind the gain, it's reasonable to have to accept some pain.

>Yes, that's another problem.

Fairly straightforward to solve using standard techniques.

>resegmentation is also widespread (though evil, IMO).

Well, resegmentation isn't usually a problem on a SYN anyway.

We can't improve on the general pain caused by resegmentation. I was 
only talking about the delta pain that my strawman would suffer, 
where a resegmenting function sees data on my SYN and doesn't 
understand my strawman, so it won't patch up the damage in my 
strawman protocol that it would have patched up by altering sequence 
numbers in a regular single SYN with a payload.

I think this is a corner case.

>Won't work through current NATs, which won't recalculate the 
>checksum properly.

I'm building on the general idea of using something invalid, not 
using the checksum idea specifically.

>Again, please don't assume back-to-back means anything.

See above.
Actually, even tho order isn'[t guaranteed, it is important to get 
them in the right order to optimise performance (much as TCP doesn't 
assume perfect order, but it goes smoothest with reasonable ordering).

>>middlebox would), e.g. a payload that is invalid for the application

No. You can design a payload that has headers that a NAT will ignore 
but an end-host will have to process. E.g. HTTP connection control 
headers newly defined for this protocol that would be ignored by 
NATs, without any other HTTP behaviour, so a legacy host does nothing 
at the app layer.

>>would be vulnerable to meddleboxes). Not insurmountable.

Still?

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 28 May 2014 18:16:02 +0100

Joe,

I don't think this sufficiently answers the question to justify WG 
adoption. You seem to be confirming that this is an academic exercise.

Bob

At 17:55 23/05/2014, Joe Touch wrote:
>Sometimes the answer "you can't have A, but at least here's B" is 
>more than an exercise; it educates the community. By not providing 
>either answer, we have continued to drag this issue around the block 
>for far too long -- and spent far too many cycles in this and other 
>WGs seeking solutions.
>
>The current DO field is 4 bits, with a max value of 15 = 60 bytes 
>for the total header, less 20 for the base TCP header which leaves 
>40 for options.
>

---

From: Joe Touch <[email-masked]>
Date: Wed, 28 May 2014 10:30:55 -0700

> adoption. You seem to be confirming that this is an academic exercise.

You asked for a specific example - one that the MPTCP community has 
raised. It's not a proof that there's "some set" that might overload the 
space; it's based on a *specific* request.

I gave other reasons - including educating the community as to the 
issues. If you want to call that "academic", sure - in part.

But I don't see that as interfering with WG adoption - we do that sort 
of thing all the time (that's the basis of a BCP).

Joe

>> too long -- and spent far too many cycles in this and other WGs
>> seeking solutions.
>>

---

From: Joe Touch <[email-masked]>
Date: Wed, 28 May 2014 10:58:17 -0700

On 5/28/2014 10:13 AM, Bob Briscoe wrote:
> But as I suggested initially, we can standardise the protocol so that an
> upgraded host synthesises shared fate.

Fate is dependent on path.

> Eg. a 2-octet TCP option on the
> D-type SYN that says "Hold for x [ms] to wait for the supplementary
> C-type SYN, where x is a lot less than the usual time you hold SYN
> connection state (e.g. x=2). If SYN C hasn't arrived by then, continue
> without it, and discard it if it arrives."
 >

(C=legacy; D=extended - from your earlier email, for context)

Legacy clients now experience connection delays when contacting upgraded 
servers - why is that acceptable?

FWIW, many endpoints don't hold SYN state at all if they implement SYN 
cookies, so 2ms would end up overloading the server with state much 
larger than it currently keeps.

> Unless we can think of a way for the C-SYN to be discarded by a legacy
> TCP stack (but not a middlebox), I'm assuming we need to get the C-SYN
> up to the legacy app-layer before discard... So, I was assuming they use
> different source ports (otherwise, as you say, a legacy TCP stack could
> reset the D SYN connection if it arrived second)

If they use different source ports, you could be limiting the connection 
rate. It's not enough to just "discard" late D-SYNs; they could be for 
new connections (how do you know the two are paired?), which means you 
need to RST them and keep state for 2MSL.

> The ISN on the C-SYN is redundant - we might be able to think of another
> use for that field.

But it's not redundant if you're talking to a legacy server, so you 
can't do anything with that field.

> On an upgraded host, max concurrent connections would only be slightly
> impacted, because of the v small timeout of C-SYNs.

See above; you need to know how to pair C-SYNs and D-SYNs. Consider that 
legacy hosts and upgraded hosts might be behind the same NAT, at which 
point how do you know the two SYNs are even from the same host?

> The semantics of the option-space-extension option would be to only hold
> the C-type SYN for a timeout and only the D-type SYN creates full
> connection state (I'm deferring SYN cookie behaviour to tomorrow for now
> - this is a straw man).

I don't think you can defer a widely-deployed feature like SYN cookies, 
and that seems to kill this.

> new stack to stop wasting their memory.

Both legacy clients and legacy servers are impacted. We already know 
that servers are overloaded with connections, so halving that number 
seems like a non-starter. As does adding x ms delay for *each* legacy 
client.

>> What happens when they arrive out of order? What happens when you get
>> D but not yet C? How long do you wait for C?
>
> See above.

The pain is the problem. The smaller the timeout, the smaller the pain 
to legacy clients but the lower probability the extension will be useful 
to upgraded clients. The method cuts the rate of connections for legacy 
servers in half - which is *huge*. And it adds to the state overhead of 
the upgraded clients that need to do double the work for *every* connection.

Ultimately, it's roughly equivalent to "try both and shut down the one 
you don't want" -- the dual-stack approach we already know about.

> Fairly straightforward to solve using standard techniques.

I think that warrants more than a claim. You need to explain how you 
avoid false postives through NATs.

> only talking about the delta pain that my strawman would suffer, where a
> resegmenting function sees data on my SYN and doesn't understand my
> strawman, so it won't patch up the damage in my strawman protocol that
> it would have patched up by altering sequence numbers in a regular
> single SYN with a payload.

You're right, in the sense that SYN data would typically be discarded 
anyway at the endpoint if it implemented SYN cookies.

> I think this is a corner case.

It is, but you raised it in a general sense; were you thinking of it in 
terms of just SYNs or other segments?

> I'm building on the general idea of using something invalid, not using
> the checksum idea specifically.

Anything considered invalid by an endpoint might be checked by an 
overzealous midbox.

> Actually, even tho order isn'[t guaranteed, it is important to get them
> in the right order to optimise performance (much as TCP doesn't assume
> perfect order, but it goes smoothest with reasonable ordering).

Right, but I want to emphasize that things emitted back-to-back could 
end up being delivered days later (e.g., when a link goes down, some 
routers just keep the queues there). The impact in those cases needs to 
be considered.

> No. You can design a payload that has headers that a NAT will ignore but
> an end-host will have to process.

You can try, but if an end-host looks at it you can be sure that a 
midbox will do the same - if not today, then soon.

 > E.g. HTTP connection control headers
> newly defined for this protocol that would be ignored by NATs, without
> any other HTTP behaviour, so a legacy host does nothing at the app layer.

Those are rewritten by "helpful" midboxes all the time, FWIW.

>>> would be vulnerable to meddleboxes). Not insurmountable.

If it means halving the performance of legacy servers and delaying 
legacy clients, yes.

Joe

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 30 May 2014 10:55:10 +0100

Joe,

I was asking for an example of something useful that /can/ be done with EDO.

I'm sure you can come up with one. However, you have shown 
conclusively that the example in the draft (SACK + MPTCP + TCP-AO), 
when added to the widespread basic options, /cannot/ be done with 
EDO, because it needs 7 more bytes of options than a SYN allows.

Bob

At 18:30 28/05/2014, Joe Touch wrote:

>raised. It's not a proof that there's "some set" that might overload 
>the space; it's based on a *specific* request.
>
>But I don't see that as interfering with WG adoption - we do that 
>sort of thing all the time (that's the basis of a BCP).
>
>Joe
>

---

From: John Leslie <[email-masked]>
Date: Fri, 30 May 2014 09:50:44 -0400

> EDO, because it needs 7 more bytes of options than a SYN allows.

   The claim that MPTCP needs 20 bytes in the SYN is getting on my
nerves...

   As I read RFC 6824, it seems pretty clear that the initial SYN calls
for 12 bytes, not 20. It's the SYN-ACK that uses 20. Somebody please
correct me if I'm reading this wrong.

---

From: Christoph Paasch <[email-masked]>
Date: Fri, 30 May 2014 16:14:11 +0200

> correct me if I'm reading this wrong.

Indeed, it uses 12 bytes on both the SYN and the SYN/ACK.

Only the third ACK carries 20 bytes.

Cheers,
Christoph

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 30 May 2014 17:42:37 +0100

Joe,

>(C=legacy; D=extended - from your earlier email, for context)

THis is not a good way to describe them, because neither SYN is 
legacy, but even if you did think this way, it would be the other way round
(mnemonic: D = primary *D*ata connection, C = additional *C*ontrol options).

>Legacy clients now experience connection delays when contacting 
>upgraded servers - why is that acceptable?

Nope. A legacy client experiences no extra delay. As I said, both 
non-legacy SYNs carry a flag TCP option that says that the server 
should wait for the other if it hasn't seen it yet. A legacy client 
wouldn't add this option, so there would never be any delay.

Only upgraded clients see any extra delay, and then only if the C-SYN is lost.
In the 'regular' case where the two SYNs both arrive, even upgraded 
clients see no delay. There's only additional delay if the C-SYN is 
lost AND both client and server are upgraded. If the D-SYN is lost, 
there would have been a delay anyway.

>FWIW, many endpoints don't hold SYN state at all if they implement 
>SYN cookies, so 2ms would end up overloading the server with state 
>much larger than it currently keeps.

Let's deal with whether SYN cookies can be supported alongside this 
dual-SYN scheme first...

Because both client and server have to support using a pair of SYNs, 
the server doesn't have to exactly mimic the SYN cookie mechanism, it 
just has to achieve the same outcome as a SYN cookie. So the 
generalised question is: how does a server accept C-SYNs and D-SYNs, 
but also reflect all its connection state back off the client, 
without holding any state itself, while at the same time responding 
to any negotiation requests in the TCP options on both SYNs, so as 
not to delay negotiations.

==SYN Cookie v2==
The high level answer would surely be for the server to use an EDO 
TCP option on the SYN-ACK, and put all the connection state in a 
suitably large TCP option, for the client to reflect back. On the 
SYN-ACK, the server would also include any responses to TCP options 
(on either SYN) that expected responses.

Back to your original question, paraphrasing: "Doesn't holding state 
for 2ms risk more overload?"
The answer comes in two parts:

1) The server reduces y (the time it holds C-SYNs) the more it is 
under pressure. Then if necessary it reduces x (the time it holds 
D-SYNs), until both are zero.

2) This seems to allow a flooding attack to make a server remove the 
SYN extension facility, but we can prevent that... For either SYN, a 
server doesn't just discard it when it reaches the timeout. For SYN-D 
it returns a SYN cookie, and for SYN-C it returns a SYN Cookie v2 
(described eariler). For a genuine client this doesn't mean that an 
attack prevents the use of dual SYNs, it merely defers binding the 
two SYNs together until the next round. This only affects any options 
that depend on being combined with information in the other SYN, 
which should be a corner case.

(...I'm making this up as I go along, but it's still holding water 
pretty well isn't it.)

Perhaps this points to an alternative solution, where SYN cookie v2 
is always used, and the binding between the two SYNs only happens on 
the second round.

>If they use different source ports, you could be limiting the 
>connection rate. It's not enough to just "discard" late D-SYNs; they 
>could be for new connections (how do you know the two are paired?), 
>which means you need to RST them and keep state for 2MSL.

The upgraded listener knows which D-SYN that a C-SYN is paired with as follows:
1) On the D-SYN, the TCP option declaring that it is a D-SYN would 
have to carry a parameter duplicating the src port in the main TCP 
header of the D-SYN (so it would survive NATing, unlike the src port itself).
2) On the C-SYN, the TCP option declaring that it is a C-SYN makes 
the server read the payload of the C-SYN, which contains the original 
value of the source port of the D-SYN.
3) The server can then match the src port inside the C-SYN to the 
original src port of the D-SYN.

Once the server has both SYNs, it can check they are 
cryptographically paired, using other material that would have to be 
included in the C-SYN payload (tbd).

Assuming you really meant discarding late C-SYNs,...

Two cases, both don't need a 2MSL wait:

1) A legacy server doesn't have to RST the connection, because the 
app-layer payload is a valid app-layer header. Normally, of course, 
the server will hold back the payload on a SYN so it won't get to the 
app-layer unless the client responds to the SYN-ACK with an ACK. But 
that's OK, 'cos we don't need the paylkoad to get to the app-layer, 
we just need it to look like it would be valid if it did. The 
upgraded client could complete the 3WHS with a regular ACK, or it 
could respond with a RST. When a TCP server in SYN_SENT state 
receives a RST it returns to LISTEN state, and I believe it 
immediately discards connection state, because it never synchronised 
the connection.

2) If an upgraded host gets the C-SYN late, it knows it's a C-SYN, so 
when it tries to match it to a D-SYN it finds it already gave up 
waiting and went ahead with a connection based on the D-SYN alone. So 
it just silently discards the C-SYN with no need to keep to the TCP 
timeout rules (because it understands that a C-SYN doesn't open a 
connection itself).

>can't do anything with that field.

The ISN can be any number, and legacy TCPs will accept it. I just 
meant the field could be used for a number with some other useful 
meaning, because it never has to be used to synch any TCP sequence space.

>See above; you need to know how to pair C-SYNs and D-SYNs. Consider 
>that legacy hosts and upgraded hosts might be behind the same NAT, 
>at which point how do you know the two SYNs are even from the same host?

The crypto binding info in the C-SYN payload (tbd) would distinguish 
two hosts that a NAT had given the same src IP address. If the NAT 
gives different IP addresses to two connections from the same host, 
that would be a problem. We could get round that using a similar 
trick to that we already used to protect the src port value, but 
perhaps we don't need to, 'cos I think NATs like this are fairly rare (?).

>I don't think you can defer a widely-deployed feature like SYN 
>cookies, and that seems to kill this.

Solved above, I think.

>seems like a non-starter. As does adding x ms delay for *each* legacy client.

Legacy clients are not impacted at all (as explained above).
Legacy servers impact upgraded clients with delay ~2ms only if a 
C-SYN is lost. Zero otherwise.

Legacy servers have to hold more connection state, but half is the 
worst case for the last legacy server on the planet after all clients 
have upgraded.

If fraction f of clients have upgraded, a legacy server's connection 
state for n connections increases to n(1-f) + n(2f) = n(1+f).
Ie. Legacy server connection state increases by fraction f, and at 
any time it can solve this by upgrading to coupled SYNs. Usually 
servers upgrade earlier than clients anyway.

(Assumption: connection durations used by upgraded and legacy clients 
follow a similar distribution.)

>The pain is the problem. The smaller the timeout, the smaller the 
>pain to legacy clients but the lower probability the extension will 
>be useful to upgraded clients.

Solved using the SYN cookie 2 idea above.

>The method cuts the rate of connections for legacy servers in half - 
>which is *huge*.

Nope. It's cut by 1/(1+f) where f is the fraction of upgraded 
clients. And the legacy server can itself upgrade to avoid this completely.

>And it adds to the state overhead of the upgraded clients that need 
>to do double the work for *every* connection.

Accepted. However, state overhead is rarely a concern on clients. 
Anyway, they don't need to use more TCP options on every connection - 
only those connections that need them.

>Ultimately, it's roughly equivalent to "try both and shut down the 
>one you don't want" -- the dual-stack approach we already know about.

That's a bit of a wild dismissal of a scheme that so far seems to 
have pretty useful properties.

>avoid false postives through NATs.

Certainly. But I'm not going to do that now. I have no need for 
bigger TCP options in my day job (or my night job), so I would hope 
someone else would pick this up and specify it fully if they need it. 
I was just irked by your impossibility claim.

I won't believe this dual SYN idea is useful myself until it's fully 
specified, implemented and tested. Bitter experience of trying to 
extend major protocols like IP has taught me that one little niggle 
can ruin years of work, even if it looked promising until the last moment.

However, I still don't give up, or discourage others by burning time 
writing RFCs that claim it's impossible.

>You're right, in the sense that SYN data would typically be 
>discarded anyway at the endpoint if it implemented SYN cookies.
>
>It is, but you raised it in a general sense; were you thinking of it 
>in terms of just SYNs or other segments?

It was a general point that I figured might affect any scheme that 
enables TCP options that might want to refer to byte offsets, when 
those bytes are in a different packet. For the scheme that has 
developed over the course of this email thread, I don't think it's a 
concern now.

>Anything considered invalid by an endpoint might be checked by an 

True. But in this case, the client can put something perfectly valid 
in the app-layer payload, because it can even RST the connection to 
stop the payload getting to the app.

Theoretically, we could even encode the extended TCP options in HTML 
as the argument to an HTTP PUT, so that if an HTTP server did ever 
parse them, it might even display a prettily rendered list of 
extended TCP options :)

>Right, but I want to emphasize that things emitted back-to-back 
>could end up being delivered days later (e.g., when a link goes 
>down, some routers just keep the queues there). The impact in those 
>cases needs to be considered.

Considered, certainly.

>midbox will do the same - if not today, then soon.

Agree in general. But see above re HTML & PUT.

>Those are rewritten by "helpful" midboxes all the time, FWIW.

Again, see HTML-PUT commment.

>If it means halving the performance of legacy servers and delaying 

Not so. See above.

Regards

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 30 May 2014 18:13:26 +0100

John,

We needed to lose 7B and we've lost 8B.
So (SACK + MPTCP + TCP-AO + widespread basic options, incl 
TCPtimestamp) do fit in a SYN, therefore the example in the draft is valid.

However, as everyone has pointed out, jubilation may be short-lived. 
For instance, even now, people are working on a lower latency MPTCP 
that will need more option space.

Thx.

Bob

---

From: Joe Touch <[email-masked]>
Date: Fri, 30 May 2014 10:27:45 -0700

I'll update the doc accordingly.

I had also been thinking of SACK not being as limited in its use of 
space, nor MPTCP.

Joe

On 5/30/2014 10:13 AM, Bob Briscoe wrote:
> So (SACK + MPTCP + TCP-AO + widespread basic options, incl TCPtimestamp)
> do fit in a SYN, therefore the example in the draft is valid.
>
> However, as everyone has pointed out, jubilation may be short-lived. For
> instance, even now, people are working on a lower latency MPTCP that
> will need more option space.
>
> Thx.
>
>
> Bob
>
>

---

From: Olivier Bonaventure <[email-masked]>
Date: Fri, 30 May 2014 19:42:22 +0200

Bob,
...
> agree on the problem we are trying to solve first. Boring, but

I was offline this week and am now catching up the email thread. I agree 
that starting from this draft would be an excellent idea to document in 
an information draft the existing solutions and their advantages and 
drawbacks. With this documentation, we could then build another solution 
that works efficiently.

Olivier

---

From: Olivier Bonaventure <[email-masked]>
Date: Fri, 30 May 2014 19:53:27 +0200

David, Joe, Bob,

>> What is impossible is to expand the option space in the initial SYN
>> packet and maintain 100% backwards compatibility with the following
>> restrictions:

Let me make two rough proposals that could be worth being explored or at 
least documented.

1. The TCP option space is limited, but not the IPv6 destination options 
in the IPv6 header. One possibility to extend the TCP option space, 
notably (but not only) in the SYN would be to place part of the TCP 
option inside an IPv6 destination option. Given that the IPv6 
destination option is only limited by the IPv6 packet size, we are safe 
for the long term and the solution seems to be clean.

  This approach has two limitations :

  - It is a violation of the layering principle since the receiving host 
needs to provide the content of the IPv6 destination option to the TCP 
stack.

  - It only works with IPv6 and does not work with IPv4. In the short 
term, IPv4 support is mandatory, but in the long term IPv6 support is 
clearly more important.

2. TCP assumes that the client proposes the TCP extensions and the 
server selects the ones that it supports. In practice, measurements show 
that the TCP stack running on servers is usually upgraded after the 
stack running on clients. It might be interesting to allow the client to 
insert in the SYN a special option that indicates that it would accept 
new options proposed in the SYN+ACK (or perhaps later). These options 
proposed in the SYN+ACK would be confirmed in the third ACK.

If one of these solutions does not seems completely crazy, I can write a 
more detailed proposal to see how it would work in practice

Olivier

---

From: Joe Touch <[email-masked]>
Date: Fri, 30 May 2014 13:34:55 -0700

Hi, Bob,

Let's get back to the core, in a simpler fashion, so other can follow it.

I stand by my "there's no way to extend the space in the initial SYN", 
but you've convinced me there *might* be a way to provide extended space 
that can occur during the first phase of the TWHS. I think the dual-SYN 
approach still isn't viable, but I've outlined an alternative below 
that's similar but doesn't have the same baggage, IMO.

Again, I'm still concerned by what midboxes might do to this...

What do others think??

Joe

For quick review, here's what I understand:

		dso = dual-syn option
			dso-D = data
			dso-C = control
		conn_id = identifier to link the two SYNs together
		extra_opt = options that didn't fit in legacy SYN
		fit_opt = options that do fit in the legacy SYN

	new endpoint sends
		port A SYN + dso-D + conn_id + fit_opt
		port B SYN + dso-C + conn_id + extra_opt

			legacy endpoint sends back two connections:
				port A SYN-ACK + fit_opt
				port B SYN-ACK + ??
				(it's interpretation of extra_opt)
			new endpoint responds:
				port A ACK (established)
				port B RST

			Notes about legacy servers:
				- they do twice the work on SYNs
				- they might keep twice the state
				(if not using cookies)
				- they might clean state if the RST
				is received, but that state might
				persist indefinitely (until the next
				connection, depending on timeouts, etc.)

			-----

			new endpoint sends back one connection:
				port A SYN=ACK + dso-D + ....
			
			Notes:
				- can stall when dso-D SYN arrives
				before dso-C SYN, up to some limit
				- twice the work on SYNs (or more)

Here's what I was assuming, though admittedly it's not documented (yet):

	- no significant impact on TCP connection rate for
	legacy servers

	- no significant impact on TCP connection rate for
	legacy clients

	- impact dominated by processing the extended option space
	for extended clients

	- impact dominated by processing the extended option space
	for extended servers

	- compatible with typical TCP processing optimizations,
	notably SYN cookies
		you did provide a potential way forward for these

	- capable of successfully traversing typical NATs

Your approach has the following properties:

	- halves the server connection rate for updated servers
	from legacy clients when this option is in use

	- lowers (to some extent, if not halves) the client
	connection rate of updated clients to all servers
	when this option is in use

	- halves (roughly) the server rate for all servers
	when this option is in use

It also:

	- doubles the number of SYNs in the network

	- susceptible to lack of fate-sharing problems, e.g.,
	if the two SYNs experience different firewall configurations

	- reduces the space available for fit_opt due to the need
	for the conn_id even in the fall-back D-SYN, which means
	less option space in the SYNs for fall-back connections

	the conn_id which may need to be very large because it
	needs to be unique per source port and source IP address
	because that information is lost during NAT translation

	- requires the ISNs to be related (see RFC6528 - if there's
	a rule to generate it, there will be code to validate that
	rule, and eventually a BCP to encourage that validation -
	typically from the same RFC author)

I agree that you have proposed potentially viable ways to deal with the 
SYN cookie, and that RST state is not an issue.

However, there are too many problems with this, IMO, to call it viable.

> That's a bit of a wild dismissal of a scheme that so far seems to have
> pretty useful properties.

I said it was roughly equivalent - if you call that "wild dismissal", so 
be it.

Here's another trick that might clean up the above a little:

	aso - after SYN option

	FBP - front bumper packet (best I could do on names today)
		a packet with a sequence number BEFORE the ISN of
		the SYN

	new endpoint sends:

		SYN + aso + fix_opt
		FBP + aso + extra_opt + seqno-outsidewindow

			legacy endpoint sends back one connections:
				SYN-ACK + fix_opt

				if seg arrives before SYN, legacy
				will send a RST - which the new endpoint
				can ignore

				if seg arrives after SYN, it'll be
				discarded as out of window

			----

			new endpoint sends back one connection:

				SYN-ACK + options + ....

			a) if FBP arrives before SYN, it will generate
			a RST - which the new endpoint will ignore

			b) if FPB arrives with the SYN, they can be
			processed together

				the SYN-ACK can include responses to
				the extra_opts in addition to the
				fix_opts, and says "FBP received"

			c) if FPB arrives after the SYN:

				SYN-ACK proceeds, but sends
				back "wait for option response".

				at this point, the source re-sends FBP
				until an ACK is received that indicates
				"FBP received", or times-out as with
				any connection that doesn't finish TWHS

			I'm still thinking as to whether the ACK number
			might indicate whether FBP has been received,
			e.g., send back ACK = ISN -1 until FBP,
			at which point it sends back ISN. The connection
			proceeds when ISN is ACK'd.

This is cleaner as follows:

	- no need for conn_id coordination

	- no need for conn_id to consume option space for fall-back

	- avoids double-load for legacy servers

	- no problem with fate-sharing

	- traverses a NAT just fine

Upgraded servers still need to wait for the 'seg', but they could get 
that retransmitted if necessary.

It causes additional processing for legacy endpoints, but not 
necessarily in a way that decreases the rate because it's not trying to 
start a second connection; the RSTs sent back are lightweight, and the 
additional ACK.

----

---

From: Joe Touch <[email-masked]>
Date: Fri, 30 May 2014 14:11:03 -0700

Hi, Oliver,

On 5/30/2014 10:53 AM, Olivier Bonaventure wrote:
...
> in the IPv6 header.

IP option space would not be protected by TCP-AO, and likely not by 
whatever the TCPUSEC WG creates (e.g.,my proposal is here:
http://tools.ietf.org/html/draft-touch-tcp-ao-encrypt-01)

Additionally, DPI, NAT processing, etc. are all impacted by IPv6 
options, so they're generally discouraged AFAIR.

Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 30 May 2014 14:11:35 -0700

> I was asking for an example of something useful that /can/ be done with

The obvious one is larger amounts of SACK feedback and space for MPTCP 
information.

Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 30 May 2014 17:45:48 -0700

Hi, all,

Some additional information below about "another trick" I proposed, 
inspired by Bob's dual-SYN mechanism, courtesy a few long discussions 
today with Ted Faber.

I'll be glad to offer this as a potential solution in the doc.

Joe

On 5/30/2014 1:34 PM, Joe Touch wrote:
...
> Here's another trick that might clean up the above a little:

FWIW, I had explained it below as being based on sending out-of-window 
data; Ted pointed out that I had been assuming that the FBP ACK bit 
wasn't set - which means the sequence number might be more usefully 
matched to that of the SYN.

See below...

>      aso - after SYN option

		length = 2 (just a flag)
		length = 3 (or 4) indicating the length of the
			FBP expected

>          a packet
		ISN = same as the associated SYN
		ACK = cleared (i.e., a data packet NOT part of
		synchronized connection - see why that's useful below)

>          FBP + aso + extra_opt

		extra_opt in the data field
		total length < min MTU (576 for IPv4, 1280 for IPv6)
		again ACK bit is zero

>                  if seg arrives before SYN,

			it is silently dropped, because
			the ACK bit is clear (this is
			explicit in RFC793)

>                  if seg arrives after SYN,

			it is silently dropped, because
			the ACK bit is clear (this is also
			explicit in RFC793)

>              a) if FBP arrives before SYN,

			it can be silently dropped, but
			it's probably useful for new endpoints
			to hold onto these (without action)
			for a while; they can be silently
			discarded if there are too many
			(which will just result in a
			retransmission and an extra RTT)

>              might indicate whether FBP has been received,

There are a few ways to handle this, but IMO the best is to have:

		SYN-ACK aso with length=2 means "waiting for FBP"

		SYN-ACK aso with length=3 (or 4) could mean
		"got the FBP", or might even indicate how
		many bytes the FBP extension contained

Note - the SYN-ACK and all subsequent segments can assume that aso == 
edo, i.e., they can use the same EDO as spec'd in version 01 of this draft.

> that retransmitted if necessary.

And there's a small amount of additional processing to discard the FBP 
at legacy endpoints, but silently discarding one packet per connection 
doesn't seem like a huge effort to me.

Note - there's no special RST processing, based on Ted's observation 
about "data without ACK" being considered "data sent in the 
unsynchronzed state" -- something legacy TCP explicitly silently discards.

-----

---

From: Joe Touch <[email-masked]>
Date: Fri, 30 May 2014 17:58:09 -0700

> I'm wondering if you presume that using the existing code point for late

I had.

> Suppose If we use new code point, do we still have problems you
> mentioned below?

It depends:

- I don't think it matters vs. using the existing codepoint with a 
different length (e.g., as a flag in the SYN); they are equivalent in 
being uniquely identifiable.

- I don't like the idea of setting new TCP flags or changing the meaning 
of flags in the middle of a connection. That undermines the notion of a 
connection IMO.

To do so, the source would have to keep track of the last byte of the 
segment in which the change occurs, and wait for a confirmation in the 
ACK of that byte; ditto for the other end.

They'd need to resend things if lost *with the same flags* to indicate 
the change.

I.e., you'd need to implement a new TWHS per option that changes state. 
That's a mess. And I don't even want to think about what happens when 
you have multiple options that change state at the same time.

IMO, it's not worth the trouble.

Joe

>
> Thanks,
> --
> Yoshi
>
> On Tue, May 27, 2014 at 1:04 PM, Joe Touch <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     Hi, all,
>
>     Late negotiation has too many problems, as have been repeatedly
>     raised on this list:
>
>         http://tools.ietf.org/html/__draft-nishida-tcpm-apaws-00
>         <http://tools.ietf.org/html/draft-nishida-tcpm-apaws-00>
>         <[email-masked]
>         <mailto:[email-masked]>
>         <mailto:[email-masked]
>         <mailto:[email-masked]>>> wrote:
>
>               > Maintaining the same option value and this would be
>         rather straight
>               > forward.
>
>               > SACK is negotiated on the same session as well, to allow
>         more
>              efficient
>         <mailto:[email-masked]>
>              <mailto:[email-masked]
>         <mailto:[email-masked]>>__] On Behalf Of Scharf,
>               > Michael
>               > > (Michael)
>         <mailto:[email-masked] <mailto:[email-masked]>>
>               > > for draft-touch-tcpm-tcp-edo-01.__txt)
>               > >
>               > > > one approach to save SYN option space would be to
>         negotiate
>              up to 4
>               > > > binary
>               > > I might be wrong, but I've thought so far that
>         "replacing" the
>              window
>               > > they do not parse the option even if they should, or
>         they have
>               > entirely
>               > > broken logic to mess up TCP...). I think one example
>         are firewalls
>               > that
>               > > try to detect whether segments are in-window. Another
>         example could
>               > be
>               > > My understanding so far was that a solution only for
>         timestamps
>              would
>               > be
>               > > _________________________________________________
>               > > tcpm mailing list
>         <mailto:[email-masked] <mailto:[email-masked]>>
>
>               > > https://www.ietf.org/mailman/__listinfo/tcpm
>         <https://www.ietf.org/mailman/listinfo/tcpm>
>
>              _________________________________________________
>              tcpm mailing list
>         [email-masked] <mailto:[email-masked]> <mailto:[email-masked]
>         <mailto:[email-masked]>>
>         https://www.ietf.org/mailman/__listinfo/tcpm
>         <https://www.ietf.org/mailman/listinfo/tcpm>
>
>
>

---

From: Joe Touch <[email-masked]>
Date: Fri, 30 May 2014 18:07:29 -0700

On 5/28/2014 1:17 AM, Scheffenegger, Richard wrote:
> This one is genuine. However, given the prevalence of non-TS TCP
> sessions and no wide-spread havoc caused by NOT having PAWS on the SYN,
> it appears to me that the impact is manageable. (Of course it would be
> nicer to have PAWS on SYN, but as long as SYN option space is limited...)

I don't know the stats, but I'd prefer if initial connections were 
PAWS-protected.

> Well, as this would be a new mechanism, there should be some broad
> guidance given; I would assume that the latest onset - if late TS is
> negotiated for - would be when either direction has sent at most 2^31 -
> (2^16 << Window.scale) bytes; and a receiver that sees the full TS must
> then start to reflect TS as usual, independent of its own relative
> offset into the session. And never relinquish from sending TS even when
> some packets - ie. retransmissions, or due to middleboxes on a changed
> path - not contain a TS from the other side.

I had thought of a similar mechanism, e.g., where there would be an 
extended sequence number (ESN), and the upper bits would be assumed zero 
until actually needed.

The advantage to that would be:
	ESN flag would occur in all segments
		length = 2 until needed
		length can increase as additional bytes/words are needed

i.e., the meaning and use of the flag would never change; it would be 
synchronized at the start in both directions. It would just be 
compressed until actually needed. If you have a recent connection that 
already wrapped, you might start with ESN of length = 4 rather than 
length = 2 (0 in both directions), so you can sync non-zero ESNs.

The only benefit would be that this is a lot shorter than timestamps - 
i.e., it'd be 4 rather than 10 bytes.

(is this "Extended Sequence Number" option worth explaining in detail, 
e.g., in a separate draft?)

---

Doing real TS's later in the connection requires a new sort of 
synchronization in the middle of the connection - which, as I noted in 
my other post, seems to require a TWHS mechanism, and I think that's too 
cumbersome.

> As for the semantic change, I propose that when the session has SACK
> enabled as well as late TS, then the TS should reflect the TS of the
> segment triggering the ACK (we can discuss what should happen during
> in-sequence delayed ACK processing though; the relevant aspect for me is
> during loss - SACK recovery - to not mask the timestamps of
> retransmitted packets as per 1323bis; For legacy RTTM processing, the
> presence of a SACK option is indicative enough to NOT do normal RTTM
> processing - although a sender could choose to do so particularly for
> the loss recovery episodes. Having the (unique) Timestamp/SACK block
> combinations during loss recovery would allow more efficient, faster and
> also packet conserving recovery of lost retransmissions - unlike the
> current method of last resort, RTO.

Perhaps, but I don't see this paragraph as being related to late TS.

Joe

---

From: John Leslie <[email-masked]>
Date: Fri, 30 May 2014 21:46:12 -0400

> inspired by Bob's dual-SYN mechanism, courtesy a few long discussions 

   This _is- as interesting idea...

> I'll be glad to offer this as a potential solution in the doc.

   I'd _much_ rather this be detailed in a separate document. The edo
proposal is reasonably easy to understand, while this contains multiple
layers of complexity.

   I _don't_ claim to understand it all -- if we try to publish this
in an RFC, a _lot_ of work will be needed to clarify everything...

> matched to that of the SYN.

   I see you now give the "second" packet the same sequence number.
I feel a need to describe the FBP more exactly. I'm guessing it's the
same port pair (but I very much dislike guessing!). Obviously it's
the same IP-address pair (but can we really depend upon middleboxes
to translate both packets the same?). I won't venture a guess as to
whether it has the SYN bit set (but thanks for saying the ACK bit
is clear). Et cetera...

>>     aso - after SYN option

   This is an option within the Options field of the first SYN, right?

> 		length = 3 (or 4) indicating the length of the

   What does length==2 tell us about the FBP?

> 		synchronized connection - see why that's useful below)

   Does this have SYN set?

   How does the sender of the FBP learn what the receiver of the FBP
took it to mean? (Assuming, of course, it was received at all...)

>>     new endpoint sends:

   What's a "new endpoint"?

> 		total length < min MTU (576 for IPv4, 1280 for IPv6)

   I get very nervous about using the data field before the handshake
is complete.

>>             legacy endpoint sends back one connections:

   Presumably the legacy endpoint ignores the FBP... but what happens
if only the FBP arrives (the initial SYN being lost or mangled by a
middlebox)?

>>                 if seg arrives before SYN,

   Am I supposed to assume "seg" here means FBP?

> 			retransmission and an extra RTT)

   Your hands are waving...

>>                 fix_opts, and says "FBP received"

   How would it say "FBP received"?

>>                 back "wait for option response".

   How would it send that?

>>                 any connection that doesn't finish TWHS

   This sounds like the timeout wouldn't be rare -- a middlebox might
pretty automatically drop the FBP, depending...

>>             might indicate whether FBP has been received,

   That sounds strange...

> 		many bytes the FBP extension contained

   That's limiting...

> edo, i.e., they can use the same EDO as spec'd in version 01 of this draft.

   Please! No!

   (Option numbers aren't that scarce.)

---

From: Joe Touch <[email-masked]>
Date: Fri, 30 May 2014 20:58:52 -0700

> in an RFC, a _lot_ of work will be needed to clarify everything...

Sure...

Yes.

> I feel a need to describe the FBP more exactly.

Of course.

> I'm guessing it's the
> same port pair (but I very much dislike guessing!).

Oh, yes - that's the way it ensures fate-sharing with the SYN.

> Obviously it's
> the same IP-address pair

Yes.

> (but can we really depend upon middleboxes
> to translate both packets the same?).

If the FBP comes after the SYN, yes. If not, the FBP will be dropped, 
and retransmitted when the SYN-ACK indicates the FBP hasn't been 
received (for upgraded servers).

 > I won't venture a guess as to

It can't have the SYN bit - that would be two SYNs to the same place, 
and if they have different options that'd be confusing for legacy 
servers. If they had the same options, there'd be no increase in space - 
it's not possible to use the data field of a SYN for options because 
legacy hosts will interpret that as user data.

>     What does length==2 tell us about the FBP?

Here's more specifically what I was thinking:

	Length = 2 (1 byte), Kind = fpb-option-codepoint (1 byte)

or

	Length = 3, Kind = fbp-option-codepoint, fbp-length (1 byte)

Length == 4 just means a longer fbp-length field, or maybe it means:

	Length, Kind, Flags, FBP-length
		(you'll see below where the flags might come in)

Length == 2 just means there's no fbp-length field. That could mean that 
the length of the FBP should be determined from the FBP packet itself 
(using fbp-length operates as a double-check that the FBP packet 
matches); alternately, it could mean that there's no FBP - i.e., that 
this is just the EDO that extends non-SYN packets.

Alternately, we could use different option codepoints for FBP and EDO, 
in which case the FBP option just means "FBP packet should be expected".

>> 		synchronized connection - see why that's useful below)

No. that would be a second SYN, which causes load problems for legacy 
servers.

> took it to mean? (Assuming, of course, it was received at all...)

The ACK that comes back, either in the SYN-ACK (if the FBP has been 
received) or a second ACK after that will have the response to the FBP 
options.

>     What's a "new endpoint"?

An upgraded one, i.e., one that supports this new option...

>     I get very nervous about using the data field before the handshake

It's safe - as per RFC793. This packet is outside the connection because 
the ACK isn't set. That's effectively saying "this packet is for the 
unsynchronized state". RFC793 says all such packets get silently dropped 
(we're talking about a packet with ACK clear, and NO other control flags 
set - no SYN, no RST, no FIN).

> if only the FBP arrives (the initial SYN being lost or mangled by a

They ignore it at all times, except if received in CLOSED at which point 
they send a RST (just like receiving any other packet in that state).

If the SYN is lost, the same thing happens as any time the SYN is lost - 
the client resends the SYN after a timeout, and after some number of 
failed timeouts it gives up.

Upgraded endpoints might keep FBPs for a short time in the hope of 
getting a matching SYN, but if they don't or discard it before it comes, 
the client will resend it when the SYN-ACK indicates its missing.

>     Am I supposed to assume "seg" here means FBP?

Sorry - typed too fast. Yes, here and throughout the rest...

>> 			retransmission and an extra RTT)

Not that hard. Keeping a cache of recently received FBPs is a simple 
thing to do, and the mechanism is robust to that cache either not 
existing or being small.

>     How would it say "FBP received"?

I thought of a few ways - they all end up being equivalent in spirit to 
a flag in an option field in the SYN-ACK, so let's say it's that for now:

	set FBP-received flag in the aso option

>>>                  back "wait for option response".

The FBP-received flag in the SYN-ACK aso option isn't set.

> pretty automatically drop the FBP, depending...

If the FBP comes before the SYN, yes. If not, it shouldn't AFAICT.

>>>              might indicate whether FBP has been received,

Yeah - nevermind that.

>> 		many bytes the FBP extension contained

I don't see why (FBP can indicate 32-bit words, just like the current 
Data Offset field, which means 255 = 1020 bytes of option space in the 
FBP -- note that this is all contained in the data portion of the FBP 
segment).

>     (Option numbers aren't that scarce.)

See above. We could say:
	
	EDO length = 0 in a SYN means no FBP
	EDO length = N in a SYN means FBP with length N

But let's not get hung on that detail either, IMO.

Joe

---

From: Olivier Bonaventure <[email-masked]>
Date: Sat, 31 May 2014 14:57:32 +0200

Joe,
> The obvious one is larger amounts of SACK feedback and space for MPTCP

It would be intersting to look at packet traces to see how often the 
SACK option is not large enough to fit into x bytes. This would give an 
indication of its impact. Has someone already seen such a study in the 
literature ?

Olivier

---

From: Olivier Bonaventure <[email-masked]>
Date: Sat, 31 May 2014 14:55:55 +0200

Joe,
> http://tools.ietf.org/html/draft-touch-tcp-ao-encrypt-01)

If the destination option is used to transport TCP options, then it 
would define how AO would need to take into account the TCP option 
inserted by the sender in the IPv6 option.

> options, so they're generally discouraged AFAIR.

IPv6 options are much cleaner than IPv4 options. IPv4 options are 
discouraged because they impact the forwarding on routers. For IPv6 
options, this is easier since the option type indicates that the option 
should only be processed by the destination. For NATs or DPI, supporting 
TCP options as an IPv6 destination option or as any other proposal would 
be the same problem

Olivier

---

From: Bob Briscoe <[email-masked]>
Date: Sat, 31 May 2014 19:19:53 +0100

Joe,

Thx for consolidating this thread. I've given it a new subject line.

1) You've silently made an important alteration to the proposed 
protocol. You've put the extra-options directly in the TCP option 
space of the C-SYN, not within the payload. This creates problems:
a) it limits additional options to another 40B, beyond which we will 
need a third SYN, then a fourth.
b) it perpetuates the deployment problem that every newly defined TCP 
option will have: when deployed in year y, there will be a new crop 
of middleboxes that only forward options defined up to year y-1.

I had deliberately squirrelled the options away in the app layer to:
a) provide expansion space for options on a SYN, limited only by the 
max segment size
b) reduce the chances that middleboxes will alter the extra options, 
given there is a higher bar to altering the payload.
c) allow for future structured ways to make extra options invisible 
and/or immutable by middleboxes.

Yes, your altered proposal is cleaner. However, don't imagine I 
didn't think of this. I did and I deliberately didn't do it this way. 
We have a choice:
         clean and vulnerable vs. messy but robust.

I'm not wedded to using port 80 and http headers, but this is perhaps 
the most pragmatic approach. It will be really unorthodox to define 
such a protocol I know. We would have to say something like

         "The dst port of the C-SYN MUST be 80, and the payload MUST start
         with the constant magic_token, where
         magic_token = 'PUT / HTTP/1.1<CRLF>Connection : DSO<CRLF><CRLF>'
         "

I'm sorry if even thinking about this makes you feel dirty :|

Other suggestions for inner protocols are welcome, including 
tunnelled protocols, as long as middleboxes widely forward them, 
given their dst port.

2) The main problem with your notation is it doesn't say /where/ the 
info is placed.
I've added notation as follows:
TCP(base header [TCP options [APP(header[payload])]])

And for the record I've made the if-else logic clearer.

Where I've made more than clarifying edits inline, I've described 
them and tagged them with [BB].

At 21:34 30/05/2014, Joe Touch wrote:
>I stand by my "there's no way to extend the space in the initial 
>SYN", but you've convinced me there *might* be a way to provide 
>extended space that can occur during the first phase of the TWHS. I 
>think the dual-SYN approach still isn't viable, but I've outlined an 
>alternative below that's similar but doesn't have the same baggage, IMO.
>
         new client endpoint sends
                 TCP(port A SYN [dso-D(conn_id) + fit_opt] )
                 TCP(port B SYN [dso-C [APP(headers [conn_id + 
extra_opt] ) ] ] )

[BB]: i/APP(headers...)/

                         if (legacy server endpoint) { sends back two 
connections:
                                 TCP(port A SYN-ACK [fit_opt] )
                                 TCP(port B SYN-ACK [??] )
>                                 (it's interpretation of extra_opt)
                                         new client endpoint responds:
                                         TCP(port A ACK) (established)
                                         TCP(port B RST)

>                                 connection, depending on timeouts, etc.)
                         } elif (new server endpoint) { sends back 
one connection:
                                 TCP(port A SYN-ACK [edo + fit_opt + 
extra_opt] )

[BB]: s/dso-d/edo/
                                         new client endpoint responds:
                                         TCP(port A ACK) (established)

>                                 - twice the work on SYNs (or more)
                         }

>Your approach has the following properties:

The 3 bullets below are not useful ways to describe performance 
impact. They selectively describe whichever gives the most 
pessimistic picture out of:
a) either the instantaneous performance change at the moment of connection
b) or the worst-case long-run performance impact

They don't describe the average long-run performance impact, which is 
important for sizing machines.

Worse, the instantaneous performance impact is only significant when 
a machine's SYN processing time is large relative to the e2e delay, 
which would be a highly unusual scenario on public networks (even in 
scenarios such as intra-data-centre, it's hard to reduce e2e delay to 
approach SYN processing time, but you could for intra-machine connections).

>         from legacy clients when this option is in use

Eh? The long-run server connection rate will be fractionally 
decreased due to updated clients using extra options (which is your 
third case below), but the instantaneous server connection rate seen 
by a legacy client is unchanged, because it only sends one SYN.

>         when this option is in use

Nope. All long-run server rates are reduced by 1/(1+e), where e is 
the fraction of connections using extra options.

>         - doubles the number of SYNs in the network

Nope. The number of SYNs in the network is inflated by e where e is 
the fraction of connections using extra options.

>         if the two SYNs experience different firewall configurations

Nope. It's fairer to say it's potentially susceptible to second-order 
fate-sharing problems like your firewall example (the first-order 
fate sharing problems have been addressed).

>         less option space in the SYNs for fall-back connections

Yup.

>         because that information is lost during NAT translation

Given many NATs will typically make the src IPs of both SYNs the 
same, I suggest a larger conn_id should be a fall-back option for the 
client, not a default.

Even if the src IPs of both SYNs are different once they reach the 
server, the high end bits will invariably be the same. So the max 
size of the contents of the DSO TCP option can be 6B, and the server 
can take the rest of conn_id from the higher bits of the src IP addr 
of each SYN. This is a variant of the idea in <draft-wing-nat-reveal-option>.

In fact, the server doesn't even need a small conn_id for clients 
that know they are not behind a NAT and that want more option space 
in the D-SYN - then the server could use the src port & src IP for the conn_id.

To summarise, these options could be distinguished by the length 
field of the dual-syn option.
Length = 2B                             => conn_id = src netaddr + src port
Length = 6B = 2B +4B conn_id_short      => conn_id = src netaddr + 
conn_id_short
Length = 8B = 2B +6B conn_id_long       => conn_id = hsb(src netaddr) 
+ conn_id_long

Given there have been numerous other attempts to reveal a connection 
ID that is preserved through middleboxes [RFC6967], rather than 
defining a dual-syn option that carries a conn_id, we might want to 
design a TCP connection ID option with a flag to say whether it is 
also part of a dual SYN pair or not.

Where
* hsb(src netaddr) is the netaddr with the lowest 16 bits truncated
* src netaddr is the network address (IPv4, IPv6, or any other 
network protocol)

To reduce latency, a host could use the default short_conn_id for all 
connections at first, then:
- if it finds that DSO persistently doesn't work it falls back to the 
long_conn_id for all connections
- it occasionally tests the short and zero options to see if it can 
use shorter DSO options.

>         typically from the same RFC author)

Eh? The ISNs can and should be independent. To be robust against 
middleboxes that rewrite sequence numbers, we must not required ISNs 
to be related.

>I agree that you have proposed potentially viable ways to deal with 
>the SYN cookie, and that RST state is not an issue.

A feature that I think it's fair to add:
         - Good chance of passing through app-layer middleboxes that forward
         unrecognised TCP options unchanged, but not those that discard them.

>However, there are too many problems with this, IMO, to call it viable.

Once your over-pessimistic analyses of the performance impact are 
corrected, and my ideas to reduce the size of the conn_id are taken 
into account, it's a different story.

But it's up to the WG to decide whether this is worth taking further. 
Not just you or I.

>Here's another trick that might clean up the above a little:

<snip - I'll respond separately to your later updates on this ASO 
idea, with ACK=0>

Cheers

Bob

---

From: Joe Touch <[email-masked]>
Date: Sat, 31 May 2014 12:20:54 -0700

Hi, Bob,

On 5/31/2014 11:19 AM, Bob Briscoe wrote:
> protocol. You've put the extra-options directly in the TCP option space
> of the C-SYN, not within the payload.

I hadn't assumed that; I was describing only which SYN carried which 
options, not where they were. As you do, I was assuming the C-SYN 
options were in the data space.

To be clear, in my variant - using data outside the sync'd connection 
(with ACK clear), the option space is similarly in the data portion of 
the 'extra' segment.

> Yes, your altered proposal is cleaner. However, don't imagine I didn't
> think of this. I did and I deliberately didn't do it this way.We have a
> choice:
> the most pragmatic approach.

That's fraught with a whole slew of other problems, notably the 
middleboxes that currently try to translate, proxy, or otherwise modify 
anything they see as port 80.

Further, that would limit the number of pending connections.

> It will be really unorthodox to define such
> a protocol I know. We would have to say something like
>
> Other suggestions for inner protocols are welcome, including tunnelled
> protocols, as long as middleboxes widely forward them, given their dst
> port.

I don't see what this gets us other than potential interactions. There's 
no reason to use HTTP to format TCP options, and it would necessitate 
new encodings and new processing for existing options that might end up 
in the extended space.

> 2) The main problem with your notation is it doesn't say /where/ the

EDO says the options are at the head of the data segment, just with a 
longer pointer - the same way as current options.

For extending the SYN using a separate segment - whether a C-SYN or 
unsync'd data segment - the data similarly comes at the head of the 
data, just that there's no user data permitted after it.

> Where I've made more than clarifying edits inline, I've described them
> and tagged them with [BB].
>
>> but you've convinced me there *might* be a way to provide extended
>> space that can occur during the first phase of the TWHS. I think the
>> dual-SYN approach still isn't viable, but I've outlined an alternative
>> below that's similar but doesn't have the same baggage, IMO.
>>

IMO, the conn_ID in the C-SYN should similarly be in the regular option 
space, but I don't think it matters.

>>                                 (it's interpretation of extra_opt)

I'm assuming EDO after the SYN, which means the SYN-ACK has as much 
space as it wants for options, as would every other segment.

>                          } elif (new server endpoint) { sends back one
> connection:

Sure. At this point we're basically in EDO-land.

> The 3 bullets below are not useful ways to describe performance impact.
> They selectively describe whichever gives the most pessimistic picture
> out of:
> Worse, the instantaneous performance impact is only significant when a
> machine's SYN processing time is large relative to the e2e delay, which
> would be a highly unusual scenario on public networks (even in scenarios
> such as intra-data-centre, it's hard to reduce e2e delay to approach SYN
> processing time, but you could for intra-machine connections).

SYN processing is a known load - and potential overload - on machines. 
That's why it's reasonable to focus on it.

> Eh? The long-run server connection rate will be fractionally decreased
> due to updated clients using extra options (which is your third case
> below), but the instantaneous server connection rate seen by a legacy
> client is unchanged, because it only sends one SYN.

Agreed. I was doing too many of these at once...

> Nope. All long-run server rates are reduced by 1/(1+e), where e is the
> fraction of connections using extra options.

the rate for connections when this option is in use is halved - I didn't 
see that the sentence above could be parsed the other way. the overall 
rate depends on the fraction of connections using the option.

> Nope. The number of SYNs in the network is inflated by e where e is the
> fraction of connections using extra options.

You keep bringing this up - I don't like a solution that should be used 
only a little. If it works, we should assume it works everywhere all the 
time.

> fate-sharing problems like your firewall example (the first-order fate
> sharing problems have been addressed).

I have no idea why you think firewalls are a second-order fate-sharing, 
or what that even means. Fate sharing means fate is shared. When fate is 
not shared by the SYNs, this method has problems. There's no mechanism 
to recover the missing D-SYN or C-SYN.

> Given many NATs will typically make the src IPs of both SYNs the same, I
> suggest a larger conn_id should be a fall-back option for the client,
> not a default.

Multiple NATs are quite common - carrier grade NATs in the net, and user 
NATs at the home WIFI point.

Any multiple-NAT, or any path through even a single carrier-grade NAT 
will end up potentially un-doing or over-doing source IP overlap, 
creating false positives and false negatives that break this mechanism.

That's really the crux - fate sharing and the need to bind the two SYNs 
- that make this untenable, IMO.

> server, the high end bits will invariably be the same.

You can't know this - the two SYNs could have gone through different 
numbers of NATs or different NATs entirely, so the bits could change 
arbitrarily.

> So the max size
> of the contents of the DSO TCP option can be 6B, and the server can take
> the rest of conn_id from the higher bits of the src IP addr of each SYN.
> This is a variant of the idea in <draft-wing-nat-reveal-option>.

FWIW, you're now in the land where we would have to start explaining the 
privacy and tracking implications of those bits. I'd really like to 
avoid that.

> In fact, the server doesn't even need a small conn_id for clients that
> know they are not behind a NAT and that want more option space in the
> D-SYN - then the server could use the src port & src IP for the conn_id.
>
> To summarise, these options could be distinguished by the length field
> of the dual-syn option.
> Length = 8B = 2B +6B conn_id_long       => conn_id = hsb(src netaddr) +
> conn_id_long
>
> Given there have been numerous other attempts to reveal a connection ID
> that is preserved through middleboxes [RFC6967], rather than defining a
> dual-syn option that carries a conn_id, we might want to design a TCP
> connection ID option with a flag to say whether it is also part of a
> dual SYN pair or not.
>
> Where
> * src netaddr is the network address (IPv4, IPv6, or any other network
> protocol)
>
> - it occasionally tests the short and zero options to see if it can use
> shorter DSO options.
>
>
> middleboxes that rewrite sequence numbers, we must not required ISNs to
> be related.
>
>
> corrected, and my ideas to reduce the size of the conn_id are taken into
> account, it's a different story.
>

Agreed.

> <snip - I'll respond separately to your later updates on this ASO idea,
> with ACK=0>
>
> Cheers
>
>
> Bob
>
>

---

From: Joe Touch <[email-masked]>
Date: Sat, 31 May 2014 13:42:25 -0700

On May 31, 2014, at 5:55 AM, Olivier Bonaventure <[email-masked]> wrote:

> If the destination option is used to transport TCP options,
> then it would define how AO would need to take into account the TCP option inserted by the sender in the IPv6 option.

TCP doesn't necessarily get the entire IP header. It's not practical to claim that TCP-AO can protect something it might never see - and certainly might not have control over once it's inserted in the IPv6 header.

> IPv6 options are much cleaner than IPv4 options. IPv4 options are discouraged because they impact the forwarding on routers. For IPv6 options, this is easier since the option type indicates that the option should only be processed by the destination. For NATs or DPI, supporting TCP options as an IPv6 destination option or as any other proposal would be the same problem

IPv6 options push the location of TCP port numbers to places that need to be computed, rather than being known in advance - and wired into hardware.

Anything that looks at ports might toss something with *any* IP options - v4 or v6 - to slow-path.

Joe

---

From: Joe Touch <[email-masked]>
Date: Sat, 31 May 2014 13:43:38 -0700

On May 31, 2014, at 5:57 AM, Olivier Bonaventure <[email-masked]> wrote:

> It would be intersting to look at packet traces to see how often the SACK option is not large enough to fit into x bytes.

You'll never see that in a trace - what you'll see is SACK options that fit, of course.

> This would give an indication of its impact. Has someone already seen such a study in the literature ?

I don't know, though there are armies that mine traces for all sorts of stuff, so I wouldn't be surprised.

Joe

---

From: Bob Briscoe <[email-masked]>
Date: Sat, 31 May 2014 22:13:13 +0100

Joe,

Hope it's ok to have changed the subject line, with tcpm still in cc.

I'm afraid I'm not as excited about ACK=0 as you are. It's certainly 
cleaner than anything we've come up with so far.

However, I see the goal as finding a way to send a supplement to a 
SYN that is invalid in some way to legacy TCP servers, but likely to 
appear valid to many/most middleboxes. I suspect many middleboxes and 
firewalls will discard the ACK=0 segment.

When assessing the DSO scheme, you were adamant that anything invalid 
to an endpoint would always eventually become invalid to middleboxes. 
In the excitement of finding a nice clean way of doing ASO, that 
critique seems to have suddenly become unimportant.

Whatever, there may be a place for both solutions:
a) ASO (ACK=0) for paths where a NAT might be the worst middlebox on the path
b) DSO for paths with stateful firewalls, TCP normalisers etc.

The cellular world is certainly more like (b). Whether there are many 
parts of the public Internet left like (a) is unclear. (b) seems to 
have become the norm for most paths.

More inline...

At 01:45 31/05/2014, Joe Touch wrote:
>inspired by Bob's dual-SYN mechanism, courtesy a few long 
>discussions today with Ted Faber.
>
>FWIW, I had explained it below as being based on sending 
>out-of-window data; Ted pointed out that I had been assuming that 
>the FBP ACK bit wasn't set - which means the sequence number might 
>be more usefully matched to that of the SYN.
>
>See below...
>
>>      aso - after SYN option
>
>                 length = 2 (just a flag)

For the avoidance of doubt,
I assume these two packets have the same src port.
and I assume the FBP has SYN=0.

Coincidentally, a couple of weeks ago, when I was looking for places 
to find more bits in the TCP header, I noted that during an 
established connection ACK=0 is never used, so I looked into setting 
ACK=0 and overloading the ack_number field.

>                         the ACK bit is clear (this is also

Two important questions:
1) are most/all legacy TCP implementations faithful to the spec?
2) if a TCP endpoint is meant to drop SYN=0, ACK=0, then many 
middleboxes surely will.

Both these will need experimental testing.

Nit: I wouldn't exactly say RFC793 is clear. You have to follow 2 
pages of quite imprecise descriptive logic, starting from p66. But, 
yes it is eventually fairly unambiguous. Paraphrasing, it says:

SEGMENT ARRIVES
         ...
         if state = SYN-SENT
                 1. Check ACK bit
                         if ACK = 1
                                 do stuff
                 2. Check RST bit
                         if RST = 1
                                 if ACK OK enter CLOSED state
                                 elif no ACK, drop
                 3. Check security & precedence
                         Do stuff
                 4. Check SYN bit
                         Should only get here if ACK OK, or no ACK and no RST
                         if SYN = 1
                                 do stuff
                 5. if SYN = 0 and RST = 0
                         drop

>>              b) if FPB arrives with the SYN, they can be

By 'arrives with' I assume you mean 'within some time duration'.

>>                  any connection that doesn't finish TWHS

There's a dilemma whether the server:
* prioritises latency and goes ahead without the extra options,
* or prioritise completeness and blocks until the extra options arrive.

I would take the view that if the FBP is late there's a chance it got 
snarled up in a middlebox, and it will never get through no matter 
how many times it's retransmitted.

Rather than make the choice between latency and completeness at 
protocol design time, we need a flag in either scheme (on the DSO or 
ASO option) for the client to say which it wants. Then:
* the client can choose latency if it knows the extra_opts are not critical.
* otherwise it can choose completeness, and if it doesn't work after 
a retransmit or two, the client won't block for ever; it can work out 
the compromise set of options that fit in a single SYN but still 'work'.

>Note - the SYN-ACK and all subsequent segments can assume that aso 
>== edo, i.e., they can use the same EDO as spec'd in version 01 of this draft.
>

With a typical legacy server that reflects SYN cookies, less activity 
is doubled:
* my scheme: sends out two SYN cookies
* your scheme: sends out one SYN cookie and drops one packet, 
probably after a long chain of logic, because the FBP is an unusual packet.

>>      - no problem with fate-sharing

To be as consistently pessimistic as you were with my scheme, I think 
you mean that you have created machinery to synthesise fate sharing 
between a connection and an invalid segment, and it may not work in 
cases that you may not have thought of.

>>      - traverses a NAT just fine

As above, surely TCP normalising middleboxes and incoming stateful 
firewalls at the server end are likely to discard the FBP.

>>that retransmitted if necessary.

See discussion of latency vs completeness dilemma above.

>And there's a small amount of additional processing to discard the 
>FBP at legacy endpoints, but silently discarding one packet per 
>connection doesn't seem like a huge effort to me.

Despite this being an exceptional drop (see earlier), this is still a 
reasonably fair statement.

>unsynchronzed state" -- something legacy TCP explicitly silently discards.

...at least in theory.

Cheers

Bob

---

From: Joe Touch <[email-masked]>
Date: Sat, 31 May 2014 15:01:25 -0700

Hi, Bob,

On May 31, 2014, at 2:13 PM, Bob Briscoe <[email-masked]> wrote:

> I'm afraid I'm not as excited about ACK=0 as you are. It's certainly cleaner than anything we've come up with so far.
> 
> However, I see the goal as finding a way to send a supplement to a SYN that is invalid in some way to legacy TCP servers, but likely to appear valid to many/most middleboxes. I suspect many middleboxes and firewalls will discard the ACK=0 segment.
> 
> When assessing the DSO scheme, you were adamant that anything invalid to an endpoint would always eventually become invalid to middleboxes. In the excitement of finding a nice clean way of doing ASO, that critique seems to have suddenly become unimportant.

I worry about checksums in particular - which get recalculated, so using a different checksum means we're sure to fail through a middlebox that doesn't recalculate it properly (if it's stored in a new place), or that won't validate it (if it's in the current checksum field).

I can't say that middleboxes don't check for ACK=0 packets, but it seems a lot of work to do unless they're perceived to be an issue. I doubt they look at the ACK at all unless the SYN or FIN is set.

> b) DSO for paths with stateful firewalls, TCP normalisers etc.

DSO is better through stateful firewalls only when the FBP packet precedes the SYN. Keep in mind that the FBP can also be sent multiple times after the SYN anyway, with small (1ms) delays; only the first will end up being used, and loss of all of them would still be recoverable.

> The cellular world is certainly more like (b).

Cellular uses GGN (carrier-grade NATs), which could make DSO SYN pairing more difficult.

> Whether there are many parts of the public Internet left like (a) is unclear. (b) seems to have become the norm for most paths.

If we're counting numbers, there are a *lot* more hosts behind NATs. The ones behind CGNs probably swamp all others combined.

>> Some additional information below about "another trick" I proposed, inspired by Bob's dual-SYN mechanism, courtesy a few long discussions today with Ted Faber.
>> 
>> FWIW, I had explained it below as being based on sending out-of-window data; Ted pointed out that I had been assuming that the FBP ACK bit wasn't set - which means the sequence number might be more usefully matched to that of the SYN.
>> 
>> See below...
>> 
>>>     aso - after SYN option
>> 
>>                length = 2 (just a flag)

Same IP addresses, same ports, *because* they're associated with the same connection.

> and I assume the FBP has SYN=0.

Yes. All control bits are 0, including ACK.

> Coincidentally, a couple of weeks ago, when I was looking for places to find more bits in the TCP header, I noted that during an established connection ACK=0 is never used, so I looked into setting ACK=0 and overloading the ack_number field.
> 
> 
>>>     new endpoint sends:
>>> 
>>>         SYN + aso + fix_opt
>>>         FBP + aso + extra_opt
>> 

That's something I'll take a look at. 

> 2) if a TCP endpoint is meant to drop SYN=0, ACK=0, then many middleboxes surely will.

Middleboxes drop things whose checksum fails, or when a packet comes in for a connection that hasn't been established. They don't go out of their way to do a lot of other work AFAICT.

> Both these will need experimental testing.

Certainly.

> Nit: I wouldn't exactly say RFC793 is clear. You have to follow 2 pages of quite imprecise descriptive logic, starting from p66. But, yes it is eventually fairly unambiguous. Paraphrasing, it says:
> 
> SEGMENT ARRIVES
>        ...
>        if state = SYN-SENT
>                1. Check ACK bit
>                        if ACK = 1
>                                do stuff
>                2. Check RST bit
>                        if RST = 1

In CLOSED, it says to send a RST (like any other packet for a non-connection).

In LISTEN, it says:

        Any other control or text-bearing segment (not containing SYN)
        must have an ACK and thus would be discarded by the ACK
        processing.  An incoming RST segment could not be valid, since
        it could not have been sent in response to anything sent by this
        incarnation of the connection.  So you are unlikely to get here,
(where 'here' is a segment with ACK=0 and no other control bits set)
        but if you do, drop the segment, and return.

In SYN-SENT, it says:

      fifth, if neither of the SYN or RST bits is set then drop the
      segment and return.

In all other states, it says:

      if the ACK bit is off drop the segment and return

If that's not direct and explicit, I don't know what is.

> By 'arrives with' I assume you mean 'within some time duration'.

Yes - I was thinking of the case where the FBP is either cached (as per (a)), or is in the segment queue at the time the SYN is being processed.

> * or prioritise completeness and blocks until the extra options arrive.

There's no dilemma - if the SYN says "FBP is coming", then it means that an upgraded server MUST wait for the FBP.

I.e., the FBP is paired with the SYN-ACK, and the handshake doesn't complete at the client - the client should NOT send the final ACK of the TWHS - until the SYN-ACK is received *and* confirmation that the FBP has been received. That can happen together inside the SYN-ACK if known, or the SYN-ACK would say "FBP missing" and the client would retransmit the FBP *until* the server confirms it (it would send an option confirmation, not an ACK).

> I would take the view that if the FBP is late there's a chance it got snarled up in a middlebox, and it will never get through no matter how many times it's retransmitted.

The client can timeout, and can retry without the extended SYN space in that case.

> Rather than make the choice between latency and completeness at protocol design time, we need a flag in either scheme (on the DSO or ASO option) for the client to say which it wants. Then:
> * otherwise it can choose completeness, and if it doesn't work after a retransmit or two, the client won't block for ever; it can work out the compromise set of options that fit in a single SYN but still 'work'.

A flag that says "do what a legacy server would do if you have to wait X ms" seems fine to me too.

>> Note - the SYN-ACK and all subsequent segments can assume that aso == edo, i.e., they can use the same EDO as spec'd in version 01 of this draft.
>> 
> With a typical legacy server that reflects SYN cookies, less activity is doubled:
> * your scheme: sends out one SYN cookie and drops one packet, probably after a long chain of logic, because the FBP is an unusual packet.

The activity involved in parsing a packet is small compared to that of setting up a SYN cookie or creating TCB state (where SYN cookies aren't used).

> To be as consistently pessimistic as you were with my scheme, I think you mean that you have created machinery to synthesise fate sharing between a connection and an invalid segment, and it may not work in cases that you may not have thought of.

Your scheme uses two different SYNs on different ports - that's a recipe for fates NOT being shared.

Mine uses two segments on the same addresses with the same ports - if they're not fate-shared, then neither will the rest of the TCP connection be, and TCP won't work (if that fate matters, e.g., through different NATs).

> As above, surely TCP normalising middleboxes and incoming stateful firewalls at the server end are likely to discard the FBP.

I don't think so, but I agree we'll have to try. This is a completely different situation, IMO, than expecting boxes that *already* are known to validate TCP checksums to do otherwise. In this case, we don't *know* the solution won't work from the start.

> See discussion of latency vs completeness dilemma above.

Latency vs. completeness is true for all variants that use multiple packets - that's one reason I don't like that aspect of any of these approaches.

>> And there's a small amount of additional processing to discard the FBP at legacy endpoints, but silently discarding one packet per connection doesn't seem like a huge effort to me.
> 
> Despite this being an exceptional drop (see earlier), this is still a reasonably fair statement.

And I'm glad to concede that we don't know the cost on the code cache, etc. of exercising a dusty, dark path of processing. 

>> Note - there's no special RST processing, based on Ted's observation about "data without ACK" being considered "data sent in the unsynchronzed state" -- something legacy TCP explicitly silently discards.
> 
> ...at least in theory.

And in requirements. Yes, I'll see if I can find out what BSD and Linux do - though I have more hope of BSD being correct than Linux.

Joe

---

From: Bob Briscoe <[email-masked]>
Date: Sat, 31 May 2014 23:34:56 +0100

Joe,

>options were in the data space.

OK.

>To be clear, in my variant - using data outside the sync'd 
>connection (with ACK clear), the option space is similarly in the 
>data portion of the 'extra' segment.

Yup, that was understood already.

>middleboxes that currently try to translate, proxy, or otherwise 
>modify anything they see as port 80.
>
>I don't see what this gets us other than potential interactions. 
>There's no reason to use HTTP to format TCP options, and it would 
>necessitate new encodings and new processing for existing options 
>that might end up in the extended space.

To be clear, I would put the general requirement as :

"       Pick a port number, then make sure the app-layer headers at 
the start of the TCP payload appear valid for that port number, so 
that middleboxes will forward it, then include the extra TCP options 
as the body of this faked up app-layer protocol.
"

Then the dilemma becomes: pick a number, any number as long as it's 
80. The alternative is often to be blocked. Particularly if you 
connect from inside a company that has bought one of those myopic Web 
gateways, or you connect to a cellular network, or... you just 
connect to the Internet via a box from a vendor beginning with a 
letter in the alphabet,... or a digit.

>EDO says the options are at the head of the data segment, just with 
>a longer pointer - the same way as current options.
>

I understood all this. I just meant that the notation you introduced 
in your email didn't have the facility to specify at which layer 
fields were placed.

>IMO, the conn_ID in the C-SYN should similarly be in the regular 
>option space, but I don't think it matters.
>
>>[BB]: i/APP(headers...)/
>>
>SYN processing is a known load - and potential overload - on 
>machines. That's why it's reasonable to focus on it.

But we should primarily think in terms of SYN cookie processing for 
legacy hosts, and the equivalent for upgraded hosts. We should design 
thia as the norm, not the exception. Possibly the /only/ way to 
implement this new form of SYN processing.

Then, for the dual-SYN scheme, the upgraded host doesn't have a 
normal SYN cookie processing load with a C-SYN, so we can no longer 
rely on known-load formulae.

>the rate for connections when this option is in use is halved - I 
>didn't see that the sentence above could be parsed the other way. 
>the overall rate depends on the fraction of connections using the option.
>
>>>It also:
>>>
>You keep bringing this up - I don't like a solution that should be 
>used only a little. If it works, we should assume it works 
>everywhere all the time.

It doesn't make sense to predict that all connections will always use 
MSS, timestamp, SACK, window-scale, MPTCP, TCP-AO, new option X, new 
option y, etc etc.

For a start, less than 1% of connections contain enough packets to 
make MPTCP worth starting.

The word option means optional.

>I have no idea why you think firewalls are a second-order 
>fate-sharing, or what that even means. Fate sharing means fate is 
>shared. When fate is not shared by the SYNs, this method has 
>problems. There's no mechanism to recover the missing D-SYN or C-SYN.

What I mean by first-order fate-sharing problems is those obvious 
cases (reordering, drop) that we have addressed by resynthesising 
fate sharing.

There is certainly a possibility that two SYNs between the same 
endpoints will pass through different firewalls, but it is small in 
the Internet as a whole. There is a further possibility that these 
two firewalls will be configured differently, but it is small. There 
is a further possibility that the differences in configuration will 
be significant enough to distinguish the behaviour wrt the two SYNs, 
but it is small. When three small probabilities are multiplied 
together, you get a really small probability. This is what I call a 
second-order fate-sharing problem.

You must surely see that it is inconsistent to say:
* "this method has problems" about the DSO idea, based on such a corner case
* "no problem with fate-sharing" about the ASO idea, when all you 
mean is that the FBP will follow the same path as the SYN, and they 
will therefore likely pass through the same devices.

In the ASO proposal, splitting the options on a SYN over two packets, 
one of which is considered invalid by TCP, will cause the FBP to be 
discarded by numerous TCP normalisers and stateful firewalls. That is 
a fate sharing problem of the first order. Likely to be on a much 
larger scale than the 'second-order' firewall problem above.

>Multiple NATs are quite common - carrier grade NATs in the net, and 
>user NATs at the home WIFI point.
>
>Any multiple-NAT, or any path through even a single carrier-grade 
>NAT will end up potentially un-doing or over-doing source IP 
>overlap, creating false positives and false negatives that break 
>this mechanism.
>
>That's really the crux - fate sharing and the need to bind the two 
>SYNs - that make this untenable, IMO.
>

That's what 'invariably' means. I'm not claiming this works 100%. I 
am saying I expect it to work sufficiently to be worth trying it out 
to see if I'm right. If it doesn't work, on some paths, the fall-back 
has been described - the client has to rely on the TCP options it can 
fit in a single SYN.

>FWIW, you're now in the land where we would have to start explaining 
>the privacy and tracking implications of those bits. I'd really like 
>to avoid that.

Nope. I think you're reading this the wrong way round. I'm not 
revealing private IDs in TCP option fields. I'm reducing the size of 
the (random) conn_id in cases where the /public/ src port or the 
upper bits of the /public/ src IP have sufficient entropy to be used 
to construct the rest of the conn_id.

I was acknowledging Dan Wing 'cos he gave us the idea. But it's a 
/variant/ that doesn't reveal anything private.

I'm going offline, possibly until Tue now.

Regards

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Sat, 31 May 2014 23:45:34 +0100

Joe,

Nothing further to add on this thread.
I do think you're seeing the ASO scheme thru rose-coloured spectacles.

Cheers

Bob

At 23:01 31/05/2014, Joe Touch wrote:
> > I'm afraid I'm not as excited about ACK=0 as you are. It's 
> certainly cleaner than anything we've come up with so far.
> >
> > However, I see the goal as finding a way to send a supplement to 
> a SYN that is invalid in some way to legacy TCP servers, but likely 
> to appear valid to many/most middleboxes. I suspect many 
> middleboxes and firewalls will discard the ACK=0 segment.
> >
> > When assessing the DSO scheme, you were adamant that anything 
> invalid to an endpoint would always eventually become invalid to 
> middleboxes. In the excitement of finding a nice clean way of doing 
> ASO, that critique seems to have suddenly become unimportant.
>
>I worry about checksums in particular - which get recalculated, so 
>using a different checksum means we're sure to fail through a 
>middlebox that doesn't recalculate it properly (if it's stored in a 
>new place), or that won't validate it (if it's in the current checksum field).
>
>I can't say that middleboxes don't check for ACK=0 packets, but it 
>seems a lot of work to do unless they're perceived to be an issue. I 
>doubt they look at the ACK at all unless the SYN or FIN is set.
>
>DSO is better through stateful firewalls only when the FBP packet 
>precedes the SYN. Keep in mind that the FBP can also be sent 
>multiple times after the SYN anyway, with small (1ms) delays; only 
>the first will end up being used, and loss of all of them would 
>still be recoverable.
>
> > Whether there are many parts of the public Internet left like (a) 
> is unclear. (b) seems to have become the norm for most paths.
>
>If we're counting numbers, there are a *lot* more hosts behind NATs. 
>The ones behind CGNs probably swamp all others combined.
>
> > More inline...
> >
> >> Some additional information below about "another trick" I 
> proposed, inspired by Bob's dual-SYN mechanism, courtesy a few long 
> > Coincidentally, a couple of weeks ago, when I was looking for 
> places to find more bits in the TCP header, I noted that during an 
> established connection ACK=0 is never used, so I looked into 
> setting ACK=0 and overloading the ack_number field.
> >
> >
> >>>     new endpoint sends:
> >>>
> >>>         SYN + aso + fix_opt
> >>>         FBP + aso + extra_opt
> >>
>Middleboxes drop things whose checksum fails, or when a packet comes 
>in for a connection that hasn't been established. They don't go out 
>of their way to do a lot of other work AFAICT.
>
>Yes - I was thinking of the case where the FBP is either cached (as 
>per (a)), or is in the segment queue at the time the SYN is being processed.
>
>There's no dilemma - if the SYN says "FBP is coming", then it means 
>that an upgraded server MUST wait for the FBP.
>
>I.e., the FBP is paired with the SYN-ACK, and the handshake doesn't 
>complete at the client - the client should NOT send the final ACK of 
>the TWHS - until the SYN-ACK is received *and* confirmation that the 
>FBP has been received. That can happen together inside the SYN-ACK 
>if known, or the SYN-ACK would say "FBP missing" and the client 
>would retransmit the FBP *until* the server confirms it (it would 
>send an option confirmation, not an ACK).
>
> > I would take the view that if the FBP is late there's a chance it 
> got snarled up in a middlebox, and it will never get through no 
> matter how many times it's retransmitted.
>
> protocol design time, we need a flag in either scheme (on the DSO 
> or ASO option) for the client to say which it wants. Then:
> > * otherwise it can choose completeness, and if it doesn't work 
> after a retransmit or two, the client won't block for ever; it can 
> work out the compromise set of options that fit in a single SYN but 
> still 'work'.
>
> >> Note - the SYN-ACK and all subsequent segments can assume that 
> aso == edo, i.e., they can use the same EDO as spec'd in version 01 
> of this draft.
> >>
>The activity involved in parsing a packet is small compared to that 
>of setting up a SYN cookie or creating TCB state (where SYN cookies 
>aren't used).
>
> > To be as consistently pessimistic as you were with my scheme, I 
> think you mean that you have created machinery to synthesise fate 
> sharing between a connection and an invalid segment, and it may not 
> work in cases that you may not have thought of.
>
>Mine uses two segments on the same addresses with the same ports - 
>if they're not fate-shared, then neither will the rest of the TCP 
>connection be, and TCP won't work (if that fate matters, e.g., 
>through different NATs).
>
> > As above, surely TCP normalising middleboxes and incoming 
> stateful firewalls at the server end are likely to discard the FBP.
>
>I don't think so, but I agree we'll have to try. This is a 
>completely different situation, IMO, than expecting boxes that 
>*already* are known to validate TCP checksums to do otherwise. In 
>this case, we don't *know* the solution won't work from the start.
>
>Latency vs. completeness is true for all variants that use multiple 
>packets - that's one reason I don't like that aspect of any of these 
>approaches.
>
> >> And there's a small amount of additional processing to discard 
> the FBP at legacy endpoints, but silently discarding one packet per 
>And I'm glad to concede that we don't know the cost on the code 
>cache, etc. of exercising a dusty, dark path of processing.
>
> >> Note - there's no special RST processing, based on Ted's 
> observation about "data without ACK" being considered "data sent in 
> the unsynchronzed state" -- something legacy TCP explicitly silently discards.
> >
> > ...at least in theory.
>
>And in requirements. Yes, I'll see if I can find out what BSD and 
>Linux do - though I have more hope of BSD being correct than Linux.
>
>Joe