Subject: [tcpm] Asymmetric Middleboxes and Extended Data Offset (EDO)

From: Bob Briscoe <[email-masked]>
Date: Thu, 2 Oct 2014 12:50:12 +0100

Joe, Wes,

It is known that new TCP options are removed on x% of paths.
It is also known that y% of paths are asymmteric
Let's assume z% of the TCP-option-removing middleboxes are on the asymmetric part of a path.

(x y and z are left as an exercise for the experimenter. I'm offline at the mo, but I know there are papers that give good estimates of x and y at least).

Also some middleboxes behave asymmetrically, for instance firewalls have stricter rules for traffic arriving from the public side vs the private side. Such a device might remove unknown TCP options in one direction, but not the other.

==Legend==
C               = TCP client
S               = TCP server
EDO             = EDO delivered
!X-EDO-X!       = Middlebox removes EDO TCP option
Extra-opts      = TCP options beyond the Data Offset (preserved)
???!!!  = confused or crashes

==Scenarios==

1. EDO not removed from SYN, but middlebox removes EDO from SYN/ACK 
1.1 Extra Options on SYN/ACK

   C ->                    EDO   -> S
   C <-   !X-EDO-X!   Extra-opts <- S
   C ???!!!

1.2 Extra Options on subsequent segment from server

   C ->                   EDO    -> S
   C <-   !X-EDO-X!              <- S
     ...
   C <-   !X-EDO-X!   Extra-opts <- S
   C ???!!!

2. EDO not removed from SYN or SYN/ACK
2.1 Middlebox removes EDO from subsequent segment from server

   C ->                   EDO    -> S
   C <-      EDO      Extra-opts <- S
     ...
   C <-   !X-EDO-X!   Extra-opts <- S
   C ???!!!

2.2 Middlebox removes EDO from subsequent segment from client

   C ->                   EDO    -> S
   C <-      EDO      Extra-opts <- S
     ...
   C ->   Extra-opts   !X-EDO-X! -> S
                             ???!!! S

==Commentary==

Case #1:
The passive opener (server) thinks that the connection supports EDO, so it sends a SYN/ACK confirming this. But before it reaches the client, a middlebox removes unknown TCP options, but forwards the payload unchanged, including the extra TCP options.

Without the EDO option, the client believes everything beyond the Data Offset in the SYN/ACK is payload, so it passes both the extra TCP options and the actual payload to the app. 

Outcome in both cases under #1: 
        TCP delivers corrupt user-data to the client.

Case #2:
The 3WHS negotiating EDO completes correctly, so both ends correctly think that the connection supports EDO. But later in the connection, a middlebox starts removing unknown TCP options, but continues to forward the payload unchanged. This is probably much less likely than case 1.1, but might be due to a path change, or the behaviour of a middlbox being different for segments with SYN=0.

Again, without the EDO option, the receiving end believes everything beyond the Data Offset is payload, so it passes both the extra TCP options and the actual payload to the app. 

Outcome in case #2.1: 
        TCP delivers corrupt user-data to the client.
Outcome in case #2.2: 
        TCP delivers corrupt user-data to the server.

==Implications==

I found the above problem when I was working through all the possible problems if I was to integrate EDO with draft-briscoe-tcpm-syn-op-sis (to extend TCP option space on non-SYN packets subsequent to extending space on the SYN).

In the light of the above, I won't be integrating EDO with syn-op-sis. I don't think we should introduce anything to TCP with a known risk of delivering corrupt user-data, even if the risk is low. If there were a middlebox behaviour that 'just' blocked the new feature, it would be another matter. But if asymmetry does cause TCP to deliver corrupt data to certain users, the same users will continually experience bizarre and unreliable behaviour - very expensive for customer support people to diagnose.

I shall be adding a section to the syn-op-sis draft to define how syn-op-sis can be used to extend the TCP option space on any packet as well as the SYN (it's fairly obvious). I believe it will be immune to the above problem. (Joe, I'll also add an Appendix for the alternative protocol structure I mentioned in response to you not liking the trailer-based structure.)

As you know, I was amazed when it was proposed that EDO should go forward for proposed standard. I said in the TCPM meeting that it should be at least experimental, because these days any new TCP options need to be tested against a wide range of middlebox behaviours. With the above problems, I think we should revisit the decision even to adopt EDO.

Sorry guys, EDO looked promising, but our job is to ensure TCP continues to be robust. 

Regards

Bob

________________________________________________________________
Bob Briscoe,                                                  BT

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 2 Oct 2014 15:31:12 +0100

Wes,

I agree that combining EDO with generic middlebox hardening would be 
nice. However, it's hard to provide a general-purpose middlebox 
hardening solution for all cases; most generic solutions introduce 
overhead - complexity, processing or extra round-trip latency - and 
each case has different sensitivities to different types of overhead.

For instance, in the design of MPTCP, there's a 3-way check to 
protect against asymmetric option removal 
<http://tools.ietf.org/html/rfc6824#section-2.1>. It's MPTCP-specific 
because MPTCP is perceived as more relevant to longer flows, so the 
extra start-up latency of this 3-way check is less of a concern. 
Also, if EDO used a 3-way check, it would not be able to extend the 
option space on any of the 3WHS segments, which would reduce its usefulness.

Section 6 of the above MPTCP RFC 
<http://tools.ietf.org/html/rfc6824#section-6> discusses asymmetric 
option removal (see Fig 16) and how MPTCP was hardened against it. It 
is also a useful enumeration of other possible middlebox problems 
(and how MPTCP was designed to work round them and to work with them).

==Segment Coalescing==

A colleague (Phil Eardley) has just pointed out that segment 
coalescing could be a problem for EDO. I suspect segments in the 3WHS 
won't get coalesced because there's nothing to coalesce with. But 
then later segments might be coalesced.

If this were the case, it would mean even a 3-way EDO capability 
negotiation would appear to have worked. But then subsequent segments 
could end up with a corrupted coalesced payload, with TCP options 
embedded at regular intervals throughout.

Regards

Bob

At 14:29 02/10/2014, Eddy, Wesley M. (GRC-LCI0)[MTI SYSTEMS, INC.] wrote:
>Hi Bob,  I think there's a generic problem that many options share 
>of detecting middlebox compatibility issues, and that for some 
>subset of them (EDO included) those issues could cause corruption of user data.
>
>This motivates stronger ways to validate segments (e.g. the form of 
>TCP-AO that's compatible with NAT, with the TCP option flag set ... 
>possibly with some new unkeyed algorithm).  It should be done in 
>common and not re-invented on a per-option basis IMHO.
>
>
>
>-----Original Message-----
>From: Bob Briscoe [mailto:[email-masked]]
>Sent: Thursday, October 02, 2014 7:50 AM
>To: Joe Touch; Eddy, Wesley M. (GRC-LCI0)[MTI SYSTEMS, INC.]
>Cc: tcpm IETF list
>Subject: Asymmetric Middleboxes and Extended Data Offset (EDO)
>
>Joe, Wes,
>
>It is also known that y% of paths are asymmteric Let's assume z% of 
>the TCP-option-removing middleboxes are on the asymmetric part of a path.
>
>(x y and z are left as an exercise for the experimenter. I'm offline 
>at the mo, but I know there are papers that give good estimates of x 
>and y at least).
>
>Also some middleboxes behave asymmetrically, for instance firewalls 
>have stricter rules for traffic arriving from the public side vs the 
>private side. Such a device might remove unknown TCP options in one 
>direction, but not the other.
>
>==Legend==
>C               = TCP client
>S               = TCP server
>EDO             = EDO delivered
>The passive opener (server) thinks that the connection supports EDO, 
>so it sends a SYN/ACK confirming this. But before it reaches the 
>client, a middlebox removes unknown TCP options, but forwards the 
>payload unchanged, including the extra TCP options.
>
>Without the EDO option, the client believes everything beyond the 
>Data Offset in the SYN/ACK is payload, so it passes both the extra 
>TCP options and the actual payload to the app.
>
>The 3WHS negotiating EDO completes correctly, so both ends correctly 
>think that the connection supports EDO. But later in the connection, 
>a middlebox starts removing unknown TCP options, but continues to 
>forward the payload unchanged. This is probably much less likely 
>than case 1.1, but might be due to a path change, or the behaviour 
>of a middlbox being different for segments with SYN=0.
>
>Again, without the EDO option, the receiving end believes everything 
>beyond the Data Offset is payload, so it passes both the extra TCP 
>options and the actual payload to the app.
>
>Outcome in case #2.1:
>I found the above problem when I was working through all the 
>possible problems if I was to integrate EDO with 
>draft-briscoe-tcpm-syn-op-sis (to extend TCP option space on non-SYN 
>packets subsequent to extending space on the SYN).
>
>In the light of the above, I won't be integrating EDO with 
>syn-op-sis. I don't think we should introduce anything to TCP with a 
>known risk of delivering corrupt user-data, even if the risk is low. 
>If there were a middlebox behaviour that 'just' blocked the new 
>feature, it would be another matter. But if asymmetry does cause TCP 
>to deliver corrupt data to certain users, the same users will 
>continually experience bizarre and unreliable behaviour - very 
>expensive for customer support people to diagnose.
>
>I shall be adding a section to the syn-op-sis draft to define how 
>syn-op-sis can be used to extend the TCP option space on any packet 
>as well as the SYN (it's fairly obvious). I believe it will be 
>immune to the above problem. (Joe, I'll also add an Appendix for the 
>alternative protocol structure I mentioned in response to you not 
>liking the trailer-based structure.)
>
>As you know, I was amazed when it was proposed that EDO should go 
>forward for proposed standard. I said in the TCPM meeting that it 
>should be at least experimental, because these days any new TCP 
>options need to be tested against a wide range of middlebox 
>behaviours. With the above problems, I think we should revisit the 
>decision even to adopt EDO.
>

---

From: Joe Touch <[email-masked]>
Date: Thu, 02 Oct 2014 08:00:56 -0700

Hi, Bob and Wes (et al.),

First, IMO "option-removing" middleboxes ought to be considered broken
and fixed. Let's not treat this otherwise on that end.

Second, it's trivial to protect against that event by simply:

	a) not using the EDO space until after the SYN/ACK
	b) requiring it on every segment once enabled

(b) makes EDO robust to erroneous middleboxes, regardless of asymmetry.

(a) is required to differentiate asymmetric errors like this and
legitimate legacy servers.

FWIW, I don't think we need to do anything in the final ACK of the TWHS.
We treat it like any other segment - once the option makes it in both
directions, removal of the option effectively kills the packet (and
terminates the connection).

IF this is a significant concern, we can add it to EDO trivially, but
I'd like to know how prevalent this is and whether it can be corrected
by fixing the error first.

However, I don't quite see a way around avoiding SYN/ACK use of any
extension unless we pull the same trick with SYN/ACKs we use with SYNs
(OOB segment or dual-SYN/ACK).

Joe

On 10/2/2014 7:31 AM, Bob Briscoe wrote:
> overhead - complexity, processing or extra round-trip latency - and each
> case has different sensitivities to different types of overhead.
> 
> For instance, in the design of MPTCP, there's a 3-way check to protect
> against asymmetric option removal
> extra start-up latency of this 3-way check is less of a concern. Also,
> if EDO used a 3-way check, it would not be able to extend the option
> space on any of the 3WHS segments, which would reduce its usefulness.
> 
> option removal (see Fig 16) and how MPTCP was hardened against it. It is
> also a useful enumeration of other possible middlebox problems (and how
> MPTCP was designed to work round them and to work with them).
> 
> 
> ==Segment Coalescing==
> 
> A colleague (Phil Eardley) has just pointed out that segment coalescing
> could be a problem for EDO. I suspect segments in the 3WHS won't get
> coalesced because there's nothing to coalesce with. But then later
> segments might be coalesced.
> 
>> Hi Bob,  I think there's a generic problem that many options share of
>> detecting middlebox compatibility issues, and that for some subset of
>> them (EDO included) those issues could cause corruption of user data.
>>
>> Without the EDO option, the client believes everything beyond the Data
>> Offset in the SYN/ACK is payload, so it passes both the extra TCP
>> think that the connection supports EDO. But later in the connection, a
>> middlebox starts removing unknown TCP options, but continues to
>> forward the payload unchanged. This is probably much less likely than
>> case 1.1, but might be due to a path change, or the behaviour of a
>> middlbox being different for segments with SYN=0.
>>
>> I found the above problem when I was working through all the possible
>> problems if I was to integrate EDO with draft-briscoe-tcpm-syn-op-sis
>> (to extend TCP option space on non-SYN packets subsequent to extending
>> space on the SYN).
>>
>> In the light of the above, I won't be integrating EDO with syn-op-sis.
>> I don't think we should introduce anything to TCP with a known risk of
>> delivering corrupt user-data, even if the risk is low. If there were a
>> middlebox behaviour that 'just' blocked the new feature, it would be
>> another matter. But if asymmetry does cause TCP to deliver corrupt
>> data to certain users, the same users will continually experience
>> bizarre and unreliable behaviour - very expensive for customer support
>> people to diagnose.
>>
>> syn-op-sis can be used to extend the TCP option space on any packet as
>> well as the SYN (it's fairly obvious). I believe it will be immune to
>> the above problem. (Joe, I'll also add an Appendix for the alternative
>> protocol structure I mentioned in response to you not liking the
>> trailer-based structure.)
>>

---

From: "Eddy, Wesley M. (GRC-LCI0)[MTI SYSTEMS, INC.]" <[email-masked]>
Date: Thu, 2 Oct 2014 13:29:12 +0000

Hi Bob,  I think there's a generic problem that many options share of detecting middlebox compatibility issues, and that for some subset of them (EDO included) those issues could cause corruption of user data.

This motivates stronger ways to validate segments (e.g. the form of TCP-AO that's compatible with NAT, with the TCP option flag set ... possibly with some new unkeyed algorithm).  It should be done in common and not re-invented on a per-option basis IMHO.

-----Original Message-----
From: Bob Briscoe [mailto:[email-masked]] 
Sent: Thursday, October 02, 2014 7:50 AM
To: Joe Touch; Eddy, Wesley M. (GRC-LCI0)[MTI SYSTEMS, INC.]
Cc: tcpm IETF list
Subject: Asymmetric Middleboxes and Extended Data Offset (EDO)

Joe, Wes,

It is known that new TCP options are removed on x% of paths.
It is also known that y% of paths are asymmteric Let's assume z% of the TCP-option-removing middleboxes are on the asymmetric part of a path.

==Legend==
C               = TCP client
S               = TCP server
EDO             = EDO delivered

==Scenarios==

   C ->                    EDO   -> S

   C ->                   EDO    -> S
   C <-   !X-EDO-X!              <- S
     ...

   C ->                   EDO    -> S
   C <-      EDO      Extra-opts <- S
     ...

   C ->                   EDO    -> S
   C <-      EDO      Extra-opts <- S
     ...

==Commentary==

Case #1:

Case #2:

Outcome in case #2.1: 

==Implications==

Regards

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 2 Oct 2014 17:14:49 +0100

Joe,

>and fixed. Let's not treat this otherwise on that end.

It is our job to make the IETF's protocols robust against attack - at 
least against low cost attacks. If our protocols can be corrupted by 
simple middlebox behaviours that we already know about, they will 
certainly not be robust against deliberate attacks.

One can think of an erroneous middlebox as an attack on users by a 
company intent on profiting while complying with just enough of the 
protocols to make their product saleable.

Saying such behaviour "ought to be fixed" is equivalent to saying 
"self-interest ought to be fixed". It is also similar to saying 
"attackers ought not to be self-interested, so we shouldn't have to 
secure our protocols".

>terminates the connection).

Correct.

>(OOB segment or dual-SYN/ACK).

And what about the second part of my email on segment coalescing? 
That's much tougher, and probably much more prevalent than asymmetric 
middlebox behaviour.

I've had the core of an idea on how we can detect coalescing when 
extending option space beyond the initial SYN. It's based on the 
non-deterministic option structure within the payload that I used in 
syn-op-sis. I'll think it through further before I claim it might 
work (bearing in mind option space extension is beyond my day-job).

Cheers

Bob

---

From: Joe Touch <[email-masked]>
Date: Thu, 02 Oct 2014 09:25:47 -0700

> certainly not be robust against deliberate attacks.

None of our protocols is robust in this way except those that add
authentication. It isn't useful to try to build non-authentication
extensions to be robust to those sorts of attacks - those extensions
need to be compatible with real security protection (which they are).

> protocols to make their product saleable.

That's a fine game, but it ends up backfiring. There were a half-dozen
companies in the late 1990s that violated TCP rules to "accelerate"
transfers. Customers found out what was going on and they're no longer
in business.

> "attackers ought not to be self-interested, so we shouldn't have to

I'm saying that bugs should be treated as such. None of our non-security
protocols are designed to be robust to byzantine attacks.

> And what about the second part of my email on segment coalescing? That's
> much tougher, and probably much more prevalent than asymmetric middlebox
> behaviour.

Yes, but we're talking about segment coalescing where the middlebox
deliberately copies options it doesn't understand. That's a recipe for
disaster for options anyway.

Yes, we need to understand how prevalent that is and whether to
recommend existing methods that would either prevent (cause to fail) or
detect such rewriting, but I don't think it's useful to build that into
each option.

Joe

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 3 Oct 2014 01:57:25 +0100

Joe,

At 17:25 02/10/2014, Joe Touch wrote:

> >> First, IMO "option-removing" middleboxes ought to be considerd broken
>extensions to be robust to those sorts of attacks

So you are saying that the 'trivial' protection you added to EDO was 
not useful, even tho it solved the problem?

1) Altho carefully designed protocol structure does not address /all 
possible/ byzantine behaviours, it can harden a protocol against most 
or even all known middlebox behaviours...
2) ...without the overhead of authentication...
3) ...and authentication is not a panacea anyway - an authenticated 
message proves nothing if it's not delivered - all authentication 
solutions sit within a broader scheme that has to be /structurally/ secure.

>- those extensions

I think you're using the term 'real security' when you mean 
cryptography. Security is a much broader art than just cryptography.

I recently gave a presentation about the overlloked role of 
structural security. Many security professionals unconsciously allow 
their attention to stray from any security problems that are not 
amenable to cryptographic solutions. And they blinker themselves into 
using crytography as the hammer for every nail to the exclusion of 
more appropriate techniques.

Using protocol structure to protect against the self-interested 
behaviour of middlebox manufacturers /is/ real security. As your own 
'trivial' solution demonstrates.

>transfers. Customers found out what was going on and they're no longer

I think you mean "It ended up backfiring in half-a-dozen cases, and I 
can't prove why they went bust, but I wish it was because they 
violated TCP rules".

There are dozens, probably hundreds, of companies still making 
healthy profits out of middleboxes that violate TCP rules.

>protocols are designed to be robust to byzantine attacks.

In this thread alone, three have been mentioned:
- MPTCP
- EDO with your recent trivial alteration.
- Syn-op-sis

>detect such rewriting, but I don't think it's useful to build that into

So, for instance, if we added a payload_size field to EDO, would that 
be 'not useful' for detecting segment coalescing? Should we deny that 
payload-size is useful and require EDO to use authentication instead?

For a minimum protocol, we have to protect against muddle but not 
meddle. Then, if the implementer chooses, s/he can use authentication 
to protect against meddle as an optional extra.

Bob

---

From: Wesley Eddy <[email-masked]>
Date: Thu, 02 Oct 2014 21:07:06 -0400

> detect such rewriting, but I don't think it's useful to build that into

Segment coalescing with blind copying of options will definitely be a
problem.  I assume that's a real behavior someone may do, but haven't
actually noticed it.

To my knowledge, none of the various RFCs that talk about different
middlebox behaviors have a "don't do that" message about this.
Sort of like RFC 3360 for RST generation
(https://tools.ietf.org/html/rfc3360), maybe a quick list of very
harmful option-handling practices would be in order (this would include
blind-copy coalescing, asymmetric filtering, and others).

-- 
Wes Eddy
MTI Systems

---

From: Wesley Eddy <[email-masked]>
Date: Thu, 02 Oct 2014 21:11:33 -0400

> blind-copy coalescing, asymmetric filtering, and others).

And, yes, I know that legislating the problem away is not realistic,
but it's hard to expect people to implement the correct practices if
they aren't written down, or if we don't even necessarily agree on
what those correct practices are.  It may be easier to say what NOT
to do, which we can probably agree on.

-- 
Wes Eddy
MTI Systems

---

From: Joe Touch <[email-masked]>
Date: Thu, 02 Oct 2014 18:13:50 -0700

On 10/2/2014 5:57 PM, Bob Briscoe wrote:
> So you are saying that the 'trivial' protection you added to EDO was not
> useful, even tho it solved the problem?

It won't prevent the "attack"; it turns the attack into a DOS event.

> possible/ byzantine behaviours, it can harden a protocol against most or
> even all known middlebox behaviours...

I agree it's useful to put in reasonable safeguards, but I don't think
we should be designing against byzantine or malicious events.

> cryptography. Security is a much broader art than just cryptography.

I was being informal. Cryptography isn't the right term, unless the
intent is to hide the packet format from the middlebox.

Ultimately, only integrity protection will suffice.

> I recently gave a presentation about the overlloked role of structural
> security. Many security professionals unconsciously allow their
> attention to stray from any security problems that are not amenable to
> cryptographic solutions. And they blinker themselves into using
> crytography as the hammer for every nail to the exclusion of more
> appropriate techniques.
>

It's not security, IMO. It's error avoidance.

> can't prove why they went bust, but I wish it was because they violated
> TCP rules".

They got tarred and feathered for it at the time. Maybe not all of them,
but some did.

> There are dozens, probably hundreds, of companies still making healthy
> profits out of middleboxes that violate TCP rules.
> 
> 

None of them are robust to byzantine attacks, one of which could easily
combine the SYN solutions with an inserted TFO that isn't where you
think it would be safe.

>> That's
> So, for instance, if we added a payload_size field to EDO, would that be
> 'not useful' for detecting segment coalescing? 

Strictly, so would a new TCP option called "the real segment length", so
long as the coalescing device didn't know about it. That's what you're
basically doing by adding the info to EDO.

> Should we deny that

If we want to build an option to detect and/or protect against
coalescing, fine. But I don't see why we should bundle that in another
option. It might be useful for other options besides ours.

> meddle. Then, if the implementer chooses, s/he can use authentication to
> protect against meddle as an optional extra.

I support the idea of an option to detect and/or protect against
middleboxes, but again I don't see why we need that bundled with EDO.

Joe

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 3 Oct 2014 02:48:05 +0100

Wes,

Personally, I wouldn't spend time on writing a list of bad practices 
for an audience that won't read the list.

A better way to reach the indended audience is to write how they 
/should/ do practice X (e.g. segmentation offloading). Then we're 
producing a spec that is useful for tendering exercises. It then 
becomes worth these vendors complying, otherwise they cannot win the tenders.

Perhaps a 'Requirements for Internet Middleboxes' would also be 
something that could be tendered against. Rather than listing bad 
practices, it would give required practice. We (speaking as an 
operator) could then add this RFC to our template list of RFCs for 
which compliance is required in all our tenders. Then non-compliance 
would be punishable by loss of business.

Another practice I have started to do in recent drafts I'm involved 
in is to include directives on required middlebox behaviour (even for 
an e2e protocol). Example:
<http://tools.ietf.org/html/draft-kuehlewind-tcpm-accurate-ecn-03>
(search through for 'middlebox')

Bob

---

From: Joe Touch <[email-masked]>
Date: Thu, 02 Oct 2014 21:13:28 -0700

On 10/2/2014 6:48 PM, Bob Briscoe wrote:
> Wes,
> 
> Personally, I wouldn't spend time on writing a list of bad practices for
> an audience that won't read the list.
> 
> /should/ do practice X (e.g. segmentation offloading).

That's easy:

	If you're not the endpoint, don't modify the segment.

Was there something in RFCs 1122 and 1812 that was unclear on this?

Everything else we do are "laws for the lawless", or - at best - things
that might work today (like segment coalescing) but *will* have unknown
consequences for the future (like EDO). There's no real utility in doing
that.

Joe

---

From: Yoshifumi Nishida <[email-masked]>
Date: Thu, 2 Oct 2014 21:56:18 -0700

> consequences for the future (like EDO). There's no real utility in doing

My concern is it might not be very rare case as we have observed a certain
number of middleboxes that coalesce segments. it may look like an ex post
facto law.
--
Yoshi

---

From: Joe Touch <[email-masked]>
Date: Thu, 02 Oct 2014 22:32:44 -0700

On 10/2/2014 9:56 PM, Yoshifumi Nishida wrote:
> On Thu, Oct 2, 2014 at 9:13 PM, Joe Touch <[email-masked]
> <mailto:[email-masked]>> wrote:
> 
...
> My concern is it might not be very rare case as we have observed a
> certain number of middleboxes that coalesce segments. it may look like
> an ex post facto law.

The same was true for "nodes that don't speak IP".

Let's find them and get them fixed or removed, because they interfere
with TCP on many levels (ACK clocking, Nagle if enabled, etc.).

Joe

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 3 Oct 2014 12:14:06 +0100

Joe,

It might appear as if we still disagree, but that is either due to 
differences of terminology or differences of preference. This rather 
adversarial conversation has nonetheless been useful and produced 
useful insights.

We differ on:
- whether the field of cryptography excludes integrity checking and 
authentication
- whether the field of security includes measures to protect against 
self-interested but non-malicious parties
- whether self-interested actions are byzantine
- for protocol safety, where to draw the line between trivial 
protocol-specific measures and heavier protocol-independent measures.

We agree that:
- basic protocols do not need to be designed against malicious or 
devious adversaries
- protection against malice should be made available in additional protocols
- it is worth including simple measures in a basic protocol to 
protect against known or likely behaviour of middleboxes (but we need 
measures to get them fixed too)
- it would be better if a general-purpose protocol were available to 
detect or even prevent known middlebox behaviours

There are three suggested additions to EDO:
a) defer extra options until after the SYN/ACK
b) require EDO on all segments
c) include the payload size in the EDO option

- a) and b) together should detect asymmetric (or intermittent) 
option removal.
- c) should detect non-malicious segment coalescing.
- Once either of these behaviours is detected, the connection has to abort.
- Detection-only solutions turn middlebox corruption into DoS (ie. 
denying the extended option service).

===Next Steps===

I shall produce an update to syn-op-sis to extend it to all segments 
of a connection. I believe syn-op-sis has more potential to be 
enhanced to improve on EDO:
- it should enable extra options on the SYN/ACK (ie. all segments).
- it already /prevents/ corruption due to option removal
- however, I think it will only /detect/ corruption due to coalescing

I guess you'll update EDO with whichever of a)b)c) you decide on.

We need experiments (including those already written up) to determine 
how prevalent various middlebox behaviours are.

I don't know where this leaves SYN-EOS OOB. It is currently 
integrated with EDO, but if EDO defers extra option until after the 
SYN/ACK, then OOB + EDO together will not give any extra space on the SYN/ACK.

Bob

---

From: John Leslie <[email-masked]>
Date: Sat, 4 Oct 2014 16:24:31 -0400

Bob Briscoe <[email-masked]> wrote:
> <snipP 

   Please don't!

> c) include the payload size in the EDO option

   what about:

  d) set the Data Offset field to zero when using EDO?

   (I may have missed something, but it seems to me that would prevent
the horror Bob mentioned.)

>   denying the extended option service).

   I'm not sure we need to limit our thinking to detection-only -- but
that seems sufficient for our first RFC (if it's Experimental status).
  
> - however, I think it will only /detect/ corruption due to coalescing

   I will be very interested to read this...

> I guess you'll update EDO with whichever of a)b)c) you decide on.

   Um... tcp-edo is now a Working Group document.

   I believe we should discuss what changes are appropriate before the
Document Editor posts a new version.

> how prevalent various middlebox behaviours are.

   I agree experiments are needed. I prefer getting an Experimental
document out fairly quickly to trying to do all needed Experiments
before publishing.

> SYN/ACK, then OOB + EDO together will not give any extra space on the 

   syn-eos-oob should be a separate document. (IMHO)

   Joe should be free to do whatever he wants with it (As Bob should
be free to do whatever he wants with syn-op-sis).

--
John Leslie <[email-masked]>

---

From: Joe Touch <[email-masked]>
Date: Sat, 04 Oct 2014 21:32:59 -0700

>> b) require EDO on all segments

I'm not sure about your concern; we're talking about putting EDO in all
segments of a connection *on which* EDO has been negotiated. Nobody is
expecting EDO to be turned on for all connections.

> the horror Bob mentioned.)

That just means that middleboxes won't see *any* of the options,
including whether a segment is using TCP-AO (and thus modifying the
segment would trash it).

It doesn't address the scenario Bob described. Extended options could
still be merged with user data if the segments are merged or rewritten
in without consideration for EDO.

> that seems sufficient for our first RFC (if it's Experimental status).

It would help if you could give an example of any other option that
corrects middlebox corruption; otherwise, we should set our expectations
for this option accordingly.

> Document Editor posts a new version.

That's what we're doing...

> document out fairly quickly to trying to do all needed Experiments

By that logic, all work on all new TCP options should halt dead in its
tracks. What we do need is to understand the prevalence of the byzantine
failures we've been discussing (and no, let's not call them just
"behaviors"; there needs to be a limit to what we allow on-path. at some
point, that "behavior" is an attack or a bug and should be fixed rather
than "observed").

>    syn-eos-oob should be a separate document. (IMHO)

It is. Bob's referring to the mechanism in syn-eos-oob that turns on EDO
for the rest of the connection; that has no impact on the EDO-only document.

Joe

---

From: John Leslie <[email-masked]>
Date: Sun, 5 Oct 2014 21:31:04 -0400

> expecting EDO to be turned on for all connections.

   It's hard to know at every connection start whether at some later
point extra options will prove useful.

> including whether a segment is using TCP-AO (and thus modifying the

   Middleboxes which don't understand EDO, indeed, won't see the extra
options. But this really doesn't matter, because they'll either pass
it correctly or drop it -- the same choices as if they _did_ understand
EDO.

   (I could have sworn it was Joe who said broken middleboxes need to
be replaced. ;)

> in without consideration for EDO.

   The Data-Offset-Zero case can't lead to merging, because clueless
middleboxes won't be able to find _any_ options to merge.

====

   It might help for me to detail what I think Data-Offset-Zero means.

   The original SYN includes a two-byte option announcing that the
sender speaks EDO -- that is, it will understand if it receives an
EDO packet and it knows how to send one.

   The SYN-ACK MAY include a four-byte option announcing the option
space is larger than 40 bytes. Or it may include in the regular
option space a two-byte option simply announcing that it speaks EDO
but doesn't want to use it yet.

   If either of these doesn't happen, the connection is not EDO-capable.
But recall that the original SYN-sender knows how to speak EDO; thus
the SYN-ACK, upon sending an EDO option, MAY assume the connection to
be EDO-capable. (This assumption may prove false if there is a middlebox
which doesn't understand EDO. The ambiguity will be short-lived in that
case because the SYN-ACK with the EDO option won't arrive.)

   If the connection _is_ EDO-capable, both endpoints will know that
a Data Offset of zero indicates the presence of a four-byte EDO option
stating the actual length of the option space. Thus they can start
parsing the option space before learning its actual length.

   Middleboxes which don't understand EDO will of necessity drop the
packet instead of misinterpreting some of the actual options as
application data. The packet drop will indicate the probable presence
of such a middlebox. (I need not go into detail here about how to
deal with that, but questions will be welcomed.)

   Middleboxes which aren't trying to violate layering will simply
pass the IP packet without any damage (or drop it, if it violates
some rule they're trying to enforce).

   If a meddlebox drops the SYN or SYN-ACK EDO option, the endpoints
won't even try to speak EDO. If paths change and a different middlebox
drops a packet, we simply detect the problem when it arises.

====

> for this option accordingly.

   One _can_ only "correct" middlebox corruption by detecting it and
routing around it. Data-Offset-Zero will certainly detect it; and the
endpoints can route around it by avoiding EDO.

> tracks.

   I can't think of a way to parse what I wrote so as to lead to that
conclusion. (I have to guess you missed the "to" before "trying to do".)

   Let me restate it:

   I prefer getting an Experimental document out fairly quickly for EDO.
This threatens to drag on interminably if we try to do all the needed
experiments before publishing an Experimental-track document. Also,
if we describe the experiments in such a document, it will be easier
for folks to help us with them.

> What we do need is to understand the prevalence of the byzantine

   I sympathize!

   But eliminating bad behavior in-the-wild just doesn't happen. We
_can_ raise the awareness of it; and that will help _reduce_ it.
Just bad-mouthing it general won't even reach the folk who can help
us reduce the bad behavior. :^(

---

From: Joe Touch <[email-masked]>
Date: Mon, 06 Oct 2014 10:08:04 -0700

> point extra options will prove useful.

TCP option capabilities are negotiated during the SYN exchange. The only
benefit to making them optional afterwards is to save option space - a
problem EDO doesn't face.

We've discussed turning on options mid-connection before, and that's a
separate issue we're not considering here.

> it correctly or drop it -- the same choices as if they _did_ understand

Or ignore it but merge segments - or 'adjust' segment boundaries. Either
of these additional operations would affect the extended option contents
unless the middlebox knew about EDO.

>    (I could have sworn it was Joe who said broken middleboxes need to

I still do.

> middleboxes won't be able to find _any_ options to merge.

The ones we've been discussing merge *segments* which will corrupt the
options.

> case because the SYN-ACK with the EDO option won't arrive.)

FWIW, that's already how EDO works as per the draft.

> parsing the option space before learning its actual length.

There are some options that might warrant handling first - e.g., TCP-AO.

> application data.

We can't assume that. These are already boxes that don't follow spec.

> The packet drop will indicate the probable presence

How do we differentiate this from a packet drop due to congestion? And
what can we do when it happens anyway?

> endpoints can route around it by avoiding EDO.

The endpoints will see packet loss only. Endpoints don't route around
anything per se, and they treat loss via backoff and retransmission.

>    I prefer getting an Experimental document out fairly quickly for EDO.

The track has nothing to do with the speed of WG handling. There are no
experiments we're waiting for in the case of EDO - it's the OOB case
that definitely needs experiments, and probably also the SYNOPSIS case too.

> us reduce the bad behavior. :^(

Please post us your draft on how to have routers handle IP packets whose
type isn't 4, or better - whose Ethertype isn't 0x0800.

At some point we need to declare a system "broken" and get it fixed. We
don't make cars that fly just because bridges fail.

Joe

---

From: John Leslie <[email-masked]>
Date: Mon, 6 Oct 2014 21:08:39 -0400

> problem EDO doesn't face.

   The legacy limit on space for TCP options is not the only reason
to use options sparingly -- least of all for options that haven't yet
been specified.

> separate issue we're not considering here.

   I'm not trying to solve that issue (though it would be easy enough
to do so) -- I'm trying to cater to options which will be useful for
only part of the time a TCP connection is open.

> unless the middlebox knew about EDO.

   A middlebox would have to be _rather_ braindead to believe that a
Data Offset of zero pointed to an actual start of data. Unless you can
supply an existence-proof of such middleboxes, I think it quite safe
to rule out such behavior.

> The ones we've been discussing merge *segments* which will corrupt the

   That causes problems long before we start considering expanded
option space. The problems which _now_ cause undiagnosed strangeness
would become impossible to ignore if we use Data Offset Zero.

   But again, I have not seen evidence that middleboxes braindead in
_this_ particular way exist. (I do think that Experimental status is
appropriate to weed out _any_ unlikely flavors of braindead, though.)

>> It might help for me to detail what I think Data-Offset-Zero means.

   Note that this is aimed at WG members who are not as familiar
with the subject of this discussion as Joe and Bob...

> FWIW, that's already how EDO works as per the draft.

   Exactly. (But not everyone can grok that from a brief reading.)

> There are some options that might warrant handling first - e.g., TCP-AO.

   Indeed. (If we adopt Data-Offset-Zero, we should mention this. But
nothing about Data-Offset-Zero requires that the EDO option come first,
or even within the first 40 bytes.)

>> packets instead of misinterpreting some of the actual options as
>> application data.
> 

   If middleboxes as braindead as you stipulate are found to exist,
we should mention ways to deal with this. But please understand that
the situation can't arise without the endpoint receiving a EDO option
with Data-Offset-Zero having announced that it understands EDO: thus
we could easily make detection of a braindead middlebox part of the
protocol.

> what can we do when it happens anyway?

   A fair question.

   The simplest answer is that dropping will persist until a packet is
sent without the EDO option (assuming, of course, that some options
_don't_ have to be sent in every packet -- that's why I want to avoid
any rule that an EDO option must be in every packet).

   We could, for example, specify that a retransmitted packet must not
use the EDO option.

> anything per se, and they treat loss via backoff and retransmission.

   But of course, if the retransmission doesn't use EDO, that already
amounts to routing around the problem. If we were dealing with an MPTCP
connection-set, we would literally route-around a braindead middlebox.

> The track has nothing to do with the speed of WG handling. There are
> no experiments we're waiting for in the case of EDO - it's the OOB
> case that definitely needs experiments, and probably also the SYNOPSIS
> case too.

   I don't get the impression that we have consensus to issue tcp-edo
(as-is or with changes suggested so far) as Proposed Standard. YMMV.

> don't make cars that fly just because bridges fail.

   (Of course, we _do_ make cars that fly; but that's not the question!)

   _Somebody_ with authority to demand the fix has to notice the problem
first...

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 7 Oct 2014 11:37:33 +0100

Joe,

>that definitely needs experiments, and probably also the SYNOPSIS case too.

I hope the examples I gave at the start of this thread show that EDO 
is not immune to middlebox problems, and therefore should be 
conditional on middlebox traversal experiments (as should OOB and SYNOPSIS).

The chairs qualified the WG's intention to go straight for Proposed 
Standard status for EDO with (paraphrasing) "We can always decide to 
downgrade to Experimental, but it's best to start aiming high."

Bob

---

From: Joe Touch <[email-masked]>
Date: Tue, 07 Oct 2014 07:58:56 -0700

On 10/7/2014 3:37 AM, Bob Briscoe wrote:
> I hope the examples I gave at the start of this thread show that EDO is
> not immune to middlebox problems, and therefore should be conditional on
> middlebox traversal experiments (as should OOB and SYNOPSIS).

IMO, these experiments should be conditional on some evidence they're a
real concern, i.e.:

	- deployed in public networks in more than rare exceptions

	- a device the WG agrees that protocols should traverse
	(vs. a device considered incorrectly implemented that should
	be fixed or replaced)

It's easy to posit device properties that can interfere with a proposal,
but I don't agree that we need to deal with every possible byzantine case.

Hesmans,et al., "Are TCP Extensions Middlebox-proof?", Conext 2013
explores how options react to middlebox behaviors that include splitting
and coalescing, but the only middlebox behavior seen is a failure of ALG
translation of active FTP through NATs. Splitting and coalescing are
described, but (unlike other behaviors) their description has no citations.

The citations in that paper go back to this:
Honda, et al., "Is it still possible to extend TCP?", IMC 2011.
"Although middleboxes do split and coalesce segments, *NONE* [emphasis
mine] did so while passing unknown options...".

Can we please stop chasing phantoms?

Joe

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 19 Oct 2014 23:43:00 +0100

Joe,

>mine] did so while passing unknown options...".

A) If we were to write guidelines on middlebox design, I suspect we 
would say that a middlebox SHOULD/MUST forward unknown options.

B) It is fairly obvious that two TCP stacks either side of a proxy 
will sometimes split or coalesce segments:
* If the origin is app-limited so that small segments arrive into a 
buffer at the proxy, then the proxy is likely to coalesce them into a 
larger segment when it sends them out.
* If a sequence of coalesced segments overflows the MTU, the proxy 
will then split the last segment and start the next segment with the remainder.

So what would you advise this proxy to do? Not resegment? That's not 
easy to implement, because the outgoing side has to somehow remember 
where the segment boundaries were when they arrived.

'Ideally' the proxy shouldn't intervene uninvited. Then the sender is 
talking to the proxy as a proper endpoint, and the proxy is similarly 
talking to the receiver.

However:
* that requires extra complexity configuring the sender with the 
address of the appropriate proxy, which changes if the sender is 
mobile, or changes to connect via a VPN, etc etc.
* that still greatly slows the deployment of new options. If 10% of 
endpoints and 10% of proxies have deployed a new option, the chance 
of finding a fully deployed path is 10%*10% = 1%.

...this isn't as easy as just banging a fist on the table and shouting "No!"

I prefer to make TCP options robust to resegmentation and robust to 
option stripping. This would constructively simplify the whole system.

>Can we please stop chasing phantoms?

I'm fine with this. I am trying to find out what is out there myself 
(Michio's paper was an output of one of our projects), and go deeper 
by understanding why middleboxes are like they are.

Anyway, I don't think I ever said that we need to cater for 
middleboxes that resegment AND forward unknown options. So I think 
you might be arguing against an invented adversary (chasing phantoms?).

Perhaps we have different goals. You are saying all we have to do is 
take account of what is out there. I believe we have to do that and more.

We have to propose rules that middleboxes can follow, and there must 
be no reason why they wouldn't follow the rules. That means the rules 
have to work in their interest, and allow them to be lazy. It is 
certainly not realistic to expect these vendors to follow rules that 
entirely rule out their business model, no matter how much I don't 
like their architecture.

Bob

---

From: Joe Touch <[email-masked]>
Date: Mon, 20 Oct 2014 13:50:56 -0700

...
> would say that a middlebox SHOULD/MUST forward unknown options.

I have better guidelines; I'm writing them up for the IAB (which, IMO,
dropped the ball in not doing so properly a looong time ago).

> B) It is fairly obvious that two TCP stacks either side of a proxy will
> sometimes split or coalesce segments:
> * If a sequence of coalesced segments overflows the MTU, the proxy will
> then split the last segment and start the next segment with the remainder.
> 
> So what would you advise this proxy to do? 

IMO they're performing as a transparent proxy. Transparent may be bad,
but the least they can do is correctly implement a proxy - i.e., source
and sink options on either side of the connection separately.

I don't think the Internet is a monument to "do what you want and we'll
accommodate it". There *MUST* be limits.

> Not resegment? That's not
> * that requires extra complexity configuring the sender with the address
> of the appropriate proxy, which changes if the sender is mobile, or
> changes to connect via a VPN, etc etc.
> endpoints and 10% of proxies have deployed a new option, the chance of
> finding a fully deployed path is 10%*10% = 1%.
> 

If you intend on accommodating anything a proxy wants to do, that's what
you'll end up doing anyway.

...
> Anyway, I don't think I ever said that we need to cater for middleboxes
> that resegment AND forward unknown options. So I think you might be
> arguing against an invented adversary (chasing phantoms?).

I hope so.

IMO, as long as we have a middlebox that terminates options on either
side we're fine.

...
> We have to propose rules that middleboxes can follow, and there must be
> no reason why they wouldn't follow the rules.

Middleboxes are built by the lazy and/or greedy who want to benefit from
the rules everyone *else* plays. It's time to allow them to break and
take the heat for their own misbehavior.

> That means the rules have
> to work in their interest, and allow them to be lazy. 

I have no intention of making an Internet that runs over the Internet
solely to ensure that middlebox vendors have a job.

> It is certainly
> not realistic to expect these vendors to follow rules that entirely rule
> out their business model, no matter how much I don't like their
> architecture.

Not all business models NEED to succeed, and the it's not clear the
Internet CAN support them all. We need to stop acting like accommodating
their model isn't damaging the rest of us.

Joe