Subject: Re: [tcpm] draft-bonaventure-mptcp-long-options-00

From: Joe Touch <[email-masked]>
Date: Thu, 23 Jul 2015 14:01:04 -0700

Hi, all,

The above draft discusses several concerns with EDO, of which most have
already been addressed in Section 7.3 of the EDO-03 draft.

As it notes, EDO is for more than just MPTCP.

Other observations below.

Joe

In summary, there are 5 cases:

	2 of which are hypothetical only

	1 of which was fixed with the original EDO variant
	but breaks when the space isn't used in the SYN-ACK
	(i.e., when the option is reflected), but is very easy to fix

	1 of which is byzantine and IMO should not be fixed
	(and for which even in-band would require extreme measures)

	all others of which EDO already fixes with the Seg_Length field
	(the only issue is how to handle seg_length failures, and that
	too is easy to fix)

I raised concerns with in-band options when MPTCP was being proposed in
Stockholm. It's worrisome that it's gone back to including that because
it decouples TCP signaling from the segment boundaries, only creates
mutual escalation (once middleboxes understand it they'll surely tamper
with it) and invites "TCP over TCP" ultimately.

Many of the assertions of what works and does not through middleboxes is
from the Honda IMC11 paper, which we already know failed to confirm many
of the issues claimed in this summary.

Joe

---

Of the issues raised:

1. blindly echoing SYN options
	already applies to other widely-deployed options (SACK, WSCALE)
	trivial to address by flipping at least one bit in the
	EDO-capable option response

2. modified SYN-ACK
	we don't use EDO until both sides confirm after the TWHS
	for this reason

3. blind removal
	despite no citation for seeing this in the wild,
	the most recent EDO includes a segment_length field that
	will detect such tampering

4. splitting
	IMC11 explicitly stated that although middleboxes do split
	and coalesce segments, none did so while passing unknown
	options

	despite no citation for seeing this in the wild,
	this would be detected by the segment_length field

5. coalescing
	see (4)

6. modifying the data
	this would interfere with in-band mechanisms as well

----

---

From: Yoshifumi Nishida <[email-masked]>
Date: Thu, 23 Jul 2015 21:13:05 -0700

Hi Joe,

One question.
The IMC paper you mentioned describes TSO cases that copied the
options to all the split segments.
Can't we count it into "seeing this in the wild"?

Thanks,
--
Yoshi

On Thu, Jul 23, 2015 at 2:01 PM, Joe Touch <[email-masked]> wrote:
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: Olivier Bonaventure <[email-masked]>
Date: Fri, 24 Jul 2015 12:05:22 +0200

Joe,

> with it) and invites "TCP over TCP" ultimately.

The growing deployment of MPTCP shows that this decision was not too bad
from a deployment viewpoint..

>         EDO-capable option response

There is a difference between known options and new options. Middleboxes do
not treat them in the same manner.

>         will detect such tampering

As an example, consider the configuration of one deployed TCP normalizer

http://www.cisco.com/c/en/us/td/docs/app_ntwk_services/data_center_app_services/ace_appliances/vA1_7_/configuration/security/guide/securgd/tcpipnrm.html#wp1010795

Clearly, it could reject packets with some options or clear the option
based on their type. This is because of this type of configuration that
MPTCP has opted for a single option type and different suboptions. EDO uses
two different option types and there is thus a risk that EDORequest passes
in the SYN but not EDO in the data segments. Note that it is common for
middleboxes to replace unsupported options by NOP.

>         this would be detected by the segment_length field

But TSO usually copies the options to all packets that are segmented.

>         this would interfere with in-band mechanisms as well
This is part of most NAT implementations that are deployed

Olivier

---

From: Joe Touch <[email-masked]>
Date: Fri, 24 Jul 2015 09:58:19 -0700

> options to all the split segments.

To quote from the discussion of that issue in that paper:

"None passed options to the split segments".

> Can't we count it into "seeing this in the wild"?

It's not that splitting or coalescing isn't seen in the wild; it's that
in all cases reported from the wild in that paper either:

	- unknown options are not reflected in the SYN-ACK

	- options are not copied to the split segments

Joe

---

From: Joe Touch <[email-masked]>
Date: Fri, 24 Jul 2015 10:48:13 -0700

> from a deployment viewpoint..

You cannot draw that conclusion from one data point over this short
period of time.

...
> There is a difference between known options and new options. Middleboxes
> do not treat them in the same manner.

Blind implies new, at least to the middlebox.

> based on their type. 

It could also change TCP into UDP.

> This is because of this type of configuration that
> MPTCP has opted for a single option type and different suboptions. 

I recall only the need to conserve TCP options for that decision.

> EDO
> uses two different option types

It uses one codepoint with a variable length. We refer to them as
different options in the text but the difference is determined by the
length only.

> and there is thus a risk that EDORequest
> passes in the SYN but not EDO in the data segments. Note that it is
> common for middleboxes to replace unsupported options by NOP.

That would be detected during the SYN-ACK and EDO would not be used on
the connection - again, because we're using only one codepoint throughout.

> But TSO usually copies the options to all packets that are segmented.

The IMC11 paper noted that no options were copied to the split segment.
That would be detected by EDO as a failure because EDO is required on
all segments once negotiated.

> This is part of most NAT implementations that are deployed

For only a very small set of protocols where the contents refer to the
endpoint addresses in-band, e.g., some teleconferencing and active FTP.

Depending on the extent of the translation, that could corrupt any
in-band information (including the protocol the NAT is trying to help
through translation).

Joe

---

From: Yoshifumi Nishida <[email-masked]>
Date: Sun, 26 Jul 2015 22:26:21 -0700

Hi Joe,

On Fri, Jul 24, 2015 at 9:58 AM, Joe Touch <[email-masked]> wrote:
>         - options are not copied to the split segments>

Well, if we don't think about TSO, the above sentences are right.
However, If EDO needs to support TSO/LRO, I'm not sure if we can
ignore the NIC behavior that have been observed.

Thanks,
--
Yoshi

---

From: Joe Touch <[email-masked]>
Date: Mon, 27 Jul 2015 13:39:02 -0700

On 7/26/2015 10:26 PM, Yoshifumi Nishida wrote:
> However, If EDO needs to support TSO/LRO, 

Let's please be very clear:

	TSO/LRO needs to be compliant with TCP

> I'm not sure if we can

If it's in direct contradiction to the software's documentation, isn't
it very clearly a bug?

Finally, TSO/LRO is an optimization that is intended to work with TCP.
If TCP already knows there's a problem using TSO/LRO it can and should
be disabled for that connection.

When we did so, we saw no measurable impact on either TCP performance or
CPU load. Which raises significant concerns as to whether TSO/LRO is
even needed in many cases.

Joe

---

From: Costin Raiciu <[email-masked]>
Date: Tue, 28 Jul 2015 16:01:27 +0300

> even needed in many cases.

You can’t just disable TSO/LRO for 10Gbps NICs. With TSO, a single TCP connection easily reaches 10Gbps; without it  you only get 4Gbps, and you max out one CPU core - as my student’s tests show.

If there’s anything we have learnt from trying to get MPTCP upstreamed in the Linux kernel, is that it can’t affect the performance of the TCP stack. If MPTCP is compiled in and not used, the TCP stack should work exactly as it did before. For EDO, in particular, this means that no-one will ever enable it because the performance hit for servers is terrible.

You can always scream abuse at TSO/LRO, but this won’t change anything I am afraid - even if NIC vendors change their TSO implementation, having a concerted hardware upgrade seems highly unlikely.

You can, however, achieve 10Gbps with software segmentation offloading / coalescing at slightly higher CPU costs than hardware offloading. This is called GSO/GRO in linux, see details here: http://www.linuxfoundation.org/collaborate/workgroups/networking/gso. If you really want to use EDO in its current form, changing GSO/LRO support and upstreaming it is the best bet. However, this deployment path is also difficult - unless everyone upgrades their kernel at the same, you will either have a) fraction of servers with EDO support but broken by TSO or b) EDO servers that are dead slow.

So, my advice would also be to redesign EDO to play nice with TSO/LRO - that’s the only hope for adoption.

Cheers.
Costin

---

From: Yoshifumi Nishida <[email-masked]>
Date: Tue, 28 Jul 2015 08:34:36 -0700

Hi Joe,

> If it's in direct contradiction to the software's documentation, isn't

Hmm. But, I'm not sure which document you are referring.. Could you point
it out?

Thanks,
--
Yoshi

---

From: Joe Touch <[email-masked]>
Date: Tue, 28 Jul 2015 10:23:32 -0700

I think we're in agreement. Here is a summary:

	- don't use optimizations when you don't actually need them
	(they represent unnecessary complexity and make assumptions
	that might be false)

	- fix bugs in the optimizations that are needed

I completely agree that TCP and offload engines need better coordination
and the offload engines need to make only the most conservative of
assumptions possible.

Again, for the case we saw for Linux, we're continuing to look into it
this fall.

Joe

---

From: Joe Touch <[email-masked]>
Date: Tue, 28 Jul 2015 10:27:30 -0700

...
> Hmm. But, I'm not sure which document you are referring.. Could you
> point it out?

I'm shifting students (the last one graduated in May; the new one starts
in Sept), so I don't have that handy.

The rule is "don't coalesce if the option values don't match". It
clearly makes no sense to coalesce under those conditions unless you
KNOW you can pick among the option values. But picking means you know
what the options mean. The docs we had said (somewhere) that if there
were unknown options that varied from segment to segment then coalescing
would not happen. We tested that with dummy option whose values varied
and coalescing still happened.

That would spell disaster for a large number of options, e.g., TCP MD5,
TCP-AO, MPTCP, etc. It's clearly not intended operation.

Again, we're working to figure out whether we're seeing a bug in the
kernel or whether this might be due to incorrect data structures we're
using.

Joe