Subject: [tcpm] EDO implementation issues

From: Pasi Sarolahti <[email-masked]>
Date: Fri, 10 Oct 2014 00:03:37 +0300

Hi,

To refresh my rusty kernel hacking skills, I took a shot at sketching a Linux implementation of draft-ietf-tcpm-tcp-edo-00.txt . In case anyone is interested, the current (work-in-progress) state is at https://github.com/PasiSa/linux . Currently the only use is to extend max nr. of SACK blocks to 8, and a test feature that adds <N> NOP options to the segment. EDO can be enabled with ‘sysctl -w net.ipv4.tcp_edo=1’. Larger values turn on the test feature that will add the given number of NOP options before possible timestamps and SACKs.

Be warned that the implementation is not fully working yet (and quite quickly made), but I wanted to share some observations at this point nevertheless. In my tests, for data offset values up to 72 or so things are ok, but beyond that the receiver option handling starts facing problems. I haven’t fully analyzed this yet, but I believe the issue is that incoming data is not always in contiguous memory area in sk_buffs, but can be stored in multiple ‘page buffers’. Until now TCP header processing has been easy, because the protocol headers have easily fitted in the first part of the sk_buff. With extended option space, the header processing becomes a fair bit more complicated, because the option parsing implementation needs to follow through the page buffers (that may be split in the middle of an option). But my Linux kernel expertise is limited, so people more knowledgeable of the kernel can correct me, if the above did not make sense. Anyways, I think BSD mbufs might have a similar property.

(I’m happy to discuss and be educated about this more, and get implementation feedback, but detailed implementation discussion should be taken off-list, as this is not a Linux dev mailing list)

A couple of other comments that came up while I was working on this:

* I didn’t want to pick random option numbers, so I decided to use ExID. In case someone else is also interested in trying an EDO implementation, would it make sense to specify common ExIDs in the draft, with a note saying that the ExIDs are intended as temporary, and actual codepoints will be assigned later?

* "The EDO Header_length SHOULD be a multiple of 4 to simplify processing." — 
how about then just using 32-bit units, similar to the original data offset? Although one-byte precision might become useful somewhere, who knows.

* If incoming EDO option has invalid values, for example more than the segment length, or less than the TCP header data offset, what should be done in that case? Just ignoring the option might not be safe, because the (buggy) sender might have assumed a changed data offset. So best thing might be to drop the entire segment in that case, no? The draft should say more about this (with normative language).

* "The EDO length option is required in SYN/ACKs when confirming
   support for EDO. The SYN/ACK thus can take advantage of EDO, using
   it to extend its option space. If such extension is not required,
   then EDO would be equal to 4 * Data Offset (i.e., EDO would indicate
   in bytes the same length indicated by Data Offset in words).” — but on the other hand, the text later says that EDO should be the last option covered by TCP DO. So how about just making the TCP data offset always point to the end of the EDO length option when such is present, also in SYN/ACKs?

- Pasi

---

From: Joe Touch <[email-masked]>
Date: Thu, 09 Oct 2014 14:32:24 -0700

Hi, all,

FWIW, we have a parallel implementation experiment here at ISI as a
student project (Harry Trieu - cc'd because he's not yet on the list,
with Ted Faber co-advising), also in Linux. We'd be glad to do
interoperability testing once we're further along.

Comments below...

Joe

On 10/9/2014 2:03 PM, Pasi Sarolahti wrote:
> Hi,
> 
> To refresh my rusty kernel hacking skills, I took a shot at 
> sketching a Linux implementation of draft-ietf-tcpm-tcp-edo-00.txt .
> In case anyone is interested, the current (work-in-progress) state is
> at https://github.com/PasiSa/linux . Currently the only use is to
> extend max nr. of SACK blocks to 8, and a test feature that adds <N>
> NOP options to the segment. EDO can be enabled with ‘sysctl -w 
> net.ipv4.tcp_edo=1’. Larger values turn on the test feature that
> will add the given number of NOP options before possible timestamps
> and SACKs.

We created another JUNK option (for testing purposes only) that can be
configured with any length, basically to fill the option space for
similar reasons.

Our version is already per-connection configurable, and is planned to
have a similar system-wide default.

> Be warned that the implementation is not fully working yet (and quite
> quickly made), but I wanted to share some observations at this point
> nevertheless. In my tests, for data offset values up to 72 or so
> things are ok, but beyond that the receiver option handling starts 
> facing problems. I haven’t fully analyzed this yet, but I believe
> the issue is that incoming data is not always in contiguous memory
> area in sk_buffs, but can be stored in multiple ‘page buffers’. Until
> now TCP header processing has been easy, because the protocol
> headers have easily fitted in the first part of the sk_buff. With
> extended option space, the header processing becomes a fair bit more 
> complicated, because the option parsing implementation needs to 
> follow through the page buffers (that may be split in the middle of 
> an option). But my Linux kernel expertise is limited, so people more 
> knowledgeable of the kernel can correct me, if the above did not make
> sense. Anyways, I think BSD mbufs might have a similar property.

Good to know; we haven't tripped over that issue yet. For now, we've
been trying to just trip over the DO boundary by one word (i.e.,
TCP+options = 64 bytes), but the final implementation should allow the
total size to grow to MSS-4 (if it is MSS, you might end up never having
room for any data).

> (I’m happy to discuss and be educated about this more, and get
> implementation feedback, but detailed implementation discussion should
> be taken off-list, as this is not a Linux dev mailing list)
> 
> * I didn’t want to pick random option numbers, so I decided to use
> ExID. In case someone else is also interested in trying an EDO
> implementation, would it make sense to specify common ExIDs in the
> draft, with a note saying that the ExIDs are intended as temporary, and
> actual codepoints will be assigned later?

Yes. Let's use the 16-bit one that matches the name: 0x0ED0

I'll grab that in the ExID registry, and we can mark it as deprecated
later presuming an option codepoint is assigned.

> * "The EDO Header_length SHOULD be a multiple of 4 to simplify 
> processing." — how about then just using 32-bit units, similar to the
> original data offset? Although one-byte precision might become useful
> somewhere, who knows.

Sure.

> * If incoming EDO option has invalid values, for example more than 
> the segment length, or less than the TCP header data offset, what
> should be done in that case? Just ignoring the option might not be
> safe, because the (buggy) sender might have assumed a changed data
> offset. So best thing might be to drop the entire segment in that
> case, no? The draft should say more about this (with normative
> language).

Silently drop. Yes, we'll add the language.

> support for EDO. The SYN/ACK thus can take advantage of EDO, using it
> to extend its option space.

That is pending some points raised by Bob Briscoe. We may want to allow
that or might want to avoid it in the SYN-ACK but allow it in later
packets (i.e., it may be important to have all segments where SYN is set
to avoid using EDO).

However, for now let's assume it's OK (it either will be or not, but the
rest of the answer below won't change)...

> If such extension is not required, then EDO would be equal to 4 *
> Data Offset (i.e., EDO would indicate in bytes the same length
> indicated by Data Offset in words).” — but on the other hand, the
> text later says that EDO should be the last option covered by TCP DO.
> So how about just making the TCP data offset always point to the end
> of the EDO length option when such is present, also in SYN/ACKs?

Yes - we're considering making EDO mandatory once enabled, and if that's
true (in any segment), then EDO should point to the same place as DO --
and since we're making EDO count in words, the two would just have the
same value.

Joe

---

From: Pasi Sarolahti <[email-masked]>
Date: Fri, 10 Oct 2014 11:07:38 +0300

> interoperability testing once we're further along.

Ah, nice to hear! My cycles are quite limited on this, but I’d be happy to collaborate, and hopefully it is possible for you to make your code available, once you get to that point.

> total size to grow to MSS-4 (if it is MSS, you might end up never having

And I already got off-list response from Ilpo Järvinen (thanks!), that the skbuff operations have tools to tackle this issue (pskb_may_pull), but it may get quite expensive (in worst case memory reallocation and copying).

> later presuming an option codepoint is assigned.

Works for me, and will update my code accordingly. (and I only now realized that it is in fact only one codepoint that is needed, because length can be used to distinguish between the two)

- Pasi

---

From: Joe Touch <[email-masked]>
Date: Fri, 10 Oct 2014 11:17:01 -0700

> On 10 Oct 2014, at 00:32, Joe Touch <[email-masked]> wrote:
...
>>> ...In my tests, for data offset values up to 72 or so
>>> area in sk_buffs, but can be stored in multiple ‘page buffers’...
> 
> And I already got off-list response from Ilpo Järvinen (thanks!), 
> that the skbuff operations have tools to tackle this issue
> (pskb_may_pull), but it may get quite expensive (in worst case memory
> reallocation and copying).

Good to know. I'll note that in an update to the doc. IMO, that leads to
a broad recommendation to be conservative in using this extension space
but I don't think it'd be appropriate to add artificial,
implementation-dependent limits.

Joe