Subject: [tcpm] feedback on WG adoption of draft-cheng-tcpm-fastopen-00

From: Rick Jones <[email-masked]>
Date: Fri, 26 Aug 2011 12:01:12 -0700

I've been meaning to get this out for some time but "other things" have 
kept getting in the way.

Anyhow, at the highest level I find the draft useful and think it would 
be a worthwhile WG item.

Getting a bit lower, some comments/questions, one or two of which stray 
beyond the specifics of the draft and might be philosophical, but are 
triggered by it.

*) Just how many more angelic options can dance on the header of the TCP 
SYN?  How many options being present nearly all the time call for a 
revision of the standard header (Yes, that is an intractable Layer 8 and 
9 issue but still...).

*) MSS, Window Scale, Timestamp, SACK OK are all quite desirable no?  We 
want to avoid the dreaded 536 byte MSS, handle bandwidth delay products 
beyond the limit of a 64K window, not corrupt data when we do so, and be 
able to quickly recover from the inevitable segment losses while we do 
so - not just in the interests of bandwidth, but in the interests of 
time.  IIRC that is 2 + 3 + 10 + 2 or 17 bytes of option space in the 
SYN (assuming fully packed).  40 bytes of option space is the limit no?

Section 4.1.1 of the draft talks about the cookie being up to 16 bytes 
in size, consuming a total of 18 bytes of option space. So, I suppose 
that doesn't completely consume the option space, but it starts to look 
pretty full - only 5 bytes left, perhaps only 4 depending on option 
alignment.  How would one prioritize the cookie vs the other options 
once options space was exhausted?  Section 4.2.2 makes a passing mention 
of forgoing TFO but expanding on that would be goodness.

*) Section 2.1 mentions that TFO must be disabled by default and only 
enabled explicitly by applications which want it and can tolerate the 
prospect of duplicate SYN data.  If it is inappropriate to include 
discussion of the mechanism in the "protocol" draft (perhaps as an 
appendix), is there another draft with a proposed interface for this 
enabling?

*) Section 4.2.1 gives an example of AES_128 for generating the cookie 
and says it takes "only a few hundred nanoseconds."  I think it would be 
goodness to be a bit more specific, and perhaps include some 
contemporary service demands for TCP connection establishment. 
Particularly since a TCP SYN with the cookie request could arrive every 
68 nanoseconds through a 10GbE pipe into the server, and since 40 and 
100 GbE are virtually upon us.  More specific numbers for validating 
cookies up to 16 bytes would be goodness as well.

*) Doesn't Section 4.2.2 (or some other) need to specify what should 
happen when a TCP SYN with a TFO cookie arrives at an endpoint where the 
application has not explicitly enabled TFO?  Particularly since the 
cache of cookies is per-IP and not per IP-port-pair?  A restart of the 
server application could change the setting while there was still an 
entry cached in the client(s).  Unless automagically generating a new 
cookie upon receipt of an invalid one is dropped, presumably the 
behaviour of the server would be slightly different in that it would 
drop the cookie and not make a new one, as there is no point.

I suppose for (pedantic?) completeness, explicitly stating what a client 
should do when it receives an unexpected cookie could be included as well.

*) Given the ability to spoof IP addresses, I think the order of server 
processing given in 4.2.2 " Server: Receiving SYN and responding with 
SYN-ACK" should be changed.  PendingFastOpenRequests should be checked 
first, before even attempting to validate the cookie.  Further, until 
cookie generation is shown to indeed be epsilon compared to the rest of 
TCP connection establishment on the server, the server should not 
include a new cookie when the cookie was invalid and instead require the 
client to make another "from scratch" cookie request.

Also, given that the application isn't notified until step 4, "The 
server MAY include data in the SYN-ACK packet (sic  pedant mode, TCP 
sends segments not packets)" in step 3 seems out of place unless we are 
assuming there are applications which will use this that want data sent 
before receiving anything, in which case that is another bit of user 
interface to describe somewhere.

*) 4.1.3 reads "Beside the cookie, we RECOMMEND that the client caches 
the MSS and RTT to the server to enhance performance."  Should that be 
the MSS as based (possibly) on later PMTU events during the life of the 
connection?  Meaning going to the cache not just on connection 
establishment, but also when there is a PTMU event.

*) 4.1.3 reads "For a multi-homed client, the cookies are both client 
and server IP dependent."  I ass-u-me that means they should be cached 
based on both client and server IP.  That is I suppose implicit with 
cookies being generated by servers but it rarely hurts to be explicit. 
However, why does it have to be client,server IP at the client? 
Particularly since section 4.1.2 point 5 says a client only needs to 
remember one valid cookie per server IP?  And if the client is only 
going to remember one, to what end would a server produce multiple 
cookies per client?

*) 4.1.3 "The client MUST cache cookies from different servers for 
later..." - isn't "different" somewhat redundant?

happy benchmarking,

rick jones

---

From: Jerry Chu <[email-masked]>
Date: Fri, 26 Aug 2011 21:07:45 -0700

Hi Rick,

Thanks for you thoughtful input. My comments are embedded below.

On Fri, Aug 26, 2011 at 12:01 PM, Rick Jones <[email-masked]> wrote:
> I've been meaning to get this out for some time but "other things" have kept
> getting in the way.
>
> Anyhow, at the highest level I find the draft useful and think it would be a
> worthwhile WG item.
>
> SYN?  How many options being present nearly all the time call for a revision
> of the standard header (Yes, that is an intractable Layer 8 and 9 issue but
> still...).

Yes it seems that this option space problem will one day become the choke point
for all future TCP innovations...

That's why we've carefully evaluated the tradeoff between the security risk vs
cookie size and decided cookies as small as 4 bytes may be sufficient for some
cases/implementations..., etc. We've even wondered if cookies can be made
optional (or treated as the degenerate case of server advertising a zero-lenghth
cookie) for some cases where the major security risk for TFO, i.e.,
the reflection
attack, does not exist. (See
http://www.ietf.org/proceedings/81/slides/tcpm-0.pdf.)

> able to quickly recover from the inevitable segment losses while we do so -
> not just in the interests of bandwidth, but in the interests of time.  IIRC
> that is 2 + 3 + 10 + 2 or 17 bytes of option space in the SYN (assuming
> fully packed).  40 bytes of option space is the limit no?

Correct and since we believe the potential damage from an amplified reflection
attack is somewhat limited we do not suggest an insanely large cookie size
hence all our suggested sizes fit together with other, existing commonly used
options. Yes going forward will be an issue but it's a general issue not limited
to TFO that needs to be addressed separately.

>
> Section 4.1.1 of the draft talks about the cookie being up to 16 bytes in
> size, consuming a total of 18 bytes of option space. So, I suppose that
> doesn't completely consume the option space, but it starts to look pretty
> full - only 5 bytes left, perhaps only 4 depending on option alignment.  How
> would one prioritize the cookie vs the other options once options space was
> exhausted?  Section 4.2.2 makes a passing mention of forgoing TFO but
> expanding on that would be goodness.

Sure and we've already come across that problem when trying to enable the
TCP MD5 options internally.

>From the top of my head that could be an deployment decision. Each site will
need to prioritize their option preference but unfortunately options
are initiated
by the client side... Sigh (a train wreck is bound to happen).

> appendix), is there another draft with a proposed interface for this

Well, the obvious candidate for interface is socket option, right?

Years ago IETF shunned any API discussion. But I guess that have
changed with IPng and SCTP. (?) If so we can publish a separate draft
on our proposed API. I have presented part of it in Quebec City.

>
> *) Section 4.2.1 gives an example of AES_128 for generating the cookie and
> says it takes "only a few hundred nanoseconds."  I think it would be
> goodness to be a bit more specific, and perhaps include some contemporary
> service demands for TCP connection establishment. Particularly since a TCP
> SYN with the cookie request could arrive every 68 nanoseconds through a
> 10GbE pipe into the server, and since 40 and 100 GbE are virtually upon us.
>  More specific numbers for validating cookies up to 16 bytes would be
> goodness as well.

We have actually done some study on the server cookie processing overhead
and will revise the draft with the data once we are cleared for publication. In
our test the # of sustained HTTP transaction rate was higher with TFO than w/o.
We attributed this to the fact that the reduced in pkt count (due to carrying
data in SYN) from TFO and its CPU saving more than compensating for the
extra CPU processing cost from AES_128.

>
> *) Doesn't Section 4.2.2 (or some other) need to specify what should happen
> when a TCP SYN with a TFO cookie arrives at an endpoint where the
> application has not explicitly enabled TFO?  Particularly since the cache of
> cookies is per-IP and not per IP-port-pair?  A restart of the server
> application could change the setting while there was still an entry cached
> in the client(s).  Unless automagically generating a new cookie upon receipt
> of an invalid one is dropped, presumably the behaviour of the server would
> be slightly different in that it would drop the cookie and not make a new
> one, as there is no point.

Yes TFO is negotiated implicitly by client including a cookie option and the
server acknowledging explicitly the cookie option (by either acking the data in
SYN or replying with a new cookie). All other cases will cause TFO to
be disabled.
So if the listening app has not enabled TFO, TCP won't reply with a
cookie option
nor acking the data in the SYN pkt.

There is probably some details on how often the client wants to
refresh the cookie
cache etc but some of them can be left to the implementation.

One assumption we've made is the cookie generation is cheap. Otherwise it'd
open up for another form of burning server CPU attack.

> should do when it receives an unexpected cookie could be included as well.

I don't remember we allow this case, i.e., "unexpected" cookie option received
by the client will be treated as an unrecognized TCP option.

> first, before even attempting to validate the cookie.

Aha, you've caught some good corner case. It think my colleague wrote in that
order because not replying a cookie nor acking the data will be
treated by the client
as the server disabling/de-supporting TFO. But I'm sure we can further fine tune
the cookie protocol to address any ambiguity and performance optimization.

>Further, until cookie
> generation is shown to indeed be epsilon compared to the rest of TCP
> connection establishment on the server, the server should not include a new
> cookie when the cookie was invalid and instead require the client to make
> another "from scratch" cookie request.

Yes we've thought about these performance issues and have gone through a
few rounds of the seemingly simple cookie protocol design cycle. But I'm not
sure your suggested "from scratch" approach will help much to defend against
a malicious client if the cookie generation is expensive. I think it
boils down to
as long as the cookie generation is no more expensive than the processing
overhead of a SYN pkt (upto checking the listener's qlimit), this case is no
worse than the SYN attack.

>
> Also, given that the application isn't notified until step 4, "The server
> MAY include data in the SYN-ACK packet (sic  pedant mode, TCP sends segments
> not packets)"

The draft was written for those majority, non-pedantic audience, who might not
appreciate the subtle difference between "segments" and "packets" :)

>in step 3 seems out of place unless we are assuming there are
> applications which will use this that want data sent before receiving
> anything, in which case that is another bit of user interface to describe
> somewhere.

No, this is more like the regular delayed ack case when the SYN-ACK is
delayed giving the app some time to respond. (Remember TFO will send up
the data included in the SYN pkt right away before the final ack.)

>
> *) 4.1.3 reads "Beside the cookie, we RECOMMEND that the client caches the
> MSS and RTT to the server to enhance performance."  Should that be the MSS
> as based (possibly) on later PMTU events during the life of the connection?
>  Meaning going to the cache not just on connection establishment, but also
> when there is a PTMU event.

Sure.

>
> *) 4.1.3 reads "For a multi-homed client, the cookies are both client and
> server IP dependent."  I ass-u-me that means they should be cached based on
> both client and server IP.  That is I suppose implicit with cookies being
> generated by servers but it rarely hurts to be explicit. However, why does
> it have to be client,server IP at the client?

Well for a multi-homed client it needs to pick cookies based on client & server
IP. (Maybe I've missed your question?)

>Particularly since section
> 4.1.2 point 5 says a client only needs to remember one valid cookie per
> server IP?

I guess you may have quoted the above out of context. The full text is

"5. A server may encode other information in the cookie, and allow
       more than one valid cookie per client at any given time. But this
       is all server implementation dependent and transparent to the
       client. A client only needs to remember one valid cookie per
       server IP."

The last sentence can be considered redundant.

>And if the client is only going to remember one, to what end
> would a server produce multiple cookies per client?

There was some usage case where the server could encode more info
than just client IP address in the cookie (if one picks a large cookie size).
But you are right if our cookie protocol implicitly allows only one valid
cookie visible to the client then I don't see/remember why server would
allow multiple valid cookies per client. Will defer this to my colleagues.

>
> *) 4.1.3 "The client MUST cache cookies from different servers for later..."
> - isn't "different" somewhat redundant?

It looks like the whole sentence is redundant.

Thanks again,

Jerry

> happy benchmarking,
>
> rick jones
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm
>

---

From: Rick Jones <[email-masked]>
Date: Mon, 29 Aug 2011 15:36:21 -0700

going through the draft again while composing this some new stuff:

*) section 2.1 - probably better to say ""i.e., to restrict TFO to a 
class of applications that are tolerant"

> http://www.ietf.org/proceedings/81/slides/tcpm-0.pdf.)

Slide five makes it sound like the definition of an amplification attack 
is one where what is emitted is more than one packet's worth of data.  I 
do not think that is correct.

Going well back in time, there was the issue of the way that the HP-UX 
(and I believe Solaris and a given version of Mac OS, all of which had a 
stack with a common ancestor) did PathMTU discovery - by sending a full 
link-local MTU ICMP echo request with the DF bit set in parallel with 
its attempted response to the remote IP, with the DF bit cleared. (The 
idea was to find the PathMTU without incurring the requisite packet loss 
on the data that "mattered") This was an amplification attack which sent 
only one packet.  And only needed a small inbound packet to trigger it.

As such, I don't think that one can avoid the taste of cookies if the 
server is to send data back before the 3WHS is complete.

Slide nine - presumably the client side will also employ the "TCP_TFO" 
option.

Slide 10 - what about server processing overhead in addition to latency. 
  The slide almost suggests that rather than teach them to eat cookies, 
we want give applications/stacks further incentives to keep warm - say 
with a blanket :)

Slide 12 - if the bufferbloat guys are "successful" where will RTTs be - 
will they be in the 200ms range from whcih the 41% claim comes or will 
it be closer to 100ms?  Also, those whole pages - is the w/ and w/o TFO 
based on IW3 or IW10?  And another question - is there a great deal of 
connection serialization there - the 200ms RTT example for the wikipedia 
page shows saving > 2 seconds. TFO gets rid of one RTT at the beginning 
of a connection, so that page had 10, serial TCP connections established 
to load it???  That seems as much a critique of the page design of the 
TCP wikipedia page as anything else given that the other three sites 
shown saw much less benefit.

Is there any way to separate "fundamential" serialization from "bad 
design?"  I like the idea of speeding things up, but I want to make 
certain that TFO isn't abused the way TCP_NODELAY can be.

> to TFO that needs to be addressed separately.

Though concurrently.  Leaving that to the next poor schmuck who needs 6 
bytes of option space doesn't seem all that, well, polite.  Pitty we 
cannot smash several common TCP options together into one uber-option... 
or can we?  Just how random does the initial timestamp *have* to be? 
But perhaps I digress...

> are initiated by the client side... Sigh (a train wreck is bound to happen).

All the more reason to describe why TCP option A is higher priority than 
TCP option B.  So when the trains do wreck it will be possible to 
point-out that it could have been avoided - well assuming that the 
advice would have prevented it :)

> extra CPU processing cost from AES_128.

I don't doubt that in the "good" case that sustained transaction rate 
would be higher.  If it weren't I suspect we wouldn't be discussing TFO 
in the first place :)

I'm wondering about the attack case, when the attacker is sending SYN 
segments with bogus cookies, or even just cookie requests, all from 
spoofed IPs.  That is a case where TFO does not reduce the number of 
segments sent, but does increase the processing overhead.  Particularly 
if the specified behaviour ends-up being to treat an invalid cookie as 
an implicit cookie request.

Yes, that is what the global counter is all about, but once that counter 
is reached, the server is forced to fall-back on pre-TFO behaviour, 
which means that server operators are going to want to make that limit 
rather high.  An invalid cookie is Really Cheap (tm) to bake.

Ten years ago I would have called worrying about the attack case an 
example of optimizing for the non-expected case, but today it seems that 
attacks are to be expected. :(

> cookie option nor acking the data in the SYN pkt.
>
> refresh the cookie cache etc but some of them can be left to the implementation.

I would think that if the client is caching effective MSS with the 
cookie, then it should be on a timescale matching that of the cache of 
discovered PathMTUs.

> Aha, you've caught some good corner case. It think my colleague wrote
> in that order because not replying a cookie nor acking the data will
> be treated by the client as the server disabling/de-supporting TFO.
> But I'm sure we can further fine tune the cookie protocol to address
> any ambiguity and performance optimization.

Unless the client side is caching a "negative response," which I don't 
recall the draft suggesting, the next connection attempt to the server 
by a TFO-enabling application on that client will naturally attempt to 
get a cookie again anyway right?  I don't think you need to address it.

>> Further, until cookie generation is shown to indeed be epsilon
>> compared to the rest of TCP connection establishment on the
>> server, the server should not include a new cookie when the cookie
>> was invalid and instead require the client to make another "from
>> scratch" cookie request.
>
> Yes we've thought about these performance issues and have gone
> through a few rounds of the seemingly simple cookie protocol design
> cycle. But I'm not sure your suggested "from scratch" approach will
> help much to defend against a malicious client if the cookie
> generation is expensive. I think it boils down to as long as the
> cookie generation is no more expensive than the processing overhead
> of a SYN pkt (upto checking the listener's qlimit), this case is no

Cookie generation, no matter how cheap, is still more than the server 
was doing on each SYN before no?  The calculus seems to be that the 
savings from avoiding an RTT (not simply avoiding packets - at least 
some of that could come from merely accepting data on the SYN and 
awaiting the 3WHS) is going to out-weigh what happens when under attack.

Presumably, an attacker seeking to extract maximum effect on the server 
is going to send streams of SYN segments with a bogus cookie and no more 
than one byte of data.  That will maximize his send rate and effect on 
the server.  Adding the cookie to the SYN only costs the attacker a 
tenth of a nanosecond of transmission time per 10 bits at the server's 
10GbE pipe, a cost that is going down all the time as servers get fatter 
pipes but it costs the server however many nanoseconds it takes to 
invalidate the cookie.  Attackers will almost always have more client 
systems than the good guys have servers no?

> appreciate the subtle difference between "segments" and "packets" :)

Never underestimate the value of pedantry in matters of details :)

> the data included in the SYN pkt right away before the final ack.)

I guess it is a good thing that the delayed ACK timer would not be 
started until after the cookie was known to be valid, or the time for 
that would have to be added into the calculus too :)

> Well for a multi-homed client it needs to pick cookies based on client&  server

No, I simply missed the point.

I was going to ask about NAT, but found section 5.  However, when that 
section says that there is no performance penalty it points towards 
"Client receiving SYN-ACK" - my pedantic :) search didn't find that, but 
I did find "Client: Receiving SYN-ACK"  Assuming that is the section, 
when carrier-grade NAT changes the client system's IP address as seen by 
the server, presumably the cookie the client sends will be invalid, 
which means unless the server holds-on to data in the SYN upon receipt 
of an invalid cookie, the client has to retransmit what it sent in the 
SYN - should step 2 of "Client: Receiving SYN-ACK" reflect that?

The server retaining data in a SYN upon receipt of an invalid cookie is 
another possible attack vector isn't it?  One that ties-up server memory 
for the time it takes the SYN-ACK to retransmit time-out.  True, that is 
no different than if there was a classic TCP which accepted data in the 
SYN - how many of them actually do that today?  TCP theory says a TCP 
today "should" not drop data in the SYN but still...  Since any hope of 
saving an RTT is already lost in the case of an invalid cookie, and 
since CKO makes receiving that data by the server TCP a second time 
essentially free, and since an attacker isn't going to retransmit anyway 
(?), I don't think you lose anything if you have the server drop the 
data in the SYN in the case of an invalid cookie and modify that step 2 
to immediately retransmit the data upon receipt of the SYN-ACK that does 
not contain a cookie option.

Getting back to carrier NAT - how often does the clients perceived IP 
address change?  Do the carriers make any attempt to make the IPs sticky 
or are they just round-robining them?

rick jones

---

From: rick jones <[email-masked]>
Date: Mon, 29 Aug 2011 19:25:36 -0700

> 
> Presumably, an attacker seeking to extract maximum effect on the server is going to send streams of SYN segments with a bogus cookie and no more than one byte of data.  That will maximize his send rate and effect on the server.  Adding the cookie to the SYN only costs the attacker a tenth of a nanosecond of transmission time per 10 bits at the server's 10GbE pipe, a cost that is going down all the time as servers get fatter pipes but it costs the server however many nanoseconds it takes to invalidate the cookie.  Attackers will almost always have more client systems than the good guys have servers no?

Math/unit fail - 10 bits per nanosecond at 10 GbE, so costs the attacker a nanosecond per 10 bits at the server's 10GbE pipe....

rick jones
http://homepage.mac.com/perfgeek

---

From: Jerry Chu <[email-masked]>
Date: Mon, 29 Aug 2011 22:33:12 -0700

On Mon, Aug 29, 2011 at 3:36 PM, Rick Jones <[email-masked]> wrote:
> *) section 2.1 - probably better to say ""i.e., to restrict TFO to a class
> of applications that are tolerant"

Ok.

>>> revision
> Slide five makes it sound like the definition of an amplification attack is
> one where what is emitted is more than one packet's worth of data.  I do not
> think that is correct.
>
> Going well back in time, there was the issue of the way that the HP-UX (and
> I believe Solaris and a given version of Mac OS, all of which had a stack
> with a common ancestor) did PathMTU discovery - by sending a full link-local
> MTU ICMP echo request with the DF bit set in parallel with its attempted
> response to the remote IP, with the DF bit cleared. (The idea was to find
> the PathMTU without incurring the requisite packet loss on the data that
> "mattered") This was an amplification attack which sent only one packet.
>  And only needed a small inbound packet to trigger it.
>
> As such, I don't think that one can avoid the taste of cookies if the server
> is to send data back before the 3WHS is complete.

But do you know what stack still does this? I believe Solaris was the
first stack that supported PMTUD in early to mid 90' by using the DF
bit while I was the main maintainer and I don't remember it ever sent
a parallel ICMP pkt.

I tend to believe the percentage of PMTU != interface MTU is small
plus caching past MTU info making the above mentioned optimization
unnecessary.

> Slide nine - presumably the client side will also employ the "TCP_TFO"

We've assumed client doesn't need an additional call to enable TFO
because it's sort of implied in its use of TFO's sendto() call.

But this brings up an issue with application portability. On system
where TFO is not supported or disabled, one will have to provide a
version of sendto() that is API compatible with the one running on TFO
enabled stack.

So maybe it's worthwhile for the app to find out whether TFO is
enabled on a local kernel through a socket option call before it
attempts sendto()/sendmsg().

> Slide 10 - what about server processing overhead in addition to latency.

I mentioned in previous email from one test mix we did it showed the saving
from one less pkt more than covers the additional processing overhead caused
by cookies.

>  The slide almost suggests that rather than teach them to eat cookies, we
> want give applications/stacks further incentives to keep warm - say with a
> blanket :)

Yes but persistent HTTP has been problematic (otherwise we won't have
bothered with TFO).

> will they be in the 200ms range from whcih the 41% claim comes or will it be
> closer to 100ms?  Also, those whole pages - is the w/ and w/o TFO based on
> IW3 or IW10?

The table in slide 12 was produced using an emulation environment with RTT
was injected by Dummynet so is exclusive of the queuing delay. I believe
the tests used IW10 but need to double check.

> And another question - is there a great deal of connection
> serialization there - the 200ms RTT example for the wikipedia page shows
> saving > 2 seconds. TFO gets rid of one RTT at the beginning of a
> connection, so that page had 10, serial TCP connections established to load
> it???  That seems as much a critique of the page design of the TCP wikipedia
> page as anything else given that the other three sites shown saw much less
> benefit.

We used the Chrome browser and a replay tool created by the Chrome team so
there is likely more serialization. We hope to publish more details in
an upcoming
paper.

>
> Is there any way to separate "fundamential" serialization from "bad design?"
>  I like the idea of speeding things up, but I want to make certain that TFO
> isn't abused the way TCP_NODELAY can be.

Again the Chrome team has invested lots of cycles gaining insight on PLT and we
were relying on them for meaningful testcases.

> bytes of option space doesn't seem all that, well, polite.  Pitty we cannot
> smash several common TCP options together into one uber-option... or can we?
>  Just how random does the initial timestamp *have* to be? But perhaps I
> digress...

It looks like you just brought up some great idea for future drafts.

> All the more reason to describe why TCP option A is higher priority than TCP
> option B.  So when the trains do wreck it will be possible to point-out that
> it could have been avoided - well assuming that the advice would have
> prevented it :)
>
>>> and
> I don't doubt that in the "good" case that sustained transaction rate would
> be higher.  If it weren't I suspect we wouldn't be discussing TFO in the
> first place :)
>
> segments with bogus cookies, or even just cookie requests, all from spoofed
> IPs.  That is a case where TFO does not reduce the number of segments sent,
> but does increase the processing overhead.  Particularly if the specified
> behaviour ends-up being to treat an invalid cookie as an implicit cookie
> request.
>
> Yes, that is what the global counter is all about, but once that counter is
> reached, the server is forced to fall-back on pre-TFO behaviour, which means
> that server operators are going to want to make that limit rather high.  An

No they won't because these potentially bogus TFO requests take up memory
just like pending SYN requests so they should and will  be treated similarly,
may even be subject to the same max qlen as regular SYN requests.

> invalid cookie is Really Cheap (tm) to bake.
>
> Ten years ago I would have called worrying about the attack case an example
> of optimizing for the non-expected case, but today it seems that attacks are
> to be expected. :(

Yup. Like I mentioned in my previous reply there is plenty of
opportunities for fine
tuning the protocol. E.g., the qlen check could happen before a cookie
is validated
like you suggested. This will cap the cookie validation overhead to no more than
is allowed by max qlen's worth of TFO requests. In order to enable the client to
differentiate between a TFO request hitting a non-TFO server vs a TFO request
hitting server's max qlen limit, a different TFO opcode can be returned by the
server in the latter case.

>>> happen
> I would think that if the client is caching effective MSS with the cookie,
> then it should be on a timescale matching that of the cache of discovered
> PathMTUs.
>
> recall the draft suggesting, the next connection attempt to the server by a
> TFO-enabling application on that client will naturally attempt to get a
> cookie again anyway right?  I don't think you need to address it.

Well some of these details are left to the implementation but for those
important ones we will certainly love to take input like yours and continue to
refine.

> Cookie generation, no matter how cheap, is still more than the server was
> doing on each SYN before no?  The calculus seems to be that the savings from
> avoiding an RTT (not simply avoiding packets - at least some of that could
> come from merely accepting data on the SYN and awaiting the 3WHS) is going
> to out-weigh what happens when under attack.
>
> Presumably, an attacker seeking to extract maximum effect on the server is
> going to send streams of SYN segments with a bogus cookie and no more than
> one byte of data.  That will maximize his send rate and effect on the
> server.  Adding the cookie to the SYN only costs the attacker a tenth of a
> nanosecond of transmission time per 10 bits at the server's 10GbE pipe, a
> cost that is going down all the time as servers get fatter pipes but it
> costs the server however many nanoseconds it takes to invalidate the cookie.
>  Attackers will almost always have more client systems than the good guys
> have servers no?

Again if cookie validation cost becomes a real issue we can also do what I
described earlier to subject it to the same max qlen check.

Moreover, if the cost of cookie generation also becomes a concern, we can
tweak the protocol to not return a valid cookie, when asked, as part of SYN-ACK,
but instead the server will wait till 3WHS finishes before handing out
cookie as part
of ack (unreliable), or data segment (reliable). Details need to be
worked out and
it's also a tradeoff between a little more complexity vs addressing the concern
about cookie processing overhead under attack.

> I guess it is a good thing that the delayed ACK timer would not be started
> until after the cookie was known to be valid, or the time for that would
> have to be added into the calculus too :)
>
>
> section says that there is no performance penalty it points towards "Client
> receiving SYN-ACK" - my pedantic :) search didn't find that, but I did find
> "Client: Receiving SYN-ACK"  Assuming that is the section, when
> carrier-grade NAT changes the client system's IP address as seen by the
> server, presumably the cookie the client sends will be invalid, which means
> unless the server holds-on to data in the SYN upon receipt of an invalid

No, the server will always drop data if the cookie is invalid.

> cookie, the client has to retransmit what it sent in the SYN - should step 2
> of "Client: Receiving SYN-ACK" reflect that?

It's sort of implied in the paragraph followed 3:
"Note there is no latency penalty if the server does not acknowledge
the data in the original SYN packet. The client will retransmit it in
the ACK packet..."

> another possible attack vector isn't it?

Hmm, not sure where in the draft gives the impression the server will retain
the data. Oh, it looks like the draft only stated to ack the SYN only but
never explicitly stated to drop the data. Will fix this.

>One that ties-up server memory for
> the time it takes the SYN-ACK to retransmit time-out.  True, that is no
> different than if there was a classic TCP which accepted data in the SYN -
> how many of them actually do that today?  TCP theory says a TCP today
> "should" not drop data in the SYN but still...  Since any hope of saving an
> RTT is already lost in the case of an invalid cookie, and since CKO makes
> receiving that data by the server TCP a second time essentially free, and
> since an attacker isn't going to retransmit anyway (?), I don't think you
> lose anything if you have the server drop the data in the SYN in the case of
> an invalid cookie and modify that step 2 to immediately retransmit the data
> upon receipt of the SYN-ACK that does not contain a cookie option.

Sorry for the confusion. We need intend for the server to retain data upon
a invalid cookie - the whole point why a cookie is used.
> address change?  Do the carriers make any attempt to make the IPs sticky or
> are they just round-robining them?

Will have to go back to the data we collected, or collect more data to answer
this question.

Thanks,

Jerry