Subject: [tcpm] draft-ietf-tcpm-fast-open

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Mon, 11 Mar 2013 20:05:30 +0100

Hi all,

With the chair hat off:

I've read draft-ietf-tcpm-fast-open. I think that the draft significantly improved, but I still came up with a number of questions:

* Section 3, "performing TCP fast open" (and other places): I recall both some meeting as well as mailing list discussion what will happen if draft-ietf-tcpm-fast-open is combined with draft-ietf-tcpm-initcwnd. I really wonder why there is hardly any text on congestion control implications in the draft. Even if the impact may be minor and congestion control is not the major focus of this draft, I think that implementors and users will wonder like me about that question and some explanation would help them. The way the draft is written right now seems to imply that an application developer may have to decide whether to experiment with IW10 or with fast open. If an app developer has to decide this, this will be a non-trivial tradeoff for him, in particular at design time.

* Section 4.1.1: I don't recall a discussion about cookie option size, right? Out of my head, 4 bytes seem reasonable; beyond that I wonder about the tradeoff between option size consumption and security. I am not sure, but maybe even 2 bytes could be sufficient in some cases (see also below)? At least, some guidance on the length would really make sense.

* Section 4.1.3: I wonder why the client "MUST" cache cookies. Why is not a "SHOULD"? At least if the client is short of memory, there might be valid reasons to "forget" cookies?

* Section 4.1.3: "Since TFO is enabled on a per-service port basis but cookies are independent of service ports, clients' cache should include remote port numbers too." Is it possible that there are different cookies for different service ports on the same server IP? This is probably not what the specification intends, but I wonder whether it would be a possible system design. I also wonder if that could improve security (e. g., possibly a shorter cookie)?

* Section 4.2.2: Thanks for adding the text on simultaneous open! I think that this scenario can only occur if fast open is activated on the corresponding ports on both endpoints, i. e., it is really a corner case. But are the fast open cookie semantics in that corner case indeed clear? In particular, I wonder whether one can savely distinguish between a client "Fast Open Cookie Request Option" (Sect. 4.1.1) and a server "null cookie" (Section 4.1.2).

* General: The draft explicitly states that API changes are outside the scope of the document. Still, I wonder if that could be discussed briefly in an non-normative appendix.

* General: This draft is heading for experimental track. Recent IESG feedback suggests that the IESG is interested in statements about the scope of such experiment. Maybe some text on that could be added? (Very obvious is further real-world experince on the performance benefit.)

Editorial nit: Section 2.1, second paragraph: "applications in section 2.1" looks like a self-reference

Thanks

Michael

---

From: Yuchung Cheng <[email-masked]>
Date: Mon, 11 Mar 2013 15:29:23 -0400

On Mon, Mar 11, 2013 at 3:05 PM, Scharf, Michael (Michael) <
[email-masked]> wrote:

> Hi all,
>
> With the chair hat off:
>
> I've read draft-ietf-tcpm-fast-open. I think that the draft significantly
> improved, but I still came up with a number of questions:
>
> * Section 3, "performing TCP fast open" (and other places): I recall both
> some meeting as well as mailing list discussion what will happen if
> draft-ietf-tcpm-fast-open is combined with draft-ietf-tcpm-initcwnd. I
> really wonder why there is hardly any text on congestion control
> implications in the draft. Even if the impact may be minor and congestion
> control is not the major focus of this draft, I think that implementors and
> users will wonder like me about that question and some explanation would
> help them. The way the draft is written right now seems to imply that an
> application developer may have to decide whether to experiment with IW10 or
> with fast open. If an app developer has to decide this, this will be a
> non-trivial tradeoff for him, in particular at design time.
>
What's the problem of experimenting both? whatever issue identified in IW10
draft will occur with TFO except it'll happen an RTT earlier, except for
this case, which is addressed in the new draft in 4.2.2

"" Note that if SYN-ACK is lost, regular TCP reduces the initial

   congestion window before sending any data. In this case TFO is
   slightly more aggressive in the first data round trip even though
   it does not change the congestion control.

"""

Maybe you can suggest what exactly to add in the draft because I am not
sure what to do.

>
> * Section 4.1.1: I don't recall a discussion about cookie option size,
> right? Out of my head, 4 bytes seem reasonable; beyond that I wonder about
> the tradeoff between option size consumption and security. I am not sure,
> but maybe even 2 bytes could be sufficient in some cases (see also below)?
> At least, some guidance on the length would really make sense.
>
> * Section 4.1.3: I wonder why the client "MUST" cache cookies. Why is not
> a "SHOULD"? At least if the client is short of memory, there might be valid
> reasons to "forget" cookies?
>
> if the client is short of memory, does it matter if  a MUST or a SHOULD? :(
Happy to change to SHOULD if other co-authors agree.

* Section 4.1.3: "Since TFO is enabled on a per-service port basis but
> cookies are independent of service ports, clients' cache should include
> remote port numbers too." Is it possible that there are different cookies
> for different service ports on the same server IP? This is probably not
> what the specification intends, but I wonder whether it would be a possible

of course, why not? the server applications (on different) may prefer to
use different secrets, whatever reason they think. in our Linux
implementation but we'd like to keep the option open.

> system design. I also wonder if that could improve security (e. g.,
> possibly a shorter cookie)?
>
> * Section 4.2.2: Thanks for adding the text on simultaneous open! I think
> that this scenario can only occur if fast open is activated on the
> corresponding ports on both endpoints, i. e., it is really a corner case.
> But are the fast open cookie semantics in that corner case indeed clear? In
> particular, I wonder whether one can savely distinguish between a client
> "Fast Open Cookie Request Option" (Sect. 4.1.1) and a server "null cookie"
> (Section 4.1.2).
>
I am not sure I understand the problem. When a (simul open) client sends a
SYN w/ FO cookie request option, he gets a SYN-ACK that may have the Fast
Open cookie option (from the other simul-open client)

The null cookie is only used by server responding SYN-ACK to a (simul open)
client sending SYN-data with Fast Open cookie option, not FO cookie
"request" option.

>
> * General: The draft explicitly states that API changes are outside the
> scope of the document. Still, I wonder if that could be discussed briefly
> in an non-normative appendix.
>
I actually consider doing that but I don't know any previous TCP RFC has
done that. I am happy to do that if tcpm list considers this appropriate.

>
> * General: This draft is heading for experimental track. Recent IESG
> feedback suggests that the IESG is interested in statements about the scope
> of such experiment. Maybe some text on that could be added? (Very obvious
> is further real-world experince on the performance benefit.)
>

I need some help on this text. maybe an example exp RFC that exactly scopes
its experiment.

> 2.1" looks like a self-reference

will fix that thx

>
> Thanks
>
> Michael
>
>
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm
>

---

From: Wesley Eddy <[email-masked]>
Date: Mon, 11 Mar 2013 15:56:05 -0400

On 3/11/2013 3:29 PM, Yuchung Cheng wrote:
>     feedback suggests that the IESG is interested in statements about
>     the scope of such experiment. Maybe some text on that could be
>     added? (Very obvious is further real-world experince on the
>     performance benefit.)
> 
>  
> I need some help on this text. maybe an example exp RFC that exactly
> scopes its experiment.  
>  

One example I can give you is the LEDBAT spec:
http://datatracker.ietf.org/doc/rfc6817/

See section 5.

It gives a summary of what some of the open questions were that the
WG felt were interesting, and what to look for while using LEDBAT.

(I don't know what the FastOpen document does/doesn't say at the
moment, but do know that the LEDBAT one was "good enough" for the
IESG at the time ... which was relatively recently)

-- 
Wes Eddy
MTI Systems

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Mon, 11 Mar 2013 21:30:32 +0100

> 	* Section 3, "performing TCP fast open" (and other 
> places): I recall both some meeting as well as mailing list 
> discussion what will happen if draft-ietf-tcpm-fast-open is 
> combined with draft-ietf-tcpm-initcwnd. I really wonder why 
> there is hardly any text on congestion control implications 
> in the draft. Even if the impact may be minor and congestion 
> control is not the major focus of this draft, I think that 
> implementors and users will wonder like me about that 
> question and some explanation would help them. The way the 
> draft is written right now seems to imply that an application 
> developer may have to decide whether to experiment with IW10 
> or with fast open. If an app developer has to decide this, 
> this will be a non-trivial tradeoff for him, in particular at 
> design time.
> 	
> 
> What's the problem of experimenting both? whatever issue 
> identified in IW10 draft will occur with TFO except it'll 
> happen an RTT earlier, except for this case, which is 
> addressed in the new draft in 4.2.2 
> 

This statement is at some rather "random" position in the protocol spec and (at least to me) hard to parse. Our past discussion http://www.ietf.org/mail-archive/web/tcpm/current/msg07329.html describes more verbosely what this issue is, and whether it matters (or not).

Why don't you add a dedicated subsection "Congestion Control Implications" to Section 5?

> Maybe you can suggest what exactly to add in the draft 
> because I am not sure what to do.

This statement 

" 6. Advance to the SYN-RCVD state. If the FastOpened flag is set, the
    server MUST follow the congestion control [RFC5681], in particular
   the initial congestion window [RFC3390], to send more data   packets."

seems to imply to me that a stack either uses fast open or IW10, but MUST not use them both, right? If so, I think that this requires guidance for the app developer do decide whether to select either IW10 or fast open.

If this is not your intention, i. e., it should be allowed that IW10 and fast open are combined, this should be explicitly explained in the document, imho. IW10 will probably be an RFC when this document gets published, and their relationship is pretty apparent. As I said, I really think that people will ask this question.

> 	* Section 4.1.3: "Since TFO is enabled on a per-service 
> port basis but cookies are independent of service ports, 
> clients' cache should include remote port numbers too." Is it 
> possible that there are different cookies for different 
> service ports on the same server IP? This is probably not 
> of course, why not? the server applications (on different) 
> may prefer to use different secrets, whatever reason they 
> think. in our Linux implementation but we'd like to keep the 
> option open.

... but that question implies interoperability, because the client SW design depends on what the cookie is about. Not every TCP endpoint is Linux ;)

> 	system design. I also wonder if that could improve 
> security (e. g., possibly a shorter cookie)?
> 	
> 	* Section 4.2.2: Thanks for adding the text on 
> simultaneous open! I think that this scenario can only occur 
> if fast open is activated on the corresponding ports on both 
> endpoints, i. e., it is really a corner case. But are the 
> fast open cookie semantics in that corner case indeed clear? 
> In particular, I wonder whether one can savely distinguish 
> between a client "Fast Open Cookie Request Option" (Sect. 
> 4.1.1) and a server "null cookie" (Section 4.1.2).
> 	
> 
> I am not sure I understand the problem. When a (simul open) 
> client sends a SYN w/ FO cookie request option, he gets a 
> SYN-ACK that may have the Fast Open cookie option (from the 
> other simul-open client)

If in a simulatenous open both SYNs carry an empty FO cookie option, what is the correct behavior?
 
> The null cookie is only used by server responding SYN-ACK to 
> a (simul open) client sending SYN-data with Fast Open cookie 
> option, not FO cookie "request" option.

I don't understand. In simulateous open, the role of client and server is fuzzy. And the distinction between client and server is quite inherent to fast open. This is why I wonder about that corner case..

Michael

---

From: Yuchung Cheng <[email-masked]>
Date: Mon, 11 Mar 2013 17:07:23 -0400

On Mon, Mar 11, 2013 at 4:30 PM, Scharf, Michael (Michael) <
[email-masked]> wrote:

> This statement is at some rather "random" position in the protocol spec
> and (at least to me) hard to parse. Our past discussion
> http://www.ietf.org/mail-archive/web/tcpm/current/msg07329.html describes
> more verbosely what this issue is, and whether it matters (or not).
>
personally I am fine w/ a dedicated subsection but have to consult other
co-authors.

> seems to imply to me that a stack either uses fast open or IW10, but MUST
> not use them both, right? If so, I think that this requires guidance for
> the app developer do decide whether to select either IW10 or fast open.
>

> If this is not your intention, i. e., it should be allowed that IW10 and
> fast open are combined, this should be explicitly explained in the
> document, imho. IW10 will probably be an RFC when this document gets
> published, and their relationship is pretty apparent. As I said, I really
> think that people will ask this question.
>
No I do not imply that. My intention is to say TFO should follow TCP CC,
whatever IETF RFC recommends.  This is the answer to your or similar
questions like can TFO work with any other changes to CC).

> ... but that question implies interoperability, because the client SW
> design depends on what the cookie is about. Not every TCP endpoint is Linux
> ;)

Please help me understand by an example of how sever applications on port
17483 and port 432 using two masters keys talking to different TFO client
implementation may break.

A TFO cookie is only meaningful for server applications that permit TFO.

> If in a simulatenous open both SYNs carry an empty FO cookie option, what

On receiving the empty FO cookie option, aka FO cookie request, the server
(also a simul-open client) responds the cookie in SYN-ACK if it supports
TFO on that port.
details are in  Section  4.2.2 " Server: Receiving SYN and responding with
SYN-ACK"

> I don't understand. In simulateous open, the role of client and server is
> fuzzy. And the distinction between client and server is quite inherent to
> fast open. This is why I wonder about that corner case..
>
If not too much trouble please give a exact case of how simul. fast open
won't work. I don't understand your case. sorry.

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Tue, 12 Mar 2013 00:07:00 +0100

> 	seems to imply to me that a stack either uses fast open 
> or IW10, but MUST not use them both, right? If so, I think 
> that this requires guidance for the app developer do decide 
> whether to select either IW10 or fast open.
> 	
> 
> 
> 	If this is not your intention, i. e., it should be 
> allowed that IW10 and fast open are combined, this should be 
> explicitly explained in the document, imho. IW10 will 
> probably be an RFC when this document gets published, and 
> their relationship is pretty apparent. As I said, I really 
> No I do not imply that. My intention is to say TFO should 
> follow TCP CC, whatever IETF RFC recommends.  This is the 
> answer to your or similar questions like can TFO work with 
> any other changes to CC). 

The text has a MUST for RFC5681 and RFC3390. This prevents IW10, imho. As a result, I think that your text does not describe what you want. But if you replace that MUST for IW3 to something less restricitive, you will need to have to comment on congestion control.

> 	... but that question implies interoperability, because 
> the client SW design depends on what the cookie is about. Not 
> every TCP endpoint is Linux ;)
> 
> Please help me understand by an example of how sever 
> applications on port 17483 and port 432 using two masters 
> keys talking to different TFO client implementation may break.
> 

I was thinking of one BSD client talking to ports 17483 and 432 on the same destination IP address. That BSD client has to know whether to query for one or two cookies.

And actually I still struggle with understanding which of the two options is the better one from a protocol perspective.

> 	If in a simulatenous open both SYNs carry an empty FO 
> cookie option, what is the correct behavior?
> 	
> 
> 
> On receiving the empty FO cookie option, aka FO cookie 
> request, the server (also a simul-open client) responds the 
> cookie in SYN-ACK if it supports TFO on that port.
> details are in  Section  4.2.2 " Server: Receiving SYN and 
> responding with SYN-ACK"

With simulatous open, I think speaking of client and server should be avoided.

> 	I don't understand. In simulateous open, the role of 
> client and server is fuzzy. And the distinction between 
> client and server is quite inherent to fast open. This is why 
> I wonder about that corner case..
> 	
> 
> If not too much trouble please give a exact case of how 
> simul. fast open won't work. I don't understand your case. sorry.

I don't claim that it does not work. I just want to understand whether more text is needed or not. And whether the procotol state engines are designed for that case, or not. Maybe it is trivial. Maybe not. 

For instance, this pattern seems to result in the situation that both endpoints get a valid cookie, i. e., both A and B are client and server at the same time:

      TCP A                                                     TCP B
      ______________                                    _____________
      CLOSED                                                   CLOSED

  #1  SYN-SENT       ----> <SYN,CookieOpt=NIL>  ...........    CLOSED

  #2  SYN-SENT       ..... <SYN,CookieOpt=NIL>  <----------  SYN-SENT

  #3  SYN-SENT       ..... <SYN,CookieOpt=NIL>  ---------->  SYN-RCVD

  #4  SYN-RECV       <---- <SYN,CookieOpt=NIL>  ...........  SYN-RCVD

  #5 ESTABLISHED     <---- <SYN,ACK,CookieOpt=C1> ----------  ESTABLISHED

  #6 ESTABLISHED     ----- <SYN,ACK,CookieOpt=C2> --------->  SYN-RCVD

At least, the document does not state right now that one TCP endpoint can be both fast open client and server for one connection at the same time (even if it is unlikely in practice). This seems also imply that in the next connection, if simultaneous open is used again, there could be bi-directional fast open with data in SYNs.

Furthermore, it is unclear to me what happens if #4 and #5 get reordered. Does TCP A process the cookie request from B, or not?

Sorry, I haven't thought of more complicated cases so far. I've never studied the simulateous open code in real stacks.

As a side note, there seems to be a general inconsistency issue in the draft: Section 4.1.1 does apparently not foresee that a client gets an empty cookie option, while this is allowed in Section 4.1.2, right?

Michael

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Wed, 13 Mar 2013 20:07:49 +0000

A few nits:

Section 2.1, 3rd paragraph should be a properly formatted list:

   a) the receiver host receives data in a duplicate SYN after it has
   forgotten it received the original SYN (e.g. due to a reboot); b) the
   duplicate is received after the connection created by the    original
   SYN has been closed and the close was initiated by the    sender (so
   the receiver will not be protected by the 2MSL TIMEWAIT    state).

Page 5, first paragraph:

   TFO goes one step further to allow server-side TCP to process and
   send up data to the application layer before 3WHS is completed.

"send data up to the" reads better...

Section 3:
Line breaks before the list would look better:

   future TCP connections to exchange data during 3WHS.

   Requesting a Fast Open Cookie:

   1. The client sends a SYN with a Fast Open Cookie Request option.

Section 4.1.2:
the implementation is often server
   specific.

Remove the "often" (alternatively, an example of a not server specific key rotation scheme would be needed. But that somehow collides with the text before this).

The Null cookie is not defined (last paragraph, same section). As mentioned earlier, if there exists a special entity (NULL cookie, ie. filled with all zeros or something), this should be documented if the client is to do anything with it (like purging it's TFO cookie cache for that server). In that context however, the null cookie can be any arbitrary binary blob that is just not verified on the server, for debugging purposes. Further, if the null cookie might look like a regular cookie (all zeros may also be a valid cookie, for example), this may require documentation of a special case when a collision between a valid cookie and the null cookie happens (albeit, the impact would be minimal, I believe; an impacted client would simply not be allowed TFO until a key change; so this may not even require any code change at all.

The abbreviation TCB (TCP control block) is not defined in the document (but I think writing it out before first use and not pointing to rfc793 there should do).

Section 4.2.2, Step 1b). Is the text stating 536 bytes clear enough, that this does not necessarily mean 536 data bytes (TCP option size must be subtracted)? Perhaps a reference to RFC6691 might be good here...

Also Section 4.2.2 the formatting (indentation) seem to have gone awry...

Section 6.2: 3., 4. Paragraph
"But she may be" - while I like the thought that a malicious adversary is a female (to some extent :), I think that may be s/she/the attacker/g. That's still gender neutral.

Section 6.3: s/ outweigh/ outweight/ ?

Best regards,

Richard Scheffenegger

---

From: Yuchung Cheng <[email-masked]>
Date: Wed, 13 Mar 2013 15:37:26 -0700

> The text has a MUST for RFC5681 and RFC3390. This prevents IW10, imho. As a result, I think that your text does not describe what you want. But if you replace that MUST for IW3 to something less re…

could you suggest some text if you agree on my intention?
I am not sure why citing the current RFCs in a draft precludes future
updates or related experimental RFCs. By that rationale, any draft
with text "MUST follow RFCxxx" should change to "MUST follow RFCxxx
but also the future updated version but does not preclude using
related experimental RFCs".

> I was thinking of one BSD client talking to ports 17483 and 432 on the same destination IP address. That BSD client has to know whether to query for one or two cookies.
The TFO is per dst-ip-port, so the client TCP stack should request
cookies from both port.

> For instance, this pattern seems to result in the situation that both endpoints get a valid cookie, i. e., both A and B are client and server at the same time:
in a simul. open #4 should be
#4  SYN-RECV       <---- <SYN,ACK,CookieOpt=C1>  ...........  SYN-RCVD

It seems you are looking at cookie request during a "regular" TCP
simultaneous open? not simultaneous Fast Open?
Since cookie request is just passing options, and the receiver of the
SYN always respond with SYN-ACK with the cookie (null or non-null),
there shouldn't be any problem, just like other option negotiation.

---

From: Yuchung Cheng <[email-masked]>
Date: Wed, 13 Mar 2013 23:04:31 -0700

> The Null cookie is not defined (last paragraph, same section). As mentioned earlier, if there exists a special entity (NULL cookie, ie. filled with all zeros or something), this should be documente…

null cookie is a cookie option with zero option length. if the active
SYN sender sends such an option, it is called a Fast Open Cookie
Request. For a SYN-receiver (that sends such an option in SYN-ACK),
it's called null-cookie. I will add that to the document and fix other
formatting and wording issues mentioned in this email. thx