Subject: [tcpm] ] WGLC for draft-ietf-tcpm-fastopen-05

From: Gorry <[email-masked]>
Date: Sat, 2 Nov 2013 16:09:41 -0700

I have reviewed the current version of TFO and have the comments below. This is rather long ...

Nits in Section 1 - I think the document here talks of servers and clients from a TCP endpoint perspective, this is normal for TCP, but the document also raises implications for apps designers who may have a very different view on what is intended by the word "server" and "client". To avoid doubt maybe you should explain this in the intro?

Page 3, para 2 is not clear. I think Otis just the detail of the wording, but it would be good to see clear text.

Last para section 2: to me, the requirements are not clear. 

I think this piece of text means: TCP stacks should NOT enable TCP-TFO by default. The stack should provide a global flag to enable this. An application wishing to use this must enable each TCP socket on a per service/per port basis.

I think the other comments/questions maybe more significant.

section 2 also suggests the main change is to allow data in SYN, I have noted that some people think this is the only change --- it is not. The bigger implication is that the document proposes to allow new data sent during the 3WHS. This change needs to be more clearly noted,  sending an IW of data before there is an ACK on the return path is a significant change to TCP semantics and it's behaviour under heavy load.

Page 5 fast open, bullet 3. : This says the server can send data before the 3WHS completes. I am curious what this actually means, is this implied that the use is extricated to standards-track mechanisms, I.e. I am looking here for some explicit understanding of whether this implies IW3. Or is the experiment linked to both a proposal for iW10 and this new experiment?

Last para 4.1.3 the text says /should include remote port numbers too/ Can we be clear which port numbers are used. Is this the received server port number by the client? Actually, do not understand what is intended by the paragraph, can you provide an example to be sure what is meant?

Section 4.3.2. I hoped this would say something explicit about not sending SYN segments that are larger than the default MTU. This seems like a bad thing to do.

Why are the examples only for web? This is a TCP spec.

There are two sections 6.3.2.

Second Section 6.3.2 - I think this applies to SSL/TLS in general, doesn't it? ... I think the section should address the general case, and then provide an example for https, rather than assuming a web-centric use.

Section 7.1  first para last line.please check text, this pathology is not necessarily malicious!!!
The issue seems to be mainly related to NAPT where ports are used to discriminate senders?
Also the shared NAPT can result in a range of RTT and PMTU to the same IP, but this is already possible with ECMP routing.

The lack of issues listed in section 7 raises serious concerns to me. I do not list that is provided currently as the only, or even most significant reasons why this is deemed to be experimental work. Indeed, the lack of identification of potential CC issues is why this document is in my opinion NOT ready to complete a WGLC - if the issues are not listed how can we evaluate if the method is safe enough to experiment or can later be evaluated properly.

Here are some additional issues I would like considered by the group and if the group is happy these may be safe enough to recommend testing in the general internet, then I think we should document that these are things to be confirmed.

1) I raised on the list that a negative impact of using the SYN as a probe to check for new option support. The examples I gave were ECN interactions and IPv6 probing. ECN if used will probe for ECN support in the SYN segment. If we include data on the SYN and then there is a lack of response, it is assumed by the sender that both TFO and ECN are not supported. This is an implication of the model we use to negotiate for features that may be black-holed. It is not an issue with the method, but I think we need tone clear this is something that may happen. A similar case exists if a client attempts TFO to a server using IPv6 ... Is it assumed that the path does not support IPv6 or TFO (or actually both). It does not happen when multiple options are negotiated.

2) the draft proposes caching RTT per path and using this for SYN timeout. This updates a standards track RFC, and will have implications in some network paths that may seriously degrade performance, in other cases it may improve performance. Examples of negative performance are cases where an initial packet can take much longer to process than a subsequent packet. This can and does occur in bandwidth-on-demand L2 networks, where the first packet causes path setup. It also occurs to some extent with policing and routing devices that build cached state from the first packet in a flow. I think this is one of the reasons why the RFC-series has to date not described a shortened RTT method for the SYN. 

Another issue is that previous RTT (or PMTU) is not necessarily a good indication of future value in a different 5tuple, since ECMP etc can lead to very different path characteristics for a session with a different port number. The current text therefore proposes an experiment that is beyond what was previously considered safe. It does not specify a way to avoid the server making the same mistake next connection.

3) seeding the SYN RTT from a lower value makes the sender more aggressive in heavily congested networks. The sender is also made more aggressive by sending IW data packets before there is any indication the path can contain even a single data segment. This is significant change to standard behaviour. If the proposal is to use IW 3 (as standard) it still needs to be called out. If the proposal is to use a larger experimental number then I have concerns here that this is a significant change that needs an automated recovery method to prevent significant collateral damage on capacity-limited paths - there needs to be a way to stop a server doing this each time resulting in recurring loss!

4) Are there any changes to PMTUD behaviour that need experience? PMTU was naturally cached and I am not sure if this changes the behaviour or not. It does of course allow a packet with a larger MTU to be sent before the path has been initially validated via the 3WHS (I.e. After a path change while idle, this can generate TCP segments in a SYN that are larger than the  receiver advertised MSS <I presume the recipient will reset the connection?> - I suspect this is not crucial, but I would also assume that the SYN itself should not use an increased PMTU value to carry the data. Is this correct? 

Are there other case where this change will impact the network for specific path characteristics? 

Have others thought through these issues and feel they are ok? Or at least ok for experimental deployment?

Gorry

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Wed, 6 Nov 2013 11:48:08 +0000

Hi all,

Our AD asked in the meeting on Monday whether there are further planned implementations of draft-ietf-tcpm-fastopen, beyond the Linux one.

Therefore, the TCPM chairs would really appreciate feedback (possibly privately) on potential implementations of draft-ietf-tcpm-fastopen. General interest in implementing the mechanism at some point in time would be useful as well.

Thanks!

Michael

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Scharf, Michael (Michael)
> Sent: Friday, October 25, 2013 2:21 PM
> To: [email-masked]
> Cc: [email-masked]
> Subject: [tcpm] WGLC for draft-ietf-tcpm-fastopen-05
> 
> Hi all,
> 
> The TCPM chairs believe that draft-ietf-tcpm-fastopen-05 addresses all
> comments received so far. This email therefore starts a working group
> last call running until ***November 10, 2013***.
> 
> IETF datatracker status page for this draft is:
> https://datatracker.ietf.org/doc/draft-ietf-tcpm-fastopen
> 
> There's also a htmlized version available at:
> http://tools.ietf.org/html/draft-ietf-tcpm-fastopen-05
> 
> The intended status of the document is Experimental.
> 
> The document asks for IANA allocation of a new TCP option number. The
> chairs' understanding is that it is TCPM consensus to ask for IESG
> approval of such a codepoint allocation.
> 
> Please send any comments on the TCPM mailing list until November 10.
> Notifications of approving the current version (even without other
> comments) are helpful as well.
> 
> Thanks!
> 
> Michael
> on behalf of the TCPM chairs
> 
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: Yuchung Cheng <[email-masked]>
Date: Sat, 9 Nov 2013 10:06:20 -0800

Hi Gorry,

Sorry for the last response. Too many good sessions and dinner parties
during this IETF for me to respond promptly.

> Nits in Section 1 - I think the document here talks of servers and clients from a TCP endpoint perspective, this is normal for TCP, but the document also raises implications for apps designers who…
They are already explained in the "Terminology" section.

> section 2 also suggests the main change is to allow data in SYN, I have noted that some people think this is the only change --- it is not. The bigger implication is that the document proposes to a…

will revise sec2 and discuss w/ you before we publish the revision.

> Page 5 fast open, bullet 3. : This says the server can send data before the 3WHS completes. I am curious what this actually means, is this implied that the use is extricated to standards-track mech…
will add "initial congestion window of " before the "... the
congestion control in .." to make it more clear.
or

> Last para 4.1.3 the text says /should include remote port numbers too/ Can we be clear which port numbers are used. Is this the received server port number by the client? Actually, do not understan…
a "remote" port number of the client means the "server port". but will
replace remote/server to avoid confusion.

The paragraph is added because of your suggestion to add negative caching.

> Section 4.3.2. I hoped this would say something explicit about not sending SYN segments that are larger than the default MTU. This seems like a bad thing to do.
It's a bad thing to send any segment larger than the MTU. Why is that
specific to Fast Open?

> Why are the examples only for web? This is a TCP spec.
Sure there are a lot more applications can benefit from that, but Web
is what motivates our work and the application is complex enough to
demonstrate many aspects of Fast Open. If you have another good
examples, you are welcome to suggest one.

> Second Section 6.3.2 - I think this applies to SSL/TLS in general, doesn't it? ... I think the section should address the general case, and then provide an example for https, rather than assuming a…
Yes it is generic, but anyone who knows SSL/TLS good enough knows that.

> Section 7.1  first para last line.please check text, this pathology is not necessarily malicious!!!
sure we can s/malicious/pathological. does this deserves three !!!?

> 1) I raised on the list that a negative impact of using the SYN as a probe to check for new option support. The examples I gave were ECN interactions and IPv6 probing. ECN if used will probe for EC…
if a SYN-data is dropped, TFO will retransmit a regular SYN (with ECN if used).

> Another issue is that previous RTT (or PMTU) is not necessarily a good indication of future value in a different 5tuple, since ECMP etc can lead to very different path characteristics for a session…

We'll just take RTT caching out of the text if that makes people happy.

> 3) seeding the SYN RTT from a lower value makes the sender more aggressive in heavily congested networks. The sender is also made more aggressive by sending IW data packets before there is any indi…

when the IW of packets are not all acked, loss recovery is triggered
and window is reduced. this is part of standard CC.

also when you use "significant" four times, please back up with a good
theory or data or both. and abuse that word does not it make more
significant.

> 4) Are there any changes to PMTUD behaviour that need experience? PMTU was naturally cached and I am not sure if this changes the behaviour or not. It does of course allow a packet with a larger MT…
in our implementation, we use what ever MTU (possibly discovered in a
prior PMTU) cached to send SYN-data.

---

From: Yuchung Cheng <[email-masked]>
Date: Wed, 13 Nov 2013 10:56:08 -0800

On Mon, Oct 28, 2013 at 4:35 PM, Smith, Donald
<[email-masked]> wrote:
Hi Donald,

Sorry for the late reply. I dropped your email b/c it was buried in
the busy discussion of code point.

> This doesn't work.
> Example Implementation: a simple implementation is to use AES_128 to
>    encrypt the IPv4 (with padding) or IPv6 address and truncate to 64
>    bits. The server can periodically update the key to expire the
>    cookies. AES encryption on recent processors is fast and takes only
>    few hundred nanoseconds [RCCJR11].
>
>    If only one valid cookie is allowed per-IP and the server can
>    regenerate the cookie independently, the best validation process is
>    to simply regenerate a valid cookie and compare it against the
>    incoming cookie. In that case if the incoming cookie fails the check,
>    a valid cookie is readily available to be sent to the client.
>
> That would allow someone doing reflection attacks to spoof an syn wait a few seconds for the server to respond (which the attacker wouldn't see) but the attacker knows what src ip he used in the syn so he can generate the syn/ack with a correct TFO cookie.

I am not sure I understand the issue. the attack can generate the
syn/ack with a correct TFO cookie if he got the secrete key in the
server. The defense is compromised if the secret key is not longer a
secret just like many systems ...

>
> And maybe I am missing something but this seems to have an extra step in it.
>
> Performing TCP Fast Open in connection 2:
>
>       TCP A (Client)                                    TCP B(Server)
>       ______________                                    _____________
>       CLOSED                                                   LISTEN
>
>    #1 SYN-SENT       ----- <SYN=x,CookieOpt=C,DATA_A> ---->  SYN-RCVD
>
>    #2 ESTABLISHED    <---- <SYN=y,ACK=x+len(DATA_A)+1> ----  SYN-RCVD
>
>    #3 ESTABLISHED    <---- <ACK=x+len(DATA_A)+1,DATA_B>----  SYN-RCVD
>
>    #4 ESTABLISHED    ----- <ACK=y+1>--------------------> ESTABLISHED
>
>    #5 ESTABLISHED    --- <ACK=y+len(DATA_B)+1>----------> ESTABLISHED
>
>
>
> I think you could skip 4 because 5 would be a valid ack so 4 shouldn't be required?
yes skip 4 can be skipped but this is just an illustration/example. We
do not imply #4 is required.

>
> Just one example:
> The server can periodically update the key to expire the cookies.
>
> So that should be TFO_server_cookie_expiration (or something like that) and probably should be configurable by the users.
>
>
> Next there are several timers defined or recommended but they weren't given names.
> I find when working with vendors if you have a standard name for a tcp wait or time out value it helps a lot.
>
> Beside the cookie, we RECOMMEND that the
>    client caches the MSS and RTT to the server to enhance performance.
>
> Just minor but RECOMMENDED is a key word but afaik RECOMMEND is not.
> It could easily be rewritten
> "Besides the cookie, it is RECOMMENDED that ..."
will fix that in next revision. thanks.

>
> This is just my first set of comments, based on a quick review. I believe there are more areas of concern but wanted to get these posted asap.
>
>
> (coffee != sleep) & (!coffee == sleep)
>  [email-masked]
>

---

From: Gorry Fairhurst <[email-masked]>
Date: Tue, 19 Nov 2013 17:21:26 +0000

See response in-line...

Many of these are small items,

Somewhere down there is the significant point that I do not yet 
understand, i.e. what are the potential CC issues that need to be 
evaluated - especially how TFO  respond to severe path overload? (see below)

Gorry

> They are already explained in the "Terminology" section.
OK, but it's terse!

>>
>> Page 3, para 2 is not clear. I think it is just the detail of the wording, but it would be good to see clear text.
>>
In RFC 6936 we wrote (albeit slightly different problem, with a 
different protocol):
         "IPv6 nodes MUST provide a way for the application/protocol to
         indicate the set of ports that will be enabled to send datagrams
         with a zero UDP checksum.  This may be implemented by enabling a
         transport mode using a socket API call when the socket is
         established, or by a similar mechanism.  It may also be
         implemented by enabling the method for a pre-assigned static
         port"

> will revise sec2 and discuss w/ you before we publish the revision.
Let's do that.
> replace remote/server to avoid confusion.
That works.

> The paragraph is added because of your suggestion to add negative caching.
OK

> It's a bad thing to send any segment larger than the MTU. Why is that
... Let me check:
1. TFO could allow PMTU to be cached across sessions.
2. Multi-homing/ Multipath routing could result in a different path for 
a new connection.
3. If TFO used a cahched PMTU with the SYN (or before the SYN-ACK), then
these could be black-holded.

- While local MTU is known, remote (or path) MTU can only be validated 
once there is a path... To me, allowing TFO to use larger MTU seems like 
an unnecessary extra complication to have to deal with.

> examples, you are welcome to suggest one.
.. I could OR it mya be wise to just say something similar to what you 
have written above at the end of section 1?

> Yes it is generic, but anyone who knows SSL/TLS good enough knows that.
So could you name the section TLS over TCP ... and then give HTTP over 
TLS as an example?

> sure we can s/malicious/pathological. does this deserves three !!!?
Fine (indeed, we want those nice people to fix all their middleboxes).

>> Also the shared NAPT can result in a range of RTT and PMTU to the same IP, but this is already possible with ECMP routing.
So In this case there is need to understand the implications of using a 
cached RTT and cached PMTU value. What concerns me is that we need 
experimental data to show that this is safe - not just to realise that 
this may have no benefit for TFO.

> if a SYN-data is dropped, TFO will retransmit a regular SYN (with ECN if used).
May be good, I agree - but I do not see this in the text (RFC 3168 would 
allow this, it kind of suggests clearing ECT on retx).
> We'll just take RTT caching out of the text if that makes people happy.
Maybe, although I'm not pushing for that, but it may be OK - if it is an 
orthogonal issue that TFO does not rely upon. If you want to mention 
caching these, I think the doc needs to explain more.

> theory or data or both. and abuse that word does not it make more
So let's see if we can see agree on what happens:

- If the path is very lossey - severe congestion:
TCP standard sends one SYN segment with some probability of loss, and if 
it sees loss backs off and retransmits the SYN. As (over)load increases, 
new sessions add 40B and the new sessions will often defer start-up, 
controlling their rate.

- If TFO path is very lossey - severe congestion: IW full-sized segments 
are sent. Each new session does this, adding to the load.
Each new session adds  MSS*IW = 6000B (IW=3) or 15000B (IW=10).

I concede this applies to severe congestion. Under this case, it seems 
150x more traffic to me and this before before CC is engaged. Or is this 
wrong?

> prior PMTU) cached to send SYN-data.
One option could be that the document could identify this as a topic to 
be explored (like RTT-caching, this does not seem to be a core technique 
required for FastOpen, but if you mention cachingthese, I think the WG 
should highlight that this
can also say there are potential concerns and a need for experimentation?

>> Have others thought through these issues and feel they are ok? Or at least ok for experimental deployment?
P.S. A tiny NiT on 7.2:

"Careful experimentation is necessary to evaluate if cookie-less TFO
    is practical."
- I think the experiment that is needed by the IETF is "to evaluate if 
cookie-less TFO could be safe for deployment in the general Internet."

---

From: Yuchung Cheng <[email-masked]>
Date: Sat, 23 Nov 2013 16:41:34 -0800

On Tue, Nov 19, 2013 at 9:21 AM, Gorry Fairhurst <[email-masked]> wrote:
> Somewhere down there is the significant point that I do not yet understand,
> i.e. what are the potential CC issues that need to be evaluated - especially
> how TFO  respond to severe path overload? (see below)
>
> Gorry
>
>
>>> Nits in Section 1 - I think the document here talks of servers and
>>> clients from a TCP endpoint perspective, this is normal for TCP, but the
>>> document also raises implications for apps designers who may have a very
>>> different view on what is intended by the word "server" and "client". To
>>> avoid doubt maybe you should explain this in the intro?
>>
>>> Page 3, para 2 is not clear. I think it is just the detail of the
>>> wording, but it would be good to see clear text.
>>>
> In RFC 6936 we wrote (albeit slightly different problem, with a different
> protocol):
>>> I think this piece of text means: TCP stacks should NOT enable TCP-TFO by
>>> default. The stack should provide a global flag to enable this. An
>>> application wishing to use this must enable each TCP socket on a per
>>> service/per port basis.
>>>
>>> section 2 also suggests the main change is to allow data in SYN, I have
>>> noted that some people think this is the only change --- it is not. The
>>> bigger implication is that the document proposes to allow new data sent
>>> during the 3WHS. This change needs to be more clearly noted,  sending an IW
>>> of data before there is an ACK on the return path is a significant change to
>>> TCP semantics and it's behaviour under heavy load.
>>
>>
>>> Page 5 fast open, bullet 3. : This says the server can send data before
>>> the 3WHS completes. I am curious what this actually means, is this implied
>>> that the use is extricated to standards-track mechanisms, I.e. I am looking
>>> here for some explicit understanding of whether this implies IW3. Or is the
>>> experiment linked to both a proposal for iW10 and this new experiment?
>>
>>> Last para 4.1.3 the text says /should include remote port numbers too/
>>> Can we be clear which port numbers are used. Is this the received server
>>> port number by the client? Actually, do not understand what is intended by
>>> the paragraph, can you provide an example to be sure what is meant?
>>
>>> Section 4.3.2. I hoped this would say something explicit about not
>>> sending SYN segments that are larger than the default MTU. This seems like a
>>> bad thing to do.
>>
> 2. Multi-homing/ Multipath routing could result in a different path for a
> new connection.
> - While local MTU is known, remote (or path) MTU can only be validated once
> there is a path... To me, allowing TFO to use larger MTU seems like an
> unnecessary extra complication to have to deal with.
I am still not getting your concern. Path MTU is cached on a per
"path" basis. If you use a different/new path you start with some
default MTU. btw, there is no Section 4.3.2 in the latest draft.

We only send SYN-data when we've done a previous handshake w/ the
server, and have both the cookie and path information. The path (mtu)
can change anyway, just like a normal data session, and the risk
holds. If SYN-data is lost, we will retransmit a regular SYN (and
cache the new MTU).

> .. I could OR it mya be wise to just say something similar to what you have
> written above at the end of section 1?
>
>
>>>
>>> Second Section 6.3.2 - I think this applies to SSL/TLS in general,
>>> doesn't it? ... I think the section should address the general case, and
>>> then provide an example for https, rather than assuming a web-centric use.
>>
>>
> So could you name the section TLS over TCP ... and then give HTTP over TLS
> as an example?
Sure and that's a good idea.

>>> Also the shared NAPT can result in a range of RTT and PMTU to the same
>>> IP, but this is already possible with ECMP routing.
>>>
> experimental data to show that this is safe - not just to realise that this
> may have no benefit for TFO.
>
>
>>> The lack of issues listed in section 7 raises serious concerns to me. I
>>> do not list that is provided currently as the only, or even most significant
>>> reasons why this is deemed to be experimental work. Indeed, the lack of
>>> identification of potential CC issues is why this document is in my opinion
>>> NOT ready to complete a WGLC - if the issues are not listed how can we
>>> evaluate if the method is safe enough to experiment or can later be
>>> evaluated properly.
>>>
>>> Here are some additional issues I would like considered by the group and
>>> if the group is happy these may be safe enough to recommend testing in the
>>> general internet, then I think we should document that these are things to
>>> be confirmed.
>>>
>>> 1) I raised on the list that a negative impact of using the SYN as a
>>> probe to check for new option support. The examples I gave were ECN
>>> interactions and IPv6 probing. ECN if used will probe for ECN support in the
>>> SYN segment. If we include data on the SYN and then there is a lack of
>>> response, it is assumed by the sender that both TFO and ECN are not
>>> supported. This is an implication of the model we use to negotiate for
>>> features that may be black-holed. It is not an issue with the method, but I
>>> think we need tone clear this is something that may happen. A similar case
>>> exists if a client attempts TFO to a server using IPv6 ... Is it assumed
>>> that the path does not support IPv6 or TFO (or actually both). It does not
>>> happen when multiple options are negotiated.
>>
>>> 2) the draft proposes caching RTT per path and using this for SYN
>>> timeout. This updates a standards track RFC, and will have implications in
>>> some network paths that may seriously degrade performance, in other cases it
>>> may improve performance. Examples of negative performance are cases where an
>>> initial packet can take much longer to process than a subsequent packet.
>>> This can and does occur in bandwidth-on-demand L2 networks, where the first
>>> packet causes path setup. It also occurs to some extent with policing and
>>> routing devices that build cached state from the first packet in a flow. I
>>> think this is one of the reasons why the RFC-series has to date not
>>> described a shortened RTT method for the SYN.
>>
>>
>>>
>>> Another issue is that previous RTT (or PMTU) is not necessarily a good
>>> indication of future value in a different 5tuple, since ECMP etc can lead to
>>> very different path characteristics for a session with a different port
>>> number. The current text therefore proposes an experiment that is beyond
>>> what was previously considered safe. It does not specify a way to avoid the
>>> server making the same mistake next connection.
>>
>>
> orthogonal issue that TFO does not rely upon. If you want to mention caching
> these, I think the doc needs to explain more.
>
>
>>>
>>> 3) seeding the SYN RTT from a lower value makes the sender more
>>> aggressive in heavily congested networks. The sender is also made more
>>> aggressive by sending IW data packets before there is any indication the
>>> path can contain even a single data segment. This is significant change to
>>> standard behaviour. If the proposal is to use IW 3 (as standard) it still
>>> needs to be called out. If the proposal is to use a larger experimental
>>> number then I have concerns here that this is a significant change that
>>> needs an automated recovery method to prevent significant collateral damage
>>> on capacity-limited paths - there needs to be a way to stop a server doing
>>> this each time resulting in recurring loss!
>>
>>
> TCP standard sends one SYN segment with some probability of loss, and if it
> sees loss backs off and retransmits the SYN. As (over)load increases, new
> sessions add 40B and the new sessions will often defer start-up, controlling
> their rate.
>
> - If TFO path is very lossey - severe congestion: IW full-sized segments are
> sent. Each new session does this, adding to the load.
> I concede this applies to severe congestion. Under this case, it seems 150x
> more traffic to me and this before before CC is engaged. Or is this wrong?
I can make similar case for any recent proposal that will blow up the
network, e.g., rto-restart, newcwv. The question is how practical and
common those cases are. if the link is thin, not doing Fast Open with
same IW will experience similar heavy losses. Let's face it: if the
capacity is not enough to handle the demand, there is little can be
done.

The concern has been raised by Michael Scharf's, and the step 6 in
Section 4.2.2 does mention that. I did ask the list for any better
solution for that but I've never heard back any suggestion.
>
>
>>>
>>> 4) Are there any changes to PMTUD behaviour that need experience? PMTU
>>> was naturally cached and I am not sure if this changes the behaviour or not.
>>> It does of course allow a packet with a larger MTU to be sent before the
>>> path has been initially validated via the 3WHS (I.e. After a path change
>>> while idle, this can generate TCP segments in a SYN that are larger than the
>>> receiver advertised MSS <I presume the recipient will reset the connection?>
>>> - I suspect this is not crucial, but I would also assume that the SYN itself
>>> should not use an increased PMTU value to carry the data. Is this correct?
>>
> One option could be that the document could identify this as a topic to be
> explored (like RTT-caching, this does not seem to be a core technique
Not sure RTT or PMTU caching should be part of TFO experimentation b/c
the potential concerns seem orthogonal to the core protocol of Fast
Open.

> cookie-less TFO could be safe for deployment in the general Internet."
Will make that change.

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Tue, 26 Nov 2013 14:05:50 +0000

> solution for that but I've never heard back any suggestion.

Indeed, I was (and I still am) worried by the combination of TFO and IW10. Because of that, we have already had similar list discussions about highly congested links. I am less concerned about TFO with IW3, because bursts of that size are sent by TCP anyway pretty frequently.

However, given the explicit reference to RFC 3390 in the current draft, I have not further objected during the WGLC.
 
Since this question still seems to trigger discussion, what about some additional explanation of "slightly more aggressive"? For instance:

OLD:

      Note that if SYN-ACK is lost, regular TCP reduces the initial
      congestion window before sending any data. In this case TFO is
      slightly more aggressive in the first data round trip even though
      it does not change the congestion control.

NEW:

      Note that in some cases TFO can be slightly more aggressive than regular TCP.
      If SYN-ACK is lost, regular TCP reduces the initial congestion window before sending any data.
      On such a congested link, TFO could first send a burst according to the initial window [RFC3390],
	and the server would then back-off after one round trip time if one or more packets are lost. In this case TFO is
      slightly more aggressive in the first data round trip even though it does not change the congestion control.

My 2 cents, with chair hat off...

Michael

---

From: Yuchung Cheng <[email-masked]>
Date: Tue, 26 Nov 2013 11:38:28 -0800

On Tue, Nov 26, 2013 at 6:05 AM, Scharf, Michael (Michael) <
[email-masked]> wrote:

> Indeed, I was (and I still am) worried by the combination of TFO and IW10.
> Because of that, we have already had similar list discussions about highly
> congested links. I am less concerned about TFO with IW3, because bursts of
> that size are sent by TCP anyway pretty frequently.
>
> However, given the explicit reference to RFC 3390 in the current draft, I
> have not further objected during the WGLC.
>
> Since this question still seems to trigger discussion, what about some
> additional explanation of "slightly more aggressive"? For instance:
>
> OLD:
>
>         and the server would then back-off after one round trip time if
> one or more packets are lost. In this case TFO is
sure

---

From: [email-masked]
Date: Tue, 26 Nov 2013 19:51:52 -0000

See below...

>> > >>> standard behaviour. If the proposal is to use IW 3 (as standard)
>> it
>> > still
>> > > TCP standard sends one SYN segment with some probability of loss,
>> and
>> > if it
>> > > sees loss backs off and retransmits the SYN. As (over)load
>> increases,
>> > new
>>       On such a congested link, TFO could first send a burst according
>> to
>> the initial window [RFC3390],

The description is basically OK - but I don't agree on two points:

My point is that this DOES change the congestion control in the most
critical starvation prevention mode (i.e. under heavy load): It allows a
single TCP session to send up to IW bytes into the network without
confirming the path is available. This is not a trivial change. It makes
significant assumptions about the buffering on the path, and the level of
current congestion - for high rate links this may be OK (likely no impact)
for low rate links this is a change that I would expect could make a
significant difference.

I don't understand why this is described as  "slightly" when a standard
TCP sends 40B in 1 packet before it has validated the path, and this
proposed change can send up to IW (10?) packets - that 10x the rate - in
bytes this increases to x250 the sending rate (which may be what matters
for slow links).

Whether this matters I think needs to be determined, especially if there
are robust ways to prevent this happening repeatedly over the same path.

Gorry

---

From: Yuchung Cheng <[email-masked]>
Date: Tue, 26 Nov 2013 12:00:01 -0800

On Tue, Nov 26, 2013 at 11:51 AM, <[email-masked]> wrote:

> >> does not change the congestion control.
> >>
> > sure
> >
>

> bytes this increases to x250 the sending rate (which may be what matters
I will remove the word "slightly" if you can stop using "significantly",
b/c both are subjective w/o good data to back-up

> are robust ways to prevent this happening repeatedly over the same path.
What's your idea?

---

From: Gorry Fairhurst <[email-masked]>
Date: Sat, 30 Nov 2013 17:07:49 +0000

Sorry for the delay (I was unwell), so this email is to restart.
I think we got a long way through discussing things, so I've
summarised below all the points as I understand from my side,
so we and others know where we have arrived and can fix others.

I support the experimentation with TFO, although I would argue
we do need to write more about how the proposed changes in this
ID will modify STD behaviour and explicitly say how to "negatively"
cache path info that indicates TFO should not be used.

I would like to see documented the cases where this could
@@potentially@@ be worse than STD mechanisms. The ID already
has some good documentation, but I think it needs a little
more detail especially the set of potential issues that we
expect to be mitigated by a cache at both the client and server
- especially when and why do we need to revoke the cookie
to prevent potentially bad pathologies from being
repeated in a short time between a pair of clients and servers
that keep starting up new connections.

---

The points below hopefully help us see what we can agree on:

a) Case: cached RTT greater than minRTO
    The current text says "RECOMMEND that the client caches the MSS
    and RTT to the server to enhance performance."

    I think ideally we should also explicitly say if the RTT is
    not cached TCP SHOULD disable use of TFO (make the cookie
    invalid) when the RTT>MinRTO.

    ... & note something about  "this prevents a premature RTO
    for flows with a long RTT and also provides an acceptable RTT
    for pacing" - I feel the current text focuses on performance
    improvement for a low RTT, but does not explain why it is
    important for large RTTs.

b) Case: cached RTT less than minRTO and path change
    We could just note that this ID does not change the behaviour
    that a path change can result in an invalid RTT/RTO value,
    and that normal TCP behaviour is expected to recover.

c) Case: receiver-advertised MSS less than 536 bytes.

    This is an unusual corner-case and is not one that I think we need
    to engineer for. However, I do think we need to note that an IPv4
    receiver advertised MSS less than 536 would result in
    transmission of an unexpected large segment to the receiver.
    (I.e. say we noted this, but don't change the recommendation).

d) Case: receiver-advertised MSS greater than 1460 B,
    i.e. allowing TFO with large segments.

    I think we should explicitly say MUST NOT use an MSS that
    results in a packet size greater than 1500B.

    - AND I think we should caution that we do not have current
    experience of the effect of this behaviour. We could note
    it as an area for further experimentation? I'm not sure what
    we agreed?

e) Case: A sender overwhelms a path with limited capacity.
    This is the corner case where a SYN may fail. The issue is most
    severe for links that suffer major overload (think low-speed
    large multiplexing).

    First SORRY - I'd like to avoid "slightly" and "significantly"
    wording  - we discussed on the thread - instead what matters
    to me really is:

    This draft updates the congestion-control behaviour while the
    sender is waiting acknowledgement for a SYN. This will result in
    more data (up to one IW of segments) being sent before the sender
    receives the ACK for the SYN. This could result in additional
    congestion on links that have a high loss rate where the initial
    ACK would have been lost, which in standard behaviour would have
    resulted in a lower initial congestion window.

    - personally I see the effect as important to document,
    but is OK for experimentation with an IW of 3 (see (g)).

f) The cookie-less experimental method will likely also need a negative
    cache to disable inappropriate TFO usage.

    I'd like to see an explicit note for the future experimentation
    something a bit like this would be good:

   "Even if cookie-less methods are found, it is expected that a
    future method will still need a way to detect conditions where
    TFO should not be used due to path properties and some
    equivalent method may then be needed to disable TFO."

g) I am unsure that allowing TFO to use IW=10 is safe!

Best wishes,

Gorry