Subject: Re: [tcpm] Summary of mt comments/questions WGLC for draft-ietf-tcpm-fastopen-05

From: Yuchung Cheng <[email-masked]>
Date: Sun, 1 Dec 2013 10:07:58 -0800

On Sat, Nov 30, 2013 at 9:07 AM, Gorry Fairhurst <[email-masked]>wrote:

>
> Sorry for the delay (I was unwell), so this email is to restart.
> I think we got a long way through discussing things, so I've
> summarised below all the points as I understand from my side,
> so we and others know where we have arrived and can fix others.
>
> I support the experimentation with TFO, although I would argue
> we do need to write more about how the proposed changes in this
> ID will modify STD behaviour and explicitly say how to "negatively"
> cache path info that indicates TFO should not be used.
>
> I would like to see documented the cases where this could
> @@potentially@@ be worse than STD mechanisms. The ID already
> has some good documentation, but I think it needs a little
> more detail especially the set of potential issues that we
> expect to be mitigated by a cache at both the client and server
> - especially when and why do we need to revoke the cookie
> to prevent potentially bad pathologies from being
> repeated in a short time between a pair of clients and servers
> that keep starting up new connections.
>
> ---
>
> The points below hopefully help us see what we can agree on:
>
> a) Case: cached RTT greater than minRTO
>    The current text says "RECOMMEND that the client caches the MSS
>    and RTT to the server to enhance performance."
>
>    I think ideally we should also explicitly say if the RTT is
>    not cached TCP SHOULD disable use of TFO (make the cookie
>    invalid) when the RTT>MinRTO.
>
>    ... & note something about  "this prevents a premature RTO
>    for flows with a long RTT and also provides an acceptable RTT
>    for pacing" - I feel the current text focuses on performance
>    improvement for a low RTT, but does not explain why it is
>    important for large RTTs.
>

> b) Case: cached RTT less than minRTO and path change
>    We could just note that this ID does not change the behaviour
>    that a path change can result in an invalid RTT/RTO value,
>    and that normal TCP behaviour is expected to recover.
>
no on a and b.

The agreed change is to remove RTT caching in TFO spec.
TFO will benefit higher RTT client more. Do I need to explain such a simple
logic further?

min-RTO is not defined in IETF RFCs, nor do I see your new logic improves
TFO b/c ultimately cached RTT can be out-dated. I don't see how tweaking
TFO with historical RTT values can make it safer.

>
> c) Case: receiver-advertised MSS less than 536 bytes.
>
>    This is an unusual corner-case and is not one that I think we need
>    to engineer for. However, I do think we need to note that an IPv4
>    receiver advertised MSS less than 536 would result in
>    transmission of an unexpected large segment to the receiver.
>    (I.e. say we noted this, but don't change the recommendation).
>
Why is that specific to TFO?

>
> d) Case: receiver-advertised MSS greater than 1460 B,
>    i.e. allowing TFO with large segments.
>
>    I think we should explicitly say MUST NOT use an MSS that
>    results in a packet size greater than 1500B.
>
No. if you send a packet too big you risk it getting lost and wish you get
some ICMP warnings. Why is that specific to TFO?

>
>    - AND I think we should caution that we do not have current
>    experience of the effect of this behaviour. We could note
>    it as an area for further experimentation? I'm not sure what
>    we agreed?
>
ok. Sending SYN-data is indeed a new behavior and an area for further
experimentation. I think that's why TFO-draft is experimental. I could add
more words about this in the "areas for experimentation".

>
>
> e) Case: A sender overwhelms a path with limited capacity.
>    This is the corner case where a SYN may fail. The issue is most
>    severe for links that suffer major overload (think low-speed
>    large multiplexing).
>
>    First SORRY - I'd like to avoid "slightly" and "significantly"
>    wording  - we discussed on the thread - instead what matters
>    to me really is:
>
>    This draft updates the congestion-control behaviour while the
>    sender is waiting acknowledgement for a SYN. This will result in
>    more data (up to one IW of segments) being sent before the sender
>    receives the ACK for the SYN. This could result in additional
>    congestion on links that have a high loss rate where the initial
>    ACK would have been lost, which in standard behaviour would have
>    resulted in a lower initial congestion window.
>
I can merge the text above into the draft.

>
>    - personally I see the effect as important to document,
>    but is OK for experimentation with an IW of 3 (see (g)).
>
> f) The cookie-less experimental method will likely also need a negative
>    cache to disable inappropriate TFO usage.
>
>    I'd like to see an explicit note for the future experimentation
>    something a bit like this would be good:
>
>   "Even if cookie-less methods are found, it is expected that a
>    future method will still need a way to detect conditions where
>    TFO should not be used due to path properties and some
>    equivalent method may then be needed to disable TFO."
>
ok.

>
> g) I am unsure that allowing TFO to use IW=10 is safe!
>

> Best wishes,
> Gorry

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Mon, 2 Dec 2013 13:49:11 +0000

Some comments, with chair hat off...

a)+b):

I have no strong opinion on whether the connection initiator (client) should cache the RTT, or not. It seems partly orthogonal to TFO. If the RTO expires spuriously, the document recommends to retransmit without data in SYN, i.e., the amount of bytes sent into the network is the same as for regular TCP. From a congestion control perspective, the result is more or less the same.

However, after having followed this discussion, I wonder why the document does not comment at all about RTT estimation at the connection acceptor (server). With TFO, the connection acceptor (server) is allowed to send a full IW of data back to the connection initiator (client), following the SYN-ACK. However, at that point in time, the connection acceptor (server) has no recent RTT measurement, right? This is different to the traditional TCP handshake, in which an RTT measurement is available before data is sent into the network.  The fact that TFO allows sending of a full IW of data before an RTT estimation is available should be mentioned as a warning sign, IMHO. For instance, it will make it more difficult to apply pacing.

c)+d):

In almost all existing use of regular TCP, the endpoints negotiate the MSS before actually sending any data, because adding data to SYNs doesn't make a lot of sense without TFO. Therefore, "MSS clamping" in middleboxes usually works for regular TCP, and it may avoid PMTUD procedures in many cases. With TFO, the situation can be more complex. In addition to the case of MSS<536 bytes raised by Gorry, it is also not clear to me what a middlebox should do if it receives a TFO SYN with data, and if this segment is larger than the "MSS clamping" value of that middlebox. This corner case can occur e.g. after path routing changes. Therefore, I agree with Gorry that the document should at least mention that MSS issues may require further experimentation.

Thanks

Michael

From: tcpm [mailto:[email-masked]] On Behalf Of Yuchung Cheng
Sent: Sunday, December 01, 2013 7:08 PM
To: Gorry Fairhurst
Cc: [email-masked]; [email-masked]
Subject: Re: [tcpm] Summary of mt comments/questions WGLC for draft-ietf-tcpm-fastopen-05

On Sat, Nov 30, 2013 at 9:07 AM, Gorry Fairhurst <[email-masked]<mailto:[email-masked]>> wrote:

---

The agreed change is to remove RTT caching in TFO spec.
TFO will benefit higher RTT client more. Do I need to explain such a simple logic further?

min-RTO is not defined in IETF RFCs, nor do I see your new logic improves TFO b/c ultimately cached RTT can be out-dated. I don't see how tweaking TFO with historical RTT values can make it safer.

   I think we should explicitly say MUST NOT use an MSS that
   results in a packet size greater than 1500B.
No. if you send a packet too big you risk it getting lost and wish you get some ICMP warnings. Why is that specific to TFO?

   - AND I think we should caution that we do not have current
   experience of the effect of this behaviour. We could note
   it as an area for further experimentation? I'm not sure what
   we agreed?
ok. Sending SYN-data is indeed a new behavior and an area for further experimentation. I think that's why TFO-draft is experimental. I could add more words about this in the "areas for experimentation".

Best wishes,

Gorry

---

From: Yuchung Cheng <[email-masked]>
Date: Mon, 2 Dec 2013 16:32:24 -0800

On Mon, Dec 2, 2013 at 5:49 AM, Scharf, Michael (Michael) <
[email-masked]> wrote:

>  Some comments, with chair hat off…
>
>
>
> a)+b):
>
>
>
> I have no strong opinion on whether the connection initiator (client)
> should cache the RTT, or not. It seems partly orthogonal to TFO. If the RTO
> expires spuriously, the document recommends to retransmit without data in
> SYN, i.e., the amount of bytes sent into the network is the same as for
> regular TCP. From a congestion control perspective, the result is more or
> less the same.
>
>
>
> However, after having followed this discussion, I wonder why the document
> does not comment at all about RTT estimation at the connection acceptor
> (server). With TFO, the connection acceptor (server) is allowed to send a
> full IW of data back to the connection initiator (client), following the
> SYN-ACK. However, at that point in time, the connection acceptor (server)
> has no recent RTT measurement, right? This is different to the traditional
> TCP handshake, in which
>
correct

> an RTT measurement is available before data is sent into the network.  The
> fact that TFO allows sending of a full IW of data before an RTT estimation
> is available should be mentioned as a warning sign, IMHO. For instance, it
> will make it more difficult to apply pacing.
>
I can add that point about RTT estimation.

>
>
> c)+d):
>
>
>
> In almost all existing use of regular TCP, the endpoints negotiate the MSS
> before actually sending any data, because adding data to SYNs doesn’t make
> a lot of sense without TFO. Therefore, “MSS clamping” in middleboxes
> usually works for regular TCP, and it may avoid PMTUD procedures in many
> cases. With TFO, the situation can be more
>
I need more information about the MSS clamping feature.

My understanding (which can be wrong) of MSS clamping is that the
middle-box is transparently reducing the value in the MSS option in the SYN
packet. If so the connection that negotiate the cookie, should receive and
cache the reduced value, which will be used by the later Fast Open
connection.

It is possible the middle-box lowers the MSS prior the to SYN-data. In that
case it is equivalent to sending a SYN-data, or a data packet on a new path
that has lower MTU. In that case, the remedy is ICMP error or ultimately a
timeout. When recurring SYN-data timeouts happen, then the client should
disable Fast Open (temporarily) due to the negative caching suggested in
the draft.

> complex. In addition to the case of MSS<536 bytes raised by Gorry, it is
> also not clear to me what a middlebox should do if it receives a TFO SYN
> with data, and if this segment is larger than the “MSS clamping” value of
> that middlebox. This corner case can occur e.g. after path routing changes.
> Therefore, I agree with Gorry that the document should at least mention
> that MSS issues may require further experimentation.
>
As replied earlier, I will document SYN-data / middlebox issue as the "area
of experimentation" in the previous email.

>
>
> Thanks
>
>
>
> Michael
>
>
>
>
>
> *From:* tcpm [mailto:[email-masked]] *On Behalf Of *Yuchung Cheng
> *Sent:* Sunday, December 01, 2013 7:08 PM
> *To:* Gorry Fairhurst
> *Cc:* [email-masked]; [email-masked]
> *Subject:* Re: [tcpm] Summary of mt comments/questions WGLC for
> draft-ietf-tcpm-fastopen-05
>
>
>
>
>
>
>
>  No. if you send a packet too big you risk it getting lost and wish you
> get some ICMP warnings. Why is that specific to TFO?
>
>
>
>

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Tue, 3 Dec 2013 09:31:48 +0000

Hi Yuchung,

I guess one of the key points here is that in corner cases the cached MSS value might be outdated. For instance, this could happen if a load balancer sits in front of two middleboxes that are (mis-) configured with two different MSS clamping values.

And legacy middlebox may not expect data in the SYN and behave different to what one could expect. I am not sure whether this is a real-world problem. But, indeed, it seems like an area where further experimentation could provide better insight (including the potential outcome that this is not a problem). I strongly agree that this should be added to the experimentation section.

Thanks

Michael

From: Yuchung Cheng [mailto:[email-masked]]
Sent: Tuesday, December 03, 2013 1:32 AM
To: Scharf, Michael (Michael)
Cc: Gorry Fairhurst; [email-masked]; [email-masked]
Subject: Re: [tcpm] Summary of mt comments/questions WGLC for draft-ietf-tcpm-fastopen-05

On Mon, Dec 2, 2013 at 5:49 AM, Scharf, Michael (Michael) <[email-masked]<mailto:[email-masked]>> wrote:
Some comments, with chair hat off...

a)+b):

However, after having followed this discussion, I wonder why the document does not comment at all about RTT estimation at the connection acceptor (server). With TFO, the connection acceptor (server) is allowed to send a full IW of data back to the connection initiator (client), following the SYN-ACK. However, at that point in time, the connection acceptor (server) has no recent RTT measurement, right? This is different to the traditional TCP handshake, in which
correct
an RTT measurement is available before data is sent into the network.  The fact that TFO allows sending of a full IW of data before an RTT estimation is available should be mentioned as a warning sign, IMHO. For instance, it will make it more difficult to apply pacing.
I can add that point about RTT estimation.

c)+d):

In almost all existing use of regular TCP, the endpoints negotiate the MSS before actually sending any data, because adding data to SYNs doesn't make a lot of sense without TFO. Therefore, "MSS clamping" in middleboxes usually works for regular TCP, and it may avoid PMTUD procedures in many cases. With TFO, the situation can be more
I need more information about the MSS clamping feature.

My understanding (which can be wrong) of MSS clamping is that the middle-box is transparently reducing the value in the MSS option in the SYN packet. If so the connection that negotiate the cookie, should receive and cache the reduced value, which will be used by the later Fast Open connection.

It is possible the middle-box lowers the MSS prior the to SYN-data. In that case it is equivalent to sending a SYN-data, or a data packet on a new path that has lower MTU. In that case, the remedy is ICMP error or ultimately a timeout. When recurring SYN-data timeouts happen, then the client should disable Fast Open (temporarily) due to the negative caching suggested in the draft.

complex. In addition to the case of MSS<536 bytes raised by Gorry, it is also not clear to me what a middlebox should do if it receives a TFO SYN with data, and if this segment is larger than the "MSS clamping" value of that middlebox. This corner case can occur e.g. after path routing changes. Therefore, I agree with Gorry that the document should at least mention that MSS issues may require further experimentation.
As replied earlier, I will document SYN-data / middlebox issue as the "area of experimentation" in the previous email.

Thanks

Michael

From: tcpm [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Yuchung Cheng
Sent: Sunday, December 01, 2013 7:08 PM
To: Gorry Fairhurst
Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [tcpm] Summary of mt comments/questions WGLC for draft-ietf-tcpm-fastopen-05

---

Best wishes,

Gorry