Subject: [tcpm] comments for draft-cheng-tcpm-fastopen-00.txt

From: Yoshifumi Nishida <[email-masked]>
Date: Tue, 25 Oct 2011 02:39:48 -0700

Hi,
I've thought about this draft for a while. I think there should be
some useful situations for this idea, although
I have the following comments on the draft.

  1: The draft presumes the primary target for TFO is HTTP, although
it looks more generic approach to me.
      I'm wondering why applicability or limitations to other
protocols are not mentioned at all.

  2:  If TFO is used for HTTP, I think there might be a risk that TFO
promotes the use of
      concurrent connections since there's no 3WS overhead for them.
I'm wondering it might be better
      to provide a certain limit for concurrent use for one TFO cookie.

  3: According to http://dl.acm.org/citation.cfm?id=1879174, many
middleboxs keep TCP bindings over 4mins.
      this doesn't seem to match the explanation in the section1 of
the draft. Do you have any comments on this?

Thanks,
--
Yoshifumi Nishida
[email-masked]

---

From: Yuchung Cheng <[email-masked]>
Date: Tue, 25 Oct 2011 10:02:24 -0700

Hi Yoshifumi,

These are great questions. The third one is highly related to Joe's
questions regarding HTTP persistent connections. We collected measurement
data and are updating the draft to address previous questions asked on the
list. We will post our update soon.

> protocols are not mentioned at all.
sure, we have talked about other usage in the last meeting presentations.
We'll include them in the new draft.

>      to provide a certain limit for concurrent use for one TFO cookie.
Do you mean TFO promotes "more" concurrent connections?

The reverse is more likely: browsers no longer needs to keep tons of active
parallel connections for future use to save 3WHS, especially on tiny
transactions like ads, probes, and redirects.

But this limit seems too application-specific and out of scope of this
draft. Plus, unfortunately vendor/servers already ignore HTTP spec on
parallel connections, they certain won't bother about another one :(

> the draft. Do you have any comments on this?
Our new draft will have a section on this part.

> Thanks,
Thanks for reviewing our draft

> --
> Yoshifumi Nishida
> [email-masked]
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm
>

---

From: Jerry Chu <[email-masked]>
Date: Tue, 25 Oct 2011 12:27:11 -0700

Hi,

> protocols are not mentioned at all.

TFO's primary target is for short-lived connections for the obvious reason.
HTTP just happens to fit in that short-lived connection category (because
unfortunately the supposed-to-be long-lived, persistent HTTP has too many
limitations limiting the level of persistency we've seen in the real world.
More
in our upcoming -01 draft revision).

The real limitation of TFO is with duplicate SYN and its implication on TCP
semantic as described in section 2.1.

If the current draft gives the impression the primary target of TFO is HTTP
we can fix it.

>      concurrent connections since there's no 3WS overhead for them.

TFO may promote the use of "short-lived" connections, not "concurrent"
connections because it removes some of the overhead with short-lived
connections.

>      to provide a certain limit for concurrent use for one TFO cookie.

See above. Also the current cookie design tries to keep the amount of
states on the server side at a minimum. Your suggestion above will
require some book-keeping. Another problem is one cookie may be
shared by many machines behind a single IP address due to NAT.

> the draft. Do you have any comments on this?

As Yuchung mentioned we've done lots of studies/measurements on the
effectiveness and limitation of P-HTTP since the IW10 days when
questions were raised about why P-HTTP is not a viable solution to our
IW10 proposal then and now the TFO proposal. The details will be better
covered by our upcoming -01 draft and hopefully it can address or at least
clarify the P-HTTP question raised by Joe Touch and others.

Thanks for your thoughtful comments.

Jerry

---

From: Yoshifumi Nishida <[email-masked]>
Date: Wed, 26 Oct 2011 02:54:39 -0700

Hi Jerry, Yuchung,
Thanks for the response.

> connections because it removes some of the overhead with short-lived

I agree that TFO can promote the use of short-lived connections.

My concern here is there might be no guarantee that TFO is used only for
short-lived connection. (also, we have no clear definition for
short-lived connection)
I'm thinking that some TFO connections can be long and it might increase
concurrent connections.

Thanks,
--
Yoshifumi Nishida
[email-masked]

---

From: rick jones <[email-masked]>
Date: Wed, 26 Oct 2011 07:15:21 -0700

> I'm thinking that some TFO connections can be long and it might increase

While it is but anecdote I have heard people say "I'd use TCP if I could establish connections more quickly" but I don't think I've ever heard anyone say "If only I could establish connections more quickly, I would increase my concurrent connections."

As for a definition of a short-lived connection, I'll go ahead and toss a stake in the ground at "less than or equal to 4 RTT" and/or perhaps "exchanges less than or equal to 2 window's worth of data" (ignoring those stacks which like to have a dynamic window size) No specific logic to it, just a starting point for discussion, if any.

rick jones
there is no rest for the wicked, yet the virtuous have no pillows

---

From: Jerry Chu <[email-masked]>
Date: Wed, 26 Oct 2011 10:24:34 -0700

Hi Yoshifumi, Rick,

> > short-lived connection. (also, we have no clear definition for

Indeed. But people will quickly realize there is little or no benefit from
TFO.

> > I'm thinking that some TFO connections can be long and it might increase

Still I don't see a strong correlation between TFO and increasing concurrent
connections...

> While it is but anecdote I have heard people say "I'd use TCP if I could
> establish connections more quickly" but I don't think I've ever heard anyone
> say "If only I could establish connections more quickly, I would increase my
> concurrent connections."
>

Yep. TFO is ideal for latency sensitive, short-lived, request/response type
of protocols
(remember the "Transaction/TCP in the 90'), which are often based on UDP,
with HTTP
being one exception. (In fact our SPDY engrs often threaten to switch to UDP
if we
TCP engrs do not make TCP faster :)). Our goal is to make TFO a viable
alternative
to UDP in terms of latency. We're investigating, e.g., to replace our
internal cluster
heart-beat msgs over UDP to TFO.

Jerry

> As for a definition of a short-lived connection, I'll go ahead and toss a
> stake in the ground at "less than or equal to 4 RTT" and/or perhaps
> "exchanges less than or equal to 2 window's worth of data" (ignoring those
> stacks which like to have a dynamic window size) No specific logic to it,
> just a starting point for discussion, if any.
>
> rick jones

---

From: Lars Eggert <[email-masked]>
Date: Wed, 26 Oct 2011 20:34:51 +0300

Hi,

> heart-beat msgs over UDP to TFO.

I'd be thrilled if an IETF activity in this space also addressed the DNS-over-TCP use case, which is the other scenario where folks see many repeated small connections.

Lars

---

From: Jerry Chu <[email-masked]>
Date: Wed, 26 Oct 2011 10:45:40 -0700

Hi Lars,

> I'd be thrilled if an IETF activity in this space also addressed the DNS-over-TCP use case, which is the other scenario where folks see many repeated small connections.

Yep, DNSSEC/TCP is on our radar screen too. We're thinking hard if the
TFO cookie can be made optional for some use cases where the
reflection attack is not an issue. TFO cookies require a prior regular
3WHS to acquire cookies hence limiting the benefit only to later
connections to the same server.

Jerry

---

From: Jerry Chu <[email-masked]>
Date: Wed, 26 Oct 2011 11:27:33 -0700

Hi Joe,

On Wed, Oct 26, 2011 at 10:34 AM, Joe Touch <[email-masked]> wrote:
>
>
> On 10/26/2011 10:24 AM, Jerry Chu wrote:
> ...
>>
>> Indeed. But people will quickly realize there is little or no benefit
>> from TFO.
>
> That's my concern. Both to play nice with NATs and as a simpler mechanism in
> general, it's easier to use persistent connections AFAICT.

Obviously my statement above (TFO provides little or no benefit) was meant for
long-lived connections.

It turns out persistent connections are not any easier for NAT than
TFO is (with SYN
pkts carrying data). Persistent connections can tie up large amount of
resources,
both on the end hosts and on the middleboxes. Our Chrome engrs for Android
discovered a serious problem recently with long-lived HTTP connections when
middleboxes, under pressure for free mapping space, either silently dropped, or
sent RST to handsets already in power-saving mode when removing existing
translation from persistent connections. The end result is HTTP connections
from mobile devices may hang for a long time until TCP abort timeout (many
middleboxes do not generate RST when receiving pkts that do not have any
mapping entries for some reason).

This problem was serious (browser hung) enough that our Chrome engrs are
pondering about disabling P-HTTP. More on why persistent connections are not
panacea can be found in our upcoming -01 draft revision.

I have said this before for our IW10 proposal as well - if persistent
connections
had been a viable solution to our latency problem we would not have wasted our
time looking for alternative solutions.

Jerry

> Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 26 Oct 2011 12:32:01 -0700

Hi, Jerry,

A lot of what you describe below sounds like clutching onto a single 
persistent connection too long.

I wonder if a 90% solution would have resulted in the bulk of the 
benefit and avoided these issues.

Joe

On 10/26/2011 11:23 AM, Jerry Chu wrote:
> Hi Joe,
>
> On Wed, Oct 26, 2011 at 10:34 AM, Joe Touch<[email-masked]>  wrote:
>>
>>

---

From: Yoshifumi Nishida <[email-masked]>
Date: Wed, 26 Oct 2011 13:24:46 -0700

Hi Jerry,

> Still I don't see a strong correlation between TFO and increasing concurrent

Probably, I should have asked how P-HTTP and TFO co-exist.
If TFO pushes out P-HTTP, I'm guessing that it promotes concurrent connections
as a consequence. I guess I should take a look at 01 version.

Thanks,
--
Yoshifumi Nishida
[email-masked]

---

From: Neal Cardwell <[email-masked]>
Date: Wed, 26 Oct 2011 16:45:22 -0400

> as a consequence. I guess I should take a look at 01 version.

I doubt TFO will displace P-HTTP where P-HTTP is already used widely. An
already-open connection will almost always outperform a TFO connection, due
to cwnd, RTT, RTO, etc. being "warmed-up" in the already-open case.

In fact, browsers are already starting to choose between already-open
connections based on estimated cwnd:

http://blog.httpwatch.com/2011/06/10/investigating-the-network-performance-of-firefox-5/

Given the effort developers are already investing in picking the warmest of
the already-open connections, it seems unlikely many applications will
choose to cold-start using TFO unless they were forced to close a P-HTTP
connection due to memory pressure.

It seems like if/when TFO is in place most applications engineered for
performance will have some sort of preference hierarchy like:

1) "warmed-up" P-HTTP connection
2) "cold" P-HTTP connection
3) new TFO connection
4) new 3-way-handshake connection

neal

---

From: Hagen Paul Pfeifer <[email-masked]>
Date: Wed, 26 Oct 2011 23:27:05 +0200

>to cwnd, RTT, RTO, etc. being "warmed-up" in the already-open case.

For Linux the connection is often "warmed-up" because of the metric cache (TCP
congestion window, RTT estimates). Furthermore, IW10 will relax this one more
time and IW10 may adequate for a lot of HTTP transfers. Not sure if an open
connection will outperform a new established connection nowadays (except the
3WHS, of course - which is addresses by TFO ;).

One question Yuchung, Jerry: are there any limitation not to exchange data
bidirectional in the SYN AND SYN-ACK packet? To cite the abstract: "allows
data to be carried in the SYN or SYN-ACK packets and consumed"

                              ^^
I see no real limitation that the "or" cannot be replaced by "and"? Did I miss
something? ;)

Hagen

---

From: Jerry Chu <[email-masked]>
Date: Wed, 26 Oct 2011 14:42:41 -0700

Hi Joe,

> persistent connection too long.

Correct except what is considered "too long" is getting shorter and shorter.

>
> I wonder if a 90% solution would have resulted in the bulk of the benefit
> and avoided these issues.

Many practical considerations over the year have constrained how long a
connection can remain open without getting into trouble. In the past the
effectiveness of persistent connections were limited by the locality of url
references, memory constraints on the endhosts, and IP address/port number
space and translation table size on the middleboxes. (We have observed from
Chrome browers more than one third of HTTP requests were made on new TCP
connections.)

With mobile network, aggressive power saving algorithms are putting more
constraint on the length of TCP connections in order to avoid any
unnecessary traffic (http ping, keepalive...) to wake up radio links...

Jerry

---

From: Jerry Chu <[email-masked]>
Date: Wed, 26 Oct 2011 14:49:19 -0700

Hi Hagen,

> I see no real limitation that the "or" cannot be replaced by "and"? Did I

Yes it should be "and" not "or".

Thanks for the correction.

Jerry

---

From: Jerry Chu <[email-masked]>
Date: Wed, 26 Oct 2011 15:00:27 -0700

Hi Yoshifumi,

On Wed, Oct 26, 2011 at 1:24 PM, Yoshifumi Nishida
<[email-masked]>wrote:

> from

A past paper "The Case for Persistent-Connection HTTP" by J. Mogul
considered
Transaction TCP, which enables data-in-SYN like TFO, a competing proposal.
The main hurdle for T/TCP then and TFO now is deployment since both require
non-trivial amount of stack changes. Personally I see P-HTTP and HTTP/TFO
are complementary solutions.

Jerry

If TFO pushes out P-HTTP, I'm guessing that it promotes concurrent

---

From: Joe Touch <[email-masked]>
Date: Thu, 27 Oct 2011 07:37:22 -0700

Hi, Jerry,

On 10/26/2011 2:42 PM, Jerry Chu wrote:
> Hi Joe,
>
> On Wed, Oct 26, 2011 at 12:32 PM, Joe Touch <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     Hi, Jerry,
>

Some of the same problems with keeping a single connection open too long 
are shared by fastopen and persistent connection approaches, e.g., at 
the endpoints.

> effectiveness of persistent connections were limited by the locality of
> url references, memory constraints on the endhosts, and IP address/port
> number space and translation table size on the middleboxes. (We have
> observed from Chrome browers more than one third of HTTP requests were
> made on new TCP connections.)

Those same locality issues also limit the utility of fastopen approaches.

> unnecessary traffic (http ping, keepalive...) to wake up radio links...

That sounds again like "too long".

We do need to see how this plays out on real traffic in an experiment. 
There's a tradeoff between persistent connections and fastopen that 
needs to be explored better - esp. since persistency is free (already 
supported) and fastopen requires TCP mods. E.g., if we're talking about 
a TCP mod to help the first of 10 connections in every 'burst', it's not 
worth the effort/complexity IMO.

The jury is still out until we see more substantial data, IMO.

Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 26 Oct 2011 10:34:11 -0700

> Indeed. But people will quickly realize there is little or no benefit

That's my concern. Both to play nice with NATs and as a simpler 
mechanism in general, it's easier to use persistent connections AFAICT.

Joe

---

From: Jerry Chu <[email-masked]>
Date: Thu, 27 Oct 2011 15:39:35 -0700

Hi Joe,

On Thu, Oct 27, 2011 at 7:37 AM, Joe Touch <[email-masked]> wrote:
> Some of the same problems with keeping a single connection open too long are
> shared by fastopen and persistent connection approaches, e.g., at the
> endpoints.
>

Correct if TFO cookie is mandatory - it will require one regular 3WHS to acquire
a valid cookie per local/remote IP addr pair before data is allowed to be
consumed during 3WHS.

We have been exploring the possibility of making the TFO cookie optional,
only required by those servers where a amplified reflection attack is possible
(see my slides at the last IETF but we haven't added anything in the
-01 draft as
we need more time to firm up the idea). If we succeed, TFO will be truly
competitive against other RR protocol running on top of UDP.

> That sounds again like "too long".

Here what is considered "too long" has come down to just a few seconds and
a handful of HTTP transactions, not very "persistent" IMO.

> There's a tradeoff between persistent connections and fastopen that needs to
> be explored better - esp. since persistency is free (already supported) and
> fastopen requires TCP mods. E.g., if we're talking about a TCP mod to help
> the first of 10 connections in every 'burst', it's not worth the
> effort/complexity IMO.

First I think it's better to have the technology ready so applications can have
a choice, rather than us debating here forever based on our own, possibly biased
opinions and incomplete/erroneous data. BTW, from our own data Google has
decided TFO is a worthy technology to pursue. (That's why we are here.)

Second, 15 years ago when T/TCP was considered a serious contender in
addition to P-HTTP as a solution to the unnecessary delay of HTTP caused by
TCP's 3WHS, deployment problem was cited as #1 hurdle to the viability of
T/TCP. Had we taken some action then, we've had long gone over the
deployment hurdle in the past 15 years.

Granted T/TCP has some fatal design flaw, focusing itself too much on the
intractable duplication SYN problem hence adding too much complexity.
I believe we've taken the right approach, to treat the data-in-SYN as an
engineering problem hence proposing an imperfect solution balancing the
right tradoffs, rather than to treat it as a math or science problem and try to
find a perfect but incomprehensible solution.

Jerry