Subject: [tcpm] comments for draft-cheng-tcpm-fastopen-01.txt

From: Hagen Paul Pfeifer <[email-masked]>
Date: Wed, 2 Nov 2011 02:44:41 +0100

Hey TFO'lers

after reading 7.1 I don't really understand why pushing TFO forward.  The
benefit-cost-ratio seems unfavorable for TFO, compared to HTTP persistent
connections.

In the end: the major challenge is to increase the transactions per connection
ratio. This can be reached by not closing the connection by frequent TCP
keep-alive probes.

The big advantage of HTTP persistent connections and TCP keep-alive probes is
that they already widely deployed and the small TCP option space is not
touched. Is power consumption really a real issue? A keep alive interval of
several minutes should be fine for every mobile device doing HTTP. I.e. mobile
webbrowsers can implement an adaptive keep-alive mechanism. Stop sending keep
alive probes if the user stop browsing for several minutes. Therefore
keep-alive probes would be send only when the browser is used.

The second disadvantage is a larger memory consumption on server side,
alright. But then maybe the Web Server should be optimized and state
information reduced.

Aside from all the disadvantages which occur by introducing new options. Like
broken middleboxes dropping unknown TCP options, time until all vendors
implement TFO, implementation bugs et cetera. I prefer to push HTTP persistent
connection forward. IMHO it is a HTTP thing - it should be solved at HTTP
level.

Said that: I support the basic concept of TFO. It will work, I'm sure. But is
it really worth that much if you can reuse HTTP connections?

Hagen

---

From: Jerry Chu <[email-masked]>
Date: Tue, 1 Nov 2011 23:00:24 -0700

> benefit-cost-ratio seems unfavorable for TFO, compared to HTTP persistent

I don't understand why section 7.1 led you to the above conclusion, after giving
a long list of limitations of P-HTTP, and data showing P-HTTP in the real world
are not so persistent (averaging only 2-4 transactions per connection,
likely due
to one or more limitations described in the section).

> ratio. This can be reached by not closing the connection by frequent TCP

But keep-alive probes are a non-starter for power conscious mobile devices.

> that they already widely deployed and the small TCP option space is not

This was the same argument by J. Mogul 16 years ago against T/TCP. 16 years
later P-HTTP only attains 2-4 transaction per connection and you still believe
P-HTTP is the way to go?

> touched. Is power consumption really a real issue?

Yes from our Android partners.

> A keep alive interval of

Several minutes are eternity for mobile network.

> webbrowsers can implement an adaptive keep-alive mechanism.

And you think our Android/Chrome browser folks haven't tried?

>Stop sending keep

It's always easier said than done. Keep in mind keep alive probes
serve to prevent
NAT entries from being removed and it looks like you believe one can design an
adaptive algorithm to both minimize keep-alive frequency, while still preventing
entry expiration, and avoiding NAT table and port/addr space depletion?

> broken middleboxes dropping unknown TCP options,

Aren't there real data showing the above is rare? Perhaps you are
suggesting IETF
should stop publishing new TCP options altogether?

>From our study P-HTTP seems much more problematic to middleboxes than unknown
TCP options.

>time until all vendors
> implement TFO, implementation bugs et cetera.

We've already got a near-production, Linux-based TFO implementation. If
we stop debating now and commit to TFO perhaps we'll see meaningful deployment
in a couple of years.

>I prefer to push HTTP persistent
> connection forward.

I admire your patience but wonder after 16 years why you still believe there is
room for P-HTTP to go further.

>IMHO it is a HTTP thing - it should be solved at HTTP
> level.

I beg to disagree. If you look closer none of the debates of P-HTTP vs TFO have
to do with HTTP, it's all about TCP, NAT, when to close a connection,..., etc.
In fact when we designed TFO we never thought of HTTP.
> it really worth that much if you can reuse HTTP connections?

We have some real performance data showing the benefit of TFO in an upcoming
ACM CoNext paper. Also we never intended TFO for HTTP only. Any transaction
protocol with constraints on connection lifetime may benefit. Even TCP
connections
in our internal network may not have a long lifetime due to the
resource constraint
created by the large number of nodes and the N square problem.

Jerry

>
>
> Hagen
>
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm
>

---

From: Joe Touch <[email-masked]>
Date: Tue, 1 Nov 2011 23:35:45 -0700

> later P-HTTP only attains 2-4 transaction per connection and you still believe

A better question is what the significant advantage is for fastopen *assuming* P-HTTP.

If you assume 2-4 transactions per connection, aren't you also getting the bulk of the benefit too?

> entry expiration, and avoiding NAT table and port/addr space depletion?

The socket pair isn't supposed to be reused for 2MSL anyway, so by using persistent connections you're *preserving* the port space of the NAT, rather than consuming it needlessly with a large number of short connections.

> In fact when we designed TFO we never thought of HTTP.

The error is in using TCP CLOSE as the dominant way to indicate EOF. This is an error in the design of HTTP, as it is in many other applications.

There are other situations that need persistent connections as well - notably the DNS, more lately with DNSSEC warranting the use of TCP rather than UDP.

In every case I've seen to date, the error seems much more like a disconnect in the use of TCP than in the design of TCP itself - *especially* considering that all the fast-open mechanisms (this one, tcpcrypt, tcpct) reduce latency only for the "subsequent" connections to a site, rather than the first one.

Joe

---

From: Hagen Paul Pfeifer <[email-masked]>
Date: Wed, 02 Nov 2011 11:40:22 +0100

On Tue, 1 Nov 2011 23:00:24 -0700, Jerry Chu <[email-masked]> wrote:

> I don't understand why section 7.1 led you to the above conclusion,
after
> giving
> a long list of limitations of P-HTTP, and data showing P-HTTP in the
real
> world

To make it short: the major argument against P-HTTP is that only 2-4
transactions per connection are handled, right? This flaw is caused by
"black-hearted" middleboxes not following the 124 minutes idle timeout,
right? But this can be solved by sending TCP keep-alive probes (I address
energy consumption later in this email). If it is solved, P-HTTP can do 10,
20, 100 transaction per connection. So if we can agree that this is the
only disadvantage, then I see no real drawback of P-HTTP if the client sent
keep-alive probes regularly. Jerry, I tried to find other real world
blockers for P-HTTP, but I didn't found any additional blockers - except
the NAT problem.

> later P-HTTP only attains 2-4 transaction per connection and you still
> believe P-HTTP is the way to go?

IMHO I imagine that HTTP performance was not on the list for several
years. Everything over IP, I mean everything over HTTP generate some new
pressure. ;-)

> Several minutes are eternity for mobile network.

I mean sending a TCP keep-alive probe every several minutes. My "adaptive"
approach say something like this: if the webbrowser is a foreground process
group then send TCP-keep-alive probes (the browser assume that within the
next couple of minutes a new HTTP request are generated), if the mobile
phone is in power saving mode turn keep-alive probes off (the browser
assume that within the next couple of minutes NO new HTTP request are
generated, the connection can close). In other words: send TCP keep alive
probes when the heuristic say that the browser is used or turn keep alive
probes off otherwise. Or take the backlight as an indicator: if the
backlight is on then generate keep-alive packets, turn off otherwise. I say
sent Keep-alive probes context aware, not blindly generates them every 10
seconds.

> And you think our Android/Chrome browser folks haven't tried?

Yes ;-) For Google folks it is easier to modify TCP: you have control over
_both_ TCP endpoints: server side and Android side. No annoying middle box
ppeoblmes (TFO works fine, except that middleboxes eventually drop unknown
TCP options, but this should be rare today), less HTTP server state, no
interoperability problems - a fresh technology which just works.  ;-)

Jerry, as I wrote earlier: TFO is fine and functional - no doubt. But I
can imagine that you can reach similar goals with P-HTTP without major TCP
stack modifications.

Hagen

---

From: Jerry Chu <[email-masked]>
Date: Wed, 2 Nov 2011 15:24:32 -0700

> A better question is what the significant advantage is for fastopen *assuming* P-HTTP.

As I've mentioned in the -01 draft and previous email, we observed
10-40% improvement
from TFO in the PLT in an emulated environment using a replay tool. And this is
WITH P-HTTP (or more accurately, with the default Chrome browser
setting, which has
P-HTTP enabled). We've seen in general about 90% of responses with
Connection: Keep-Alive header.

> If you assume 2-4 transactions per connection, aren't you also getting the bulk of the benefit too?

We don't have data on how much improvement P-HTTP provides over HTTP1.0. We only
know TFO provides additional saving on top of P-HTTP as described above.

We have provided more details on the performance data in an upcoming
paper to ACM
CoNEXT'11, which has just been announced at
http://conferences.sigcomm.org/co-next/2011/program.html

> The socket pair isn't supposed to be reused for 2MSL anyway, so by using persistent connections you're *preserving* the port space of the NAT, rather than consuming it needlessly with a large numbe…

What you said is correct if NAT boxes preserver TIME-WAIT connections
but I think
the latter is highly implementation dependent. For those NAT boxes
that don't preserve
TIME-WAIT, its working set size is determined by how many established
connections.
The more connections persist, the more mapping entries it will demand.

> The error is in using TCP CLOSE as the dominant way to indicate EOF. This is an error in the design of HTTP, as it is in many other applications.

Yep, seems like expediency on the part of application protocols.

> In every case I've seen to date, the error seems much more like a disconnect in the use of TCP than in the design of TCP itself - *especially* considering that all the fast-open mechanisms (this on…

We have been pondering on a "cookie-less" mode for TFO where the round
trip saving can
be attained on the 1st connect(). We may have to give back some
portion of the round trip
saving in order to avoid a security issue. Will need to get some
performance data first.

Jerry

---

From: Jerry Chu <[email-masked]>
Date: Wed, 2 Nov 2011 16:46:38 -0700

On Wed, Nov 2, 2011 at 3:40 AM, Hagen Paul Pfeifer <[email-masked]> wrote:

> right?

No, not that simple. We don't know the cause. We just report the fact
based on our logged data! The rest is all speculation. But come to think
about it there are multiple forces involved here. Either end point can
decide to close a connection due to resource constraint, some policy
restriction (e.g., browsers allowing only X # of open connections), user's
web surfing pattern, unexpected RST or timeout by the middleboxes,...,
just naming a few (and I must have missed a whole bunch).

> But this can be solved by sending TCP keep-alive probes (I address

> blockers for P-HTTP, but I didn't found any additional blockers - except

See my list above. And the more I dig, the more blockers I find. We are
living
in a complex world and more than one solutions are often needed.

> years.

Everything over IP, I mean everything over HTTP generate some new
> sent Keep-alive probes context aware, not blindly generates them every 10

Understood but that doesn't address the problem when NAT reclaims a
translation entry while the device owning the connection has hibernated
temporarily. Then, after waking up, the browser hangs until TCP timeout.
This is one of the real world issues that our Chrome/Android folks is
facing,
forcing them to considering disabling P-HTTP completely.

> can imagine that you can reach similar goals with P-HTTP without major TCP

We've gone through this circle a few times. Long before we proposed IW10 our
front end folks had tried to tune various parameters to attain better P-HTTP
performance and discovered they can only go so far. Then we applied IW10
and get more performance mileage. We're attempting the same with TFO. Keep
in mind we've never dismissed the usefulness of P-HTTP. It's the claim that
P-HTTP solves world hunger problem that we can not agree.

Best,

Jerry

---

From: Yuchung Cheng <[email-masked]>
Date: Wed, 2 Nov 2011 17:24:40 -0700

> 20, 100 transaction per connection. So if we can agree that this is the
To clarify: Chrome *already* sends keep-alive messages every 45s or so, but
we still see an average of 3 requests per transaction. [NOTE: Chrome is not
yet on Android which runs a different browser that disables keep-alive for
power issue]

> blockers for P-HTTP, but I didn't found any additional blockers - except
You are right on the point. We don't see any drawback of P-HTTP. In fact we
absoultely love it. We are arguing NAT, sadly but practically, is limiting
p-HTTP for long run.

Jerry has made the point clear, but let me restate:
TFO's goal is never to replace p-HTTP.  TFO is addressing the limitation of
p-HTTP. They are complimentary.

Joe asked the great question about how much limitation does TFO addresses.
I am working on getting the paper online so we can discussion over data
directly.

> interoperability problems - a fresh technology which just works.  ;-)
It would just work if we also control the middleboxes too :(

> can imagine that you can reach similar goals with P-HTTP without major TCP
Again I agree with u *if* p-HTTP can be used any time anywhere. Seriously
we would not want to go the TCP route if we have seen a easier exit. Sadly
we are seeing the trend is going backward: power issues, nat issues, blah
blah, are zapping idle TCP asap.

---

From: Hagen Paul Pfeifer <[email-masked]>
Date: Thu, 03 Nov 2011 11:18:50 +0100

On Wed, 2 Nov 2011 17:24:40 -0700, Yuchung Cheng wrote:

> To clarify: Chrome *already* sends keep-alive messages every 45s or so,
> but we still see an average of 3 requests per transaction. [NOTE: Chrome
> is not yet on Android which runs a different browser that disables
> keep-alive for power issue]

But then it would help to identify why only 3 req per transaction. I
assume that the P-HTTP timeout is to short (my apache run with the default
of 5 seconds).

> You are right on the point. We don't see any drawback of P-HTTP. In
> fact we absoultely love it. We are arguing NAT, sadly but practically,
is
> limiting p-HTTP for long run.  
> Jerry has made the point clear, but let me restate:TFO's goal is never
> to replace p-HTTP.  TFO is addressing the limitation of p-HTTP. They are
> complimentary.

I know, I consider TFO as a complimentary mechanism, not sticked to HTTP.

>  Joe asked the great question about how much limitation does TFO
> addresses. I am working on getting the paper online so we can discussion
> over data directly.

It seems Joe know how to ask smart questions. ;-)

Hagen

BTW: thank you Jerry and Yuchung for clarification!

---

From: Hagen Paul Pfeifer <[email-masked]>
Date: Thu, 03 Nov 2011 11:55:32 +0100

On Wed, 2 Nov 2011 19:53:09 -0700, William Chan (陈智昌) wrote:

> Note that servers often close down connections due to resource
> consumption.  More sockets mean more file descriptors to check, and more
> kernel socket buffers, and potentially other per-connection state (SSL
> context?). It's not free. Maybe we can push on servers to keep
> connections open longer, but I suspect there's some limit to how far
> they're willing to go. 

Sure, but holding state in TCP stack is also not for free (which is
required by TFO). The difference is the amount of state information. But I
am sure that there is room for optimization. But SSL state is a interesting
aspect: at HTTP level you get them for free, saving TLS turnarounds.

> I'm jumping in late, but it looks like this comment is wrt adaptive
> keep-alive mechanisms, specifically in the mobile case. I'm not sure I
> understand how an adaptive keep-alive mechanism fixes anything. You seem
> to indicate we should turn off TCP keep-alives when the browser goes
> idle. Do you propose keeping the connection open or not? If the
> connection is open, then what happens when middleboxes drop the mapping
> and blackhole the connection? If the connection is closed, then when the
> browser becomes active again, then we need to establish a new connection
> again, in which case TFO sounds useful. 

Forget the "adaptive keep alive mechanism". It was a quick example how to
send TCP keep alive probes in a more clever way to reduce battery
consumption. I just wanted to point out that there are smarter ways to send
keep-alive probes.

Hagen

---

From: Jerry Chu <[email-masked]>
Date: Thu, 3 Nov 2011 11:53:23 -0700

On Thu, Nov 3, 2011 at 3:55 AM, Hagen Paul Pfeifer <[email-masked]> wrote:
> required by TFO).

Pardon me? What states and which side are you alluding to?

We've worked hard to keep TFO server side stateless, with the exception of
perhaps 8 bytes for a per server master key. Or perhaps you're talking about
the client side TFO cache? But the # of cached connections will be at most
hundreds so why does it matter?

>The difference is the amount of state information. But I

I also want to point out there are people here whose day job is to solve these
difficult puzzles for a living therefore proposals from them have
likely been well
thought of, with some obvious alternatives already considered, so please don't
dismiss them too quickly.

Best,

Jerry

---

From: Hagen Paul Pfeifer <[email-masked]>
Date: Fri, 04 Nov 2011 12:12:31 +0100

Jerry: don't get me wrong! I wrote it in _every_ email that I like TFO. But it helps nobody if we focus blindly on TFO. Therefore I summed up all implications. And TCP state is one. I never wrote that it has a huge impact, I wrote that for Willian, case I thought he miss that. P-HTTP and remembering TLS state is another aspect which should not underestimated. 

Jerry, to make it clear: I don't judge TFO, I try to understand the implications, the mechanism from high level. Focus on the problem - not one solution.

Excuse me Jerry if you thought that I argue against TFO - that was and is not my intention. You and the other guys from Google make a really good job, pushing TCP forward! Trust me, I really like every new ID where the author is Jerry Chu.

Hagen

-- 
Hagen Paul Pfeifer <[email-masked]> || http://jauu.net
Telephone: +49 174 5455209

---

From: Jerry Chu <[email-masked]>
Date: Fri, 4 Nov 2011 12:19:26 -0700

Hi Hagen,

On Fri, Nov 4, 2011 at 4:12 AM, Hagen Paul Pfeifer <[email-masked]> wrote:
> Jerry: don't get me wrong! I wrote it in _every_ email that I like TFO.

Does that mean you support TFO adoption as WG item? ;-)

>But
> it helps nobody if we focus blindly on TFO. Therefore I summed up all
> implications. And TCP state is one. I never wrote that it has a huge impact,
> I wrote that for Willian, case I thought he miss that. P-HTTP and
> remembering TLS state is another aspect which should not underestimated.

Understood.

>
> Jerry, to make it clear: I don't judge TFO, I try to understand the
> implications, the mechanism from high level. Focus on the problem - not one
> solution.

Again like Yuchung said, we never intend to pit TFO against P-HTTP. We see
both have their limits and can complement each other. Therefore we object to
the statement that P-HTTP is ALL one will need.

The key decision for the WG right now is whether to adopt TFO as a WG item.
We have both data showing TFO adds performance benefits ON TOP OF P-HTTP,
and all the technical reasons why P-HTTP alone is not sufficient. I
hope for your
support and the WG can proceed to adopt TFO asap.

>
> Excuse me Jerry if you thought that I argue against TFO - that was and is
> not my intention. You and the other guys from Google make a really good job,
> pushing TCP forward! Trust me, I really like every new ID where the author
> is Jerry Chu.

Please don't single me out - it's a lot of hard work from my colleagues. It
has not been easy because at the end of the day Google judges our performance
at work not by how many I-Ds we produce (easier part), but how much real
performance improvement we have delivered (difficult).

Jerry

---

From: Yuchung Cheng <[email-masked]>
Date: Mon, 7 Nov 2011 14:16:50 -0800

> A better question is what the significant advantage is for fastopen *assuming* P-HTTP.

This is a great question!

In our paper (section 2.2), we measure the TCP connection overhead of
millions of HTTP transactions from the Chrome browser in 2011 over a
period of 28 days. We find TFO can potentially improves the latency by
25% for 80% connections. This assumes Chrome keeps current p-HTTP
setttings.

Our paper is available at: http://research.google.com/pubs/archive/37517.pdf

---

From: Yoshifumi Nishida <[email-masked]>
Date: Wed, 9 Nov 2011 00:28:33 -0800

Hi Yuchung, Jerry,

I think the result for TFO in the paper is very good. But, I'm wondering why
it's so good..
For example, in case of TCP wikipedia page, TFO reduces PLT 150msec
with 20msec RTT. This is 7.5 RTTs.
I can imagine the page consists of various resources from different hosts and
there are some limits for the number of concurrent connections in Chrome.
But, still.. 7.5 RTTs seems to be a bit big value to me..
Am I missing something?

Thanks,
--
Yoshifumi Nishida
[email-masked]

---

From: Yuchung Cheng <[email-masked]>
Date: Tue, 15 Nov 2011 12:54:03 -0800

Hi Yoshifumi,

Sorry for the late response. I hope it is  not too late for the meeting today.

The large improvement of the particular case in your email was due to
the compounded effect of RTT savings. Here is the network latency
waterfall graph of Chrome fetching
"http://en.wikipedia.org/wiki/Transmission_Control_Protocol" using
regular TCP with 200ms RTT.
http://3.bp.blogspot.com/-8Jk-EpTjtvo/TsLNuybnRhI/AAAAAAAAAC0/b5uDDZEp25o/s1600/tcp-wiki-200ms.png

The TCP wikipedia page has total 58 objects from 3 domains
{en,bits,upload}.wikimedia.org. The network latency of each object
download is shown as a colored bar that includes DNS/TCP/server
processing. I marked red points  on the cold requests that incur TCP
handshakes. There are 14 total such cold requests, i.e., Chrome opens
14 TCP connections total.

If TFO was used, these 14 cold requests will each gets 200ms faster. A
majority of them help later requests to load earlier and the
compounded saving becomes large.

I am happy to share the steps to produce this graph if people are interested.

---

From: Joe Touch <[email-masked]>
Date: Tue, 15 Nov 2011 14:13:26 -0800

Some questions:

- what kind of net connection are you using that has 200ms RTTs?

- do you know that the request was always completely in the first packet?
	if not, you won't save the first RTT
	(though this is likely)

- TCP-FO won't help with the truly "cold" opens
	of these 14, which ones are to places
	likely to be revisited by subsequent connections?

- what is the overall benefit?
	look at the 6 sets that line up vertically and
	label them:
	A - connection 1
	B - connection 2-4
	C - connection 8
	D - connection 11
	E - connections 15, 17-22
	F - connection 37
	-- at best, those are the 6 200-ms chunks you're saving, not 14

	However, you're ignoring the impact of other non-spedup
	connections; when you take them into account, you see a critical
	path of dependencies:

	1, 9, 10, 23

	in that path, you get to speedup only 1 *at best*

So you will save 200ms out of this entire exchange, not 14*200.

It's about time, not number.

Joe

On 11/15/2011 12:54 PM, Yuchung Cheng wrote:
>> 2011/11/7 Yuchung Cheng<[email-masked]>:
>>> On Tue, Nov 1, 2011 at 11:35 PM, Joe Touch<[email-masked]>  wrote:
>>>>