Subject: [tcpm] Can't we use newcwv for IW too?

From: Michael Welzl <[email-masked]>
Date: Tue, 28 May 2013 10:03:24 +0200

Hi,

Recently, the fact that the most recent HTTP2.0 draft lets a sender transfer TCP's cwnd to the receiver caused an interesting discussion in the HTTPBIS group. To see the thread, look for "cwnd" at:
http://lists.w3.org/Archives/Public/ietf-http-wg/2013AprJun/thread.html

I agree with folks saying that it's high time for transport people to get involved. I just subscribed to the list. Anyway, as Lars Eggert mentioned in this thread, this group (TCPM) is a better home for such discussions; this problem really is a TCP problem, not an app layer problem. This message that he wrote:
http://lists.w3.org/Archives/Public/ietf-http-wg/2013AprJun/0277.html
made me think: irrespective of the size of your "normal" IW, why can't we construct a rule like IW = max(IW, cwnd derived from new-cwv from a previous connection) for a connection to the same host as the previous one?

As Lars has pointed out in his message, this idea isn't new. However, it seems to me that the key problem of previous approaches (let alone the current HTTP2.0 one!) is related to cwnd dynamics - we will have to collapse or in some way decay it after a while of course - how exactly do we do that? What's new is that, with newcwv ( draft-fairhurst-tcpm-newcwv-05 ), we now have an approach that reuses cwnd after a rate-limited *or idle* time, and that approach has some support behind it. It's a TCPM WG item, it has gone through some evaluation-update rounds (cf. http://trac.tools.ietf.org/group/irtf/trac/wiki/ICCRG_newcwv ), and is backed by research (cf. http://dl.acm.org/citation.cfm?id=2413230 ).

There really isn't much difference between newcwv restarting after an idle period and a new connection using an IW based on a previous connection's newcwv.
We may have a new port number (although we could also be extra restrictive here, if we really must), and we had the FIN + SYN handshakes. These things may lead to different treatment in middle-boxes and they may give us a new path, using ECMP, so yes, it's not exactly the same as restarting from idle, but then again, consider:
- it takes only 1 RTT to notice the mistake and react to it; this is quite close to the Jump Start idea ( http://www.icir.org/mallman/papers/jumpstart-pfldnet07.pdf ), but backed by prior knowledge about the path whenever stuff like ECMP didn't kick in
- we do this only for one connection following a previously terminated one  (sure, if N just terminated, we could do it N times, within a certain time frame...)

Thoughts?

Michael

---

From: Joe Touch <[email-masked]>
Date: Wed, 29 May 2013 11:38:08 -0700

On 5/28/2013 1:03 AM, Michael Welzl wrote:
> I agree with folks saying that it's high time for transport people to
> get involved. I just subscribed to the list. Anyway, as Lars Eggert
> mentioned in this thread, this group (TCPM) is a better home for such
> discussions; this problem really is a TCP problem, not an app layer
> problem. This message that he wrote:
> made me think: irrespective of the size of your "normal" IW, why can't
> we construct a rule like IW = max(IW, cwnd derived from new-cwv from a
> previous connection) for a connection to the same host as the previous one?

There are reasons this might work (it's a bit like RFC2140), and reasons 
it might not (IP address != host these days, esp. for web server farms).

> There really isn't much difference between newcwv restarting after an
> idle period and a new connection using an IW based on a previous
> connection's newcwv.

That depends on how many new connections you open using this 
information, and whether the previous connection is closed. If it's not 
1:1, then yes, it's different. A more complete consideration of the 
issues of 2140 would handle that, but just importing the info from a 
previous connection isn't sufficient by itself.

You already pointed out some potential for alternate paths due to port 
number changes, but that's less of a concern AFAICT.

Joe

---

From: Michael Welzl <[email-masked]>
Date: Wed, 29 May 2013 21:55:06 +0200

Hi,

Just to clarify:

On May 29, 2013, at 8:38 PM, Joe Touch <[email-masked]> wrote:

> There are reasons this might work (it's a bit like RFC2140), and reasons it might not (IP address != host these days, esp. for web server farms).
> 
> That depends on how many new connections you open using this information, and whether the previous connection is closed. If it's not 1:1, then yes, it's different. A more complete consideration of the issues of 2140 would handle that, but just importing the info from a previous connection isn't sufficient by itself.

I did mean 1:1.

> You already pointed out some potential for alternate paths due to port number changes, but that's less of a concern AFAICT.
> 
> Joe

Cheers,
Michael

---

From: Joe Touch <[email-masked]>
Date: Wed, 29 May 2013 12:58:10 -0700

>> That depends on how many new connections you open using this information, and whether the previous connection is closed. If it's not 1:1, then yes, it's different. A more complete consideration of…

Ahh; then the algorithm would need to be more specifically described as 
"only co-opting the window of a recently closed TCP connection", and 
explain how long "recently" would imply ;-)

In that case, though, IMO it's just a subset of 2140, some versions of 
which are already in widespread use.

Joe

---

From: Michael Welzl <[email-masked]>
Date: Wed, 29 May 2013 22:02:40 +0200

On May 29, 2013, at 9:58 PM, Joe Touch <[email-masked]> wrote:

> Ahh; then the algorithm would need to be more specifically described as "only co-opting the window of a recently closed TCP connection", and explain how long "recently" would imply ;-)
> 
> In that case, though, IMO it's just a subset of 2140, some versions of which are already in widespread use.

Indeed! But using rules from new-cwv for "recently". But yes, it's not much more than that  (this being said, I don't think 2140 did specifically *recommend* re-using cwnd for later connections, it just hinted at the possibility, I thought… but I'd have to read it again).

Cheers,
Michael

---

From: Joe Touch <[email-masked]>
Date: Wed, 29 May 2013 13:09:25 -0700

On 5/29/2013 1:02 PM, Michael Welzl wrote:
> Indeed! But using rules from new-cwv for "recently".

Sure.

> But yes, it's not much more than that  (this being said, I don't
> think 2140 did specifically *recommend* re-using cwnd for later
> connections, it just hinted at the possibility, I thought… but I'd
> have to read it again).

It was informational, but it did talk about reusing cwnd both across 
concurrent connections (spatial sharing) and across sequences of 
connections (temporal sharing).

Joe

---

From: [email-masked]
Date: Thu, 30 May 2013 07:30:48 +0100

So here are some thoughts - if we were talking about "bulk" flows where
the cwnd reflects the network capacity to a specific destination, then
sharing of the cwnd is something that is being done by many stacks. In
reality the cwnd that is "shared" is often higher than that sustained on
the path, since this is inflated each RTT (the extra between 1 MSS and
cwnd).

Now, if we have a flow that does not fully utilise cwnd (variously called
thin,variable rate, bursty, intermittent, ...) then the cwnd can be much
higher than the that sustained on the path. RFC 2861, attempted to keep
the cwnd close to the actual used capacity, but this restricts the
dynamics o an application and adds latency to applications that are
bursty. Hence if we desire to enable these applications we should not
aggressively trim the cwnd to what was used.

New-cwv tries to avoid unecessary cwnd growth, but proposes to avoid
forcing a low cwnd on a bursty application. Instead it creates a new phase
(the non-validated phase) where cwnd exceeds the actual path capacity
used, and where the cwnd growth is capped, but importantly the congestion
response is changed to reduce more I think this helps motivate persistent
use of TCP.

Now, my question is what cwnd value should be "shared" when the flow is in
the  non-validated phase? - If a flow was to share non-valiudated cwnd,
and all flows implemented the new-cwv algorithm, then the flow that
received the "extra" cwnd would be non-validated, and that would be OK. If
it were a normal TCP flow, then I think this inflates cwnd in an
unreasonable way.

Gorry

>>>>> There are reasons this might work (it's a bit like RFC2140), and
>>>>> reasons it might not (IP address != host these days, esp. for web
>>>>> server farms).
>>>>>
>>>>> information, and whether the previous connection is closed. If it's
>>>>> not 1:1, then yes, it's different. A more complete consideration of
>>>>> the issues of 2140 would handle that, but just importing the info
>>>>> from a previous connection isn't sufficient by itself.
>>>>
>>>> I did mean 1:1.
>>>
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm
>

---

From: Michael Welzl <[email-masked]>
Date: Thu, 30 May 2013 15:30:01 +0200

Hi,

Even reading this 3 times doesn't make me understand it. Could it be that we have a misunderstanding and you're thinking of what RFC2140 calls "ensemble sharing", when what I mean is what RFC2140 calls "temporal sharing" ? Or are you just primarily concerned with a situation more complicated than I envisioned?

Let's see. I was only thinking of greedy applications for now. My idea was just that, if you have an application that e.g. stops sending and starts again, greedily, after 10 seconds, new-cwv has a clear rule on what cwnd should be.
And, if you replace this with an application that, as it stops, also closes the connection and after 10 seconds starts a new connection, and then immediately uses it to send, greedily, then there isn't much difference between this case and the case above, and IW could probably be set using a similar rule from new-cwv.

More below:

> sharing of the cwnd is something that is being done by many stacks. In

Until I sent my first email about this, I wasn't aware of that (but I got a private message telling me, straight away). I knew that some parts of RFC2140 are implemented, but I didn't expect cwnd to be reused beyond the end of a connection to set the next IW. Anyway, as Joe said, RFC2140 is informational, and I don't think it has clear rules about how long cwnd could be kept etc., so what I'm suggesting is to make a recommendation to let new-cwv consider a connection that terminates the same way as it considers a connection that enters the non-validated phase, and then allows re-using the cwnd in a following connection by noticing that the non-validated phase ends. If a very similar reasonable behavior is already out there, then I guess all I'm saying is "legalise it!"  :-)

> the path, since this is inflated each RTT (the extra between 1 MSS and

I don't get this; sorry if it should be obvious. We're talking about the time between the end of connection 1 and the beginning of connection 2, surely this cached value isn't then updated every RTT when there's nothing going on? And what do you mean with "the extra between 1 MSS and cwnd" ?

> it were a normal TCP flow, then I think this inflates cwnd in an

This last paragraph gives me the impression that you're thinking of ensemble sharing, not temporal sharing. I'm not suggesting giving away the cwnd of a flow while it's running (even if it's taking a break). Yes that could be reasonable as you sketch it (only for new-cwv-supporting flows, to make flows go from non-validated to validated), but it's just a more complicated beast than what I was thinking about.

Cheers,
Michael

---

From: [email-masked]
Date: Thu, 30 May 2013 15:05:09 +0100

See in-lime...

> Hi,
>
> Even reading this 3 times doesn't make me understand it. Could it be that
> we have a misunderstanding and you're thinking of what RFC2140 calls
> "ensemble sharing", when what I mean is what RFC2140 calls "temporal
> sharing" ? Or are you just primarily concerned with a situation more
> complicated than I envisioned?
>
I was initially thinking of temporal sharing when the final TCB of an
earlier session is used to initialize parameters of a new TCP connection
to the same host.

> Let's see. I was only thinking of greedy applications for now. My idea was
> just that, if you have an application that e.g. stops sending and starts
> again, greedily, after 10 seconds, new-cwv has a clear rule on what cwnd
> should be.
>
All true for a bulk flow: new-cwv would leave the flow with the same cwnd
(likely) as TCP stack without new-cwv. AND in this case cwnd would reflect
the capacity of the network path that was used - validated in the language
of CWV (within a factor of 2 or less).

> And, if you replace this with an application that, as it stops, also
> closes the connection and after 10 seconds starts a new connection, and
> then immediately uses it to send, greedily, then there isn't much
> difference between this case and the case above, and IW could probably be
> set using a similar rule from new-cwv.
>
We may agree for bulk flows.

> Until I sent my first email about this, I wasn't aware of that (but I got
> a private message telling me, straight away). I knew that some parts of
> RFC2140 are implemented, but I didn't expect cwnd to be reused beyond the
> end of a connection to set the next IW. Anyway, as Joe said, RFC2140 is
> informational, and I don't think it has clear rules about how long cwnd
> could be kept etc., so what I'm suggesting is to make a recommendation to
> let new-cwv consider a connection that terminates the same way as it
> considers a connection that enters the non-validated phase, and then
> allows re-using the cwnd in a following connection by noticing that the
> non-validated phase ends. If a very similar reasonable behavior is already
> out there, then I guess all I'm saying is "legalise it!"  :-)
>
*IF* the flow was validated when it closed, then using new-cwv rules for
decay of the value with time may be useful - this could control the size
of the contribution to the temporal pool for new flows, decaying with
time. i.e. the older the close was, the fewer the bytes you allow to be
reused in an initial window for a new flow.

If the values are not right in new-cwv, then I'd welcome that we can
discuss that...

> I don't get this; sorry if it should be obvious. We're talking about the
> time between the end of connection 1 and the beginning of connection 2,
> surely this cached value isn't then updated every RTT when there's nothing
> going on? And what do you mean with "the extra between 1 MSS and cwnd" ?
>
This is a small point that we have not discussed with respect to the final
value of cwnd in a closed flow.

cwnd reflects the estimate of what we think is safe for the next RTT,
rather than what was actually found to be safe from ACKs returned in the
last RTT (a fundamental difference between pipeACK and cwnd). I was simply
suggesting that we shouldn't restart at cwnd but 1/2 cwnd?

> This last paragraph gives me the impression that you're thinking of
> ensemble sharing, not temporal sharing. I'm not suggesting giving away the
> cwnd of a flow while it's running (even if it's taking a break). Yes that
> could be reasonable as you sketch it (only for new-cwv-supporting flows,
> to make flows go from non-validated to validated), but it's just a more
> complicated beast than what I was thinking about.
>
It applies also to temporal sharing.

Here is the issue... when an app uses TCP as currently defined, it can
send as much or as little as it chooses. The cwnd reflects an upper limit
to the maximum the app can send on one RTT. For bulk flows the effect of
sharing is clear (above).

For a flow that has been sending less than cwnd, it would not be
reasonable to say that the non-validated cwnd value of a closed flow could
be used by a new flow that did not start in a non-validated phase. If this
happens, the flow could have "far too much credit".

Of course if the new flow also used new-cwv and restored other TCB state
it would be the same as if the same TCP flow had been used persistently.
That would be nice.

Gorry

---

From: Michael Welzl <[email-masked]>
Date: Fri, 31 May 2013 17:27:54 +0200

On May 30, 2013, at 4:05 PM, [email-masked] wrote:

> See in-lime…

I even licked it, it's very sour!   :-D

> earlier session is used to initialize parameters of a new TCP connection

ok

> of CWV (within a factor of 2 or less).

yes

> We may agree for bulk flows.

It seems like it!

> reused in an initial window for a new flow.

Yes!

> discuss that…

Yes, below

> value of cwnd in a closed flow.

Got it. Now I understand what you meant -

> suggesting that we shouldn't restart at cwnd but 1/2 cwnd?

Yes, why not. I hadn't really thought of this case yet.

> happens, the flow could have "far too much credit".

Yes.

> it would be the same as if the same TCP flow had been used persistently.

Not sure I get that. The good news is, it's the only paragraph in your email that I can't follow, and it's much shorter than last time  :-)

Cheers,
Michael