Subject: [tcpm] RFC5681: why halving FlightSize not cwnd?

From: Yuchung Cheng <[email-masked]>
Date: Thu, 6 Sep 2012 11:42:37 -0700

RFC 5681 specifically calls to use FlightSize instead of cwnd for cwnd
reduction.

"""When a TCP sender detects segment loss using the retransmission timer
   and the given segment has not yet been resent by way of the
   retransmission timer, the value of ssthresh MUST be set to no more
   than the value given in equation (4):

      ssthresh = max (FlightSize / 2, 2*SMSS)            (4)

   where, as discussed above, FlightSize is the amount of outstanding
   data in the network.

   [...]

   Implementation Note: An easy mistake to make is to simply use cwnd,
   rather than FlightSize, which in some implementations may
   incidentally increase well beyond rwnd.
"""
When a sender sends cwnd==20 packets
a) fast-recovery: only 17th packet got dropped. FlightSize == 4
(pipe==1), ssthresh=2
b) timeout: 17-20th packets are dropped. FlightSize == 4(pipe==4), ssthresh=2

As a result, the next transaction, e.g., an HTTP response, will almost
exit slow-start immediately and slowly increase cwnd per RTT. But
AFAIK, ssthresh is based on slow-start overshoot by half of cwnd (of
previous round trip), not current FlightSize.

This has been implicitly addressed in Laminar, but I want to call this
out b/c I don't understand the rwnd argument in RFC5681. As long as
the sender respects rwnd, what's the problem?

---

From: Ethan Blanton <[email-masked]>
Date: Thu, 6 Sep 2012 15:08:36 -0400

> b) timeout: 17-20th packets are dropped. FlightSize == 4(pipe==4), ssthresh=2

Note that this is _only_ true if there were only 20 segments available
to send.  Otherwise, by the time the dup ACKs for 17 come in, up to 32
(remember ssthresh!) new segments would have been clocked out.  So
your sender is either data-limited or protocol-limited or some such.

What that means is, in your scenario, where FlightSize == 4, at the
time that packet 17 was dropped, your TCP's contribution to the
network traffic was only 4 packets.  Therefore, as far as it can tell,
a burden of 4 packets is "too much".

Yes, this is over simple and probably not the case -- but we're also
talking about a corner case, here.  Since you don't have any new data
to send, anyway, you'll retransmit 17, get an ACK for 20, and close
the connection or wait for new data or whatever.

Ethan

---

From: Yuchung Cheng <[email-masked]>
Date: Thu, 6 Sep 2012 12:33:02 -0700

> your sender is either data-limited or protocol-limited or some such.
This is unfortunately the case of trillions of HTTP flows or any RPC
 interactive type traffic today. It's not a corner case.

> a burden of 4 packets is "too much".

so if I send 100 packet (then stop) and only the last 1 got dropped. I
 should start from ssthresh == 2? isn't it more reasonable to assume
100 is too much, not 1 is too much?

then I am kinda unfortunate if the next some data is ready to send 1 RTT
after the one packet loss is recovered.

and I still don't get the rationale of rwnd argument in RFC 5681.

> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (GNU/Linux)
>
> iQEVAwUBUEj0tP8fixZ3H8crAQjFCgf/XRxoG+d3G1Hk7baB3tMht2Vgwx6VrTu/
> daL2bOLuabBjAySKUp1ouBtmaAJUs40Il1w9cdyQrcJ99NUxqpBywB2goLTxsdPQ
> xkdjzHXI52upeg2bgDBtamc7T+qsSiDkyCzNpWUG8FdPPAqC7lGoL0Xe8OlOFET5
> FfgeCH/K1Z+wsI05ykNe8Vv9Shx1/CJnWXjJ86DUNZl56qXqpENmLAvDdgQLe4AK
> H8BZgG6Ij66Vii8Al8Tu8ndK2HMMO8JnYIIEbj9gASHCusB2ZqdopA7xg8GCaMKM
> E641K5C5mRpyQE6yR/ORlE6xIL243ARsiM9Sfv0S30hoMmNn5ll71Q==
> =HJZ1
> -----END PGP SIGNATURE-----
>

---

From: Mark Allman <[email-masked]>
Date: Thu, 06 Sep 2012 15:54:02 -0400

A few things ...

  - I don't buy Ethan's argument that the burden on the network is 4
    packets if you lose the 17th.  It seems to me the burden is measured
    from the front of the window not the back.  So, in this case it was
    a burden of 17 packets that caused the loss.

  - The historical reason why we use FlightSize instead of cwnd is that
    cwnd was not always reflective of the burden placed on the network.
    For instance, some implementations just let the cwnd grow without
    bound regardless of whether it was used or not.  So, the cwnd might
    be (say) 100 even though you have never sent more than 20 segments
    per RTT when you go to set ssthresh.

  - (The note about the rwnd is not elegant!  All we mean is that the
    cwnd is growing without being used and, hey, it could even grow
    higher than the rwnd.  But, really who cares about the rwnd ... the
    point is that it grows unused and so is not reflective of anything.)
    
  - I may well buy that the current scheme leaves us too conservative
    due to cases like you sketch.  Exactly how important this case is
    I'll leave to you and Ethan to fight out! :-)

  - So, without additional schemes we're left with being too aggressive
    (using cwnd) or too conservative (using FlightSize).  But, if we're
    going to error that is probably the right direction.

  - I probably would not have all that much heartburn making the
    ssthresh 10 in the case you describe as long as there was some
    knowledge that a cwnd of 20 was used recently.  I.e., it isn't the
    result of some large storage of permission to send that was built up
    over time, but was in fact the result of the application's sending
    pattern.  I think one could design some rules around that notion
    that would be OK.

allman

---

From: Ethan Blanton <[email-masked]>
Date: Thu, 6 Sep 2012 17:25:30 -0400

>     a burden of 17 packets that caused the loss.

Note that this was not intended to be my argument; my argument is that
a TCP that doesn't "remember" such things (and 5681 does not) only
knows about the 4 packets at the time of the loss, so it *thinks* the
burden is 4.  This is clearly not optimal, and I would not argue that
it is.  Perhaps I stated this poorly.

>     going to error that is probably the right direction.

Agreed.

>     pattern.  I think one could design some rules around that notion

Also agreed.  I think it's reasonable to assume that the FlightSize in
effect at the time a lost packet was *sent* is safe, for sure.

Ethan

---

From: Mark Allman <[email-masked]>
Date: Thu, 06 Sep 2012 18:05:32 -0400

> it is.  Perhaps I stated this poorly.

I agree with that.

allman

---

From: Matt Mathis <[email-masked]>
Date: Thu, 6 Sep 2012 15:50:34 -0700

> it is.  Perhaps I stated this poorly.

I would say that you are using the wrong state variables:  at this
point cwnd is simultaneously being used to suppress bursts and
remember the congestion state from before the application pause.   If
you parameterize it differently (e.g. Laminar) this becomes a
non-problem.

>>     going to error that is probably the right direction.

Yes exactly.   One solution would be to pace from FS up to ccwind....
This case is described in the Laminar draft.

> effect at the time a lost packet was *sent* is safe, for sure.

I point out this logic assumes that it was the instantaneous queue
length that triggered the losses (as it is for drop tail).  With RED
or CoDel, the losses are normally triggered by a persistent queue,
which may or may not depend on the instantaneous window at the time
the packet was either sent or received.   With CoDel, drops are
triggered when the minimum window size is still large enough to
sustain a queue....  The peak window has no effect on the drops
(unless the queue overflows).

Thanks,
--MM--
The best way to predict the future is to create it.  - Alan Kay

---

From: [email-masked]
Date: Fri, 7 Sep 2012 07:53:14 +0100

If FlightSize is not ~cwnd, then at the time, the flow is not fully
utilising cwnd, but there are a wide range of (app) behaviours that result
in this condition. My thoughts are that we should update the behaviour in
both in standards-track TCP and also in Laminar.

This has been the topic of the new-cwv draft that we have been proposing
as an update to TCP. It's been discussed on the ICCRG list, and we're
re-structuring our draft and expect to publish this revision in a week or
so.

Gorry

> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm
>

---

From: Michael Welzl <[email-masked]>
Date: Fri, 7 Sep 2012 09:42:16 +0200

Hi all,

Not too long ago, Yuchung pointed out a very similar problem to the one here with our own work, draft-hurtig-tcpm-rtorestart-02.txt, which intends to make the RTO timer more aggressive if the really used window is very small. We used FlightSize in our algorithm to represent this "really used window". As with RFC 5681, our rationale is that FlightSize captures the amount of data that is actually transmitted onto the network, which may be significantly smaller than cwnd if the sender is application-limited.

Yuchung has now shown us that this choice is bad, twice. FlightSize captures what we want, but additionally, a small FlightSize is also always reached towards the end of an arbitrarily large window (yes, this is also an application-limited case - it's about the difference between an application with a low constant sending rate and the end of a transfer (or, similarly, an application with a stop-and-go behavior)). Neither our draft nor RFC 5681 really captures this situation. Indeed, it seems to me the same problem appears in draft-fairhurst-tcpm-newcwv-03.txt, which also relies on FlightSize in a similar way:

Section 4.2:
***
      Non-validated phase: FlightSize <(2/3)*cwnd.  This is the phase
      where the cwnd has a value based on a previous measurement of the
      available capacity, and the usage of this capacity has not been
      validated in the previous RTT.
***

If FlightSize reaches this value at the end of a larger window, I think this statement is wrong. Indeed, in accordance with Yuchung's examples, this would make the mechanisms in this draft kick in every time at the end of e.g. a 20-packet-long web flow (or indeed at the end of any transfer, with arbitrary length). I don't think that's the intention?

I suspect that we can find more examples of RFCs or drafts with this issue. The problem is indeed more fundamental - as Matt said, we're using the wrong state variables. I can well imagine that the problem automatically disappears with TCP Laminar, but as a smaller update to existing / ongoing work, what we really need is probably a means to differentiate Yuchung's case from the case of concern in at least RFC 5681, draft-hurtig-tcpm-rtorestart and draft-fairhurst-tcpm-newcwv-03.txt.

Here's a simple suggestion: can we identify Yuchung's case by saying that FlightSize has been continuously decreasing?

Cheers,
Michael

---

From: Gorry Fairhurst <[email-masked]>
Date: Fri, 07 Sep 2012 09:32:38 +0100

It's not easy to just say FlightSize decreases, since this can happen in 
various cases. I'm wondering how useful it is to measure FlightSize 
during a transfer. It's a reasonable measure when there is reported 
congestion to figure out what was sent in the last "pipe", but it can 
result in an unexpected value (e.g. at the end of a burst).

One of the changes we'd like to introduce in the next revision of 
new-cwv is to avoid this reliance on FlightSize to determine the window 
state of the sender. Instead we will suggest that the sender dtermines 
the state of the "pipe" by comparing the window that was ACK'ed with 
cwnd - not the volume of data was sent (FlightSize). In some cases, the 
two seem to be quite different.

Gorry

---

From: Yuchung Cheng <[email-masked]>
Date: Fri, 7 Sep 2012 15:49:01 -0700

I am glad we all identify the problem about FlightSize. We need a
better mechanism to address application-limited cases, i.e., when cwnd
is not fully used recently.

within an RTT or so:
a) loss: halve cwnd or flightsize or something else?
b) no-loss: grow cwnd or not?

longer than several RTTs (idle)
c) decay the cwnd every couple minutes or every couple RTO, slow-start
to prev cwnd, blah?

There is always a concern about potential bursts leaving cwnd >>
FlightSize. So RFCs and implementations drag cwnd to "moderate"
potential bursts. For example, Linux pulls cwnd down to FlightSize +
epsilon whenever an ACK does not look right, and refuses to grow cwnd
unless it was fully used. And most congestion control design does not
consider these events.

I don't know what the right solution is yet. Pacing + track inflight
of last RTT?

but I know app-limited cases happen a whole lot today. Beside HTTP or
RPCs (in data centers), even video deliveries are interactive, e.g.,
NetFlix or Youtube. The apps pace at second(s) interval to conserve
network bandwidth. And TCP-smart apps all use persistent connections
today. TCP usage has moved away from bulk-ftp long time...

Yuchung