Subject: [tcpm] Why is the minimum ssthresh 2*SMSS (rather than 1)?

From: Bob Briscoe <[email-masked]>
Date: Fri, 15 May 2015 13:33:28 +0100

tcpm folks,

Despite searching and asking around, I cannot find any definitive 
reasoning for the choice of 2*SMSS in eqn (4) in RFC5681.

    ssthresh = max (FlightSize / 2, 2*SMSS)            (4)

In testing a new AQM with multiple flows, now we've got the RTT 
really low, we've found that TCP's minimum window of 2 makes TCP 
ignore the AQM's signals ('cos it always rounds up to 2) so all the 
TCP flows drive up the queueing delay to give themselves a 
large-enough RTT - defeating the AQM.

Bob

________________________________________________________________
Bob Briscoe,                                                  BT

---

From: John Heffner <[email-masked]>
Date: Fri, 15 May 2015 13:12:36 -0400

On Fri, May 15, 2015 at 8:33 AM, Bob Briscoe <[email-masked]> wrote:
> tcpm folks,
>
> Despite searching and asking around, I cannot find any definitive reasoning
> for the choice of 2*SMSS in eqn (4) in RFC5681.
>
> In testing a new AQM with multiple flows, now we've got the RTT really low,
> we've found that TCP's minimum window of 2 makes TCP ignore the AQM's
> signals ('cos it always rounds up to 2) so all the TCP flows drive up the
> queueing delay to give themselves a large-enough RTT - defeating the AQM.

Delayed ack?

---

From: Yuchung Cheng <[email-masked]>
Date: Fri, 15 May 2015 14:52:56 -0700

> we've found that TCP's minimum window of 2 makes TCP ignore the AQM's
I am confused. min ssthresh != min-cwnd == 1?

> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: Ilpo Järvinen <[email-masked]>
Date: Sat, 16 May 2015 01:14:59 +0300 (EEST)

On Fri, 15 May 2015, Bob Briscoe wrote:

> we've found that TCP's minimum window of 2 makes TCP ignore the AQM's signals
> ('cos it always rounds up to 2) so all the TCP flows drive up the queueing
> delay to give themselves a large-enough RTT - defeating the AQM.

Does this really matter that much whether ssthresh would be 1 or 2 SMSS? 
Both congestion avoidance and slow start are equally aggressive for the 
first RTT: when TCP start with cwnd=1 and the sender gets the ACK, the
applied increase is cwnd = cwnd + 1 SMSS regardless of the operating
mode.

However, ssthresh == cwnd case is left open so there might be a 
difference depending on implementation:
   "When cwnd and ssthresh are equal, the sender may use either
    slow start or congestion avoidance."
...but I don't see how that would relate to "minimum window of 2" you're 
talking here (and ssthresh is not really "window" anyway).

-- 
 i.

---

From: Bob Briscoe <[email-masked]>
Date: Sat, 16 May 2015 12:52:12 +0100

Ilpo & Yuchung,

Sorry, the phrasing of my question was too curt.

I'm not specifically interested in why ssthresh 
has a min. I only phrased my question that way 
because the way TCP fast recovery is specified 
and implemented sets a min for cwnd based on the min for ssthresh.

 From RFC5681:
2.  When the third duplicate ACK is received, a TCP MUST set ssthresh
        to no more than the value given in equation (4).
...
6.  When the next ACK arrives that acknowledges previously
        unacknowledged data, a TCP MUST set cwnd to ssthresh (the value
        set in step 2).  This is termed "deflating" the window.

The only guess I've heard for why it says 2*SMSS 
in eqn (4) is delayed ACKs (John Heffner, as well 
as Michael Welzl when I was asking around before 
I posted this to tcpm). Certainly delayed ACKs 
will cause segments to bunch in twos even if the 
window were less than two. However, if the RTT 
was short enough (or the link congested enough) 
to need a low window, at least segments could 
bunch in twos at a lower rate than two per RTT.

I believe TCP would not work against AQMs at low 
RTTs if the min for ssthresh in Eqn (4) was zero, 
not 2*SMSS. But there must have been a reason for 
this minimum at some point in the past - that's what I'm trying to find out.

If there is no strong reason to keep the 2*SMSS 
minimum, the rule after a timeout would have to 
be changed too. Specifically, the two changes needed to RFC5681 would be:

CURRENT:
     ssthresh = max (FlightSize / 2, 2*SMSS)            (4)
PROPOSED:
     ssthresh = FlightSize / 2                          (4)

CURRENT
    upon a timeout [...] cwnd MUST be set to
    no more than the loss window, LW, which equals 1 full-sized segment.
PROPOSED:
    upon a timeout [...] cwnd MUST be set to
      cwnd = min(LW, ssthresh)
    where the loss window, LW, which equals 1 full-sized segment.

Bob

---

From: Yoshifumi Nishida <[email-masked]>
Date: Mon, 18 May 2015 02:35:43 -0700

Hi Bob,

I'm not very sure about this.
With your proposal, if FlightSize is zero, cwnd will be zero.
In this case, TCP doesn't have a chance to increase cwnd other than
keep-alive?

Thanks,
--
Yoshi

On Sat, May 16, 2015 at 4:52 AM, Bob Briscoe <[email-masked]> wrote:

> I'm not specifically interested in why ssthresh has a min. I only phrased
> my question that way because the way TCP fast recovery is specified and
> implemented sets a min for cwnd based on the min for ssthresh.
>
> From RFC5681:
> The only guess I've heard for why it says 2*SMSS in eqn (4) is delayed
> ACKs (John Heffner, as well as Michael Welzl when I was asking around
> before I posted this to tcpm). Certainly delayed ACKs will cause segments
> to bunch in twos even if the window were less than two. However, if the RTT
> was short enough (or the link congested enough) to need a low window, at
> least segments could bunch in twos at a lower rate than two per RTT.
>
>
> I believe TCP would not work against AQMs at low RTTs if the min for
> ssthresh in Eqn (4) was zero, not 2*SMSS. But there must have been a reason
> for this minimum at some point in the past - that's what I'm trying to find
> out.
>
> If there is no strong reason to keep the 2*SMSS minimum, the rule after a
> timeout would have to be changed too. Specifically, the two changes needed
> to RFC5681 would be:
>
> CURRENT:
>> reasoning
>> signals

---

From: John Heffner <[email-masked]>
Date: Mon, 18 May 2015 11:15:10 -0400

On Sat, May 16, 2015 at 7:52 AM, Bob Briscoe <[email-masked]> wrote:
> I'm not specifically interested in why ssthresh has a min. I only phrased my
> question that way because the way TCP fast recovery is specified and
> The only guess I've heard for why it says 2*SMSS in eqn (4) is delayed ACKs
> (John Heffner, as well as Michael Welzl when I was asking around before I
> posted this to tcpm). Certainly delayed ACKs will cause segments to bunch in
> twos even if the window were less than two. However, if the RTT was short
> enough (or the link congested enough) to need a low window, at least
> segments could bunch in twos at a lower rate than two per RTT.

With delayed ack, reducing cwnd to one breaks the ack clock.
Transmission instead will be driven by the delayed ack timer.

  -John

---

From: Joe Touch <[email-masked]>
Date: Mon, 18 May 2015 08:18:30 -0700

+1

This is also why the original initial window was 2*SMSS.

Joe

---

From: Joe Touch <[email-masked]>
Date: Mon, 18 May 2015 11:25:18 -0700

> This is also why the original initial window was 2*SMSS.

To clarify:

	the initial CWND window spec'd in RFC2001  is 1 SMSS

	I recall reports when RFC2414 was being proposed
	that BSD never really implemented this; they started
	with 2*SMSS

	the interaction with 1*SMSS and delayed ACKs was
	known back in the late 90s (Mark Allman, Sally Floyd, et al.)

	which is why RFC2414 spec'd CWND in terms of an
	even number of SMSS bounded by 4K

In general, CWND with an odd number of SMSS is a bad idea.

Joe

---

From: Matt Mathis <[email-masked]>
Date: Mon, 18 May 2015 11:44:57 -0700

A smidge more detail: it was understood that for efficiency w/ DA we almost
always want cwnd >= 2, however there are some corner cases (persistent
timeouts due to high losses in particular) where this seems a bit unsafe.
As a consequence you get to cwnd=1 in a few cases, but if that one packet
is delivered, you immediately go to cwnd=2.  This balance was carefully
thought out at one point, but may be due to be revisited.

Thanks,
--MM--
The best way to predict the future is to create it.  - Alan Kay

Privacy matters!  We know from recent events that people are using our
services to speak in defiance of unjust governments.   We treat privacy and
security as matters of life and death, because for some users, they are.

---

From: Yuchung Cheng <[email-masked]>
Date: Mon, 18 May 2015 12:07:38 -0700

> Transmission instead will be driven by the delayed ack timer.
Delayed-ack may not be a concern in fast recovery, because the
receiver does not delay ACK when there is out-of-order data in the
queue.

Bob: does reducing ssthresh lower-bound to 1 help your aqm test case?

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 19 May 2015 10:53:59 +0100

Matt, Joe, John,

Thx for explaining the history.

I am saying what Matt just hinted at. "This decision may need to be revisited."

First, this affects well designed systems, even a 
broadband network with a good set of local CDNs 
and AQM installed everywhere, so most usage will 
see a very low RTT. Not just overloaded, underprovisioned networks with no AQM.

For window, w < 2*SMSS, the sender would need to 
pace segments at 1/w per RTT (as implemented in 
Linux for TCP Nice). But only if it also implemented ABC.

Then, at a low RTT, transmissions wouldn't be 
driven by the delayed ACK timer, they would be 
driven by the delayed ACK /factor/ (ie the number 
of segments received before a delayed ACK is 
emitted). With cwnd = 1, after 2 short RTTs, the 
receiver would release an ACK, (before the DA 
timer expires), which in turn would release two 
segments. So it gets an average window of 1, but as 2 segments every 2 RTTs.

If the min cwnd of 2*SMSS has been hit, sending 
two segments per RTT but pushing up the queue to 
do so, doesn't make throughput any higher. It 
just increases the RTT for yourself and everyone else (even with an AQM).

In the light of renewed interest in deploying 
AQM, we have to review which is more important:
* avoiding TCP being clocked by the delayed ACK 
mechanism (does this make significant difference to performance?)?
* or avoiding TCP artifically inflating the 
bottleneck queue (which does make a real 
difference - it reduces delay, for the user and everyone else)?

Bob

At 19:44 18/05/2015, Matt Mathis wrote:
>A smidge more detail: it was understood that for 
>efficiency w/ DA we almost always want cwnd >= 
>2, however there are some corner cases 
>(persistent timeouts due to high losses in 
>particular) where this seems a bit unsafe.Â  As 
>a consequence you get to cwnd=1 in a few cases, 
>but if that one packet is delivered, you 
>immediately go to cwnd=2.Â  This balance was 
>carefully thought out at one point, but may be due to be revisited.
>
>Thanks,
>--MM--
>The best way to predict the future is to create it. Â - Alan Kay
>
>Privacy matters!Â  We know from recent events 
>that people are using our services to speak in 
>defiance of unjust governments. Â  We treat 
>privacy and security as matters of life and 
>death, because for some users, they are.
>
>On Mon, May 18, 2015 at 11:25 AM, Joe Touch 
><<mailto:[email-masked]>[email-masked]> wrote:
>
>
>Â  Â  Â  Â  the initial CWND window spec'd in RFC2001Â  is 1 SMSS
>
>Â  Â  Â  Â  I recall reports when RFC2414 was being proposed
>Â  Â  Â  Â  that BSD never really implemented this; they started
>Â  Â  Â  Â  with 2*SMSS
>
>Â  Â  Â  Â  the interaction with 1*SMSS and delayed ACKs was
>Â  Â  Â  Â  known back in the late 90s (Mark Allman, Sally Floyd, et al.)
>
>Â  Â  Â  Â  which is why RFC2414 spec'd CWND in terms of an
>Â  Â  Â  Â  even number of SMSS bounded by 4K
>
> >>Â  Â -John
><mailto:[email-masked]>[email-masked]

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 19 May 2015 10:59:57 +0100

Yuchung,

>Bob: does reducing ssthresh lower-bound to 1 help your aqm test case?

See email just sent to Joe, Matt, John.

I knew we would need pacing for a sub-SMSS window, but I thought we 
could at least do an initial fix to reduce the min cwnd to 1*SMSS 
without pacing. However, people's responses have made me realise that 
we would need pacing in all cases when cwnd < 2*SMSS.

You will know better than me whether pacing would be feasible to add.

Bob

---

From: John Heffner <[email-masked]>
Date: Tue, 19 May 2015 08:01:56 -0400

> On May 19, 2015, at 5:53 AM, Bob Briscoe <[email-masked]> wrote:
> 
> Matt, Joe, John,
> 
> First, this affects well designed systems, even a broadband network with a good set of local CDNs and AQM installed everywhere, so most usage will see a very low RTT. Not just overloaded, underprovisioned networks with no AQM.
> 
> For window, w < 2*SMSS, the sender would need to pace segments at 1/w per RTT (as implemented in Linux for TCP Nice). But only if it also implemented ABC.
> 
> Then, at a low RTT, transmissions wouldn't be driven by the delayed ACK timer, they would be driven by the delayed ACK /factor/ (ie the number of segments received before a delayed ACK is emitted). With cwnd = 1, after 2 short RTTs, the receiver would release an ACK, (before the DA timer expires), which in turn would release two segments. So it gets an average window of 1, but as 2 segments every 2 RTTs.
> 
> If the min cwnd of 2*SMSS has been hit, sending two segments per RTT but pushing up the queue to do so, doesn't make throughput any higher. It just increases the RTT for yourself and everyone else (even with an AQM).
> 
> In the light of renewed interest in deploying AQM, we have to review which is more important:
> * avoiding TCP being clocked by the delayed ACK mechanism (does this make significant difference to performance?)?
> * or avoiding TCP artifically inflating the bottleneck queue (which does make a real difference - it reduces delay, for the user and everyone else)?

Another degree of freedom is segment size.  For instance, if the expected fair share is less than 2 mtu a network operator could reduce mtu.  Or tcp might try to be adaptive.

  -John

---

From: Ilpo Järvinen <[email-masked]>
Date: Tue, 19 May 2015 16:40:45 +0300 (EEST)

On Tue, 19 May 2015, John Heffner wrote:

> Another degree of freedom is segment size.  For instance, if the 
> expected fair share is less than 2 mtu a network operator could reduce 
> mtu.  Or tcp might try to be adaptive.

Linux actually already does segment size reduction for the cases where 
receiver advertized window is too small to fit enough packets. However, in 
case of congestion, reducing segment size works to some extent but in the 
end of that road lurks congestion collapse due to header overhead so it 
might not be wise to proceed towards that direction.

Pacing, on the other hard, wouldn't impose similar utilization reductions 
due to overhead increase.

-- 
 i.

---

From: Joe Touch <[email-masked]>
Date: Tue, 19 May 2015 10:06:38 -0700

It might be useful to take a look at this too:

http://www.computer.org/csdl/proceedings/icnp/1997/8061/00/80610205-abs.html

On 5/19/2015 2:59 AM, Bob Briscoe wrote:
> without pacing. However, people's responses have made me realise that we
> would need pacing in all cases when cwnd < 2*SMSS.
>