Subject: [tcpm] ACK aggregation and congestion window growth

From: Ingemar Johansson S <[email-masked]>
Date: Fri, 26 Apr 2019 13:46:37 +0000

Hi

I am experimenting with a simple test setup with 2 Ubuntu 18.04 PCs (Cubic
CC). 

Between these two I have a simple setup that aggregates ACKs so that they
are forwarded to the server only every 10ms. The min RTT is 18ms. 

The problem I see is that even though I should reach 1Gbps throughput, I
only get around 200Mbps. 

I would believe that the congestion window should eventually increase enough
to handle the burstiness given by the ACK aggregation but it seems like this
is not the case. 

Is there a limitation in the Linux stack that prevents congestion window
growth when ACKs arrive in bursts like this ? 

/Ingemar

==================================

Ingemar Johansson  M.Sc. 

Master Researcher

Ericsson Research

Network Protocols & E2E Performance

Labratoriegränd 11

971 28, Luleå, Sweden

Phone +46-1071 43042

SMS/MMS +46-73 078 3289

[email-masked]

www.ericsson.com

                Nothing to stop this

             being the best day ever

                            U2

==================================

---

From: Rick Jones <[email-masked]>
Date: Fri, 26 Apr 2019 07:03:06 -0700

Does your ACK aggregator also delay the ACKs of SYNchronize segments? If not, perhaps the congestion control sees the increase (?) in round trip time after connection establishment as a signal of congestion and behaves accordingly. 

> On Apr 26, 2019, at 06:46, Ingemar Johansson S <[email-masked]> wrote:
> 
> Hi
>  
> I am experimenting with a simple test setup with 2 Ubuntu 18.04 PCs (Cubic CC).
> Between these two I have a simple setup that aggregates ACKs so that they are forwarded to the server only every 10ms. The min RTT is 18ms.
> The problem I see is that even though I should reach 1Gbps throughput, I only get around 200Mbps.
> I would believe that the congestion window should eventually increase enough to handle the burstiness given by the ACK aggregation but it seems like this is not the case.
> Is there a limitation in the Linux stack that prevents congestion window growth when ACKs arrive in bursts like this ?
>  
> /Ingemar
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: Neal Cardwell <[email-masked]>
Date: Fri, 26 Apr 2019 10:27:10 -0400

Hi Ingemar,

Thanks for the report. From the description of the issue, I suspect there
is some combination of the following:

(1) This may be related to some buggy behavior in the Linux TCP logic for
assessing whether a connection is cwnd-limited or application-limited. In
the scenario you describe, with those high speeds and long gaps between
ACKs, it sounds like the flow may have cwnd that is unused due to to TSO
deferral, and this may be causing the logic to erroneously mark the flow as
not being cwnd-limited, which would prevent cwnd growth by CUBIC.

(2) If there are entire flights of data that are ACKed with a single ACK,
then the CUBIC code may assess the long gaps between transmits and ACKs as
an idle period, and erroneously push its epoch_start forward to skip any
cwnd growth that would have been slated for that period.

(I would guess (1) is more likely, since the Reno emulation code path in
CUBIC should not be affected by (2), so CUBIC should eventually grow cwnd
in a Reno-style fashion whether or not it hits (2).)

We can provide some proposed patches for those issues. Would you be able to
apply the patches and test them in your workload?  If so, what exact kernel
version would the patches need to be generated for?

Also, would you be able to post (suitably anonymized, heads-only) tcpdump
packet traces so that we can see what the exact scenario is? This would be
particularly useful if you are unable to apply the patches to verify the
fix.

thanks,
neal

On Fri, Apr 26, 2019 at 10:03 AM Rick Jones <perfgeek=
[email-masked]> wrote:

> Does your ACK aggregator also delay the ACKs of SYNchronize segments? If
> not, perhaps the congestion control sees the increase (?) in round trip
> time after connection establishment as a signal of congestion and behaves
> accordingly.
>
> I would believe that the congestion window should eventually increase
> enough to handle the burstiness given by the ACK aggregation but it seems
> like this is not the case.
>

---

From: Ingemar Johansson S <[email-masked]>
Date: Tue, 30 Apr 2019 05:44:14 +0000

Hi

No the ACK aggregator does not delay the 3WHS, I guess this can cause Cubic to exit slow start. Still I would expect that CWND would eventually increase, even in congestion avoidance.

Regards

/Ingemar

From: Rick Jones <[email-masked]> 
Sent: den 26 april 2019 16:03
To: Ingemar Johansson S <[email-masked]>
Cc: [email-masked]
Subject: Re: [tcpm] ACK aggregation and congestion window growth

On Apr 26, 2019, at 06:46, Ingemar Johansson S <[email-masked] <mailto:[email-masked]> > wrote:

Hi

/Ingemar

Ingemar Johansson  M.Sc. 

Master Researcher

Ericsson Research

Labratoriegränd 11

971 28, Luleå, Sweden

Phone +46-1071 43042

SMS/MMS +46-73 078 3289

[email-masked] <mailto:[email-masked]> 

www.ericsson.com <http://www.ericsson.com> 

                Nothing to stop this

             being the best day ever

                            U2

_______________________________________________
tcpm mailing list
[email-masked] <mailto:[email-masked]> 
https://www.ietf.org/mailman/listinfo/tcpm

---

From: Ingemar Johansson S <[email-masked]>
Date: Tue, 30 Apr 2019 07:31:04 +0000

Hi

Thanks for the hints. I will try and fix some wireshark logs, seems though that this issue is a bit transient and admittedly it works more often than it does not. This is perhaps a corner case ?

/Ingemar 

From: Neal Cardwell <[email-masked]> 
Sent: den 26 april 2019 16:27
To: Ingemar Johansson S <[email-masked]>
Cc: [email-masked]; Yuchung Cheng <[email-masked]>; Eric Dumazet <[email-masked]>; Rick Jones <perfgeek=[email-masked]>
Subject: Re: [tcpm] ACK aggregation and congestion window growth

Hi Ingemar,

Thanks for the report. From the description of the issue, I suspect there is some combination of the following:

(1) This may be related to some buggy behavior in the Linux TCP logic for assessing whether a connection is cwnd-limited or application-limited. In the scenario you describe, with those high speeds and long gaps between ACKs, it sounds like the flow may have cwnd that is unused due to to TSO deferral, and this may be causing the logic to erroneously mark the flow as not being cwnd-limited, which would prevent cwnd growth by CUBIC.

(2) If there are entire flights of data that are ACKed with a single ACK, then the CUBIC code may assess the long gaps between transmits and ACKs as an idle period, and erroneously push its epoch_start forward to skip any cwnd growth that would have been slated for that period.

(I would guess (1) is more likely, since the Reno emulation code path in CUBIC should not be affected by (2), so CUBIC should eventually grow cwnd in a Reno-style fashion whether or not it hits (2).)

We can provide some proposed patches for those issues. Would you be able to apply the patches and test them in your workload?  If so, what exact kernel version would the patches need to be generated for?

Also, would you be able to post (suitably anonymized, heads-only) tcpdump packet traces so that we can see what the exact scenario is? This would be particularly useful if you are unable to apply the patches to verify the fix.

thanks,

neal

 

On Fri, Apr 26, 2019 at 10:03 AM Rick Jones <perfgeek= <mailto:[email-masked]> [email-masked]> wrote:

On Apr 26, 2019, at 06:46, Ingemar Johansson S < <mailto:[email-masked]> [email-masked]> wrote:

Hi

/Ingemar

Ingemar Johansson  M.Sc. 

Master Researcher

Ericsson Research

Labratoriegränd 11

971 28, Luleå, Sweden

Phone +46-1071 43042

SMS/MMS +46-73 078 3289

 <mailto:[email-masked]> [email-masked]

 <http://www.ericsson.com> www.ericsson.com

                Nothing to stop this

             being the best day ever

                            U2

_______________________________________________
tcpm mailing list
 <mailto:[email-masked]> [email-masked]
 <https://www.ietf.org/mailman/listinfo/tcpm> https://www.ietf.org/mailman/listinfo/tcpm

_______________________________________________
tcpm mailing list
 <mailto:[email-masked]> [email-masked]
 <https://www.ietf.org/mailman/listinfo/tcpm> https://www.ietf.org/mailman/listinfo/tcpm

---

From: Neal Cardwell <[email-masked]>
Date: Tue, 30 Apr 2019 15:38:54 -0400

Hi Ingemar,

Thanks. We'll see if we can reproduce this and fix it based on our working
theory. We'll follow up on this thread if we end up posting a patch for
Linux TCP. If you do happen to collect a trace that shows this problem, we
would love to take a look.

thanks,
neal

On Tue, Apr 30, 2019 at 3:31 AM Ingemar Johansson S <
[email-masked]> wrote:

> Hi
>
>
>
> Thanks for the hints. I will try and fix some wireshark logs, seems though
> that this issue is a bit transient and admittedly it works more often than
> it does not. This is perhaps a corner case ?
>
>
>
> /Ingemar
>
>
>
> *From:* Neal Cardwell <[email-masked]>
> *Sent:* den 26 april 2019 16:27
> *To:* Ingemar Johansson S <[email-masked]>
> *Cc:* [email-masked]; Yuchung Cheng <[email-masked]>; Eric Dumazet <
> [email-masked]>; Rick Jones <perfgeek=[email-masked]>
> *Subject:* Re: [tcpm] ACK aggregation and congestion window growth
>
>
>
> Hi Ingemar,
>
>
>
> We can provide some proposed patches for those issues. Would you be able
> to apply the patches and test them in your workload?  If so, what exact
> kernel version would the patches need to be generated for?
>
>
>