Subject: [tcpm] Why is Cubic said to be RTT-fair?

From: Tom Sanders <[email-masked]>
Date: Wed, 20 May 2015 18:50:59 +0530

Hi,

As per my understanding, unlike other TCP variants, CUBIC claims to be RTT
fair. This is because the window growth rate is independent of the RTT, and
hence the performance is the same as on a low and a high RTT paths. Others
are dependent upon RTT since their windows grow as they receive ACKs from
their peers, while CUBIC doesnt (window growth is a function of time). This
is theory.

In practice, CUBIC (which is the default now on most linux distributions)
also depends upon RTT. I connected to machines on a LAN and did a FTP
transfer. I got a certain bandwidth. Next, i artificially inserted delay
using "tc qdisc" in the path. I increased the delay on the ethernet
interface connecting my linux machine to the LAN to be around 100ms. I
immediately saw that the TCP performance went down.

To bring it up to the same level as before i had to use multiple TCP
streams. To me this means that CUBIC performance is dependent of the RTT.
So why do we call it RTT-fair?

Thanks, Toms

---

From: Rick Jones <[email-masked]>
Date: Wed, 20 May 2015 07:21:49 -0700

On May 20, 2015, at 6:20 AM, Tom Sanders <[email-masked]> wrote:

> Hi,
> 
> As per my understanding, unlike other TCP variants, CUBIC claims to be RTT fair. This is because the window growth rate is independent of the RTT, and hence the performance is the same as on a low and a high RTT paths. Others are dependent upon RTT since their windows grow as they receive ACKs from their peers, while CUBIC doesnt (window growth is a function of time). This is theory.
> 
> In practice, CUBIC (which is the default now on most linux distributions) also depends upon RTT. I connected to machines on a LAN and did a FTP transfer. I got a certain bandwidth. Next, i artificially inserted delay using "tc qdisc" in the path. I increased the delay on the ethernet interface connecting my linux machine to the LAN to be around 100ms. I immediately saw that the TCP performance went down.
> 
> To bring it up to the same level as before i had to use multiple TCP streams. To me this means that CUBIC performance is dependent of the RTT. So why do we call it RTT-fair?

I cannot speak to CUBIC specifically, but I would think you would want to make sure you were seeing effects of congestion window and not of the classic TCP window.  For example, if the FTP client/server you were using makes an explicit setsockopt() to set the socket buffer sizes and by extension the classic TCP window, your bumping of the RTT to around 100ms may have taken the bandwidth down as a consequence of the classic:

Throughput <= WindowSize/RTT

Similarly, since you mention Linux, if the FTP client/server didn’t make explicit setsockopt() calls,  the sysctl values for tcp_rmem and tcp_wmem may have been such that the auto tuning of the window size couldn’t allow the classic TCP window to grow “enough.”

I suspect that the experiment needs to be setup to have the same classic window size in both cases, sized per the above to be sufficient to achieve the peak throughput in the high RTT case.  Then you will have eliminated classic window as a factor.

Also, depending on the version of the Linux kernel you are using, there may be some effects from tcp small queues or perhaps (a stretch?) even byte queue limits which may not be playing well with netem.  (Just guessing there really)

happy benchmarking,

rick jones

---

From: Tom Sanders <[email-masked]>
Date: Wed, 20 May 2015 22:48:31 +0530

Thanks Rick, this was very helpful.

So if there are no setsockopts, etc then CUBIC's performance, unlike other
TCP variants, should not get impacted by RTTs, right? Is that a fair
statement to make?

Toms

On 20 May 2015 at 19:51, Rick Jones <[email-masked]> wrote:

> > As per my understanding, unlike other TCP variants, CUBIC claims to be
> RTT fair. This is because the window growth rate is independent of the RTT,
> and hence the performance is the same as on a low and a high RTT paths.
> Others are dependent upon RTT since their windows grow as they receive ACKs
> from their peers, while CUBIC doesnt (window growth is a function of time).
> This is theory.
> >
> > In practice, CUBIC (which is the default now on most linux
> distributions) also depends upon RTT. I connected to machines on a LAN and
> did a FTP transfer. I got a certain bandwidth. Next, i artificially
> inserted delay using "tc qdisc" in the path. I increased the delay on the
> ethernet interface connecting my linux machine to the LAN to be around
> 100ms. I immediately saw that the TCP performance went down.
> >
> I cannot speak to CUBIC specifically, but I would think you would want to
> make sure you were seeing effects of congestion window and not of the
> classic TCP window.  For example, if the FTP client/server you were using
> makes an explicit setsockopt() to set the socket buffer sizes and by
> extension the classic TCP window, your bumping of the RTT to around 100ms
> may have taken the bandwidth down as a consequence of the classic:
>
> Similarly, since you mention Linux, if the FTP client/server didn’t make
> explicit setsockopt() calls,  the sysctl values for tcp_rmem and tcp_wmem
> may have been such that the auto tuning of the window size couldn’t allow
> the classic TCP window to grow “enough.”
>
> I suspect that the experiment needs to be setup to have the same classic
> window size in both cases, sized per the above to be sufficient to achieve
> the peak throughput in the high RTT case.  Then you will have eliminated
> classic window as a factor.
>
> Also, depending on the version of the Linux kernel you are using, there
> may be some effects from tcp small queues or perhaps (a stretch?) even byte
> queue limits which may not be playing well with netem.  (Just guessing
> there really)
>
> happy benchmarking,
>
> rick jones

-- 
Toms.

---

From: Rick Jones <[email-masked]>
Date: Wed, 20 May 2015 10:39:24 -0700

On 05/20/2015 10:18 AM, Tom Sanders wrote:
> So if there are no setsockopts, etc then CUBIC's performance, unlike
> other TCP variants, should not get impacted by RTTs, right? Is that a
> fair statement to make?

Like I said, I cannot speak to CUBIC specifically.  But, since it is 
possible to alter the default congestion control under Linux, if you 
have a sufficiently stable test environment, you could run the different 
congestion controls through your added delay and see what you get.

I would suggest setting the likes of net.core.[rw]mem_max to a value 
large enough to allow for "link rate" for your physical network even at 
100ms of RTT, and then run the likes of either iperf or netperf (*) with 
them making the setsockopt()s to that value to eliminate concerns about 
sufficient classic TCP window and/or Linux's autotuning.  Lather, rinse, 
repeat with the different congestion controls set to default via 
net.ipv4.tcp_congestion_control or if using netperf there is the 
test-specific -K option to specify the congestion control to use 
(assuming that functionality hasn't bitrotted)

I believe that iperf can be told to emit interim results, which would 
allow you to see (perhaps) how things "ramp-up" - there is a similar, 
optional feature of netperf which will cause it to emit interim results 
at a specified cadence.

Or you could just take packet traces and run them through the likes of 
tcptrace to get pretty pictures that way.

rick

(*) rather than FTP or some other file transfer because you want to 
eliminate filesystem effects

>
> Toms
>
> On 20 May 2015 at 19:51, Rick Jones <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>
>     On May 20, 2015, at 6:20 AM, Tom Sanders <[email-masked]
>     <mailto:[email-masked]>> wrote:
>
>      > Hi,
>      >
>      > As per my understanding, unlike other TCP variants, CUBIC claims
>     to be RTT fair. This is because the window growth rate is
>     independent of the RTT, and hence the performance is the same as on
>     a low and a high RTT paths. Others are dependent upon RTT since
>     their windows grow as they receive ACKs from their peers, while
>     CUBIC doesnt (window growth is a function of time). This is theory.
>      >
>     distributions) also depends upon RTT. I connected to machines on a
>     LAN and did a FTP transfer. I got a certain bandwidth. Next, i
>     artificially inserted delay using "tc qdisc" in the path. I
>     increased the delay on the ethernet interface connecting my linux
>     machine to the LAN to be around 100ms. I immediately saw that the
>     TCP performance went down.
>      >
>      > To bring it up to the same level as before i had to use multiple
>     TCP streams. To me this means that CUBIC performance is dependent of
>     the RTT. So why do we call it RTT-fair?
>
>     I cannot speak to CUBIC specifically, but I would think you would
>     want to make sure you were seeing effects of congestion window and
>     not of the classic TCP window.  For example, if the FTP
>     client/server you were using makes an explicit setsockopt() to set
>     the socket buffer sizes and by extension the classic TCP window,
>     your bumping of the RTT to around 100ms may have taken the bandwidth
>     down as a consequence of the classic:
>
>     Similarly, since you mention Linux, if the FTP client/server didn’t
>     make explicit setsockopt() calls,  the sysctl values for tcp_rmem
>     and tcp_wmem may have been such that the auto tuning of the window
>     size couldn’t allow the classic TCP window to grow “enough.”
>
>     I suspect that the experiment needs to be setup to have the same
>     classic window size in both cases, sized per the above to be
>     sufficient to achieve the peak throughput in the high RTT case.
>     Then you will have eliminated classic window as a factor.
>
>     Also, depending on the version of the Linux kernel you are using,
>     there may be some effects from tcp small queues or perhaps (a
>     stretch?) even byte queue limits which may not be playing well with
>     netem.  (Just guessing there really)
>
>     happy benchmarking,
>
>     rick jones
>
>
>
>
> --
> Toms.
>
>
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm
>

---

From: Tom Sanders <[email-masked]>
Date: Thu, 21 May 2015 06:21:40 +0530

Hi Neal,

This is odd.

In the original Cubic paper (
http://www4.ncsu.edu/~rhee/export/bitcp/cubic-paper.pdf) by Rhee and Xu,
they explicitly state that the Cubic is independent of RTT.

"The main feature of CUBIC is that its window growth function is defined in
real-time so that its growth will be independent of RTT. Our work was
partially inspired by HTCP [5], whose window growth function is also based
on real time. The congestion epoch period of CUBIC is determined by the
packet loss rate alone. As TCP’s throughput is defined by the packet loss
rate as well as RTT, the throughput of CUBIC is defined by only the packet
loss rate. Thus, when the loss rate is high and/or RTT is short, CUBIC can
operate in a TCP mode. Moreover, since the growth function is independent
of RTT, its RTT fairness is guaranteed as different RTT flows will still
grow their windows at the same rate."

Am i missing something here?

Thanks, Toms

On 20 May 2015 at 23:13, Neal Cardwell <[email-masked]> wrote:

> On Wed, May 20, 2015 at 1:18 PM, Tom Sanders <[email-masked]>
> wrote:
> other
> CUBIC, like most TCP congestion control variants, has significant
> aspects that evolve on the scale of RTTs: for example, in slow start,
> loss recovery, or when the CUBIC function is very steep, CUBIC's
> behavior is heavily dependent on RTT (send some packets, wait for an
> ACK, send some more packets,...). So CUBIC will not be
> perfectly"RTT-fair.
>
> I think it might be fair to say something like: once a path is
> saturated with CUBIC flows, as long as loss recovery or send/receive
> buffers are not a bottleneck, then competition between CUBIC flows
> should be more RTT-fair than Reno, in the long run, since for large
> parts of the life cycle of the CUBIC flow the evolution of cwnd is
> constrained by wall clock time rather than ACK arrival.
>
> neal
>
>
> > Toms
> >
> LAN and
> inserted
> >> > delay using "tc qdisc" in the path. I increased the delay on the
> ethernet
> >> > streams. To me this means that CUBIC performance is dependent of the
> RTT. So
> >> > why do we call it RTT-fair?
> >>
> there
> >> really)

-- 
Toms.

---

From: Mirja Kühlewind <[email-masked]>
Date: Fri, 22 May 2015 12:00:32 +0200

Hi Tom,

the text below only says that if you have two cubic flows with a different base RTT competing at the same bottleneck they’ll still converge to share the link equally. This does not mean that if you change the RTT strongly during one connection that Cubic is able to catch up fast. 

In fact this is one weakness of Cubic that I have observed. If there are change in base delay/rtt or strong increases in the available capacity, Cubic needs a rather long time to catch up. This is because cubic has a ‚target value‘ to which it increases it sending rate/window quickly but then if the capacity cap is not somewhere close to this value (as expected) it will probe very conservatively around this value and it takes a rather long time until it gets back into a phase where it increases its congestion window fast.

So what do you mean by 'I immediately saw that the TCP performance went down‘. How long do you run your test? And over which period of time this the performance go down?

Mirja

> Am 21.05.2015 um 02:51 schrieb Tom Sanders <[email-masked]>:
> 
> Hi Neal,
> 
> This is odd.
> 
> In the original Cubic paper (http://www4.ncsu.edu/~rhee/export/bitcp/cubic-paper.pdf) by Rhee and Xu, they explicitly state that the Cubic is independent of RTT.
> 
> "The main feature of CUBIC is that its window growth function is defined in real-time so that its growth will be independent of RTT. Our work was partially inspired by HTCP [5], whose window growth function is also based on real time. The congestion epoch period of CUBIC is determined by the packet loss rate alone. As TCP’s throughput is defined by the packet loss rate as well as RTT, the throughput of CUBIC is defined by only the packet loss rate. Thus, when the loss rate is high and/or RTT is short, CUBIC can operate in a TCP mode. Moreover, since the growth function is independent of RTT, its RTT fairness is guaranteed as different RTT flows will still grow their windows at the same rate."
> 
> On Wed, May 20, 2015 at 1:18 PM, Tom Sanders <[email-masked]> wrote:
> >> > did a FTP transfer. I got a certain bandwidth. Next, i artificially inserted
> >> > delay using "tc qdisc" in the path. I increased the delay on the ethernet
> >> > streams. To me this means that CUBIC performance is dependent of the RTT. So
> >> queue limits which may not be playing well with netem.  (Just guessing there
> >> really)
> >>
> >> happy benchmarking,
> >>
> >> rick jones
> >
> >
> >
> >
> > --
> > Toms.
> >

---

From: Tom Sanders <[email-masked]>
Date: Sat, 23 May 2015 06:46:32 +0530

Hi Mirja,

>
>
> the text below only says that if you have two cubic flows with a different
> base RTT competing at the same bottleneck they’ll still converge to share
> the link equally. This does not mean that if you change the RTT strongly
> during one connection that Cubic is able to catch up fast.
>
> In fact this is one weakness of Cubic that I have observed. If there are
> change in base delay/rtt or strong increases in the available capacity,
> Cubic needs a rather long time to catch up. This is because cubic has a
> ‚target value‘ to which it increases it sending rate/window quickly but
> then if the capacity cap is not somewhere close to this value (as expected)
> it will probe very conservatively around this value and it takes a rather
> long time until it gets back into a phase where it increases its congestion
> window fast.
>

Then which congestion control algorithm fares better there? I thought Cubic
was suited for large fat pipes -- bigger bandwidth-delay pipes.

>
> So what do you mean by 'I immediately saw that the TCP performance went
> down‘. How long do you run your test? And over which period of time this
> the performance go down?
>

I did a simple test.

I connect two machines on my 1GB LAN.I do a tcp data transfer test and i
get around ~900Mbps. I then added a delay using qdisc on my outgoing
ethernet interface to simulate a longer RTT. So, if my ping earlier was
around 0.255ms, it then became 120ms. I repeat the test and i see that i
can only send about 200Mbps. I had to use multiple TCP flows to fill up my
1GB link with Cubic. I run this experiment for about 2 minutes, and Cubic
is not able to catch up. I dont have long standing flows, and only need to
send bursts of data, hence did not leave it running for very long.

I was under the impression, after having read the original Cubic paper that
Cubic's window growth rate is independent of RTT and hence wasnt expecting
any difference in the throughput.

Toms.

> > "The main feature of CUBIC is that its window growth function is defined
> in real-time so that its growth will be independent of RTT. Our work was
> want to
> make
> classic

-- 
Toms.

---

From: Gavin McCullagh <[email-masked]>
Date: Sat, 23 May 2015 06:36:51 +0100

Hi Tom,

What did you use to create the tcp flows?

Was it iperf or similar?

Gavin
On 23 May 2015 02:16, "Tom Sanders" <[email-masked]> wrote:

> Hi Mirja,
>
>>
>>
>> the text below only says that if you have two cubic flows with a
>> different base RTT competing at the same bottleneck they’ll still converge
>> to share the link equally. This does not mean that if you change the RTT
>> strongly during one connection that Cubic is able to catch up fast.
>>
> Then which congestion control algorithm fares better there? I thought
> Cubic was suited for large fat pipes -- bigger bandwidth-delay pipes.
>
>
>>
> I was under the impression, after having read the original Cubic paper
> that Cubic's window growth rate is independent of RTT and hence wasnt
> expecting any difference in the throughput.
>
> Toms.
>
>>
>> Mirja
>>
>>
>> > "The main feature of CUBIC is that its window growth function is
>> defined in real-time so that its growth will be independent of RTT. Our
>> work was partially inspired by HTCP [5], whose window growth function is
>> also based on real time. The congestion epoch period of CUBIC is determined
>> by the packet loss rate alone. As TCP’s throughput is defined by the packet
>> loss rate as well as RTT, the throughput of CUBIC is defined by only the
>> packet loss rate. Thus, when the loss rate is high and/or RTT is short,
>> CUBIC can operate in a TCP mode. Moreover, since the growth function is
>> independent of RTT, its RTT fairness is guaranteed as different RTT flows
>> will still grow their windows at the same rate."
>> >
>> to be
>> TCP

---

From: Tom Sanders <[email-masked]>
Date: Sat, 23 May 2015 14:23:05 +0530

You can use iperf with parallel streams and the result is the same. In my
case it was my own client/server pushing data over the TCP socket.

I have a theory that the following happens:

When i artificially induce the delay there are some packet losses since the
outgoing ethernet interface has to buffer the packets (to simulate the link
delay). Its probably of these packet losses that the throughput that i get
with one TCP stream is much lower than the available bandwidth. When i
increase the number of TCP streams, for reasons well known, i get better
bandwidth.

I think, and correct me if i am wrong here, if i were to induce latency
WITHOUT packet losses then even one TCP stream would be able to fill up the
network pipe (assuming that the kernel buffers are not the limiting issue).

Thanks, Toms

On 23 May 2015 at 11:06, Gavin McCullagh <[email-masked]> wrote:

>>> http://www4.ncsu.edu/~rhee/export/bitcp/cubic-paper.pdf) by Rhee and
>>> Xu, they explicitly state that the Cubic is independent of RTT.
>>> >

-- 
Toms.

---

From: Rick Jones <[email-masked]>
Date: Sat, 23 May 2015 08:36:12 -0700

On May 23, 2015, at 1:53 AM, Tom Sanders <[email-masked]> wrote:

> You can use iperf with parallel streams and the result is the same. In my case it was my own client/server pushing data over the TCP socket.
> 
> When i artificially induce the delay there are some packet losses since the outgoing ethernet interface has to buffer the packets (to simulate the link delay). Its probably of these packet losses that the throughput that i get with one TCP stream is much lower than the available bandwidth. When i increase the number of TCP streams, for reasons well known, i get better bandwidth.
> 
> I think, and correct me if i am wrong here, if i were to induce latency WITHOUT packet losses then even one TCP stream would be able to fill up the network pipe (assuming that the kernel buffers are not the limiting issue).

Not to be overly pedantic, but have you verified the packet losses - either through a packet trace on (I would think) the receiver or by looking at the netstat statistics on the sender before and after your run(s)?  Also, have you validated that indeed there is enough “classic” TCP window to allow for 900 Mbit/s on a single stream at 120ms RTT?  My rough calculation based on:

Throughput <= WindowSize/RTT
900 Mbit/s <= WindowSize/0.120s
108 Mbit <= WindowSize

suggests a single stream with a 120 ms RTT would require a classic TCP window of 13,500,000 bytes or more even without any packet losses.  I don’t think I’ve seen a net.ipv4.tcp_[rw]mem which defaulted to more than 4-6MB (M == 1048576) on the Linux systems I’ve seen regardless of how much memory they had in them.

happy benchmarking,

rick jones

> Then which congestion control algorithm fares better there? I thought Cubic was suited for large fat pipes -- bigger bandwidth-delay pipes. 
>  
> 
> I connect two machines on my 1GB LAN.I do a tcp data transfer test and i get around ~900Mbps. I then added a delay using qdisc on my outgoing ethernet interface to simulate a longer RTT. So, if my ping earlier was around 0.255ms, it then became 120ms. I repeat the test and i see that i can only send about 200Mbps. I had to use multiple TCP flows to fill up my 1GB link with Cubic. I run this experiment for about 2 minutes, and Cubic is not able to catch up. I dont have long standing flows, and only need to send bursts of data, hence did not leave it running for very long.
> 
> I was under the impression, after having read the original Cubic paper that Cubic's window growth rate is independent of RTT and hence wasnt expecting any difference in the throughput.
> 
> Toms.
> 
> Mirja
> 
>