Subject: Re: [tcpm] Why is Cubic said to be RTT-fair?

From: Mirja Kühlewind <[email-masked]>
Date: Mon, 1 Jun 2015 18:15:03 +0200

Hi Tom,

if your connection is 900Mbit/s and you have an RTT of 120ms, it’s likely that our sent and or received buffer is too small as mention by Rick.

However, the scenario you describe, where you suddenly increase your delay strongly and therefore the algorithm has to increase its congestion window, in your case, from 19 packets to 9000 packets, has nothing to do with the fact that window growth is independent of RTT. 

What is meant in the Cubic paper is that to flows competing with a different base RTT on the small link will still be able to share the link capacity equally. That’s a completely different scenario.

Increase the congestion window from 19 to 9000 packets in Congestion Avoidance if for the most algorithms challenging. Cubic does better than Reno but is still not optimal.

I’ve presented an own algorithm a few IETF meeting ago, where we introduce a Fast Increase phase to address this problem:

https://www.ietf.org/proceedings/91/slides/slides-91-iccrg-5.pdf

I personally think one could also apply this Fast Increase phase to other existing algorithms with only slightly changing them.

Mirja

> Am 23.05.2015 um 03:16 schrieb Tom Sanders <[email-masked]>:
> 
> Hi Mirja,
> 
> 
> the text below only says that if you have two cubic flows with a different base RTT competing at the same bottleneck they’ll still converge to share the link equally. This does not mean that if you change the RTT strongly during one connection that Cubic is able to catch up fast.
> 
> In fact this is one weakness of Cubic that I have observed. If there are change in base delay/rtt or strong increases in the available capacity, Cubic needs a rather long time to catch up. This is because cubic has a ‚target value‘ to which it increases it sending rate/window quickly but then if the capacity cap is not somewhere close to this value (as expected) it will probe very conservatively around this value and it takes a rather long time until it gets back into a phase where it increases its congestion window fast.
> 
> Then which congestion control algorithm fares better there? I thought Cubic was suited for large fat pipes -- bigger bandwidth-delay pipes. 
>  
> 
> So what do you mean by 'I immediately saw that the TCP performance went down‘. How long do you run your test? And over which period of time this the performance go down?
> 
> I did a simple test.
> 
> I connect two machines on my 1GB LAN.I do a tcp data transfer test and i get around ~900Mbps. I then added a delay using qdisc on my outgoing ethernet interface to simulate a longer RTT. So, if my ping earlier was around 0.255ms, it then became 120ms. I repeat the test and i see that i can only send about 200Mbps. I had to use multiple TCP flows to fill up my 1GB link with Cubic. I run this experiment for about 2 minutes, and Cubic is not able to catch up. I dont have long standing flows, and only need to send bursts of data, hence did not leave it running for very long.
> 
> I was under the impression, after having read the original Cubic paper that Cubic's window growth rate is independent of RTT and hence wasnt expecting any difference in the throughput.
> 
> Toms.
> 
> Mirja
> 
> 
> > Am 21.05.2015 um 02:51 schrieb Tom Sanders <[email-masked]>:
> >
> > Hi Neal,
> >
> > This is odd.
> >
> > In the original Cubic paper (http://www4.ncsu.edu/~rhee/export/bitcp/cubic-paper.pdf) by Rhee and Xu, they explicitly state that the Cubic is independent of RTT.
> >
> > "The main feature of CUBIC is that its window growth function is defined in real-time so that its growth will be independent of RTT. Our work was partially inspired by HTCP [5], whose window growth function is also based on real time. The congestion epoch period of CUBIC is determined by the packet loss rate alone. As TCP’s throughput is defined by the packet loss rate as well as RTT, the throughput of CUBIC is defined by only the packet loss rate. Thus, when the loss rate is high and/or RTT is short, CUBIC can operate in a TCP mode. Moreover, since the growth function is independent of RTT, its RTT fairness is guaranteed as different RTT flows will still grow their windows at the same rate."
> >
> > Am i missing something here?
> >
> > Thanks, Toms
> >
> > On 20 May 2015 at 23:13, Neal Cardwell <[email-masked]> wrote:
> > On Wed, May 20, 2015 at 1:18 PM, Tom Sanders <[email-masked]> wrote:
> > > Thanks Rick, this was very helpful.
> > >
> > > So if there are no setsockopts, etc then CUBIC's performance, unlike other
> > > TCP variants, should not get impacted by RTTs, right? Is that a fair
> > > statement to make?
> >
> > CUBIC, like most TCP congestion control variants, has significant
> > aspects that evolve on the scale of RTTs: for example, in slow start,
> > loss recovery, or when the CUBIC function is very steep, CUBIC's
> > behavior is heavily dependent on RTT (send some packets, wait for an
> > ACK, send some more packets,...). So CUBIC will not be
> > perfectly"RTT-fair.
> >
> > I think it might be fair to say something like: once a path is
> > saturated with CUBIC flows, as long as loss recovery or send/receive
> > buffers are not a bottleneck, then competition between CUBIC flows
> > should be more RTT-fair than Reno, in the long run, since for large
> > parts of the life cycle of the CUBIC flow the evolution of cwnd is
> > constrained by wall clock time rather than ACK arrival.
> >
> > neal
> >
> >
> > > Toms
> > >
> > > On 20 May 2015 at 19:51, Rick Jones <[email-masked]> wrote:
> > >>
> > >>
> > >> On May 20, 2015, at 6:20 AM, Tom Sanders <[email-masked]> wrote:
> > >>
> > >> > Hi,
> > >> >
> > >> > As per my understanding, unlike other TCP variants, CUBIC claims to be
> > >> > RTT fair. This is because the window growth rate is independent of the RTT,
> > >> > and hence the performance is the same as on a low and a high RTT paths.
> > >> > Others are dependent upon RTT since their windows grow as they receive ACKs
> > >> > from their peers, while CUBIC doesnt (window growth is a function of time).
> > >> > This is theory.
> > >> >
> > >> > In practice, CUBIC (which is the default now on most linux
> > >> > distributions) also depends upon RTT. I connected to machines on a LAN and
> > >> > did a FTP transfer. I got a certain bandwidth. Next, i artificially inserted
> > >> > delay using "tc qdisc" in the path. I increased the delay on the ethernet
> > >> > interface connecting my linux machine to the LAN to be around 100ms. I
> > >> > immediately saw that the TCP performance went down.
> > >> >
> > >> > To bring it up to the same level as before i had to use multiple TCP
> > >> > streams. To me this means that CUBIC performance is dependent of the RTT. So
> > >> > why do we call it RTT-fair?
> > >>
> > >> I cannot speak to CUBIC specifically, but I would think you would want to
> > >> make sure you were seeing effects of congestion window and not of the
> > >> classic TCP window.  For example, if the FTP client/server you were using
> > >> makes an explicit setsockopt() to set the socket buffer sizes and by
> > >> extension the classic TCP window, your bumping of the RTT to around 100ms
> > >> may have taken the bandwidth down as a consequence of the classic:
> > >>
> > >> Throughput <= WindowSize/RTT
> > >>
> > >> Similarly, since you mention Linux, if the FTP client/server didn’t make
> > >> explicit setsockopt() calls,  the sysctl values for tcp_rmem and tcp_wmem
> > >> may have been such that the auto tuning of the window size couldn’t allow
> > >> the classic TCP window to grow “enough.”
> > >>
> > >> I suspect that the experiment needs to be setup to have the same classic
> > >> window size in both cases, sized per the above to be sufficient to achieve
> > >> the peak throughput in the high RTT case.  Then you will have eliminated
> > >> classic window as a factor.
> > >>
> > >> Also, depending on the version of the Linux kernel you are using, there
> > >> may be some effects from tcp small queues or perhaps (a stretch?) even byte
> > >> queue limits which may not be playing well with netem.  (Just guessing there
> > >> really)
> > >>
> > >> happy benchmarking,
> > >>
> > >> rick jones
> > >
> > >
> > >
> > >
> > > --
> > > Toms.
> > >
> > > _______________________________________________
> > > tcpm mailing list
> > > [email-masked]
> > > https://www.ietf.org/mailman/listinfo/tcpm
> > >
> >
> >
> >
> > --
> > Toms.
> > _______________________________________________
> > tcpm mailing list
> > [email-masked]
> > https://www.ietf.org/mailman/listinfo/tcpm
> 
> 
> 
> 
> -- 
> Toms.