Subject: [tcpm] TCP duplex bandwidth

From: Umesh Deshpande <[email-masked]>
Date: Sat, 15 Sep 2012 22:50:06 -0400

Hi,

I'm testing TCP duplex bandwidth with the following scenario.

Consider 3 machines A, B, C.
One sender thread on A is sending data to B, multiple sender threads (2) on
B are sending data to C simultaneously.
I observed that under this scenario, (B receiving data while sending data
to C), B couldn't receive data with 1Gbps from A. However, addition of
bandwidths of individual threads from B -> C was close to 1Gbps. I was
expecting that B should be able to receive and send with 1Gbps
simultaneously. (1Gbps RX, and 1Gbps TX).

A -> B : 503 Mbps
B -> C : 490 Mbps x 2 threads = 880Mbps

If I increase A->B thread from 1 to 2, each thread can send with more
bandwidth. Also, I didn't observe this problem when A->B sender was UDP,
i.e. rate of UDP remained closed to 1Gbps.

Setup Used
All machines were running kernel 3.1.0-030100-generic of Ubuntu.
Each machine has 8 CPUs.
Broadcom BCM5721 Ethernet card.
Above test was verified with a custom program and netperf.

Sorry if this is not a relevant mailing list.

Thanks
Umesh

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Sun, 16 Sep 2012 17:30:30 +0200

The analysis of such performance issues is not trivial. And not always (in fact, seldom) the root cause is TCP. For instance, in a scenario like this, there can be issues with the NIC drivers or the Ethernet switch (e. g., Ethernet flow control).

Given that this list deals with TCP only, it would make sense to exclude all non-TCP issues first.

Reverse-engineering TCP's behavior in such a setup would really benefit from further information, e. g., the packet losses and RTT seen by each connection. Linux makes some useful information available in the /proc system and by tcp_info.

Michael

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of Umesh Deshpande
> Sent: Sunday, September 16, 2012 4:50 AM
> To: [email-masked]
> Subject: [tcpm] TCP duplex bandwidth
> 
> Hi,
> 
> One sender thread on A is sending data to B, multiple sender 
> threads (2) on B are sending data to C simultaneously.
> I observed that under this scenario, (B receiving data while 
> sending data to C), B couldn't receive data with 1Gbps from 
> A. However, addition of bandwidths of individual threads from 
> B -> C was close to 1Gbps. I was expecting that B should be 
> able to receive and send with 1Gbps simultaneously. (1Gbps 
> RX, and 1Gbps TX).
> 
> A -> B : 503 Mbps
> If I increase A->B thread from 1 to 2, each thread can send 
> with more bandwidth. Also, I didn't observe this problem when 
> A->B sender was UDP, i.e. rate of UDP remained closed to 1Gbps.
> 
> Setup Used

---

From: rick jones <[email-masked]>
Date: Sun, 16 Sep 2012 10:12:44 -0700

On Sep 15, 2012, at 7:50 PM, Umesh Deshpande wrote:

> One sender thread on A is sending data to B, multiple sender threads (2) on B are sending data to C simultaneously.
> I observed that under this scenario, (B receiving data while sending data to C), B couldn't receive data with 1Gbps from A. However, addition of bandwidths of individual threads from B -> C was close to 1Gbps. I was expecting that B should be able to receive and send with 1Gbps simultaneously. (1Gbps RX, and 1Gbps TX).
> 
> A -> B : 503 Mbps
> If I increase A->B thread from 1 to 2, each thread can send with more bandwidth. Also, I didn't observe this problem when A->B sender was UDP, i.e. rate of UDP remained closed to 1Gbps.
> 
> Setup Used

If indeed tcpm is not an appropriate list (not sure I can say really), since netperf was involved, you might consider [email-masked].  There you would probably be asked clarifying questions such as:

*) If B sending to C required two streams, what led you to believe that A sending to be would require only one?
*) What were your exact netperf command lines?
*) What were the utilizations of each of the CPUs on each system during the test?
*) What is the latency between A and B?
*) Others that I cannot think of just now :)

Alternatively, since a linux kernel is involved, it may be appropriate to ask in something like the netdev mailing list.  Though your use of what to them would be considered an "old" kernel might lead to their first asking you to use a newer one.

happy benchmarking,

rick jones

---

From: Matt Mathis <[email-masked]>
Date: Sun, 16 Sep 2012 12:59:46 -0700

You may have rediscovered a long know problem that is a fairly direct
consequence of TCP's self clock, but made worse by some combination
of: channel acquisition algorithm that have hysteresis (current sender
is most likely to be the next sender) plus the lack of AQM and
explicit scheduling (how does fq_codel play here?), and probably other
effects.

See:  Zhang, L., Shenker, S., and D. D. Clark, "Observations and
             Dynamics of a Congestion Control Algorithm: The Effects of
             Two-Way Traffic", Proc. ACM SIGCOMM, ACM Computer
             Communications Review (CCR), Vol 21, No 4, 1991, pp.133-
             147.

I point out that device driver polling (aka interrupt coalescing)
causes channel acquisition hysteresis, and that all of the link
effects discussed in the above paper might be happening within the
operating system itself.

I would start with a forward literature search from the above paper.

It would be extremely useful to see if/how fq_codel might change this
story.  At least in the Linux world there have been enough recent
changes in this space where you must be working at the HEAD to have
relevant results.  Otherwise you are at risk of doing an extensive
study of an already obsolete OS.

Thanks,
--MM--
The best way to predict the future is to create it.  - Alan Kay

On Sun, Sep 16, 2012 at 10:12 AM, rick jones <[email-masked]> wrote:
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: Jim Gettys <[email-masked]>
Date: Sun, 16 Sep 2012 16:56:51 -0400

> study of an already obsolete OS.

My memory of some data Dave Taht took two months ago was that fq_codel
did spectacularly well in this case; both directions were fully
saturated; without fq_codel, performance was asymmetric and one
direction lost 10-20% of bandwidth or more.

Eric Dumazet's TCP small queues work by itself did almost as well at
bandwidth utilisation, but left 17ms of latency, which I then complained
about to Dave, triggering his post of my complaint to G+, about it being
way too much of a decent VOIP latency budget, which 17ms is...). 

Best of all, was both in combination, saturating the NIC in both
directions while preserving low latency for other traffic and keeping
TCP buffer sizes more sane in sized.

In any case, any data taken on any version of Linux is seriously out of
date, given all the work which has been done to fix bufferbloat over the
last 18 months: e.g. fq_codel, TCP small queues, and BQL.  For any data
in this area to be relevant, you'd better be running Linus' kernel.  TSQ
I think is in the latest bits, IIRC.

                            - Jim