Subject: [tcpm] Bad GRO / TCP FACK feature interaction

From: "Bless, Roland (TM)" <[email-masked]>
Date: Wed, 12 Aug 2015 13:48:59 +0200

Hi,

[not sure that this actually belongs to this WG, but I think
it might be interesting input...]

while investigating TCP streams at 10Gbit/s we saw several
retransmissions without actual packet loss happen. The
analysis showed that it was caused by the GRO (Generic Receive
Offload) feature together with the FACK optimization
(under Linux 4.1.x).
The receiving side produced two larger out-of-order segments
(of sizes 10k and 20k respectively), i.e., they were swapped
(not on the wire, but locally).
The FACK algorithm then determines that more than three
segments have been reordered and (unnecessarily) starts
retransmission of several segments.
[Thanks to Polina Goltsman for finding this].
Turning SACK, DSACK and FACK off eliminates
such retransmissions completely.

This is IMHO another example of a bad feature
interaction between "smart" local optimizations and
protocols. Implementations get more complicated, because
they have to distinguish between what's probably on the
wire and what is locally optimized (if possible at all).

Now the question: does anyone know the reason for the
local re-ordering at the receiving side? [Presumably,
it is caused by internal re-hashing of TCP flows
to input queues, but that's a rough guess only.]

If the implementers thought, re-ordering is no problem,
because TCP will deal with it - this is a little bit
short-sighted as shown above. So either the GRO implementation
needs to be fixed to avoid re-ordered delivery or the TCP
implementation needs to be made more GRO-aware.

Regards,
 Roland

---

From: Yuchung Cheng <[email-masked]>
Date: Wed, 12 Aug 2015 13:16:34 -0700

> such retransmissions completely.
In your case, the spurious retransmission/recovery would be detected
by TCP timestamps. Subsequently Linux would 1) revert cwnd reduction,
2) raise the dupthresh and 3) disables FACK. Since cwnd is recovered
the connection won't lose tput except wasting some spurious
retransmits.

> implementation needs to be made more GRO-aware.
I am skeptical GRO is the culprit of reordering. GRO does not reorder
packets b/c it's single lane processing. It may be other input pathing
problems not GRO.

>
> Regards,
>  Roland
>
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: Eric Dumazet <[email-masked]>
Date: Wed, 12 Aug 2015 14:55:45 -0700

> packets b/c it's single lane processing. It may be other input pathing

Indeed, I am not aware of GRO being so badly broken.

We had some bugs in the past, but not of this magnitude.

More details are needed :
- Model of NIC
- RPS/RFS settings if any
- Interrupt affinities

My suspicion is that you use some kind of aRFS NIC (flow director), and
some ACK packets reprogram the NIC to receive packets on a different RX
queue.

---

From: "Bless, Roland (TM)" <[email-masked]>
Date: Thu, 13 Aug 2015 16:57:52 +0200

Hi Yuchung,

thanks for you quick response.

> the connection won't lose tput except wasting some spurious

According to our pcap file, the reordered chunks are 2*10^-6 (2µs) apart
from each other. The TCP timestamps are exactly the same in this
case, so no chance to detect reordering by using timestamps.

> packets b/c it's single lane processing. It may be other input pathing

Maybe it's happening earlier or later in the local delivery path.
We are still trying to find out the cause, currently, we are
seeing the symptoms only.

Regards,
 Roland

---

From: "Bless, Roland (TM)" <[email-masked]>
Date: Thu, 13 Aug 2015 17:19:09 +0200

Hi Eric,

> Indeed, I am not aware of GRO being so badly broken.

Maybe it's not caused by GRO and our first guess was wrong.
We are still investingating.

> - RPS/RFS settings if any

See below...

> some ACK packets reprogram the NIC to receive packets on a different RX

Here is what Polina says (she's not on the tcpm list):
----------------------------------------------------------------------
Our computers: each is equipped with Xeon-E5 (32cores) and four-port
Intel x710 NICs (probably  X710-DA4 but I am not sure) and are running
Linux 4.1.2. We test it with iperf3 (v3.1b3) - single TCP flow with
data going only in one direction and no additional parameters.

Receive Flow Steering is off:

$ cat /proc/sys/net/core/rps_sock_flow_entries
0

Receive Packet Steering is off for all queues:

$cat /sys/class/net/enp2s0f0/queues/rx-*/rps_cpus
00000000
...
00000000

As far as I can tell from /proc/interrupts and
/proc/irq/<IRQ-NUM>/smp_affinity each Rx queue is assigned to one core
(TxRx queue N (i40e-eth1-TxRx-N) is assigned to core N).

Currently, I can't figure out how to read flow director rules in Linux.

This is what I observe when I run iperf3 experiments:

At the receiver:
1. somewhere one a scale of several seconds the queue that receives
packets changes (when I check stats with ethtool -S, counters of
different queue rx-<N>.rx_packets are incremented). At about the same
time different CPU core processes packets (gets loaded with system or
softirq process).

2. another stat that gets incremented is: port.fdir_atr_match

At the sender:
I observe similar queue switching behavior at the sender. The core
processing the packets and the queue occasionally change, it sometimes
happen every 20-30 seconds and sometimes after about 3 seconds.

Time difference in tcpdump between two reordered packets is 2-3
microseconds. Both packets have the same TCP timestamp.

Here are two more observations that seem strange to me:

At the sender: If I send 512 segments with netperf [netperf -H 10.0.0.1
-l 180 -D 1 -- -m 512 ], CPU core gets 100% utilization and the
queues/cores are not switched.

At the receiver: When sender is called with  netperf -H 10.0.0.1 -l 180
-D 1 -- -M 512, CPU core gets 100% utilization and the queues/cores are
not switched.

----------------------------------------
Regards,
 Roland

---

From: Rick Jones <[email-masked]>
Date: Thu, 13 Aug 2015 09:52:06 -0700

> Here is what Polina says (she's not on the tcpm list):

I don't know the Intel NIC product line well enough to know what driver 
that NIC uses, but if the x710 uses the ixgbe driver, LRO (Large Receive 
Offload - done in the NIC) may be enabled.  For that matter, it might be 
enabled by default in the driver used for the x710 even if it isn't ixgbe.

For other reasons, a few months ago now I submitted an ixgbe patch to 
the Intel folks which would not enable LRO by default and am waiting for 
it to emerge from their trees.

happy benchmarking,

rick jones

---

From: Yuchung Cheng <[email-masked]>
Date: Thu, 13 Aug 2015 10:48:32 -0700

> case, so no chance to detect reordering by using timestamps.
In that case the second level (slower) protection is DSACKs, triggered
by the spurious retransmission. When Linux sender realizes all its
retransmission are DSACK'd, it'll undo the cwnd reduction as well. If
the spurious retransmission is bothersome you can disable FACK or
raise default dupthresh. The new echo option draft can also help the
timestamp resolution limitation.

btw we are putting up a new recovery mechanism to deal exactly with
this "tiny reordering" issue. The core idea is to replace
counter-based heuristics (dupthresh or sequence delta like FACK) with
a time-based approach (time delta between pending and delivered
packet). We plan to submit a draft before next tcpm meeting along with
linux patches.

---

From: Roland Bless <[email-masked]>
Date: Fri, 14 Aug 2015 00:27:37 +0200

Hi Rick,

> enabled by default in the driver used for the x710 even if it isn't ixgbe.

LRO is turned off in our setting by default and cannot be turned on
(fixed), so this isn't causing trouble here.

> it to emerge from their trees.

LRO doesn't seem to be the problem. Moreover, GRO probably isn't causing
the *reordering* itself, but since it merges several packets together a
single duplicate ack, which was caused by reordering, also triggered the
retransmission. Without GRO a single DupACK wouldn't
have caused the retransmission.

Regards,
 Roland

---

From: Eric Dumazet <[email-masked]>
Date: Thu, 13 Aug 2015 15:33:35 -0700

> have caused the retransmission.

Sorry, I do not get it. GRO aggregates in order segments only.

So no retransmits could have been avoided.

( There is some work done here at Google to experiment a reorder
resilient GRO engine )

---

From: Eric Dumazet <[email-masked]>
Date: Thu, 13 Aug 2015 15:41:41 -0700

> ( There is some work done here at Google to experiment a reorder

BTW, bnx2x GRO is implemented partially on the NIC. It is possible to
get some reorders because of this.

Here is what can happen :

P1 delivered as a single MSS by bnx2x. -> Stored into GRO engine
P2 delivered as a single MSS -> GRO engine appends P2 to P1

P3-P6 delivered in a 4-MSS super packet
     GRO engine is bypassed. P3-P6 delivered to upper stack, while P1-P2
still are on queue.

Arguably this is a bnx2x bug. Following patch will deliver packets
in GRO engine _before_ this bad packet.

diff --git a/net/core/dev.c b/net/core/dev.c
index a8e4dd430285..fbfb007bad45 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -4151,9 +4151,10 @@ static enum gro_result dev_gro_receive(struct napi_struct *napi, struct sk_buff
 	if (!(skb->dev->features & NETIF_F_GRO))
 		goto normal;
 
-	if (skb_is_gso(skb) || skb_has_frag_list(skb) || skb->csum_bad)
+	if (skb_is_gso(skb) || skb_has_frag_list(skb) || skb->csum_bad) {
+		napi_gro_flush(napi, false);
 		goto normal;
-
+	}
 	gro_list_prepare(napi, skb);
 
 	rcu_read_lock();

---

From: Roland Bless <[email-masked]>
Date: Fri, 14 Aug 2015 01:15:53 +0200

Hi Eric,

> Sorry, I do not get it. GRO aggregates in order segments only.

I'm sorry for the confusion, I'll try to provide more details below...
Of course you are right that GRO aggregates in order segments only.

> So no retransmits could have been avoided.

No, as the subject said, it's a bad feature interaction with FACK.
I've included an excerpt from the dump @ the receiver side:
1) we get ~10kBytes as a block of segments out of order (packet #11)
and then ~20kBytes as a block of segments (packet # 12):

10 0.000135000 10.0.2.3 10.0.2.1 TCP 66 5201→37056 [ACK] Seq=1
Ack=191137 Win=24576 Len=0 TSval=44830005 TSecr=62231400

11 0.000152000 10.0.2.1 10.0.2.3 TCP 10202 [TCP Previous segment not
captured] 37056→5201 [ACK] Seq=212857 Ack=1 Win=8192 Len=10136
TSval=62231400 TSecr=44830005

12 0.000154000 10.0.2.1 10.0.2.3 TCP 21786 [TCP Out-Of-Order] 37056→5201
[ACK] Seq=191137 Ack=1 Win=8192 Len=21720 TSval=62231400 TSecr=44830005

13 0.000156000 10.0.2.3 10.0.2.1 TCP 78 [TCP Dup ACK 10#1] 5201→37056
[ACK] Seq=1 Ack=191137 Win=24576 Len=0 TSval=44830005 TSecr=62231400
SLE=212857 SRE=222993

2) Since packet #13 is now a DupACK with SACK option, FACK is triggering
  a retransmission at the sender, because it sees a hole of more than
  3 MSS segments.

Is it now getting clearer?

Regards,
 Roland