Subject: [tcpm] RFC3465:TCP ABC and SACK'ed data

From: Erik Hugne <[email-masked]>
Date: Wed, 21 May 2014 09:36:11 +0200

TCP ABC (still in the experimental track) defines a method to increase the cwnd
based on the bytes being acknowledged, instead of the number of acks that
arrive. I am assuming that SACK'ed (RFC2018) data should be included in this
calculation, but i could find no mention of this in RFC3465.
Could someone confirm this?

What i'd like to achieve with this is to solve an issue that follows after a
TCP path change where a large portion of the window have been lost. On the
new path, the data that was successfully passed to the receiver is sacked
and the rest is being retransmitted. This prevents the cwnd from growing and the
throughput is very low.

Am i correct in that a SACK-aware ABC implementation would allow the cwnd
to grow and the "hole in the window" to be closed faster?

//E

---

From: "Zimmermann, Alexander" <[email-masked]>
Date: Wed, 21 May 2014 08:57:09 +0000

Hi,

> Could someone confirm this?

No, SACKed data is not included. RFC 5681 says that you increment the CWND for
each ACK received that *cumulatively* acknowledges new data. The same applies
for ABC.

SACKed data „counts“ as DUPACKs. C&P from RFC 6675: „For the purposes of this
specification, we define a „duplicate acknowledgment“ as a segment that arrives
carrying a SACK block that identifies previously unacknowledged and un-SACKed
octets… „

> and the rest is being retransmitted. This prevents the cwnd from growing and the

Why? For each received SACK block you will later get a cumulative ACK (as soon as
the retransmission arrives) which includes the SACK block and which increases the
CWND. In other words: as soon as a full ACK arrives (an ACK which covers the
RecoveryPoint), the CWND will increased on the basis of all newly cumulatively
acknowledged data (if ABC is on)

Alex

> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: Mark Allman <[email-masked]>
Date: Wed, 21 May 2014 07:46:28 -0400

> TCP ABC (still in the experimental track) defines a method to increase
> the cwnd based on the bytes being acknowledged, instead of the number
> of acks that arrive. I am assuming that SACK'ed (RFC2018) data should
> be included in this calculation, but i could find no mention of this
> in RFC3465.  Could someone confirm this?

ABC does not include SACK information in its calculations and it cannot
include SACK information.  ABC applies to slow start---which ends when
loss occurs.  And, (essentially) SACKs only arrive during loss recovery.
So, there is no need for ABC to deal with SACKs.

allman

---

From: Erik Hugne <[email-masked]>
Date: Wed, 21 May 2014 14:55:18 +0200

> So, there is no need for ABC to deal with SACKs.

Thanks for taking the time to explain this.

The problem (this is related to Ingemars posts around "Problem with low
SSthresh") is that following the path change, there is a large train of
segments being lost (say 150k out of a 500k window). Data at the end of the 
window did make it through and is SACK'ed. Upon receiving these SACK's the
sending side enters loss recovery. It will take a very long time before
recovering the 100k lost segments.

An aggressive recovery algorithm would mitigate the effects of this, but i was
hoping for a more elegant solution.

//E

---

From: Mark Allman <[email-masked]>
Date: Wed, 21 May 2014 09:02:20 -0400

> The problem (this is related to Ingemars posts around "Problem with
> low SSthresh") is that following the path change, there is a large
> train of segments being lost (say 150k out of a 500k window). Data at
> the end of the window did make it through and is SACK'ed. Upon
> receiving these SACK's the sending side enters loss recovery. It will
> take a very long time before recovering the 100k lost segments.
> 
> An aggressive recovery algorithm would mitigate the effects of this,
> but i was hoping for a more elegant solution.

You may be able to define some solution to this problem, but out of the
box TCP is just going to see a loss event and going to use whatever
usual loss recovery procedure it uses in this case.  ABC is just a riff
on slow start.  And, not slow start that follows an RTO because the
cumulative ACKs are not to be believed in that case (plus, delayed ACKs
should not be in play anyway, making byte counting less needed).

allman

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Thu, 22 May 2014 06:47:56 +0000

Are you perhaps observing an effect of the ACK clock loss, when you refer to a huge number of consecutive lost segments?

Standard SACK does suffer from an issue that during loss recovery, there may be a gap (SACKs don't clock out new/retransmitted packets while flightsize > cwnd (which is now 1/2); only during the 2nd half, SACK starts to clock out packets, but at the same rate as before (which may not that bright an idea if a path change with differnet characteristics is now there).

RateHalving (Linux has some kind of variant of this) / Proportional Rate Reduction RFC6937 may be the thing that you are after?

ABC really only comes into play during slowstart, to make connections which use delayed ACKs not perform much less aggressive than sessions where the receiver features QuickAck (Linux non-standard).

Best regards,

Richard Scheffenegger

> -----Original Message-----
> From: tcpm [mailto:[email-masked]] On Behalf Of Erik Hugne
> Sent: Mittwoch, 21. Mai 2014 14:55
> To: Mark Allman
> Cc: [email-masked]; [email-masked];
> [email-masked]
> Subject: Re: [tcpm] RFC3465:TCP ABC and SACK'ed data
> 
> > > TCP ABC (still in the experimental track) defines a method to
> > > increase the cwnd based on the bytes being acknowledged, instead of
> > > the number of acks that arrive. I am assuming that SACK'ed (RFC2018)
> > > data should be included in this calculation, but i could find no
> > > mention of this in RFC3465.  Could someone confirm this?
> >
> > ABC does not include SACK information in its calculations and it
> > cannot include SACK information.  ABC applies to slow start---which
> > ends when loss occurs.  And, (essentially) SACKs only arrive during loss
> recovery.
> segments being lost (say 150k out of a 500k window). Data at the end of
> the window did make it through and is SACK'ed. Upon receiving these SACK's
> the sending side enters loss recovery. It will take a very long time
> before recovering the 100k lost segments.
> 
> An aggressive recovery algorithm would mitigate the effects of this, but i
> was hoping for a more elegant solution.
> 
> //E
>

---

From: Erik Hugne <[email-masked]>
Date: Fri, 23 May 2014 15:32:51 +0200

Maybe..
This is a standard Linux 3.0 kernel with sack and frto enabled.
Additional lab experiments have shown that following the path change, a stream
of lost packets occurs (almost a full window, ~400kB). The receiver sends
dupacks on the new path to trigger fast retransmit, but then the retransmission
timer on the sender side fires and all unacked data will be resent. At the RTO,
we get the expected cwnd drop. While the receiver keeps kicking the the sender
by sending dupacks, the retransmission is reasonably fast but after a few ms it
grinds down to a stop-and-go behaviour.

I failed in my efforts to find some explanation to this in the RFC's.
What i did find was in VJ's paper from '88.

-When starting or restarting after a loss, set cwnd to one packet.
-On each ack for new data, increase cwnd by one packet

If retransmitted data are not considered "new", that would explain the behavior
i guess.

Scripted ss -ti for the connection every 3 sec.
In this test it wasn't even a path change, we just downed the interface on a
router in the middle for some 300ms to simulate the packetloss (This was done at 15:06:18)

Fri May 23 15:06:01 CEST 2014
	reno wscale:9,7 rto:1892 rtt:1678/7 ato:40 cwnd:2215 send 15.3Mbps rcv_space:14480
Fri May 23 15:06:04 CEST 2014
	reno wscale:9,7 rto:1888 rtt:1678/8 ato:40 cwnd:2215 send 15.3Mbps rcv_space:14480
Fri May 23 15:06:07 CEST 2014
	reno wscale:9,7 rto:1888 rtt:1676/9 ato:40 cwnd:2215 send 15.3Mbps rcv_space:14480
Fri May 23 15:06:10 CEST 2014
	reno wscale:9,7 rto:1884 rtt:1675.5/5 ato:40 cwnd:2215 send 15.3Mbps rcv_space:14480
Fri May 23 15:06:14 CEST 2014
	reno wscale:9,7 rto:1816 rtt:1607.5/22 ato:40 cwnd:2215 send 16.0Mbps rcv_space:14480
Fri May 23 15:06:17 CEST 2014
	reno wscale:9,7 rto:1892 rtt:1681.5/7 ato:40 cwnd:2215 send 15.3Mbps rcv_space:14480
Fri May 23 15:06:20 CEST 2014
	reno wscale:9,7 rto:1548 rtt:651/225 ato:40 cwnd:3 ssthresh:1107 send 53.4Kbps rcv_space:14480
Fri May 23 15:06:23 CEST 2014
	reno wscale:9,7 rto:1224 rtt:263/110 ato:40 cwnd:3 ssthresh:1107 send 132.1Kbps rcv_space:14480
Fri May 23 15:06:26 CEST 2014
	reno wscale:9,7 rto:1172 rtt:210.5/18 ato:40 cwnd:3 ssthresh:1107 send 165.1Kbps rcv_space:14480
Fri May 23 15:06:29 CEST 2014
	reno wscale:9,7 rto:1164 rtt:203.5/6 ato:40 cwnd:3 ssthresh:1107 send 170.8Kbps rcv_space:14480
Fri May 23 15:06:32 CEST 2014
	reno wscale:9,7 rto:1164 rtt:203.5/3 ato:40 cwnd:3 ssthresh:1107 send 170.8Kbps rcv_space:14480
Fri May 23 15:06:35 CEST 2014
	reno wscale:9,7 rto:1164 rtt:203.5/3 ato:40 cwnd:6 ssthresh:1107 send 341.5Kbps rcv_space:14480
Fri May 23 15:06:38 CEST 2014
	reno wscale:9,7 rto:1164 rtt:203.5/4 ato:40 cwnd:6 ssthresh:1107 send 341.5Kbps rcv_space:14480
Fri May 23 15:06:41 CEST 2014
	reno wscale:9,7 rto:1168 rtt:204/4 ato:40 cwnd:6 ssthresh:1107 send 340.7Kbps rcv_space:14480
Fri May 23 15:06:44 CEST 2014
	reno wscale:9,7 rto:1164 rtt:203.5/3 ato:40 cwnd:6 ssthresh:1107 send 341.5Kbps rcv_space:14480
Fri May 23 15:06:47 CEST 2014
	reno wscale:9,7 rto:1164 rtt:203.5/3 ato:40 cwnd:6 ssthresh:1107 send 341.5Kbps rcv_space:14480
Fri May 23 15:06:50 CEST 2014
	reno wscale:9,7 rto:1164 rtt:203.5/3 ato:40 cwnd:6 ssthresh:1107 send 341.5Kbps rcv_space:14480
Fri May 23 15:06:53 CEST 2014
	reno wscale:9,7 rto:1164 rtt:203.5/3 ato:40 cwnd:6 ssthresh:1107 send 341.5Kbps rcv_space:14480
Fri May 23 15:06:56 CEST 2014
	reno wscale:9,7 rto:1164 rtt:203.5/3 ato:40 cwnd:6 ssthresh:1107 send 341.5Kbps rcv_space:14480
Fri May 23 15:06:59 CEST 2014
	reno wscale:9,7 rto:1164 rtt:203.5/4 ato:40 cwnd:6 ssthresh:1107 send 341.5Kbps rcv_space:14480
Fri May 23 15:07:02 CEST 2014
	reno wscale:9,7 rto:1164 rtt:203.5/3 ato:40 cwnd:9 ssthresh:1107 send 512.3Kbps rcv_space:14480
Fri May 23 15:07:05 CEST 2014
	reno wscale:9,7 rto:1164 rtt:203.5/3 ato:40 cwnd:9 ssthresh:1107 send 512.3Kbps rcv_space:14480
Fri May 23 15:07:08 CEST 2014
	reno wscale:9,7 rto:1164 rtt:203.5/3 ato:40 cwnd:9 ssthresh:1107 send 512.3Kbps rcv_space:14480
Fri May 23 15:07:11 CEST 2014
	reno wscale:9,7 rto:1260 rtt:487.5/8 ato:40 cwnd:960 ssthresh:1107 send 22.8Mbps rcv_space:14480
Fri May 23 15:07:14 CEST 2014
	reno wscale:9,7 rto:1244 rtt:863.5/6 ato:40 cwnd:1111 ssthresh:1107 send 14.9Mbps rcv_space:14480
Fri May 23 15:07:17 CEST 2014
	reno wscale:9,7 rto:1148 rtt:864/6 ato:40 cwnd:1114 ssthresh:1107 send 14.9Mbps rcv_space:14480
Fri May 23 15:07:20 CEST 2014
	reno wscale:9,7 rto:1096 rtt:868/8 ato:40 cwnd:1118 ssthresh:1107 send 14.9Mbps rcv_space:14480

---

From: Neal Cardwell <[email-masked]>
Date: Fri, 23 May 2014 13:37:19 -0400

> grinds down to a stop-and-go behaviour.

It sounds like the big problems start with the RTO. In theory, if the
sender is getting enough SACKs then it ought to be able to repair all
those lost packets in one or a handful of RTTs, depending on just how
many were lost, and how much the RTT jumps upward in the switch to the
new path.

Can you retry your experiments with Linux 3.13 or newer? Recent Linux
kernels (3.13 and beyond) should do better in this regard: Yuchung
Cheng (cc-ed) has made some nice changes that take RTT measurements
from SACKs of unretransmitted data, and then keep updating the RTO
timer accordingly.

Regarding your scenario - what are the RTTs of the old and new paths?

cheers,
neal

---

From: Yoshifumi Nishida <[email-masked]>
Date: Fri, 23 May 2014 16:06:12 -0700

This is just out of my curiosity, but can we share the tcpdump file for
this?
--
Yoshi

On Friday, May 23, 2014, Erik Hugne
<[email-masked]<javascript:_e(%7B%7D,'cvml',[email-masked]');>>
wrote:

>         reno wscale:9,7 rto:1244 rtt:863.5/6 ato:40 cwnd:1111
> ssthresh:1107 send 14.9Mbps rcv_space:14480
> > Are you perhaps observing an effect of the ACK clock loss, when you
> refer to a huge number of consecutive lost segments?
> >
> > Standard SACK does suffer from an issue that during loss recovery, there
> may be a gap (SACKs don't clock out new/retransmitted packets while
> flightsize > cwnd (which is now 1/2); only during the 2nd half, SACK starts
> to clock out packets, but at the same rate as before (which may not that
> bright an idea if a path change with differnet characteristics is now
> there).
> >
> > RateHalving (Linux has some kind of variant of this) / Proportional Rate
> Reduction RFC6937 may be the thing that you are after?
> >
> >
> > ABC really only comes into play during slowstart, to make connections
> which use delayed ACKs not perform much less aggressive than sessions where
> the receiver features QuickAck (Linux non-standard).
> >
> > Best regards,
> >
> >
> > Richard Scheffenegger
> >
> >

---

From: Erik Hugne <[email-masked]>
Date: Mon, 26 May 2014 11:12:32 +0200

Here's from a test just this morning, wgetting a 10MB file.
https://www.dropbox.com/sh/86u3nzcwukjzpzv/AAAlp_Me7wXf7PRyuWFq9Lmwa

//E

---

From: Erik Hugne <[email-masked]>
Date: Mon, 26 May 2014 12:32:20 +0200

> Regarding your scenario - what are the RTTs of the old and new paths?

In the synthetic experiment below, there is no path change, we just "insert"
a transient loss and the RTT are the same before and after. I'll see if i
can bump the kernel version to 3.13 and retry. 

//E

---

From: Yuchung Cheng <[email-masked]>
Date: Wed, 28 May 2014 20:31:01 -0700

> https://www.dropbox.com/sh/86u3nzcwukjzpzv/AAAlp_Me7wXf7PRyuWFq9Lmwa

The problem is neither ABC nor SACK. It's bad interaction of
DSACK/undo, F-RTO, and rate-halving in the old kernel. These features
have gone through major changes since 3.0 (F-RTO re-implementation,
PRR, dozens fixes of TCP undo) so I'd recommend trying 3.12+ kernel as
well.

00:00:00.000026 IP s > c: Flags [.], seq 83359754:83368442, ack 145,
win 122, options [nop,nop,TS val 1294920005 ecr 1943986835], length
8688

// Timeout & retransmit the head
00:00:01.806873 IP s > c: Flags [.], seq 80269362:80270810, ack 145,
win 122, options [nop,nop,TS val 1294920482 ecr 1943986836], length
1448

// a DSACK indicating the timeout is spurious, however cwnd is not
reverted and F-RTO is activated to send a new data packet
00:00:00.305013 IP c > s: Flags [.], ack 80270810, win 6132, options
[nop,nop,TS val 1943987388 ecr 1294920482,nop,nop,sack 1
{80269362:80270810}], length 0
00:00:00.000070 IP s > c: Flags [.], seq 83368442:83372786, ack 145,
win 122, options [nop,nop,TS val 1294920558 ecr 1943987388], length
4344

// On receiving the SACK for the F-RTO probe, we got "stuck" in the
stop and go, possibly a bug in rate-halving/loss-recovery/f-rto?
00:00:00.200690 IP c > s: Flags [.], ack 80270810, win 6132, options
[nop,nop,TS val 1943987438 ecr 1294920482,nop,nop,sack 1
{83368442:83372786}], length 0
00:00:00.000053 IP s > c: Flags [.], seq 80270810:80272258, ack 145,
win 122, options [nop,nop,TS val 1294920608 ecr 1943987438], length
1448
00:00:00.000009 IP s > c: Flags [.], seq 80272258:80273706, ack 145,
win 122, options [nop,nop,TS val 1294920608 ecr 1943987438], length
1448
00:00:00.000003 IP s > c: Flags [.], seq 80273706:80275154, ack 145,
win 122, options [nop,nop,TS val 1294920608 ecr 1943987438], length
1448
00:00:00.200629 IP c > s: Flags [.], ack 80275154, win 6126, options
[nop,nop,TS val 1943987488 ecr 1294920608,nop,nop,sack 1
{83368442:83372786}], length 0
00:00:00.000033 IP s > c: Flags [.], seq 80275154:80276602, ack 145,
win 122, options [nop,nop,TS val 1294920658 ecr 1943987488], length
1448
00:00:00.000007 IP s > c: Flags [.], seq 80276602:80278050, ack 145,
win 122, options [nop,nop,TS val 1294920658 ecr 1943987488], length
1448
00:00:00.000004 IP s > c: Flags [.], seq 80278050:80279498, ack 145,
win 122, options [nop,nop,TS val 1294920658 ecr 1943987488], length
1448