Subject: [tcpm] A question about PTO/RTO rescheduling in RACK draft

From: Yi Huang <[email-masked]>
Date: Wed, 17 Apr 2019 01:48:03 +0000

Hi,

I have a question about PTO/RTO rescheduling logic in RACK draft. The draft states, in Section 6.5.1, "If there was a previously scheduled PTO or RTO pending, then that pending PTO or RTO should first be cancelled, and then the new PTO should be scheduled". So if an app keeps posting data in an interval less than PTO and RTO (say one byte each time and Cwnd is large enough) and no acks are coming back due to network failures or a receiver that just does not send anything back, the app will never fires RTO since the timer is pushed out each time we send this one byte data. Is this a problem? What is the intention of pushing out the timer for each send? I think in this case the sender should disconnect the connection because, without TLP, the sender would have had several RTOs and closed the connection.

Also, RFC 6298 Section 5 (5.1) states "Every time a packet containing data is sent (including a retransmission), if the timer is not running, start it running so that it will expire after RTO seconds (for the current value of RTO).", which says TCP should schedule timer only when it is not schedule yet and this kind of conflicts with (or is overridden by) what is in RACK/TLP draft.

Thanks,

Yi Huang

---

From: hiren panchasara <[email-masked]>
Date: Wed, 17 Apr 2019 11:13:44 -0700

> Hi,

Hi!
I don't see anything inherently wrong in how this works or I am missing
something.
> 
> I have a question about PTO/RTO rescheduling logic in RACK draft. The draft states, in Section 6.5.1, "If there was a previously scheduled PTO or RTO pending, then that pending PTO or RTO should first be cancelled, and then the new PTO should be scheduled". So if an app keeps posting data in an interval less than PTO and RTO (say one byte each time and Cwnd is large enough) and no acks are coming back due to network failures or a receiver that just does not send anything back, the app will never fires RTO since the timer is pushed out each time we send this one byte data. Is this a problem?
By growing CWND, TCP has decided that it is okay to send that much
without needing an ACK.
I think something along the lines of detecting that connection is
app-limited OR doing newcwv like validation may help to keep CWND in
check?

> What is the intention of pushing out the timer for each send?
6.5.1 (Step 1) indicated a few conditions for scheduling PTO (i.e.
pushing out the timer) so its not blindly doing for each send. But it
would do that for the case you described as app *is* handing new data to
be sent to TCP.

>I think in this case the sender should disconnect the connection because, without TLP, the sender would have had several RTOs and closed the connection.
I am probably not understanding something. I think you are saying TLP is
causing RTO to be pushed out unnecessarily.

One cannot schedule PTO when a TLP is out. So, on first timeout i.e. PTO
as we prefer that over RTO, you send a TLP out. And if you don't get an
ACK back for that, next timeout would be an RTO as 6.5.1.  Phase 1
condition 1 suggests.

But if there is new data to be sent, RTO would also gets pushed out,
no?
> Also, RFC 6298 Section 5 (5.1) states "Every time a packet containing data is sent (including a retransmission), if the timer is not running, start it running so that it will expire after RTO secon…

I am not sure I follow. For the connection, you need a timeout
mechanism. And RACK draft suggests keeping PTO aligned with RTO as the
former is less expensive. Try PTO and if it doesn't work, do RTO.

I believe the wordings around "first cancel the pending one and then
only schedule new one" are for clarity around implementation details.
Gist is to not keep 2 sets of timers running for the same timeout
mechanism. 

Someone with more clue on the list would probably correct me. :-)
Cheers,
Hiren

---

From: hiren panchasara <[email-masked]>
Date: Wed, 17 Apr 2019 12:22:02 -0700

On 04/17/19 at 06:52P, Matt Olson wrote:
> Previously, you'd start the RTO timer on the first packet, and the timer would run until an ACK was received. Now, the timer is reset on each new packet. So the concern is that there exist cases where previously TCP would reduce the CWnd and retransmit, but not anymore.

I see your point. I guess in this particular case, RACK (which is the
smallest/first timer to fire) is also not useful as there are no acks
coming back.

> 
> If all such cases are app-limited cases, then I like Hiren's point about this being a "keep CWnd in check while app-limited" problem. 

iirc, there was some guidance around this in previous version(s) of the
draft. I don't find that in -04. May be authors (CC'd) can chime in.

Cheers,
Hiren
> 
> -----Original Message-----
> From: hiren panchasara <[email-masked]> 
> Sent: Wednesday, April 17, 2019 11:14 AM
> To: Yi Huang <huanyi=[email-masked]>
> Cc: [email-masked]; Matt Olson <[email-masked]>
> Subject: Re: [tcpm] A question about PTO/RTO rescheduling in RACK draft
> 
> I don't see anything inherently wrong in how this works or I am missing something.
> > 
> By growing CWND, TCP has decided that it is okay to send that much without needing an ACK.
> I think something along the lines of detecting that connection is app-limited OR doing newcwv like validation may help to keep CWND in check?
> 
> pushing out the timer) so its not blindly doing for each send. But it would do that for the case you described as app *is* handing new data to be sent to TCP.
> 
> I am probably not understanding something. I think you are saying TLP is causing RTO to be pushed out unnecessarily.
> 
> One cannot schedule PTO when a TLP is out. So, on first timeout i.e. PTO as we prefer that over RTO, you send a TLP out. And if you don't get an ACK back for that, next timeout would be an RTO as 6.5.1.  Phase 1 condition 1 suggests.
> 
> But if there is new data to be sent, RTO would also gets pushed out, no?
> > 
> I am not sure I follow. For the connection, you need a timeout mechanism. And RACK draft suggests keeping PTO aligned with RTO as the former is less expensive. Try PTO and if it doesn't work, do RTO.
> 
> I believe the wordings around "first cancel the pending one and then only schedule new one" are for clarity around implementation details.
> Gist is to not keep 2 sets of timers running for the same timeout mechanism. 
> 
> Someone with more clue on the list would probably correct me. :-) Cheers, Hiren

---

From: Neal Cardwell <[email-masked]>
Date: Wed, 17 Apr 2019 16:29:40 -0400

Hi Yi and Hiren,

Thank you for raising this question and the continued discussion.

I think there are a few concrete mechanisms that kick into play in the kind
of scenario Yi lays out.

(1) The TLP is scheduled at the min of the "native" TLP time and the
"native" RTO time, as laid out in section 6.5.1
<https://tools.ietf.org/html/draft-ietf-tcpm-rack-04#section-6.5.1>.:

   TLP_timeout():
  ...

       If Now() + PTO > TCP_RTO_expire():
           PTO = TCP_RTO_expire() - Now()

   This keeps the TLP (PTO) from being pushed past the time at which an RTO
previously would have fired.

(2) After the TLP probe is sent, an RTO timer is scheduled and fires. The
intent is to not get stuck in cycles of repeated TLPs.

To illustrate more concretely, below is a (passing) packetdrill script
illustrating/documenting the Linux TCP behavior in this kind of scenario,
showing the points at which a loss probe and timeout happen.

We will need to update the draft to make these aspects more clear. Thank
you for raising this!

neal

------------------
// Test for TLP with an app that makes periodic small writes.
`nstat > /dev/null`

// Establish a connection.
    0 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
   +0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
   +0 bind(3, ..., ...) = 0
   +0 listen(3, 1) = 0

   +0 < S 0:0(0) win 32792 <mss 1000,sackOK,nop,nop,nop,wscale 7>
   +0 > S. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 8>
  +.1 < . 1:1(0) ack 1 win 257
   +0 accept(3, ..., ...) = 4

// Small write of 2*MSS.
   +0 write(4, ..., 2000) = 2000
   +0 > P. 1:2001(2000) ack 1
   +0 %{ assert tcpi_ca_state == TCP_CA_Open }%
// TLP is scheduled 200ms later, but app writes again before that.

// Small write of 2*MSS, which pushes back TLP timer.
+.150 write(4, ..., 2000) = 2000
   +0 > P. 2001:4001(2000) ack 1
   +0 %{ assert tcpi_ca_state == TCP_CA_Open }%

// Loss probe is a retransmission, scheduled at a time
// calculated by the RTO formula, 300ms after the
// first packet we sent.
+.150 > P. 3001:4001(1000) ack 1
   +0 %{ assert tcpi_ca_state == TCP_CA_Open }%
   +0 `nstat -a -z | grep LossProbes`

// Small write of 2*MSS.
+.150 write(4, ..., 2000) = 2000
   +0 > P. 4001:6001(2000) ack 1
   +0 %{ assert tcpi_ca_state == TCP_CA_Open }%

// RTO timer fires.
+.210 > . 1:1001(1000) ack 1
   +0 %{ assert tcpi_ca_state == TCP_CA_Loss }%
   +0 `nstat -a -z | grep TCPTimeouts`

---

From: hiren panchasara <[email-masked]>
Date: Wed, 17 Apr 2019 14:15:21 -0700

> Thank you for raising this question and the continued discussion.

Thanks for a quick response as always. :)

[skip]
> We will need to update the draft to make these aspects more clear. Thank

I'll keep this to the point I thought got missed.

https://tools.ietf.org/rfcdiff?url2=draft-ietf-tcpm-rack-04.txt

Is where we can see `Scheduling a loss probe` getting changed.
Specifically, following condition has been removed from -04:

   3.  The connection is either limited by congestion window (the data
       in flight matches or exceeds the cwnd) or application-limited
       (there is no unsent data that the receiver window allows to be
       sent).

And afaict, Linux code still has this condition in. If you can provide
some rationale behind this change, it'd be great.

Thanks a ton,
Hiren

---

From: Yi Huang <[email-masked]>
Date: Wed, 17 Apr 2019 22:09:03 +0000

Hi Neal,

Thanks for your quick response and also thanks Hiren for the help. It’s very clear to me now.

Yi

From: tcpm <[email-masked]> On Behalf Of Neal Cardwell
Sent: Wednesday, April 17, 2019 1:30 PM
To: hiren panchasara <[email-masked]>
Cc: [email-masked]; Matt Olson <[email-masked]>; [email-masked]; Priyaranjan Jha <[email-masked]>; Yi Huang <huanyi=[email-masked]>
Subject: Re: [tcpm] A question about PTO/RTO rescheduling in RACK draft

Hi Yi and Hiren,

I think there are a few concrete mechanisms that kick into play in the kind of scenario Yi lays out.

(1) The TLP is scheduled at the min of the "native" TLP time and the "native" RTO time, as laid out in section 6.5.1<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-ietf-tcpm-rack-04%23section-6.5.1&data=02%7C01%7Chuanyi%40microsoft.com%7Cd36a239568784d5233d908d6c3737cb2%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636911298117752987&sdata=qlPq%2FDtmZYp4zumBJZPDq%2FNX3vFumJB3qzT4ZzHYmiM%3D&reserved=0>.:

   TLP_timeout():

  ...

   This keeps the TLP (PTO) from being pushed past the time at which an RTO previously would have fired.

(2) After the TLP probe is sent, an RTO timer is scheduled and fires. The intent is to not get stuck in cycles of repeated TLPs.

To illustrate more concretely, below is a (passing) packetdrill script illustrating/documenting the Linux TCP behavior in this kind of scenario, showing the points at which a loss probe and timeout happen.

We will need to update the draft to make these aspects more clear. Thank you for raising this!

neal

------------------

// Establish a connection..
    0 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
   +0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
   +0 bind(3, ..., ...) = 0
   +0 listen(3, 1) = 0

// Small write of 2*MSS.

// Small write of 2*MSS.

// RTO timer fires.

---

From: Neal Cardwell <[email-masked]>
Date: Thu, 18 Apr 2019 09:18:57 -0400

> some rationale behind this change, it'd be great.

We removed the code for that condition from Linux around 2017-12-13,
in this commit (GPLv2 patch in this link):
  https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/commit/net/ipv4/tcp_output.c?id=b4f70c3d4ec32a2ff4c62e1e2da0da5f55fe12bd

The commit description explains the rationale for removing that
condition, but here is a paraphrased description of the rationale for
removing that code and that condition in the draft:

---

Disallowing TLP when there is unused cwnd had the primary effect of
disallowing TLP when there is TSO deferral, Nagle deferral, or we hit
the receiver window limit. Why? Because basically every application
write() or incoming ACK will cause us to run the TCP transmit loop to
see if we can send more. And then if we sent something we then see if
we should schedule a TLP. At that point, there are a few common
reasons why some cwnd budget  could still be unused:

    (a) receiver window limits
    (b) Nagle deferral
    (c) TSO deferral (deferring with the hope of sending a bigger TSO
offload burst later)
    (d) intra-send-host flow control (in Linux, TCP small queues, aka TSQ)

For (d), after the next packet tx completion the TSQ mechanism will
allow us to send more packets, so we don't really need a TLP (in
practice it shouldn't matter whether we schedule one or not). But for
(a), (b), or (c) the sender won't send any more packets until it
receives another ACK. But if the whole flight of data was lost, or all
the ACKs for the flight were lost, then the sender won't get any more
ACKs, and so in this case ideally we should schedule and send a TLP to
get more feedback.

---

best,
neal

---

From: Matt Olson <[email-masked]>
Date: Wed, 17 Apr 2019 18:52:14 +0000

-----Original Message-----
From: hiren panchasara <[email-masked]> 
Sent: Wednesday, April 17, 2019 11:14 AM
To: Yi Huang <huanyi=[email-masked]>
Cc: [email-masked]; Matt Olson <[email-masked]>
Subject: Re: [tcpm] A question about PTO/RTO rescheduling in RACK draft

Hi!

---

From: hiren panchasara <[email-masked]>
Date: Thu, 18 Apr 2019 10:53:17 -0700

[skip]

Thanks a lot. This makes sense.
Apologies for misreading the code.

Cheers,
Hiren