Subject: [tcpm] draft-ietf-tcpm-rtorestart-04

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Sun, 2 Nov 2014 23:33:32 +0000

Hi,

I've read -04 and I have a couple of questions and suggestions:

* Section 4: The "total number of outstanding and previously unsent segments" in sum of two variables, right? If so, why not use the term "sum"?

* Section 4: I also wonder if the text should define "previously unsent segments". The term "previously unsent data" is very well-known in the RFC series. A quick search revealed that "previously unsent segments" is usually only used to refer to situations when segments with this property have indeed been transmitted. Is my understanding correct that draft-ietf-tcpm-rtorestart-04 uses this term differently? I think it here refers to data that will be sent *in future*. In that case, I wonder if it is really clear how the sender can determine the number of "previously unsent segments". For instance, that number may depend on the segmentation strategy of the sender, in particular for applications that use many small write() calls. In this case a TCP stack may e.g. decide to send partial segments. I am not sure if the actual number of segments created from the "unsent data" is really known in advance always. Or do I miss something?

* Section 5.1: "Given RTOR's ability to only work when it is beneficial for the loss recovery process, it is suitable as a system-wide default mechanism for TCP traffic." I think other text in the draft asks for further experimentation regarding the actual trade-off between benefit and risk. Given that, I think another wording should be used (e.g., "Given that RTOR is a mostly conservative algorithm", ...). Given that this is not a PS document, I'd also prefer a more careful phrasing regarding the recommendation as system-wide default (e.g., "it is suitable for experimentation as system-wide default").

* Section 5.2: As mentioned in the last face-to-face meeting, spurious timeout negatively affect the network by needlessly sending data. This negative effect is not mentioned in this section. Why?

* Section 5.2: Mobile networks have a variable RTT, including e.g. longer delays during handovers (either without or with transient loss during the handover). A lot of past work on spurious timeout recovery has been motivated by these delay spikes. Reducing the RTO almost certainly has an impact in such environments; I think performance can both increase or decrease. If no data on that is available, mobile networks seem to me an area where more experimentation would be particularly useful, right?

* Section 5.2: Is there any data on the risk of spurious timeouts in networks where RTT and the minimum RTO are of the same order of magnitude? For instance, in satellite networks the RTT can be huge.

* Section 5.2: "However, with respect to RTOR spurious timeouts are only a problem for applications transmitting multiple bursts of data within a single flow." I think this section should be reworded and it should more explicitly discuss the impact of RTOR on HTTP/1.1 and HTTP/2.0 traffic, including e.g. adaptive video streaming. They all transmit "multiple bursts of data within a single flow". To me, the current sentence could imply that RTOR would be a "problem" for a vast majority of Internet traffic. A more explicit discussion on how RTOR affects HTTP/1.1 and HTTP/2.0 would IMHO make a lot of sense.

* Section 5.3: Given that not all stacks are segment-based, this section seems really relevant. However, the description how to determine outstanding segments is vague ("it is possible to exactly determine..."). I think this could be reworded to provide more guidance to implementers. The same applies to the calculation of "previously unsent segments"; for instance, the text does not explain what happens if the "number of bytes in the send queue" is not a multiple of the SMSS (see also above regarding the definition of this variable).

* Section 6: "In contrast, RTOR is trying to make the RTO more appropriate in cases where there is no need to be overly cautious." I think this sentence should use a more neutral language. I think whether an RTO value is "appropriate" is impossible to know (in advance), and whether a more aggressive timeout is "cautious" or not may depend on the network under consideration. For instance, an alternative would be to compare the complexity of RTOR and TLP.

* Section 9: I wonder if an attacker could try to send certain ACK patterns to increase the risk of timeouts e.g. at a Web Server, leveraging the smaller RTO value. If successful, the RTOs could significantly reduce application performance. Probably, such an attacker could cause much more harm by other means, i.e., this may not be a significant security risk. But the current security analysis is very short.

Thanks

Michael

> -----Original Message-----
> From: tcpm [mailto:[email-masked]] On Behalf Of Per Hurtig
> Sent: Tuesday, October 28, 2014 12:18 AM
> To: [email-masked]
> Subject: Re: [tcpm] I-D Action: draft-ietf-tcpm-rtorestart-04.txt
> 
> Hi all,
> 
> we have now updated the draft to address the outstanding comments. The
> main changes from last version are:
> 
>    o  Changed the algorithm to allow RTOR when there is unsent data
> available, but the cwnd does not allow transmission.
>    o  Changed the algorithm to not trigger if "RTO - T_earliest" <= 0,
> to avoid that ACKs to previous retransmissions trigger premature
> timeouts.
>    o  Made minor adjustments throughout the document to adjust for the
> algorithmic changes.
>    o  Improved the wording throughout the document.
> 
> 
> for experimental results and a Linux implementation, please visit:
> http://riteproject.eu/resources/rto-restart/
> 
> 
> 
> Cheers,
> Per
> 
> On 2014-10-28 00:11, [email-masked] wrote:
> >
> > A New Internet-Draft is available from the on-line Internet-Drafts
> directories.
> >   This draft is a work item of the TCP Maintenance and Minor
> Extensions Working Group of the IETF.
> >
> >          Title           : TCP and SCTP RTO Restart
> >          Authors         : Per Hurtig
> >                            Anna Brunstrom
> >                            Andreas Petlund
> >                            Michael Welzl
> > 	Filename        : draft-ietf-tcpm-rtorestart-04.txt
> > 	Pages           : 13
> > 	Date            : 2014-10-27
> >
> > Abstract:
> >     This document describes a modified algorithm for managing the TCP
> and
> >     SCTP retransmission timers that provides faster loss recovery
> when
> >     there is a small amount of outstanding data for a connection.
> The
> >     modification, RTO Restart (RTOR), allows the transport to restart
> its
> >     retransmission timer more aggressively in situations where fast
> >     retransmit cannot be used.  This enables faster loss detection
> and
> >     recovery for connections that are short-lived or application-
> limited.
> >
> >
> > The IETF datatracker status page for this draft is:
> > https://datatracker.ietf.org/doc/draft-ietf-tcpm-rtorestart/
> >
> > There's also a htmlized version available at:
> > http://tools.ietf.org/html/draft-ietf-tcpm-rtorestart-04
> >
> > A diff from the previous version is available at:
> > http://www.ietf.org/rfcdiff?url2=draft-ietf-tcpm-rtorestart-04
> >
> >
> > Please note that it may take a couple of minutes from the time of
> submission
> > until the htmlized version and diff are available at tools.ietf.org.
> >
> > Internet-Drafts are also available by anonymous FTP at:
> > ftp://ftp.ietf.org/internet-drafts/
> >
> > _______________________________________________
> > tcpm mailing list
> > [email-masked]
> > https://www.ietf.org/mailman/listinfo/tcpm
> >
> 
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

---

From: Anna Brunstrom <[email-masked]>
Date: Sun, 09 Nov 2014 13:17:53 +0100

Hi Michael,

Thanks for your comments!  Replies inline below.

> * Section 4: The "total number of outstanding and previously unsent segments" in sum of two variables, right? If so, why not use the term "sum"?

Correct, so "sum" is better. We will update this.

> * Section 4: I also wonder if the text should define "previously unsent segments". The term "previously unsent data" is very well-known in the RFC series. A quick search revealed that "previously u…

You are correct that "previously unsent segments" refers to data that 
will be sent in the future. This is the same meaning as in RFC 3517 and 
RFC 4653. A definition of this term could be added to Section 2, but I 
am not sure if it is needed? To better understand the possible conflict 
in terminology, could you point to some of RFCs that use the term to 
refer to already transmitted segments? Intuitively that seems a bit 
strange to me.

As discussed in section 5.3, the number of unsent segments can be 
estimated from the amount of unsent data and the SMSS. Note that we are 
only talking about data that is already available in the send queue, and 
later write calls can of course create additional segments. I guess a 
TCP stack that has one SMSS of data in its send queue is free to send 
this as multiple segments if it wants to, but I assume that would not be 
the normal behavior. As the estimate is not very critical it is not 
really a problem if this happens occasionally.

The addition of "previously unsent segments" is there to capture a 
corner case (see issue raised by Alex in 
http://www.ietf.org/mail-archive/web/tcpm/current/msg08780.html), so it 
will only be occasionally used. If we then get an incorrect estimate, 
say a sender with one SMSS of data would instead send this as two SMSS, 
and the algorithm triggers "incorrectly", this would be equivalent to 
using a RTOR threshold of rrthresh + 1 so nothing critical happens. (In 
some cases rrthresh + 1 may even be better.)

> * Section 5.1: "Given RTOR's ability to only work when it is beneficial for the loss recovery process, it is suitable as a system-wide default mechanism for TCP traffic." I think other text in the…

Agreed, we will update the text.

> * Section 5.2: As mentioned in the last face-to-face meeting, spurious timeout negatively affect the network by needlessly sending data. This negative effect is not mentioned in this section. Why?

The initial intention was to focus on aspects of spurious timeouts that 
were specific to RTOR. But looking at the text I see that this is not at 
all clear in the current version. The best resolution is probably to not 
try to scope it that way. We can add some text also about the increased 
network load in the next version.

> * Section 5.2: Mobile networks have a variable RTT, including e.g. longer delays during handovers (either without or with transient loss during the handover). A lot of past work on spurious timeout…

Yes, as the RTTs in mobile networks can be higher and are often variable 
it is an interesting area. It could be mentioned as an example in 5.2.

> * Section 5.2: Is there any data on the risk of spurious timeouts in networks where RTT and the minimum RTO are of the same order of magnitude? For instance, in satellite networks the RTT can be hu…

Our Linux experiments over emulated networks cover RTTs that go beyond 
the value of the minimum RTO, i.e. RTTs larger than 200 ms, see 
http://riteproject.eu/resources/rto-restart/ for some sample results. I 
am not aware of any real experiments in satellite networks or similar. A 
general problem when experimenting with this is that spurious timeouts 
happen only occasionally so a lot of data is needed before you can 
determine any trends.

> * Section 5.2: "However, with respect to RTOR spurious timeouts are only a problem for applications transmitting multiple bursts of data within a single flow." I think this section should be reword…

Ok, we will work on the wording and use HTTP as an example. And also try 
to better highlight the tradeoffs involved. In our experiments with 
HTTP/1.1 traffic RTOR performs better than the baseline.

> * Section 5.3: Given that not all stacks are segment-based, this section seems really relevant. However, the description how to determine outstanding segments is vague ("it is possible to exactly d…

OK, we can reuse the full discussion on how to determine outstanding 
segments from RFC 5827 rather than summarizing and pointing to it. We 
can of course also clarify the number of segments to be estimated when 
the "number of bytes in the send queue" is not a multiple of the SMSS. I 
somehow thought this was obvious.

> * Section 6: "In contrast, RTOR is trying to make the RTO more appropriate in cases where there is no need to be overly cautious." I think this sentence should use a more neutral language. I think…

Ok. We will look over the wording in this part.

> * Section 9: I wonder if an attacker could try to send certain ACK patterns to increase the risk of timeouts e.g. at a Web Server, leveraging the smaller RTO value. If successful, the RTOs could si…

The section is very short, but this is because we do not really see any 
new security issues.

  If I understand your example above correctly, the receiver is trying 
to somehow reduce performance for its own application by manipulating 
the ACK pattern. But if you are trying to mess up things for your own 
application you can deliver incorrect data, throw away any data received 
or do whatever you want so I do not think trying to possibly slow down 
the sender is the thing to worry about in this attack scenario. If 
someone finds a relevant security issue it should of course be 
documented, but I do not think it makes sense to put in something like 
the above just to make the section longer.

Thanks again for your comments!

Cheers,
Anna

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Mon, 10 Nov 2014 15:59:55 +0000

Hi Anna,

Some follow-up comments. I omit the parts for which there seems to be agreement on the need for a document update.

> > * Section 4: I also wonder if the text should define "previously
> unsent segments". The term "previously unsent data" is very well-known
> in the RFC series. A quick search revealed that "previously unsent
> segments" is usually only used to refer to situations when segments
> with this property have indeed been transmitted. Is my understanding
> correct that draft-ietf-tcpm-rtorestart-04 uses this term differently?
> I think it here refers to data that will be sent *in future*. In that
> case, I wonder if it is really clear how the sender can determine the
> number of "previously unsent segments". For instance, that number may
> depend on the segmentation strategy of the sender, in particular for
> applications that use many small write() calls. In this case a TCP
> stack may e.g. decide to send partial segments. I am not sure if the
> actual number of segments created from the "unsent data" is really
> known in advance always. Or do I miss something?
> 

I didn't say that there is a conflict in terminology. I might be wrong, but to me draft-ietf-tcpm-rtorestart-04 defines the term "previously unsent segments" as a *new* TCP state variable.

The value of that variable may be obvious for certain architectures of a TCP stack, in particular if segmentation into packets occurs on socket write() calls, i.e., before data is actually sent to network. As far as I recall, this is how the Linux stack works internally. But I'd like to avoid that an RFC describes an algorithm that can only be implemented for certain architecture of a TCP stack.

RFC 3517 and RFC 4653 either refer to the "amount of previously unsent data" (which doesn't need to be further defined IMHO), or the "transmission of previously unsent segments" (which is not what draft-ietf-tcpm-rtorestart-04 refers to). For instance, here is the exact wording of the closest matches I could find in RFC 3517 and RFC 4563:

RFC 3517 Section 5 Processing and Acting Upon SACK Information

     "(2) If no sequence number 'S2' per rule (1) exists but there
          exists available unsent data and the receiver's advertised
          window allows, the sequence range of one segment of up to SMSS
          octets of previously unsent data starting with sequence number
          HighData+1 MUST be returned."

   => "previously unsent data", NOT segments

RFC 3517 Section 5 Algorithm Details

  "Note: The first and second
   duplicate ACKs can also be used to trigger the transmission of
   previously unsent segments using the Limited Transmit algorithm
   [RFC3042]."

   => "transmission of previously unsent segments", NOT the number of segments queued

RFC 4653 Section 2 NCR Description

  "The first Extended Limited Transmit variant, Careful Limited
   Transmit, calls for the transmission of one previously unsent
   segment, in response to duplicate acknowledgments, for every two
   segments that are known to have left the network."

   => "transmission of previously unsent segments", NOT the number of segments queued

RFC 4653 Section 3.2 Terminating Extended Limited Transmit and Preventing Bursts

  "(T.3) A TCP is now permitted to transmit previously unsent data as
         allowed by cwnd, FlightSize, application data availability, and
         the receiver's advertised window."

   => "previously unsent data", NOT segments

RFC 4653 Section 3.3. Extended Limited Transmit

  "(E.2) If the comparison in equation (1), below, holds and there are
         SMSS bytes of previously unsent data available for
         transmission, then the sender MUST transmit one segment of SMSS
         bytes."

   => "previously unsent data", NOT segments

Unless there is a definition of "number of previously unsent segments", I think this variable has to be defined and it has to be explained how it is calculated.
 
> really a problem if this happens occasionally.

If Nagle's algorithm is disabled, sending out multiple segments would be the normal behavior in this case.

I could imagine that disabling Nagle's algorithm could be pretty common among thin stream applications. This is why the draft should IMHO discuss if disabling Nagle's algorithm affects the method.

> some cases rrthresh + 1 may even be better.)

If a wrong parameter for the estimate is possible but not critical, this should be explained in the document.

In general, the cited email from Alex stated:

"* Sec 3: IMO the algo/doc is too much Linux driven. I would like to see a
  segment-based *and* byte-based version of the algo, like RFC 5827."

I fully agree to this statement and I think it has not been completely addressed in -04 so far.
 
> > * Section 9: I wonder if an attacker could try to send certain ACK
> patterns to increase the risk of timeouts e.g. at a Web Server,
> leveraging the smaller RTO value. If successful, the RTOs could
> significantly reduce application performance. Probably, such an
> attacker could cause much more harm by other means, i.e., this may not
> be a significant security risk. But the current security analysis is
> very short.
> 

I was thinking e.g. about an off-path attacker that tries to inject ACKs into the TCP connection to trigger RTOs to harm application performance. I am not arguing that this is really a relevant attack scenario, but it seems not impossible to try that.

Michael