Subject: Re: [tcpm] draft-ietf-tcpm-rtorestart-04

From: Anna Brunstrom <[email-masked]>
Date: Mon, 01 Dec 2014 15:37:00 +0100

Hi Michael,

Sorry for my delayed response. Comments inline.

On 2014-11-10 16:59, Scharf, Michael (Michael) wrote:
> Hi Anna,
>
> Some follow-up comments. I omit the parts for which there seems to be agreement on the need for a document update.
>
>>> * Section 4: I also wonder if the text should define "previously
>> unsent segments". The term "previously unsent data" is very well-known
>> in the RFC series. A quick search revealed that "previously unsent
>> segments" is usually only used to refer to situations when segments
>> with this property have indeed been transmitted. Is my understanding
>> correct that draft-ietf-tcpm-rtorestart-04 uses this term differently?
>> I think it here refers to data that will be sent *in future*. In that
>> case, I wonder if it is really clear how the sender can determine the
>> number of "previously unsent segments". For instance, that number may
>> depend on the segmentation strategy of the sender, in particular for
>> applications that use many small write() calls. In this case a TCP
>> stack may e.g. decide to send partial segments. I am not sure if the
>> actual number of segments created from the "unsent data" is really
>> known in advance always. Or do I miss something?
>>
>> You are correct that "previously unsent segments" refers to data that
>> will be sent in the future. This is the same meaning as in RFC 3517 and
>> RFC 4653. A definition of this term could be added to Section 2, but I
>> am not sure if it is needed? To better understand the possible conflict
>> in terminology, could you point to some of RFCs that use the term to
>> refer to already transmitted segments? Intuitively that seems a bit
>> strange to me.
> I didn't say that there is a conflict in terminology. I might be wrong, but to me draft-ietf-tcpm-rtorestart-04 defines the term "previously unsent segments" as a *new* TCP state variable.

Ok, I understood your first mail as two separate issues: 1) that the 
term "previously unsent segment" had a different meaning from other 
RFCs, 2) it was not clear how to calculate the number of previously 
unsent segments. So my pointer to RFC 3517 and RFC 4653 was with respect 
to issue 1 above.  But if I understand correctly now, your question is 
mainly about the second issue of how one can estimate the number of 
previously unsent segments.

We can add "previously unsent segments" to the terminology section as a 
variable, with the discussion on how to estimate the number of 
previously unsent segments then still contained in section 5.3. Is this 
what you are looking for?

> The value of that variable may be obvious for certain architectures of a TCP stack, in particular if segmentation into packets occurs on socket write() calls, i.e., before data is actually sent to network. As far as I recall, this is how the Linux stack works internally. But I'd like to avoid that an RFC describes an algorithm that can only be implemented for certain architecture of a TCP stack.
>
> RFC 3517 and RFC 4653 either refer to the "amount of previously unsent data" (which doesn't need to be further defined IMHO), or the "transmission of previously unsent segments" (which is not what draft-ietf-tcpm-rtorestart-04 refers to). For instance, here is the exact wording of the closest matches I could find in RFC 3517 and RFC 4563:
>
> RFC 3517 Section 5 Processing and Acting Upon SACK Information
>
>       "(2) If no sequence number 'S2' per rule (1) exists but there
>            exists available unsent data and the receiver's advertised
>            window allows, the sequence range of one segment of up to SMSS
>            octets of previously unsent data starting with sequence number
>            HighData+1 MUST be returned."
>
>     => "previously unsent data", NOT segments
>
> RFC 3517 Section 5 Algorithm Details
>
>    "Note: The first and second
>     duplicate ACKs can also be used to trigger the transmission of
>     previously unsent segments using the Limited Transmit algorithm
>     [RFC3042]."
>
>     => "transmission of previously unsent segments", NOT the number of segments queued
>
> RFC 4653 Section 2 NCR Description
>
>    "The first Extended Limited Transmit variant, Careful Limited
>     Transmit, calls for the transmission of one previously unsent
>     segment, in response to duplicate acknowledgments, for every two
>     segments that are known to have left the network."
>
>     => "transmission of previously unsent segments", NOT the number of segments queued
>
> RFC 4653 Section 3.2 Terminating Extended Limited Transmit and Preventing Bursts
>
>    "(T.3) A TCP is now permitted to transmit previously unsent data as
>           allowed by cwnd, FlightSize, application data availability, and
>           the receiver's advertised window."
>
>     => "previously unsent data", NOT segments
>
> RFC 4653 Section 3.3. Extended Limited Transmit
>
>    "(E.2) If the comparison in equation (1), below, holds and there are
>           SMSS bytes of previously unsent data available for
>           transmission, then the sender MUST transmit one segment of SMSS
>           bytes."
>
>     => "previously unsent data", NOT segments
>
> Unless there is a definition of "number of previously unsent segments", I think this variable has to be defined and it has to be explained how it is calculated.
>   
>> As discussed in section 5.3, the number of unsent segments can be
>> estimated from the amount of unsent data and the SMSS. Note that we are
>> only talking about data that is already available in the send queue,
>> and
>> later write calls can of course create additional segments. I guess a
>> TCP stack that has one SMSS of data in its send queue is free to send
>> this as multiple segments if it wants to, but I assume that would not
>> be
>> the normal behavior. As the estimate is not very critical it is not
>> really a problem if this happens occasionally.
> If Nagle's algorithm is disabled, sending out multiple segments would be the normal behavior in this case.

Why would turning off Nagle's algorithm make the sender split up data 
ALREADY AVAILABLE in the send queue into multiple small-sized segments 
when it gets a chance to send? Note that the sender is blocked by the 
cwnd when the ACK arrives. While technically allowed I guess, I do not 
think this would be normal.

>
> I could imagine that disabling Nagle's algorithm could be pretty common among thin stream applications. This is why the draft should IMHO discuss if disabling Nagle's algorithm affects the method.

I do not think it affects it. We can of course say this. Do you think 
this would be good/needed? (There are of course a lot of things 
available in TCP/SCTP that does not affect RTO restart.)

>
>> The addition of "previously unsent segments" is there to capture a
>> corner case (see issue raised by Alex in
>> http://www.ietf.org/mail-archive/web/tcpm/current/msg08780.html), so it
>> will only be occasionally used. If we then get an incorrect estimate,
>> say a sender with one SMSS of data would instead send this as two SMSS,
>> and the algorithm triggers "incorrectly", this would be equivalent to
>> using a RTOR threshold of rrthresh + 1 so nothing critical happens. (In
>> some cases rrthresh + 1 may even be better.)
> If a wrong parameter for the estimate is possible but not critical, this should be explained in the document.

Ok, we can mention this in section 5.3.

>
> In general, the cited email from Alex stated:
>
> "* Sec 3: IMO the algo/doc is too much Linux driven. I would like to see a
>    segment-based *and* byte-based version of the algo, like RFC 5827."
>
> I fully agree to this statement and I think it has not been completely addressed in -04 so far.
>   
>>> * Section 9: I wonder if an attacker could try to send certain ACK
>> patterns to increase the risk of timeouts e.g. at a Web Server,
>> leveraging the smaller RTO value. If successful, the RTOs could
>> significantly reduce application performance. Probably, such an
>> attacker could cause much more harm by other means, i.e., this may not
>> be a significant security risk. But the current security analysis is
>> very short.
>>
>> The section is very short, but this is because we do not really see any
>> new security issues.
>>
>>    If I understand your example above correctly, the receiver is trying
>> to somehow reduce performance for its own application by manipulating
>> the ACK pattern. But if you are trying to mess up things for your own
>> application you can deliver incorrect data, throw away any data
>> received
>> or do whatever you want so I do not think trying to possibly slow down
>> the sender is the thing to worry about in this attack scenario. If
>> someone finds a relevant security issue it should of course be
>> documented, but I do not think it makes sense to put in something like
>> the above just to make the section longer.
> I was thinking e.g. about an off-path attacker that tries to inject ACKs into the TCP connection to trigger RTOs to harm application performance. I am not arguing that this is really a relevant attack scenario, but it seems not impossible to try that.

If you inject early ACKs you may be able to affect the RTO calculation 
and thereby increase the risk of spurious retransmissions, but this 
applies to RFC 6298 as well so there is nothing RTO restart specific 
about the attack. You do not increase the risk of a spurious RTO with 
RTO restart by sending an early ACK as this will be adjusted for in the 
RTO restart calculation, the timer always trigger one RTO after the 
transmission no matter when the ACK arrives.

What should probably be clarified in this section is that the security 
considerations in RFC 6298 still applies also with RTO restart, but that 
no additional security problems are known. Unless someone detects some 
specific problems of course. We will clarify this.

Cheers,
Anna

> Michael

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Mon, 1 Dec 2014 15:30:51 +0000

Hi Anna,

> > Some follow-up comments. I omit the parts for which there seems to be
> agreement on the need for a document update.
> >
> > I didn't say that there is a conflict in terminology. I might be
> wrong, but to me draft-ietf-tcpm-rtorestart-04 defines the term
> "previously unsent segments" as a *new* TCP state variable.
> 

Yes, I look for an exact definition of this parameter given that it is used in the algorithm. I think the document should clearly define the variable "previously unsent segments" and discuss how a TCP stack can determine this variable. That definition should cover both a TCP stack that stores unsent data internally in a segment-based buffer as well as a TCP stack that uses a byte-based send buffer. 
 
> > The value of that variable may be obvious for certain architectures
> of a TCP stack, in particular if segmentation into packets occurs on
> socket write() calls, i.e., before data is actually sent to network. As
> far as I recall, this is how the Linux stack works internally. But I'd
> like to avoid that an RFC describes an algorithm that can only be
> implemented for certain architecture of a TCP stack.
> >
> > RFC 3517 and RFC 4653 either refer to the "amount of previously
> unsent data" (which doesn't need to be further defined IMHO), or the
> "transmission of previously unsent segments" (which is not what draft-
> ietf-tcpm-rtorestart-04 refers to). For instance, here is the exact
> wording of the closest matches I could find in RFC 3517 and RFC 4563:
> >
> > Unless there is a definition of "number of previously unsent
> segments", I think this variable has to be defined and it has to be
> explained how it is calculated.
> >

As far as I can tell, you refer by a "send queue" to a byte-based stack, right?

Example: Let us assume an application that triggerd three socket write() calls of 100B after having disabled Nagle, but CWND currently does not allow sending a new packet.
  write(100B)
  write(100B)
  write(100B)

How many unsent segments does the TCP stack than have?

For a byte-based stack, the most likely answer seems to be one segment of 300B, given that the data has to be queued.

But for a segment-based stack, it depends whether the stack internally creates a segment immediately after each write() call. In this case, the stack could create *three* segments of 100B and schedule all of them for transmission. Alternatively, the stack could detect during/after the second and third write() call that there is queued data less than the MSS and merge the segments into one segment, resulting again in one segment of 300B. The latter operation could imply memory copies in the data structure that stores the segment, and this may be more expensive than scheduling three segments. Because of this effort, I could imagine some TCP stacks would have three segments of unsent data in this example, in particular if Nagle is disabled.

I have not checked recently what TCP stacks in reality do in that case, i.e., my example may not be realistic. But there are both segment- and byte-based TCP stacks in the wild, and the document should explain whether the parameter "number of previously unsent segments" depends on the stack-internal segmentation strategy, or not.

> > I could imagine that disabling Nagle's algorithm could be pretty
> common among thin stream applications. This is why the draft should
> IMHO discuss if disabling Nagle's algorithm affects the method.
> 

Unless I miss something, disabling Nagle algorithm does affect the algorithm depending on the segmentation strategy used by the stack. See the example above.

But my key question is about dependency on the segmentation strategy. Nagle is just one aspect of this.

> > If a wrong parameter for the estimate is possible but not critical,
> this should be explained in the document.
> 

OK

> > I was thinking e.g. about an off-path attacker that tries to inject
> ACKs into the TCP connection to trigger RTOs to harm application
> performance. I am not arguing that this is really a relevant attack
> scenario, but it seems not impossible to try that.
> 

Thanks

Michael

---

From: Anna Brunstrom <[email-masked]>
Date: Mon, 08 Dec 2014 00:13:09 +0100

Hi Michael,

> Yes, I look for an exact definition of this parameter given that it is used in the algorithm. I think the document should clearly define the variable "previously unsent segments" and discuss how a…

Ok.

> As far as I can tell, you refer by a "send queue" to a byte-based stack, right?

I was not really making a distinction.

> How many unsent segments does the TCP stack than have?

I would think that one segment of 300B is the normal case for both types 
of stacks, but it is true that the implementation can do what it wants.

> I have not checked recently what TCP stacks in reality do in that case, i.e., my example may not be realistic. But there are both segment- and byte-based TCP stacks in the wild, and the document sh…

Ok, we will try to address this in the next version.

BR,
Anna