Subject: [tcpm] 1323bis - preparing for WGLC

From: "Scheffenegger, Richard" <[email-masked]>
Date: Wed, 20 Mar 2013 14:07:19 +0000

Hi group,

running up to and during the IETF 86, the following points were raised w.r.t 1323bis. I hope this lengthy email makes it through your tl;dr filter...

a)      Is it necessary or useful, to have an Appendix D giving a Pseudo Code example?

     This section was added early in the process of creating the draft. As a couple of years have passed since, it was suggested to poll the WG if this text is valuable, or if the document would be better off without a strict Pseudocode example. Note that certain points that were added later  in the text of the draft, are not reflected in the Pseudocode (ie. late TS negotiation), and the Pseudocode also does NOT touch any of the interactions with other TCP subsystems (ie. Window Retraction).

     From an editorial point of view, removing an incomplete example, that may be misleading, is easier than working out a full-fledged pseudocode covering all the edge cases... (I updated my view of this from neutral to remove; the document is long enough even without an incomplete example).

b)      Guidance regarding late TS negotiation:

     This is a technical (semantic) change of timestamps; the upcoming -07 version of the draft will have this text added; is the WG satisfied with this wording, or shall more specific (2119) wording be added:

   A TCP MAY send the Timestamps option (TSopt) in an initial <SYN>
   segment (i.e., a segment containing a SYN bit and no ACK bit).  Once
   a TSopt has been sent or received in a non <SYN> segment, it MUST be
   sent in all segments.  Once a TSopt has been received in a non <SYN>
   segment, then any successive segment that is received without the RST
   bit and without a TSopt MAY be dropped without further processing,
   and an ACK of the current SND.UNA generated.

   Note that it is RECOMMENDED to negotiate the use of the timestamp
   option during the initial handshake.  Although this document is less
   restrictive regarding the use of the timestamp option when not sent
   initially, any guidance and discussion of potential issues when
   timestamp is enabled late in a session is beyond the scope of this
   document.

     I have received suggestions, to increase the "MAY" in the first sentence to a "SHOULD" (but that might indicate to implementers, that any TCP session SHOULD have TS enabled, not that TS MAY be used, and if, it SHOULD be negotiated during SYN).

     Another suggestion was to add a sentence like "Timestamps MUST be negotiated in <SYN> and <SYN,ACK> unless another mechanism allows enabling them in during an established session.  Such a  mechanism is outside the scope of this document."

c)      More appropriate introductory text; the following statement will supersede very old text (that apparently addressed WS, TS and SACK back in RFC1072); this was already presented at IETF 86 and received no objections.

   The window scale option negotiates fundamental parameters of the TCP
   session.  Therefore, it is only sent during the initial handshake.
   Furthermore, the window scale option will be sent in a <SYN,ACK>
   segment only if the corresponding option was received in the initial
   <SYN> segment.

   The timestamps option may appear in any data or ACK segment, adding
   12 bytes to the 20-byte TCP header.  It is recommended that this TCP
   option will be sent on non-SYN segments only after an exchange of
   options on the SYN segments has indicated that both sides understand
   the extension.  We believe that the bandwidth saved by reducing
   unnecessary retransmission timeouts will more than pay for the extra
   header bandwidth.

d)      Better text in the new section 2.4. describing the corner case when the Window Scale  option results in a Window Retraction during Loss Recovery; but no suggestions for better wording have been made so far. Does the WG agree with these steps, or should there be a different description around that particular case:

2.4.  Addressing Window Retraction

   When a non-zero scale factor is in use, there are instances when a
   retracted window can be offered [Mathis08].  The end of the window
   will be on a boundary based on the granularity of the scale factor
   being used.  If the sequence number is then updated by a number of
   bytes smaller than that granularity, the TCP will have to either
   advertise a new window that is beyond what it previously advertised
   (and perhaps beyond the buffer), or will have to advertise a smaller
   window, which will cause the TCP window to shrink.  Implementations
   MUST ensure that they handle a shrinking window, as specified in
   section 4.2.2.16 of [RFC1122].

   For the receiver, this implies that:

   1)  The receiver MUST honor, as in-window, any segment that would
      have been in-window for any ACK sent by the receiver.

   2)  When window scaling is in effect, the receiver SHOULD track the
      actual maximum window sequence number (which is likely to be
      greater than the window announced by the most recent ACK, if more
      than one segment has arrived since the application consumed any
      data in the receive buffer).

   On the sender side:

   3)  The initial transmission MUST honor window on most recent ACK.

   4)  On first retransmission, or if the sequence number is out-of-
      window by less than (2^Rcv.Wind.Scale) then do normal
      retransmission(s) without regard to receiver window as long as the
      original segment was in window when it was sent.

   5)  On subsequent retransmissions, treat such ACKs as zero window
      probes.

I'm currently in the process of updating the -07 "Changes " Appendix to split it into an technical/editorial part, as mentioned in my IETF 86 talk. Once that finishes, -07 will be submitted, together with any changes which the WG agrees.

Best regards,

Richard Scheffenegger

---

From: Joe Touch <[email-masked]>
Date: Wed, 20 Mar 2013 09:25:59 -0700

Hi, all,

On 3/20/2013 7:07 AM, Scheffenegger, Richard wrote:
> Hi group,
> running up to and during the IETF 86, the following points were raised
> w.r.t 1323bis. I hope this lengthy email makes it through your tl;dr filter…
>
...
>  2. Guidance regarding late TS negotiation:
>
> This is a technical (semantic) change of timestamps; the upcoming -07
> version of the draft will have this text added; is the WG satisfied with
> this wording, or shall more specific (2119) wording be added:
>
> I have received suggestions, to increase the “MAY” in the first sentence
> to a “SHOULD” (but that might indicate to implementers, that any TCP
> session SHOULD have TS enabled, not that TS MAY be used, and if, it
> SHOULD be negotiated during SYN).

IMO, the first sentence should be a MUST. TCP needs to know about all 
the options it needs to support during the initial handshake. If you 
turn on the option later and it isn't supported, there's no way to 
"handshake" to let the other end know.

> Another suggestion was to add a sentence like “Timestamps MUST be
> negotiated in <SYN> and <SYN,ACK> unless another mechanism allows
> enabling them in during an established session.  Such a  mechanism is
> outside the scope of this document.”

This is the one I expected, and is fine. It can either be confirmed by 
the option itself or by some other option; the overall assumption is 
that "the set of options that MAY be used during a TCP connection MUST 
be negotiated during the 3WHS". (that text is outside the scope of this 
doc, though)

>  3. More appropriate introductory text; the following statement will
>     supersede very old text (that apparently addressed WS, TS and SACK
>     back in RFC1072); this was already presented at IETF 86 and received
>     no objections.
>

The "*" inserted below is explained further down...

>     12 bytes to the 20-byte TCP header. *  It is recommended that this TCP

Again, IMO recommended should be REQUIRED (in all caps too).

>     the extension.

The portion below is a bit different:

>                   We believe that the bandwidth saved by reducing

it talks about the bandwidth of the overall connection; it's not 
specific to the SYN exchange. It might be useful to move this up to the 
"*" above.

Joe

---

From: Andre Oppermann <[email-masked]>
Date: Thu, 21 Mar 2013 12:35:52 +0100

On 20.03.2013 17:25, Joe Touch wrote:
> IMO, the first sentence should be a MUST. TCP needs to know about all the options it needs to
> support during the initial handshake. If you turn on the option later and it isn't supported,
> there's no way to "handshake" to let the other end know.

Seconded.

> This is the one I expected, and is fine. It can either be confirmed by the option itself or by some
> other option; the overall assumption is that "the set of options that MAY be used during a TCP
> connection MUST be negotiated during the 3WHS". (that text is outside the scope of this doc, though)
>

Is required the correct RFC keyword?  Shouldn't the sentence be rewritten
to make it use the "MUST" keyword?

-- 
Andre

> it talks about the bandwidth of the overall connection; it's not specific to the SYN exchange. It
> might be useful to move this up to the "*" above.
>
> Joe
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm
>
>

---

From: David Borman <[email-masked]>
Date: Thu, 21 Mar 2013 14:06:57 +0000

On Mar 20, 2013, at 11:25 AM, Joe Touch <[email-masked]> wrote:

> IMO, the first sentence should be a MUST. TCP needs to know about all the options it needs to support during the initial handshake. If you turn on the option later and it isn't supported, there's no way to "handshake" to let the other end know.

That really depends on the TCP option.

Turning on timestamps midstream without it being in the initial SYN isn't
technically that hard, as long as once you start sending and receiving them,
you continue to send/receive them for the rest of the connection.

The original reason for negotiating options in the SYN was twofold:
1) SYNs are reliably delivered, hence no additional negotiation
mechanism is needed.  2) At the time, all TCP implementations would
handle unknown options in the SYN, but some implementations would
crash if they received unknown TCP options in non-SYN packets.

Hopefully reason #2 is no longer needed.

Enabling TS midstream on the receiver is easy: if you receive any
packet with TS, then enable TS and use it for the duration of the
connection.  On the sender, you just start sending TS in every
packet, and remember the seq# of the first non-retransmitted packet
that was sent with TS.  If a TS is received in any packet, then enable
TS for the duration of the connection.  If no TS has been received when
an ACK that covers the initial TS seq# is received, then disable
sending TS for the duration of the connection.

As long as an unexpected TS mid-stream doesn't crash the other
side, then you're good to go, provided you take care of using
the TS for PAWS during the transition.

Other options, like WS, have to be negotiated in the SYN; trying
to change the WS value mid-stream using the current option would
require implementing additional negotiating mechanisms, since
ACKs are not reliably transmitted.

Honestly, the dictum that all TCP options have to be negotiated
in the SYN is to avoid sending TCP options mid stream to the other
side that the other side doesn't understand and isn't able to
properly ignore without crashing.  Be conservative in what you
send, liberal in what you receive.  But maybe it's time to say
that any TCP implementation that can't safely ignore unknown
options mid-stream is just plain broken and the standards shouldn't
continue to try to accommodate them anymore. 

			-David Borman

> This is the one I expected, and is fine. It can either be confirmed by the option itself or by some other option; the overall assumption is that "the set of options that MAY be used during a TCP connection MUST be negotiated during the 3WHS". (that text is outside the scope of this doc, though)
> 
> it talks about the bandwidth of the overall connection; it's not specific to the SYN exchange. It might be useful to move this up to the "*" above.
> 
> Joe

----------------------------------------------------------------------
The information contained in this transmission may be confidential. Any disclosure, copying, or further distribution of confidential information is not permitted unless such privilege is explicitly granted in writing by Quantum. Quantum reserves the right to have electronic communications, including email and attachments, sent across its networks filtered through anti virus and spam software programs and retain such messages in order to comply with applicable data security and retention requirements. Quantum is not responsible for the proper and complete transmission of the substance of this communication or for any delay in its receipt.

---

From: Andre Oppermann <[email-masked]>
Date: Thu, 21 Mar 2013 15:23:30 +0100

> Hopefully reason #2 is no longer needed.

Unlikely that any TCP implementation crashes on unknown options.
And if they do they get what they deserve.

There is a very different problem though.  Firewalls again.  Those
doing stateful inspection may drop packets when well-known but not
negotiated options show up.

Without empirical studies on the current situation we should not
allow this change in behavior of late TS enabling.

-- 
Andre

---

From: Mark Allman <[email-masked]>
Date: Thu, 21 Mar 2013 11:01:48 -0400

I agree with David.

> And if they do they get what they deserve.

Exactly.

> allow this change in behavior of late TS enabling.

Well, luckily we have some empirical study:

  Alberto Medina, Mark Allman, Sally Floyd.  Measuring the Evolution of
  Transport Protocols in the Internet.  ACM Computer Communication
  Review, 35(2), April 2005.
  http://www.icir.org/mallman/papers/tcp-evo-ccr05.ps

We injected a TS and an unknown junk option into the middle of
connections (independently) without negotiation in the 3WHS.  The
connection failure rate in both cases is 3%.  If we instead put the
options in the SYN the connection failure rate is 0.2%.  We cannot of
course tell why the connection is failing.  It could be middleboxes or
it could be receivers.

That paper is 8 years old and the data a bit older.  Perhaps someone has
newer data.  But, my guess is that this shows that this is simply not a
big problem. 

My hit here is that timestamps are simply a waste.  They take space.
They take complexity and processing at the endpoints.  And, yet, they
provide benefit on only exceedingly rare occasions.  In particular,

  - If you want to use TS for RTTM then you'll just turn it on from the
    beginning and be done with it.  (RTTM doesn't buy you anything, but
    thats beside the point.)

  - If you want to use TS for Eifel then you'll turn it on from the
    beginning and be done with it.

  - If you want to use TS for PAWS then you can get by without it until
    you've sent (nearly) 4GB (the sequence space) and you're sending
    quite fast (such that you might wrap too quickly).  Well, go look at
    flow sizes on the Internet.  Without even considering speed this is
    a Damn Small box.  And, once you throw speed in it gets even
    smaller.  And, throw on top of that the failure rate is pretty small
    (a long time ago) and I can't see where firing up TS as needed is at
    all problematic.  (Plus, we cannot engineer for the corner of the
    corner case that you found off to the side of the side road.  Or, so
    it seems to me.)

allman

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Thu, 21 Mar 2013 15:07:16 +0000

Thank you David!

You made some insightful comments, that I agree with!

> sending TS for the duration of the connection.

It's really the middleboxes that we need to work around in such a scenario...

1) In the forward path, a middlebox might start dropping segments, that contain TSopt which wasn't negotiated for. Thus a receiver never sees these data segments...

2) Or a middlebox might scrub the TSopt from the data packets before delivery...

3) In the return path, again a middle box might stop forwarding an ACK containing TSopt, if previous segments of that flow didn't have TSopt enabled.

4) Or that middlebox again scrubs the TSopt before delivery.

Cases 1+3 and 2+4 are virtually identical to the sender: 

In 1+3, the sender would not see any ACKs beyond the seq#, where TSopt was stated; Eventually multiple back-to-back RTOs would happen. 

In 2+4, the sender would see the ACKs, but without any TSopt. However, the receiver might have started using TSopt already on the receiver processing, thus it would be better to keep on sending with TSopt enabled.

Only when a RTO (or the 2nd RTO) for the data packet with the Seq# where TSopt was enabled happens, than it would be save to disable TSopt again... But once Snd.Fack or Snd.Una goes beyond the Seq# of the first TSopt segment, there should be no way to disable TSopt again...

Does that sound reasonable?

Richard Scheffenegger

> -----Original Message-----
> From: David Borman [mailto:[email-masked]]
> Sent: Donnerstag, 21. März 2013 15:07
> To: Joe Touch
> Cc: Scheffenegger, Richard; tcpm ([email-masked])
> Subject: Re: [tcpm] 1323bis - preparing for WGLC
> 
> 
> >> w.r.t 1323bis. I hope this lengthy email makes it through your tl;dr
> filter.
> >>
> > ...
> >> I have received suggestions, to increase the "MAY" in the first
> sentence
> >> to a "SHOULD" (but that might indicate to implementers, that any TCP
> the options it needs to support during the initial handshake. If you turn
> on the option later and it isn't supported, there's no way to "handshake"
> to let the other end know.
> 
> >> Another suggestion was to add a sentence like "Timestamps MUST be
> >> outside the scope of this document."
> >
> the option itself or by some other option; the overall assumption is that
> "the set of options that MAY be used during a TCP connection MUST be
> negotiated during the 3WHS". (that text is outside the scope of this doc,
> though)
> >
> The information contained in this transmission may be confidential. Any
> disclosure, copying, or further distribution of confidential information
> is not permitted unless such privilege is explicitly granted in writing by
> Quantum. Quantum reserves the right to have electronic communications,
> including email and attachments, sent across its networks filtered through
> anti virus and spam software programs and retain such messages in order to
> comply with applicable data security and retention requirements. Quantum
> is not responsible for the proper and complete transmission of the
> substance of this communication or for any delay in its receipt.

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Thu, 21 Mar 2013 16:20:32 +0100

To me, this discussion would perfectly fit into a new ID on allowing late TS enabling. It would be short and straightforward. But a separate ID could be easier than adding these kinds of thoughts to the 1323bis document, which we want to finish...

I think that such a new document would be allowed by "The Timestamp option MUST be negotiated during the initial <SYN> and <SYN,ACK> unless another mechanism allows to enable it during an established session.".

Michael (chair hat off)

> From: [email-masked] [mailto:[email-masked]] On 
> Behalf Of Scheffenegger, Richard
> Sent: Thursday, March 21, 2013 4:07 PM
> To: David Borman; Joe Touch
> Cc: tcpm ([email-masked])
> > connection.  On the sender, you just start sending TS in 
> every packet, 
> > and remember the seq# of the first non-retransmitted packet 
> that was 
> > sent with TS.  If a TS is received in any packet, then 
> enable TS for 
> > the duration of the connection.  If no TS has been received when an 
> > ACK that covers the initial TS seq# is received, then 
> disable sending 
> > TS for the duration of the connection.
> 
> 
> 1) In the forward path, a middlebox might start dropping 
> segments, that contain TSopt which wasn't negotiated for. 
> Thus a receiver never sees these data segments...
> 
> 3) In the return path, again a middle box might stop 
> forwarding an ACK containing TSopt, if previous segments of 
> that flow didn't have TSopt enabled.
> 
> In 1+3, the sender would not see any ACKs beyond the seq#, 
> where TSopt was stated; Eventually multiple back-to-back RTOs 
> would happen. 
> 
> In 2+4, the sender would see the ACKs, but without any TSopt. 
> However, the receiver might have started using TSopt already 
> on the receiver processing, thus it would be better to keep 
> on sending with TSopt enabled.
> 
> Only when a RTO (or the 2nd RTO) for the data packet with the 
> Seq# where TSopt was enabled happens, than it would be save 
> to disable TSopt again... But once Snd.Fack or Snd.Una goes 
> beyond the Seq# of the first TSopt segment, there should be 
> no way to disable TSopt again...
> 
> 
> > >> running up to and during the IETF 86, the following points were 
> > >> raised w.r.t 1323bis. I hope this lengthy email makes it through 
> > >> your tl;dr
> > filter.
> > >>
> > > ...
> > >> This is a technical (semantic) change of timestamps; the 
> upcoming 
> > >> -07 version of the draft will have this text added; is the WG 
> > >> satisfied
> > with
> > >>    segment, then any successive segment that is received without 
> > >> the
> > RST
> > >> to a "SHOULD" (but that might indicate to implementers, that any 
> > >> TCP session SHOULD have TS enabled, not that TS MAY be used, and 
> > >> if, it SHOULD be negotiated during SYN).
> > >
> > turn on the option later and it isn't supported, there's no 
> way to "handshake"
> > Turning on timestamps midstream without it being in the initial SYN 
> > isn't technically that hard, as long as once you start sending and 
> > receiving them, you continue to send/receive them for the 
> rest of the 
> > connection.
> > 
> > connection.  On the sender, you just start sending TS in 
> every packet, 
> > and remember the seq# of the first non-retransmitted packet 
> that was 
> > sent with TS.  If a TS is received in any packet, then 
> enable TS for 
> > the duration of the connection.  If no TS has been received when an 
> > ACK that covers the initial TS seq# is received, then 
> disable sending 
> > TS for the duration of the connection.
> > 
> > As long as an unexpected TS mid-stream doesn't crash the 
> other side, 
> > then you're good to go, provided you take care of using the TS for 
> > PAWS during the transition.
> > 
> > Other options, like WS, have to be negotiated in the SYN; trying to 
> > change the WS value mid-stream using the current option 
> would require 
> > implementing additional negotiating mechanisms, since ACKs are not 
> > reliably transmitted.
> > 
> > Honestly, the dictum that all TCP options have to be 
> negotiated in the 
> > SYN is to avoid sending TCP options mid stream to the other 
> side that 
> > the other side doesn't understand and isn't able to properly ignore 
> > without crashing.  Be conservative in what you send, 
> liberal in what 
> > you receive.  But maybe it's time to say that any TCP 
> implementation 
> > that can't safely ignore unknown options mid-stream is just plain 
> > broken and the standards shouldn't continue to try to 
> accommodate them 
> > anymore.
> > 
> > 			-David Borman
> > 
> > >
> > >> enabling them in during an established session.  Such a  
> mechanism 
> > >> is outside the scope of this document."
> > >
> > > This is the one I expected, and is fine. It can either be 
> confirmed 
> > > by
> > be negotiated during the 3WHS". (that text is outside the scope of 
> > this doc,
> > though)
> > >
> > >>    12 bytes to the 20-byte TCP header. *  It is recommended that 
> > >> this
> > TCP
> > >
> > specific to the SYN exchange. It might be useful to move this up to 
> > the "*" above.
> > >
> > > Joe
> > The information contained in this transmission may be confidential. 
> > Any disclosure, copying, or further distribution of confidential 
> > information is not permitted unless such privilege is explicitly 
> > granted in writing by Quantum. Quantum reserves the right to have 
> > electronic communications, including email and attachments, sent 
> > across its networks filtered through anti virus and spam software 
> > programs and retain such messages in order to comply with 
> applicable 
> > data security and retention requirements. Quantum is not 
> responsible 
> > for the proper and complete transmission of the substance 
> of this communication or for any delay in its receipt.

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Thu, 21 Mar 2013 15:21:18 +0000

Hi Mark,

One point that I would like to mention, which clearly speaks for late TS negotiation, is the severely limited option space, particularly in the <SYN>. As mentioned by David, and my previous emails, late enablement would be easy AND useful, and as you pointed out, not impact any of the current mechanisms tied to TSopt (PAWS, RTTM, Eifel). Returning 12 bytes option space during SYN (or not painting TS into a corner, where it can never be used in conjunction with larger SYN options) seems a valid reason to allow that.

(I'm still pursuing my semantic change for TS, to allow improvements in loss recovery; I think that would be a valuable use case for Timestamps).

I'm not proposing to have a full fletched mechanism for the late negotiation should be part of 1323bis (apart from a "passive" requirement to support it).

But I'm reluctant to disallow such late enablement categorically in 1323bis either.

Richard Scheffenegger

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Mark Allman
> Sent: Donnerstag, 21. März 2013 16:02
> To: Andre Oppermann
> Cc: tcpm ([email-masked]); David Borman; Joe Touch
> > Without empirical studies on the current situation we should not allow
> > this change in behavior of late TS enabling.
> 
> We injected a TS and an unknown junk option into the middle of connections
> (independently) without negotiation in the 3WHS.  The connection failure
> rate in both cases is 3%.  If we instead put the options in the SYN the
> connection failure rate is 0.2%.  We cannot of course tell why the
> connection is failing.  It could be middleboxes or it could be receivers.
>

---

From: Pasi Sarolahti <[email-masked]>
Date: Thu, 21 Mar 2013 18:00:38 +0200

> I think that such a new document would be allowed by "The Timestamp option MUST be negotiated during the initial <SYN> and <SYN,ACK> unless another mechanism allows to enable it during an establish…

To me this sounds like a proper case of SHOULD, considering the proposed "unless..." addition. The negotiation either MUST always happen during initial handshake, or if there can be exceptions, we should not use MUST. I see that original RFC1323 did not require the option to be included in SYN. Addition of MUST would turn previously RFC1323 compliant implementations to be non-compliant.

In general, I think MUST should be used sparingly, only when something is absolutely required for interoperability. RFC 2119 seems to give the same advice.

- Pasi, an individual TCPM follower

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Thu, 21 Mar 2013 16:39:57 +0000

Hi group,

While there was quite some feedback regarding the points b and c in my announcement, I'd still like to learn what the WG thinks about these two points?

a) Appendix D - Pseudo Code (incomplete)

I'm leaning to remove that entire section, any objections?

d) Window Scale reduction

The current text has been proposed by Matt Mathis on the list quite a while ago 

http://www.ietf.org/mail-archive/web/tcpm/current/msg03564.html

Is the WG in agreement with the specific text, that short summary of the necessary steps an implementer has to take? Or can anyone suggest a better wording for that section?

Best regards,
 
Richard Scheffenegger

---

From: Joe Touch <[email-masked]>
Date: Thu, 21 Mar 2013 10:44:54 -0700

Hi, Dave (et al.),

On 3/21/2013 7:06 AM, David Borman wrote:
...
> crash if they received unknown TCP options in non-SYN packets.

TCP is always supposed to ignore options it doesn't understand - whether 
in the SYN or after (RFC1122m Sec 4.2.2.5).

 > Hopefully reason #2 is no longer needed.

It isn't -- we just had a discussion about a TCP that reflects options 
it doesn't understand. That doesn't mean TCP should be designed to 
correct for that - we should declare that a serious bug and get it fixed.

However...

> sending TS for the duration of the connection.

You're describing half a handshake. You need to finish, including things 
like simultaenous initiation, what to do when if you detect loss and 
retransmit (do you retry with or without the TS? how many times? what if 
the loss is *because* of the TS?), etc.

And what happens when you set TS? Do you try the above mechanism with 
the first segment you transmit (including a retransmission)? (you can't 
because now an ACK without the TS is ambiguous - it can mean no TS 
support, or that the first segment you thought was lost got there anyway).

Even for something as simple as TS, this is a mess, and IMO should be 
avoided.

...
>...  But maybe it's time to say

I agree with this, but it doesn't affect (and isn't motivating) my view 
of enabling new capabilities mid-stream.

Joe

---

From: Joe Touch <[email-masked]>
Date: Thu, 21 Mar 2013 10:48:25 -0700

On 3/21/2013 8:21 AM, Scheffenegger, Richard wrote:
>
> Hi Mark,
>
>
> One point that I would like to mention, which clearly speaks for
> late TS negotiation, is the severely limited option space,
> particularly in the <SYN>. As mentioned by David, and my previous
> emails, late enablement would be easy AND useful, and as you pointed
> out, not impact any of the current mechanisms tied to TSopt (PAWS,
> RTTM, Eifel). Returning 12 bytes option space during SYN (or not
> painting TS into a corner, where it can never be used in conjunction
> with larger SYN options) seems a valid reason to allow that.

A better solution than mid-stream enabling is to allow the option to be
"vacuous" during the SYN exchange, e.g., to be 2 bytes long (KIND, LEN)
-- if the option that is enabled isn't something with values or
conditions to be negotiated as part of the handshake.

(and NO - I really don't want two KIND values, one for "enable X" and 
one for "use X"; the LEN field should be sufficient to distinguish 
between those two cases).

Joe

---

From: Joe Touch <[email-masked]>
Date: Thu, 21 Mar 2013 10:50:13 -0700

On 3/21/2013 8:20 AM, Scharf, Michael (Michael) wrote:
> To me, this discussion would perfectly fit into a new ID on allowing
> late TS enabling. It would be short and straightforward. But a
> separate ID could be easier than adding these kinds of thoughts to
> the 1323bis document, which we want to finish...

I agree.

> I think that such a new document would be allowed by "The Timestamp
> option MUST be negotiated during the initial <SYN> and <SYN,ACK>
> unless another mechanism allows to enable it during an established
> session.".

The current doc should say *only* "MUST be negotiated during the initial 
3WHS". The new doc can try to update that doc when - and if - it passes 
WG approval. We shouldn't be leaving doors open until we have the full 
debate on them.

Joe

---

From: Mark Allman <[email-masked]>
Date: Thu, 21 Mar 2013 14:02:03 -0400

> the first segment you transmit (including a retransmission)?  (you
> can't because now an ACK without the TS is ambiguous - it can mean no
> TS support, or that the first segment you thought was lost got there
> anyway).
> 

How is this a mess?  I don't get it.

Say we have TCP endpoints A and B.  A wants to turn on timestamps.  So,
on some non-retransmit data packet it turns them on and then just keeps
using them on every subsequent packet.  Now,

  - If the ACKs for those data packets come back with timestamps,
    great.  We have just figured out we can use timestamps.

  - If the ACKs for those data packets come back without timestamps,
    great.  We have just figured out either the path or the end host
    doesn't support timestamps (or doesn't support them when turned on
    in the middle).  We stop sending timestamps.  Big deal.

  - If no ACKs for the window of data come back then we retransmit
    without TS.  We can't tell why these packets were dropped, but
    perhaps because they had TS.  So, be conservative in what you send.
    This may then hamper the rate you can send at because you won't have
    PAWS.

We have three easy cases: we get back TS, we get back ACKs but no TS and
we get back nothing.  We ride on top of TCP's data reliability and it
all works pretty straightforwardly, it seems to me.  It isn't a mess.

The simultaneous start of TS usage is a no-op.  Both ends will figure
out they us TS and then they'll use it.

allman

---

From: Joe Touch <[email-masked]>
Date: Thu, 21 Mar 2013 11:15:40 -0700

Hi, Mark,

I gave the details in my earlier response, but here are some others:

> using them on every subsequent packet.

How long? when does it give up?

> Now,

"for those packets" means you need to keep track of that state. And, as 
Dave already noted, you need to check whether the ACK is covered 
*within* a range that is ACKd - including SACK, etc.

>      in the middle).  We stop sending timestamps.  Big deal.

Yes.

>      perhaps because they had TS.  So, be conservative in what you send.

That makes mid-stream TS enable more fragile than it would be if it were 
confirmed during the SYN.

> out they us TS and then they'll use it.

At the very least, you have to specify it.

My point is that the TS option needs to define a handshake mechanism in 
detail - including what doesn't need special handling, like simultaneous 
open.

AND you need to define how or whether this interacts with any other 
pending mid-stream handshakes.

The only reason is to avoid what should be 2 bytes (at most) in the SYN. 
If it's part of an aggregate option ("enable 1323"), then it'd be even 
less (effectively).

I don't think that's a good trade-off.

Joe

---

From: Mark Allman <[email-masked]>
Date: Thu, 21 Mar 2013 14:34:56 -0400

> > Say we have TCP endpoints A and B.  A wants to turn on timestamps.
> > So, on some non-retransmit data packet it turns them on and then
> > just keeps using them on every subsequent packet.
> 

Oh come on ... this isn't rocket science, Joe.

If A turns on TS at sequence number S1 it keeps sending them until the
ACKs for the window starting at S1 roll in.  So, when you get the first
ACK (or SACK) for something > S1 you note the largest sequence number
you have sent S2.  There is your window S1--S2.  Now, look at the ACKs
that roll in for that window.  And, you might as well keep sending TS
speculatively until S2 is ACKed---at which point you can make your final
decision.

Yeah- a little state.  Point taken.

> That makes mid-stream TS enable more fragile than it would be if it
> were confirmed during the SYN.

OK, fine.  Point taken.  But, it isn't like there isn't a path that adds
a little fragility over the current method.  See below ...

> pending mid-stream handshakes.

I don't think it does.  I mean, it just rides on packets being delivered
reliably and only needs to deal with its own option.  So, it seems
pretty self-contained to me.  And, what are those options?

> The only reason is to avoid what should be 2 bytes (at most) in the
> SYN. If it's part of an aggregate option ("enable 1323"), then it'd be
> even less (effectively).

You make it sound that is some sort of panacea.  Its not.  Its at least
as fragile as just starting to use TS in the middle:

  - Send me one of those new fangled options on the day after the RFC is
    published.  You get no timestamps!

  - So, to make sure you get timestamps you're going to have to send the
    full 10 bytes in the packet just like we do now.  At that point why
    send this new option, too?  It says the same thing and it further
    squeezes the option space in the SYN.  There is a *disincentive* to
    move to this new option ... it isn't getting you what you want *and*
    it takes more room.

So, how is this going to help us?  If we had a flux and could do this
From the start I might buy your argument.  But, I don't think its a
great path at this point.

allman

---

From: Joe Touch <[email-masked]>
Date: Thu, 21 Mar 2013 11:42:45 -0700

> Oh come on ... this isn't rocket science, Joe.

No, but it took you a paragraph to answer. And it's just the tip of what 
needs to be covered. Corner cases are why 3WHS isn't described in a 
single sentence.

...
>      published.  You get no timestamps!

I was talking about how to handle this in the future, to transition to a 
solution that is more efficient in its use of SYN space (and to design 
protocols that are more SYN-efficient now).

> great path at this point.

Well, the current TS mid-stream version isn't documented, and has a lot 
of corner cases that aren't addressed yet, so there are places where 
things won't work in current implementations even if you figure them out 
now.

Joe

---

From: "Ilpo Järvinen" <[email-masked]>
Date: Fri, 22 Mar 2013 12:44:30 +0200 (EET)

On Thu, 21 Mar 2013, Scheffenegger, Richard wrote:

> While there was quite some feedback regarding the points b and c in my 
> announcement, I'd still like to learn what the WG thinks about these two 
> points?
> 

While I don't exactly object the removal (it seems you did it already). 
There's one serious shortcoming in the document. That is, I was once 
looking for guidance on how to implement RFC793 acceptability check (3.3) 
together with RFC1323 PAWS, but the precedence was not specified anywhere.

I then found that there's the pseudocode in the 1323bis (which was then 
kind of dead, before you took over it), but even that failed to address 
the precendence, ie., no RFC793 checks were included to the pseudocode. 
I'd have suggested that RFC793 checks should be added to the pseudocode
but somewhere else in the document is just as fine as long as it is 
specified.

RFC793 checks are relevant in the case when there's 100% ACK loss for 
a window, the sender RTOs and retransmits to that out-of-synch condition. 
The interesting question is what timestamp should be echoed. It depends on 
which runs first, RFC1323 checks or RFC793 checks. I think it's important 
enough case to warrant consistent implementations, and therefore should 
be specified in 1323bis.

In addition, those ACKs that are triggered after the event should come 
with DSACK (assuming it's enabled, not detectable otherwise by the 
sender). They're similarly unreliable like those with SACK (although it 
actually depends on which of the checks take precedence). I'd amend 3.3, 
case b), to say explicitly that those with DSACK should also be excluded.

-- 
 i.

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Fri, 22 Mar 2013 11:51:30 +0000

Hi Ilpo,

> case b), to say explicitly that those with DSACK should also be excluded.

Well, the receiver should reflect the highest timestamp in that case:

  (2)  If:

            SEG.TSval >= TS.recent and SEG.SEQ <= Last.ACK.sent

        then SEG.TSval is copied to TS.Recent; otherwise, it is ignored.

Unless there was reordering between the RTO segment and the original segment, the DSACK <ACK> should carry a current measurement of the network RTT.

If the entire flight of <ACK>s before that DSACK <ACK> was lost, that <ACK> will advance SND.UNA and could be processed for RTTM (delivering a valid measurement, unless the reordering + flight of acks lost event). However, DSACK is just a special case of SACK, and I would read 

      b)  the segment does not indicate any loss or reordering, i.e.
          contains SACK options

to exclude DSACKs (as the DSACK itself is an indication of either loss or reordering (or spurious RTO), and DSACK is just a special case of regular SACK).

So, that particular case appears to me to be covered, but even if DSACKs would be processed (as they are in current stacks), there is little harm as the RTTM should mostly be up-to-date.

To the RFC0793 3.3 acceptability check: That is the area of Fernando's recent draft, I would like to hear his view of this issue.

Your scenario looks like this:

A(0) -->
      X- ack(A,0)

B(1) -->
      X- ack(B,1)

C(2) -->
      X- ack(C,2)
:
RTO
:
A(10) -->
          Ack(C,2)? Or ack(C,10)?

Correct?

When 793/3.3 runs first, the A(10) segment will be dropped, and TS.recent not updated, thus Ack(C,2) would be sent.

My view is, that this is not the correct behavior, as the TSecr in the ACK for that duplicate segment A would not reflect the most recent measurement of RTT.

Therefore, PAWS processing takes precedence. I think the steps in section 4.3 of 1323bis-07 allude to this implicit assumption.

Having a short statement at the beginning of section 4.3 (PAWS processing MUST take precedence over 793/3.3 acceptability checks) should make that clear, right?

Best regards,

Richard Scheffenegger

> From: Ilpo Järvinen [mailto:[email-masked]]
> Sent: Freitag, 22. März 2013 11:45
> To: Scheffenegger, Richard
> Cc: tcpm ([email-masked])
> > announcement, I'd still like to learn what the WG thinks about these
> > two points?
> >
> RFC793 checks are relevant in the case when there's 100% ACK loss for a
> window, the sender RTOs and retransmits to that out-of-synch condition.
> enough case to warrant consistent implementations, and therefore should be
> specified in 1323bis.
>

---

From: David Borman <[email-masked]>
Date: Fri, 22 Mar 2013 09:39:49 -0500

> Having a short statement at the beginning of section 4.3 (PAWS processing MUST take precedence over 793/3.3 acceptability checks) should make that clear, right?

This is a good point.  The intention is that TS.recent should be the largest
value received, and in this case receipt of A(10) will cause an ACK to be
generated; TS.recent should be updated and Ack(C,10) should be sent.  This
is important because when the sender receives the Ack(C,10), it will
acknowledge new data since all the previous ACKs were lost, and if Ack(C,2)
was sent, the TS information would not be accurate, defeating the whole
purpose of TS.

In addition, the retransmission of A(10) could have updated ACK information
in it for the reverse direction, and we'd want to get that.  Hence, for the
TCP Sequence draft, this is an argument for accepting ACK information as well
as updating TS.recent when receiving duplicate packets that are more than one
byte to the left of the window.  The TCP Sequence draft should be updated to
include updating TS.recent as well as the ACK information.

			-David Borman

---

From: "Ilpo Järvinen" <[email-masked]>
Date: Fri, 22 Mar 2013 18:39:13 +0200 (EET)

Hi Richard and David,

On Fri, 22 Mar 2013, David Borman wrote:

> > to exclude DSACKs (as the DSACK itself is an indication of either loss 
> > or reordering (or spurious RTO), and DSACK is just a special case of 
> > regular SACK). 
> >
> > So, that particular case appears to me to be covered, but even if 
> > DSACKs would be processed (as they are in current stacks), there is 
> > little harm as the RTTM should mostly be up-to-date. 

I agree that there's improvement over RFC1323 alone thanks to the recent 
1323bis work. I discovered this long time ago when even that SACK wording 
did not exists (when 1323bis was rather dormant), and therefore I might 
have been "spoiled" by the previous docs to judge the present wording 
clear enough. However, I'd say I'd still interpret the "does not indicate 
any loss or reordering" to mean data direction only (DSACK doesn't exactly 
"indicate" ACK loss even if that occurred in a particular scenario, IMHO 
it only "indicates" that a segment was received more than once). Thus it 
excludes ACK loss, which makes SACK option present vs DSACK ambiguous 
because there were no data dir losses nor reordering. 

IMHO, one of the most challenging thing from implementer POV is getting 
interaction between RFCs right because lack of exact and precise spec for 
the combos, and I think here's one good example of such (combination of 
SACK option, DSACK and TS, not exactly trivial combo of three). But adding 
something because of this is not as important as it used to be before the 
recent 1323bis changes.

Also note that the current Linux implementation does reply with the older 
timestamp, which is how I discovered this issue in the first place (I set 
large ACK loss % while I tried to reproduce some totally different issue 
and saw RTT drifting occassionally to max during testing). So no matter 
what is done/decided now, we'll have the legacy around for quite long time 
still. I don't know what other OSes do in this case but it would be 
interesting to know.

> > To the RFC0793 3.3 acceptability check: That is the area of Fernando's 
> > recent draft, I would like to hear his view of this issue. 
> > 

Yes, this is the scenario.

> > When 793/3.3 runs first, the A(10) segment will be dropped, and 
> > TS.recent not updated, thus Ack(C,2) would be sent. 
> > 
> > My view is, that this is not the correct behavior, as the TSecr in the 
> > ACK for that duplicate segment A would not reflect the most recent 
> > measurement of RTT.
> >
> > Therefore, PAWS processing takes precedence. I think the steps in 
> > section 4.3 of 1323bis-07 allude to this implicit assumption. 
> > 
> > Having a short statement at the beginning of section 4.3 (PAWS 
> > processing MUST take precedence over 793/3.3 acceptability checks) 
> > should make that clear, right?
>

I realized this myself when I discovered the issue, but sadly I think both 
options have downsides:

- With Ack(C,10) no window check is done for the PAWS update? It would 
remove ambiguity that otherwise remains even with TS (like you put it, 
it's the purpose of TS). But it's against the basic PAWS algo, if it is 
thought as steps in-order. But the key problem is that it opens a can of 
security worms (TSRecent-in-future attack)!? (I failed to state this in 
the 1st email as I misremembered the issue details slightly, and I had 
completely forgotten the existance of that basic PAWS algo :-)).

- With Ack(C,2): the sender mis-measures RTT. Could be mitigated by the 
sender with a check for DSACK.

Due to security, I'd err to the Ack(C,2) option which makes discarding 
DSACKed ACKs necessary. But I'd definately like to see Ack(C,10) though to 
remove ambiguity, however, only if security is not jeopardized.

Third option I was then thinking was to echo the TSVal from the 
discarded segment (Ack(C,10)), but to not update TSRecent because of 
the RFC793 check failure. It would remove the ambiguity successfully but 
protect against TSRecent-in-future attacks? But there might be other cans 
of worms with this approach, I'm not fully sure of its impact.

-- 
 i.

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Fri, 22 Mar 2013 18:59:01 +0100

> While there was quite some feedback regarding the points b 
> and c in my announcement, I'd still like to learn what the WG 
> thinks about these two points?
> 

[...]

> Is the WG in agreement with the specific text, that short 
> summary of the necessary steps an implementer has to take? Or 
> can anyone suggest a better wording for that section?

I am not sure if I can provide a better wording. But indeed in particular the sender side part is hard to parse:

- I think the meaning of "honor" is unclear in "The initial transmission MUST honor window on most recent ACK.". Does it imply "The initial transmission MUST be within the window announced by the most recent ACK"?

- I am not sure if the meaning of "such ACKs" is clear in "On subsequent retransmissions, treat such ACKs as zero window probes.". Probably, "such ACKs" refers to "ACKs shrinking the window". But even then it is unclear to me how this relates to the question whether retransmissions must be in-window.

Given that this is an entire new section in a -bis document, I could also imagine adding an example such as http://www.ietf.org/mail-archive/web/tcpm/current/msg03564.html in a new appendix, to better motivate and explain that change compared to 1323.

Michael

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Sun, 24 Mar 2013 10:48:20 +0000

Hi Ilpo,

>> This is a good point.  The intention is that TS.recent should be the
>> largest value received, and in this case receipt of A(10) will cause
>> an ACK to be generated; TS.recent should be updated and Ack(C,10)
>> should be sent.  This is important because when the sender receives
>> the Ack(C,10), it will acknowledge new data since all the previous
>> ACKs were lost, and if Ack(C,2) was sent, the TS information would not
>> be accurate, defeating the whole purpose of TS.
> 

Can you explain this a little more?

In the scenario, the A,10 packet has the TS processed first (and TSrecent is updated), as by step (2) in section 3.4 (1323bis). As the SEG.SEQ is old, only an ACK is triggered without any further processing. Someone observing the session could mount a DOS attack by resending an old packet, with a higher TSval, true...

What do you suggest? 

Swapping points R2/R3 around, and adding (to the current R3):

(RCV.NXT - RCV.WND [- 1]) <= SEG.SEQ <= Last.ACK.sent

> sender with a check for DSACK.

Expanded the RTTM rules to state: 

      b)  the segment does not indicate any potential loss or
          reordering, i.e. contains SACK [RFC2018] or D-SACK
          [RFC2883] options

> Third option I was then thinking was to echo the TSVal from the discarded
> segment (Ack(C,10)), but to not update TSRecent because of the RFC793
> check failure. It would remove the ambiguity successfully but protect
> against TSRecent-in-future attacks? But there might be other cans of worms
> with this approach, I'm not fully sure of its impact.

That would be in-line with my proposed TS nego draft, which actually changes the semantics of TSopt when SACK is also negotiated... Keeping 1323bis in line with the use of TSrecent and the semantics imposed by that would be preferable.

I would suggest an extension to the TSrecent update test as mentioned above. Any downsides with that?

Best regards,

Richard Scheffenegger

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Sun, 24 Mar 2013 11:29:43 +0000

Hi Michael,

>> d) Window Scale reduction

I reworded that section using your suggestion:

   On the sender side:

   3)  The initial transmission MUST be within the window announced by
       the most recent <ACK>.

   4)  On first retransmission, or if the sequence number is out-of-
       window by less than (2^Rcv.Wind.Scale) then do normal
       retransmission(s) without regard to receiver window as long as
       the original segment was in window when it was sent.

   5)  Subsequent retransmissions MAY only be sent, if they are within
       the window announced by the most recent <ACK>.

The zero window probe reference I removed, as regular new transmissions, which would fall beyond the most recent announced window, should not be performed anyway.

Also, I added an appendix, quoting Matt's email near verbatim (fixing typos):

Appendix F.  Window Retraction Example

   Consider a established TCP connection with WSCALE=7 (128 byte
   receiver window quantization), that is running with a very small
   windows because the receiver is bottlenecked and both ends are doing
   small reads and writes.

   Consider the ACKs coming back:

   SEG.ACK  SEG.WIN computed SND.WIN   receiver's actual window
   1000     2       1256               1300

   The sender writes 40 bytes and receiver ACKs:

   1040     2       1296               1300

   The sender writes 5 additional bytes and the receiver has a problem.
   Two choices:

   1045     2       1301               1300   - BEYOND BUFFER

   1045     1       1173               1300   - RETRACTED WINDOW

   This problems is completely general and can in principle happen any
   time the sender does a write which is smaller than the window scale
   quanta.

   In most stacks it is at least partially obscured when the window size
   is larger than some small number of segments because the stacks
   prefer to announce windows that are integral numbers of segments
   (rounded up to the next window quanta).  This plus silly window
   suppression tends to cause less frequent, larger window updates.  If
   the window was rounded down to a segment size there is more
   opportunity to advance it ("beyond buffer" case above) rather than
   retracting it.

Richard Scheffenegger

> From: Scharf, Michael (Michael) [mailto:[email-masked]]
> Sent: Freitag, 22. März 2013 18:59
> To: Scheffenegger, Richard; tcpm ([email-masked])
> Subject: RE: 1323bis - preparing for WGLC
> 
> > Is the WG in agreement with the specific text, that short summary of
> > the necessary steps an implementer has to take? Or can anyone suggest
> > a better wording for that section?
> 
> - I think the meaning of "honor" is unclear in "The initial transmission
> MUST honor window on most recent ACK.". Does it imply "The initial
> transmission MUST be within the window announced by the most recent ACK"?
> 
> - I am not sure if the meaning of "such ACKs" is clear in "On subsequent
> retransmissions, treat such ACKs as zero window probes.". Probably, "such
> ACKs" refers to "ACKs shrinking the window". But even then it is unclear
> to me how this relates to the question whether retransmissions must be in-
> window.
> 
> Given that this is an entire new section in a -bis document, I could also
> imagine adding an example such as http://www.ietf.org/mail-
> archive/web/tcpm/current/msg03564.html in a new appendix, to better
> motivate and explain that change compared to 1323.
> 
> Michael

---

From: Mark Allman <[email-masked]>
Date: Wed, 27 Mar 2013 09:38:57 -0400

[This message has been sitting here mostly written for a long time now.
 I have lost track of the discussion.  The generic subject lines don't
 really help me tease all this apart.  I am hoping a new draft will be
 forthcoming so I can re-sync.]

I want to pop up a level here.  I think the important thing in all of
this 1323bis business is to downplay the need for TS.  

  - First, there is no evidence that RTTM buys you any better RTO
    estimator (and there is evidence that it does not) so there is no
    reason to use TS to grab more RTT samples per RTT.  

    (My druthers would be to relegate RTTM to an appendix.  I.e., "here
    is something we thought would help, but it doesn't really change
    much and while you're free to still use it as it doesn't much hurt
    the RTO, we're no longer recommending it".)

  - Second, in the absence of RTTM that leaves PAWS as the sole user of
    timestamps.  But, just because we use WS does not mean we need PAWS
    and so we should divorce the use of TS from the use of WS.
    Obviously, at some point there is a need for PAWS and so we have to
    engineer a mechanism to enable TS when needed.  That might mean
    negotiating in the SYN.  It might not.  There are tradeoffs.  But,
    ultimately this need will arise quite rarely and so probably the
    specifics are not crucial.

Its easy to focus on the nitty-gritty of mechanism (I do all the time!),
but lets also keep the big picture in mind, too.

allman

---

From: Tim Shepard <[email-masked]>
Date: Wed, 27 Mar 2013 09:50:53 -0400

>     engineer a mechanism to enable TS when needed.  That might mean

Do you think that such a mechanism would need to work with the
existing installed base?  Or do you think the existing installed base
will soon be irrelevant and we can engineer such a mechanism which
does not achieve PAWS when interoperating with the existing installed
base?

			-Tim Shepard
			 [email-masked]

---

From: John Leslie <[email-masked]>
Date: Wed, 27 Mar 2013 10:07:06 -0400

> this 1323bis business is to downplay the need for TS.  

   I'd be willing to "deprecate" it entirely...

>     reason to use TS to grab more RTT samples per RTT.  

   That's the way I see it too.

>     the RTO, we're no longer recommending it".)

   If we do that, I think we need to note some of the problems we've
found since RFC 1323 was published.

>     specifics are not crucial.

   IMHO, PAWS use of TS must remain possible (in the absence of middlebox
damage, that is) -- but I agree it may not be the best way to protect
against wrapped sequence numbers.

> but lets also keep the big picture in mind, too.

   I'm pretty sure that the TS option can't be "fixed" enough to be
problem-free -- if we believe there is a need for something it does,
we're probably better off designing a (new) replacement option.

   (YMMV, obviously...)

--
John Leslie <[email-masked]>

---

From: David Borman <[email-masked]>
Date: Wed, 27 Mar 2013 14:38:41 +0000

>    reason to use TS to grab more RTT samples per RTT. 

I've said it before, and I'll say it again:

The worst case scenario using TS is equivalent to the best case
scenario for the Karn algorithm; that's that TS buys you.

As long as you are making forward progress, with TS you are guaranteed
to be able to get at least one valid RTT sample per RTT, whenever new
data is acknowledged, even in the face of packet loss.  With the Karn
algorithm over a lossy connection, in the worst case scenario you never
get any valid RTT samples, if they keep getting invalidated due to
retransmissions.

For lossy connections, that distinction is probably important.

		-David Borman

---

From: Jakob Heitz <[email-masked]>
Date: Wed, 27 Mar 2013 15:01:23 +0000

RTO is 1 second.
RTT doesn't matter, because it's always much less.

--
Jakob Heitz.

---

From: Mark Allman <[email-masked]>
Date: Wed, 27 Mar 2013 11:03:54 -0400

> RTO is 1 second.

(not in practice)

---

From: Mark Allman <[email-masked]>
Date: Wed, 27 Mar 2013 11:03:20 -0400

> algorithm over a lossy connection, in the worst case scenario you
> never get any valid RTT samples, if they keep getting invalidated due
> to retransmissions.
> 

Let me say two things ...

(1) Requiring 10 bytes of every TCP packet ever transmitted to guard
    against a pathology seems pretty dubious to me.  The 'net worked and
    works without timestamps.

(2) This: "in the worst case scenario you never get any valid RTT
    samples, if they keep getting invalidated due to retransmissions" is
    either wrong or a red herring.

    I.e., recall that you backoff the RTO until you can get a new
    segment and its ACK through---at which point you have an RTT sample
    and you can get rid of the backoff.  So, there is a mechanism to
    hack the RTO in the absence of RTT samples.

    I.e., that leaves a case like where every segment is dropped on its
    first transmission and makes it through on retransmission.  And,
    that doesn't change with backing off the RTO.  In this case it just
    isn't at all that this RTT sample is at all important in getting us
    to a better RTO.

allman

---

From: Mark Allman <[email-masked]>
Date: Wed, 27 Mar 2013 10:50:07 -0400

> does not achieve PAWS when interoperating with the existing installed

I think you could do either.  I think you get some skid greasing by
leveraging the current scheme with some tweaks to allow for the TS to be
used without requiring it to be used for the entire connection.

allman

---

From: Mark Allman <[email-masked]>
Date: Wed, 27 Mar 2013 10:48:43 -0400

> found since RFC 1323 was published.

We can simply say RTTM does not help.  Evidence:

  Mark Allman, Vern Paxson.  On Estimating End-to-End Network Path
  Properties.  Proceedings of the ACM SIGCOMM Technical Symposium,
  Cambridge, MA, September 1999.
  http://www.icir.org/mallman/papers/estimation.ps

We learned that since the publication of 1323.  And, bis documents are
supposed to reflect what we have learned (either in terms of
specification looseness or ambiguity or in terms of technical lessons).

> against wrapped sequence numbers.

My hit is that the sequence space isn't big enough for the windows
afforded by large WS factors.  So, we need some additional protection
against wrapped sequence space when we (a) actually wrap and (b) are
transmitting quite fast.  The current mechanism we have is PAWS and that
in turn is based on the timestamp option.  So, it seems to me the path
of least resistance is to tweak when the timestamp option is used.

Now,

> we're probably better off designing a (new) replacement option.

... this is also a possibility.  I am sure a 4 or 6 byte option would
get us the same thing as PAWS in less space.  But, by doing this we lose
the greasing of the skids TS gives us.  And, in the infrequent cases we
really need PAWS the bytes might not matter.

Also, note, that the TS option has been used for other things.  And,
that seems fine to me.  If you want to use Eifel or if more RTT samples
helps some delay-based CC scheme or whatnot then great, use timestamps.
But, turning on TS without an actual tangible reason (i.e., just because
you want to scale the window) is highly dubious.

allman

---

From: Jakob Heitz <[email-masked]>
Date: Wed, 27 Mar 2013 15:05:52 +0000

What is it in practice?

--
Jakob Heitz.

---

From: Mark Allman <[email-masked]>
Date: Wed, 27 Mar 2013 11:17:35 -0400

> What is it in practice?

I don't have good data at my finger tips, but I think it is pretty well
known that the 1sec min is violated routinely and the min is often a
couple hundred msec.  

(What we don't know is the implications of this change.  We found that
there is a pretty fundamental tradeoff between making the RTO aggressive
and taking more spurious RTOs.  Many times when I have talked with folks
who advocate dropping the min they are solely focused on needed RTOs
taking less time.  Well, for needed RTOs you can be as aggressive as you
want and show improvement.  However, that neglects the impact of
unneeded RTOs which drop the cwnd to one segment.  I have never seen
anyone do a nice study of both sides of this equation.)

allman

---

From: Joe Touch <[email-masked]>
Date: Wed, 27 Mar 2013 08:38:13 -0700

>    works without timestamps.

+1

More to the point, 10 bytes of option space, i.e., 25%.

Joe

---

From: Yuchung Cheng <[email-masked]>
Date: Wed, 27 Mar 2013 09:22:33 -0700

> known that the 1sec min is violated routinely and the min is often a

For Linux the 4*RTTVAR is lower-bounded at 200ms (due to the common
setting of delayed ACK). RTO = SRTT + 4*RTTVAR and does not cap RTO at
1sec. Linux samples RTT for every ACK.
http://www.cs.helsinki.fi/research/iwtcp/papers/linuxtcp.pdf

Also “A Performance Study of Loss Detection/Recovery in Real-world TCP
Implementations”, ICNP 2007

> unneeded RTOs which drop the cwnd to one segment.  I have never seen
Worse in fact. The acks of original packets after a spurious RTO can
trigger a false slow-start storm of inflight packets. with deep
buffers this pours way more than BDP into the jammed highway. With
timestamp and Eifel this can be detected and stopped early. Without TS
this is terrible.

Sorry it's rude to not read the entire thread and the document and
chime in now. but TS is helpful for RTT samples during loss recovery
where RTT varies the most due to congestion. It also key to identify
spurious retransmission and is more robust and useful than F-RTO. The
former does not require new data available to send at timeout, a
luxury that interactive request-response application does not have.

I also agree 12 bytes (not 10 b/c of option alignment) is not
negligible so a lazy trigger on dubious network condition sounds a
great solution to me.

> anyone do a nice study of both sides of this equation.)
Why don't you do that :)

---

From: Mark Allman <[email-masked]>
Date: Wed, 27 Mar 2013 12:40:28 -0400

> where RTT varies the most due to congestion.

Certainly RTTM can get you samples during this period when the Usual
Mechanism cannot (although it isn't like without TS it is impossible; it
costs more state at the sender, but you can do it).  However, our
results are that in the aggregate this doesn't impact the efficacy of
the estimator.

(In fact, if you look back at our path estimation paper you'll see that
taking only a *single* RTT sample *per connection* actually isn't too
shabby.  It isn't the best estimator, but it isn't the worst either.
I am not advocating doing this, but it shows that a reasonable estimator
does not require fine-grained updating.)

> It also key to identify spurious retransmission 

Sure.  If you are using timestamps for something tangible and they are
helping you then *fine*.  I am not at all suggesting the option go
away.  I am saying the RTTM scheme is all cost and little-to-no
benefit.  And, I am saying that PAWS is simply not needed in the vast
majority of cases and so it makes more sense to me to be able to turn
the TS option when it becomes necessary and not presume it will be
needed simply because we scale the window.

> I also agree 12 bytes (not 10 b/c of option alignment) 

(Right- I was being generous in that in theory one could use those bytes
for something else.)

> > anyone do a nice study of both sides of this equation.)

If Google has figured out cloning let me know, as I have many, many
things that are rattling around my head, but too few cycles to
pursue. :-) 

allman

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Wed, 27 Mar 2013 17:25:07 +0000

Well, I found the following mechanisms (at least partially) relying on the presence of TSopt:

Eifel (RFC3522/4015)

NewReno (RFC3782)
ACK congestion control (RFC3449, RFC5690)

Plus some of the passive measurement people use it...

Did I miss any?

(Besides, I still think that TS can be put to better use during loss recovery by changing the semantics when used in conjunction with SACK. But let's finish 1323bis first.)

BTW: Latest iteration of the disputed section:

   A TCP MAY send the Timestamp option (TSopt) in an initial <SYN>
   segment (i.e., segment containing a SYN bit and no ACK bit), and MAY
   send a TSopt in other segments only if it received a TSopt in the
   initial <SYN> or <SYN,ACK> segment for the connection.

   Once TSopt has been successfully negotiated during the <SYN>,
   <SYN,ACK> exchange, TSopt MUST be sent in every non-<RST> segment for
   the duration of the connection.  If a non-<RST> segment is received
   without a TSopt, a TCP MAY drop the segment and MAY send an <ACK> for
   the last in-sequence segment.  A TCP MUST NOT abort a TCP connection
   if a non-<RST> segment is received without a TSopt.

   If a TSopt is received on a connection where TSopt was not negotiated
   in the initial three-way handshake, the TSopt MUST be ignored and the
   packet processed normally.

   In the case of crossing <SYN> segments where one <SYN> contains a
   TSopt and the other doesn't, both sides SHOULD put a TSopt in the
   <SYN,ACK> segment.

   TSopt is required for the two mechanisms described in sections 3.3
   and 4.2.  There are also other mechanisms that rely on the presence
   of the TSopt, e.g.  [RFC3522].  If a TCP stopped sending TSopt at any
   time during an established session, it interferes with these
   mechanisms.  This update to [RFC1323] describes explicitly the
   previous implicit assumption, that each TCP segment must have TSopt,
   once negotiated.

Richard Scheffenegger

> Yuchung Cheng
> Sent: Mittwoch, 27. März 2013 17:23
> To: [email-masked]
> Subject: Re: [tcpm] timestamps usage (was Re: 1323bis - preparing for
> WGLC)
> 
> > I don't have good data at my finger tips, but I think it is pretty
> > well known that the 1sec min is violated routinely and the min is
> > often a couple hundred msec.
> 
> Also "A Performance Study of Loss Detection/Recovery in Real-world TCP
> Implementations", ICNP 2007
> 
> >
> > there is a pretty fundamental tradeoff between making the RTO
> > aggressive and taking more spurious RTOs.  Many times when I have
> > talked with folks who advocate dropping the min they are solely
> > focused on needed RTOs taking less time.  Well, for needed RTOs you
> > can be as aggressive as you want and show improvement.  However, that
> > neglects the impact of unneeded RTOs which drop the cwnd to one
> > segment.  I have never seen
> trigger a false slow-start storm of inflight packets. with deep buffers
> this pours way more than BDP into the jammed highway. With timestamp and
> Eifel this can be detected and stopped early. Without TS this is terrible.
> 
> Sorry it's rude to not read the entire thread and the document and chime
> in now. but TS is helpful for RTT samples during loss recovery where RTT
> varies the most due to congestion. It also key to identify spurious
> retransmission and is more robust and useful than F-RTO. The former does
> not require new data available to send at timeout, a luxury that
> interactive request-response application does not have.
> 
> I also agree 12 bytes (not 10 b/c of option alignment) is not negligible
> so a lazy trigger on dubious network condition sounds a great solution to
> me.
>

---

From: Mark Allman <[email-masked]>
Date: Wed, 27 Mar 2013 13:31:34 -0400

> Plus some of the passive measurement people use it...

What is your point?  I just don't follow this because it isn't like I
have seen anyone saying we should get rid of the timestamp option.  So,
what are you trying to say?

allman

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Wed, 27 Mar 2013 17:36:48 +0000

Hi Mark,

Then I must have misinterpreted your statements;

I knew you question the RTTM mechanism (for yielding better estimates of RTO; there are other uses for RTTM though), and PAWS isn't always required (particularly in short flows).

Thus I got the impression that you'd rather than 1323 obsoleted and replaced with entirely different mechanisms...

Sorry.

Richard Scheffenegger

> Sent: Mittwoch, 27. März 2013 18:32
> Cc: Joe Touch; tcpm ([email-masked]); David Borman
> What is your point?  I just don't follow this because it isn't like I have
> seen anyone saying we should get rid of the timestamp option.  So, what
> are you trying to say?
> 
> allman
> 
>

---

From: Mark Allman <[email-masked]>
Date: Wed, 27 Mar 2013 13:54:59 -0400

> I knew you question the RTTM mechanism (for yielding better estimates
> of RTO; there are other uses for RTTM though), and PAWS isn't always
> required (particularly in short flows). 
> 
> Thus I got the impression that you'd rather than 1323 obsoleted and
> replaced with entirely different mechanisms... 

I have said this at least twice today.  Maybe the third time is a
charm. 

Within 1323 and 1323bis there are three things we have been talking
about:

(1) The timestamp option.  This option conveys information.  We send our
    current timestamp and we echo a timestamp back to our peer.  It is
    nothing more than exchange of information.

(2) The RTTM algorithm.  This specifies how one takes information
    conveyed via the timestamp option and uses that to calculate an RTO
    estimator. 

(3) The PAWS algorithm.  This specifies how one takes information
    conveyed via the timestamp option and uses that to protect against
    the sequence space wrapping too quickly and segments getting
    injected into the data stream in the wrong place.

That is just what we have.

I am suggesting:

  - That we simply deprecate (2).

  - When we thought we needed RTTM we just rode on its back to get PAWS.
    Without RTTM we can be more judicious and divorce the use of the
    timestamp option from the use of the window scaling option.  We can
    decide that for PAWS it is fine to start using timestamps when we
    actually need PAWS.

Neither of these things say anything about (1).  

If TCP needs timestamps for some reason that is outside the scope of
1323(bis) then the timestamp option is still there and its still
available.  If you want to pay a per-packet price to use Eifel, great.
Go for it.  If you think its worth it to pay a per-packet cost to maybe
get some fast retransmit benefit immediately after an RTO with NewReno,
great.  Go for it.  If you need more RTT samples for your new whiz-bang
delay-based congestion controller, great.  Go for it.  If you want to
invent some new scheme for loss recovery that leverages the information
in the timestamp option, great.  Go for it.  But, none of that has
anything to do with the algorithms in 1323bis (RTTM & PAWS).

allman

---

From: John Leslie <[email-masked]>
Date: Wed, 27 Mar 2013 14:05:24 -0400

>    initial <SYN> or <SYN,ACK> segment for the connection.

   That much is the original RFC 1323 text, right?

>    the duration of the connection.

   I didn't find this MUST anywhere in RFC 1323. Did I miss something?

>    If a non-<RST> segment is received without a TSopt, a TCP MAY drop
>    the segment and MAY send an <ACK> for the last in-sequence segment.

   This text confuses me. I don't particularly want to argue it, but I'm
at a loss to understand how it helps. (There are four cases, right? And
there seem to be a couple of unnamed variables as well.)

>    A TCP MUST NOT abort a TCP connection if a non-<RST> segment is
>    received without a TSopt.

   This sounds reasonable at first blush, but I don't see how anyone
could depend on it.

>    packet processed normally.

   This would have been good to include in RFC 1323, but I wonder what
it accomplishes to add it now.

>    TSopt and the other doesn't, both sides SHOULD put a TSopt in the

   This sounds funny: very likely the one that didn't send TSopt doesn't
want to dedicate N bytes of option space in every packet.

>    previous implicit assumption, that each TCP segment must have TSopt,

   No doubt some of us believe such an assumption was "implicit"; and
quite possibly there are implementations which won't interoperate without
that assumption being valid.

   Nonetheless, I find it hard to believe that any implementations break
horribly if the assumption fails.

   Writing a 1323-bis, the question isn't simply, "What should we have
said the first time?" We must include, "How do we get here from there?"
This is hard, IMHO: which is why I suggested a new aption instead.

---

From: David Borman <[email-masked]>
Date: Wed, 27 Mar 2013 18:42:04 +0000

>  I didn't find this MUST anywhere in RFC 1323. Did I miss something?

PAWS assumes it, see section 4.2.

> there seem to be a couple of unnamed variables as well.)

Maybe it should be just "a TCP MAY drop the segment and send an <ACK>..."
The point is that if you are doing PAWS and a packet arrives without
TSopt, you can't reliably determine whether or not the packet came from
a previous wrap of the sequence space, so without any other knowledge
it's best to just drop the packet and generate an ack, just as you do
for any other unacceptable packet.

>  This sounds reasonable at first blush, but I don't see how anyone
Depend on it?  No, there is little if anything that you can depend
on.  This is telling implementors that they can't abort a TCP connection
just because a packet arrives without TSopt.  And no, I am not aware
of this being a real-world issue.

> it accomplishes to add it now.

Just covering the bases.  TCPs are already supposed to ignore unknown
TCP options, and this is just saying treat it as an unknown option,
even if you do know what it is. :-)

There are two general cases:
	A) TSopt is sent in SYN, SYN/ACK, and every non-RST packet
	B) TSopt is never sent, or sent in only the initial SYN.
that are the normal, current usages.

The previous 3 statements are addressing cases that don't fall
into those two cases.  I have no problem with just dropping all
three of them, but in trying to clarify 1323, folks on this
discussion seem to want all the different scenarios spelled out
in detail.

> want to dedicate N bytes of option space in every packet.

It's the case of an implementation that only enables TSopt if it receives
it in the SYN.  It doesn't ask the other side to turn on TSopt when
initiating a connection, but agrees to use it if the other side asks.

		-David Borman

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Thu, 28 Mar 2013 11:13:12 +0000

Hi John,

> From: John Leslie [mailto:[email-masked]]

With minor editorial changes and an update to RFC2119 wording, yes.

>    I didn't find this MUST anywhere in RFC 1323. Did I miss something?

No; as mentioned on this list, certain mechanisms assume the permanent use of Tsopt right after the successful negotiation. This text was added by David in 2007 ( http://tools.ietf.org/html/draft-borman-1323bis-00 ):

         Once a TSopt has been sent or received in a
         non <SYN> segment, it must be sent in all segments.  Once a
         TSopt has been received in a non <SYN> segment, then any
         successive segment that is received without the RST bit and
         without a TSopt may be ACKed and dropped without further
         processing.

After the discussions on this list, it seemed to be better to not leave any potential gap between the 3WHS negotiation, and the actual use of the TS option. All the implementations that I know of effectively work that way.

> there seem to be a couple of unnamed variables as well.)

Well, the 2nd MAY is superfluous, I guess at this reading. But I'm open for better wording suggestions.

>    This sounds reasonable at first blush, but I don't see how anyone could
> depend on it.
> 
>    This would have been good to include in RFC 1323, but I wonder what it
> accomplishes to add it now.

Just to make this behavior explicit. There is another example in the dreded RTTM mechanism. 1323 just stated that any ACK that indicates loss/reordering should be excluded. Most ACKs that contain SACK do indicate some loss/reordering, but when I investigated, these ACKs are still used to update RTO under certain circumstances. Thus the added text there to explicitly exclude ACK+SACK for the RTTM mechanism.

> want to dedicate N bytes of option space in every packet.

This is new text; it was added in 2009 in http://tools.ietf.org/html/draft-ietf-tcpm-1323bis-01 
We could leave that case undefined, as in 1323... 

> This is hard, IMHO: which is why I suggested a new aption instead.

I agree, things will usually not go terribly wrong if the assumptions don't hold, but the implementations seen behave as described in the latest text. I think it would be save to clearly state, that this is how to actually use TSopt...

Richard

---

From: John Leslie <[email-masked]>
Date: Fri, 29 Mar 2013 10:02:56 -0400

Scheffenegger, Richard <[email-masked]> wrote:
> No; as mentioned on this list, certain mechanisms assume the permanent
> use of Tsopt right after the successful negotiation.

   Hmm... there's that "assume" word...

> This text was added by David in 2007
> ( http://tools.ietf.org/html/draft-borman-1323bis-00 ):
> 

   It's a definite problem that IETF process sometimes leaves things
in limbo for six years. (That draft perhaps predates datatracker.ietf?)

   No doubt there are implementations out there that haven't waited
for a 1323bis to become an RFC. :^( Sorry to say: they proceeded at
their own risk, and deserve to suffer the consequences.

   This does not mean I wish to punish them. They presumably _can_
interoperate with enough other nodes to be worth the trouble. We can
quite reasonably describe their situation, whether or not it happens
in a 1323-bis. What we should NOT do is publish a 1323-bis that ONLY
caters to them.

   David Borman's 2007 text, IMHO, has been overcome by events.

> After the discussions on this list, it seemed to be better to not
> leave any potential gap between the 3WHS negotiation, and the actual
> use of the TS option. All the implementations that I know of
> effectively work that way.

   The phrase "rearranging the deck chairs on the Titanic" comes to
mind...

   We can legitimately say, "All the implementations we know..." --
but there _will_ be others we don't know. I strongly advise against
changing any behavior of TSopt from the original 1323 language.

   I'm sure that I'm regarded as a Johnny-come-lately here; and I
won't deny the name. But, alas!, it's _very_ hard to get enough
review of Internet Drafts before LastCall happens.

> Just to make this behavior explicit.

   I'm not arguing the language -- I'm thinking we have no way of
knowing whether the other end is speaking 1323 or 1323-bis. Thus the
value of a MUST to ensure interoperability seems dubious.

> There is another example in the dreded RTTM mechanism. 1323 just
> stated that any ACK that indicates loss/reordering should be
> excluded. Most ACKs that contain SACK do indicate some loss/reordering,
> but when I investigated, these ACKs are still used to update RTO
> under certain circumstances.

   "In certain circumstances" this could possibly be quite correct.
I wouldn't assign 1323 a particularly high precedence here -- but it's
probably worth noting the corner case.

   Measuring Round Trip Time is somewhat of a black art. No doubt many
in-the-field measurements are just plain wrong. I've given up any hope
of fixing them. :^(

> Thus the added text there to explicitly exclude ACK+SACK for the
> RTTM mechanism.

   If that's the desire, it should be explained. Adding one more MUST
won't do the trick.

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Fri, 29 Mar 2013 14:55:30 +0000

Hi John,

>    Hmm... there's that "assume" word...

Well, Section 4.2 in RFC1323 already states:

4.2  The PAWS Mechanism

      PAWS uses the same TCP Timestamps option as the RTTM mechanism
      described earlier, and assumes that every received TCP segment
      (including data and ACK segments) contains a timestamp SEG.TSval
      whose values are monotone non-decreasing in time.

Joe pointed out, that dubious forward references (and 1323 didn't even have a forward reference in 3.2 when talking about the TSopt signal) in a document should be avoided. 

The new text in -bis-08 now clearly states, that TSopt MUST be present, after successful negotiation. Omitting TSopt would - at best (ie. Linux) - leave PAWS in an undefined state; But then, apparently, Linux doesn't implement PAWS as intended (see Ilpo's comment) anyway.

>>  Once a TSopt has been sent or received in a  non <SYN> segment, it
>> must be sent in all segments.  Once a  TSopt has been received in a
>> non <SYN> segment, then any  successive segment that is received
>> without the RST bit and  without a TSopt may be ACKed and dropped
>> without further  processing.
> 
>    It's a definite problem that IETF process sometimes leaves things in
> limbo for six years. (That draft perhaps predates datatracker.ietf?)
> 
>    No doubt there are implementations out there that haven't waited for a
> 1323bis to become an RFC. :^( Sorry to say: they proceeded at their own
> risk, and deserve to suffer the consequences.
> 
> interoperate with enough other nodes to be worth the trouble. We can quite
> reasonably describe their situation, whether or not it happens in a 1323-
> bis. What we should NOT do is publish a 1323-bis that ONLY caters to them.

1323 already had that in it; just not explicitly worded in section 3.2 (only mentioned in passing in section 4.2). I don't see how this is really changes the mechanisms in 1323. True, it closes the door on certain potential things that 1323 left unspecified (like late TS enablement). 

>    David Borman's 2007 text, IMHO, has been overcome by events.

Such as? Do you know of an example where late enablement of Tsopt is done? Judging the comments, I think David's 2007 text is more relevant (the -08 makes it even more restrictive, disallowing any data/ACK without TSopt, which seems to be less controversial).

>    The phrase "rearranging the deck chairs on the Titanic" comes to

As long as it contributes to a peace of mind in the end... :)

>    We can legitimately say, "All the implementations we know..." -- but
> there _will_ be others we don't know. I strongly advise against changing
> any behavior of TSopt from the original 1323 language.

What is then the point of a -bis, if not giving guidance to implementers to NOT interpret too limited text in a problematic way, and fixing issues with the (internal) mechanisms described?

>    I'm sure that I'm regarded as a Johnny-come-lately here; and I won't
> deny the name. But, alas!, it's _very_ hard to get enough review of
> Internet Drafts before LastCall happens.

Better late than never; But I need to understand your concerns, and what you propose for the -bis, in more detail. I like the feedback, but I cann't make out what you are suggesting (as text in the -bis).

>>>>  If a TSopt is received on a connection where TSopt was not
>>>> negotiated  in the initial three-way handshake, the TSopt MUST be
>>>> ignored and the  packet processed normally.
>>>
>    I'm not arguing the language -- I'm thinking we have no way of knowing
> whether the other end is speaking 1323 or 1323-bis. Thus the value of a
> MUST to ensure interoperability seems dubious.

The (bug-fixed) mechanisms stay the same; 1323bis fixes the wording to avoid buggy implementations. If there was a stack that does late negotiation (as 1323 does allow this), a 1323bis stack would simply continue running all the regular (non-TSopt) mechanisms, rather than starting to use PAWS on the receiver side. 

Am I missing something here?

>> stated that any ACK that indicates loss/reordering should be excluded.
>> Most ACKs that contain SACK do indicate some loss/reordering, but when
>> I investigated, these ACKs are still used to update RTO under certain
>> circumstances.
> 

Highly doubtful. That "certain circumstance" involves one dropped packet on the forward path, and at least the last in-sequence ACK dropped on the return path; A sender taking a RTT measurement from the next ACK(SACK) will inflate the RTO, and suffer from a longer retransmission timeout...

The new wording doesn't change the spirit of the RTTM mechanism one bit, but clearly points out, that the presence of a SACK option indicates something, where the RTT measurement should better not be taken. But this is a very common implementation issue I noticed.

>    Measuring Round Trip Time is somewhat of a black art. No doubt many in-
> the-field measurements are just plain wrong. I've given up any hope of
> fixing them. :^(

RTTM and OWDV (one way delay variance) don't do that well with the TS semantics from 1323(-bis) either. But I won't return to this topic as long as 1323bis is lingering.

>> Thus the added text there to explicitly exclude ACK+SACK for the RTTM
>> mechanism.
> 

The explanation is already  in 1323:

      However, the following
      rule prevents a resulting inflation of the measured RTT:

           A TSecr value received in a segment is used to update the
           averaged RTT measurement only if the segment acknowledges
           some new data, i.e., only if it advances the left edge of the
           send window.

The omission in 1323 was to not call out the case, where the returning ACKs are partially lost too, which inflates the next measurement. Without SACK, there is not much to do about this; with SACK, that one "MUST" should suffice.

I could also come up with some text describing the corner case above, if that helps, or add a TSG...

Best regards,
  Richard

---

From: Yuchung Cheng <[email-masked]>
Date: Fri, 29 Mar 2013 12:47:52 -0700

> Just to make this behavior explicit. There is another example in the dreded RTTM mechanism. 1323 just stated that any ACK that indicates loss/reordering should be excluded. Most ACKs that contain S…

This caught my attention; I am stunned. I can't believe it :(
"""
However, the following rule prevents a resulting inflation of the measured RTT:
...
      b)  the segment does not indicate any loss or reordering, i.e.
          contains SACK options
"""

The RTT of the ack contains a SACK reflects the delay when the network
is in trouble, and we want to toss it and not update the RTO. It's
still 3 days before April 1st.

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Sat, 30 Mar 2013 06:10:33 +0000

Hi Yuchung,

In the following scenario,

what should the left side (sender) do, when the last ACK arrives?

    <A, TSval=1> --------------------->

   <---------------- <ACK(A), TSecr=1>

    <B, TSval=2> --------------------->

     X-------------- <ACK(B), TSecr=2>

    <C, TSval=3> -------------------X

    <D, TSval=5> --------------------->

   <------- <ACK(B), SACK(D), TSecr=2>

If RTT is the quantity to be measured, updating with that last ACK (which, for the sender, updates the left edge of the window - thus is acceptable in 1323, when ignoring SACK - as most stacks do) would inflate sRTT and RTTvar, increasing the RTO. And sRTT is *not* the RTT of the network.

My point being, if the mechanism is to measure RTT, it should do this (with the current semantics, the only sensible approach would be that the last ACK must be ignored).

If the point is to only come up with some conservative estimate for RTO (and an even more conservative estimate in this case), the current text and implementations are fine even though that last ACK does indicate loss...

However, the variables sRTT and RTTvar need to have a connotation that they are *only* intermediary values just used for RTO purposes and nothing else, and should be renamed that they are NOT to be used for anything but RTO estimation. Other (future, experimental) mechanisms need to derive their own state then.
                   
Best regards,                                     

Richard Scheffenegger

> From: Yuchung Cheng [mailto:[email-masked]]
> Sent: Freitag, 29. März 2013 20:48
> Cc: John Leslie; [email-masked]
> use of Tsopt right after the successful negotiation. This text was added
> by David in 2007 ( http://tools.ietf.org/html/draft-borman-1323bis-00 ):
> >
> > After the discussions on this list, it seemed to be better to not leave
> any potential gap between the 3WHS negotiation, and the actual use of the
> TS option. All the implementations that I know of effectively work that
> way.
> >
> >
> could
> negotiated
> > Just to make this behavior explicit. There is another example in the
> dreded RTTM mechanism. 1323 just stated that any ACK that indicates
> loss/reordering should be excluded. Most ACKs that contain SACK do
> indicate some loss/reordering, but when I investigated, these ACKs are
> still used to update RTO under certain circumstances. Thus the added text
> there to explicitly exclude ACK+SACK for the RTTM mechanism.
> 
> > This is new text; it was added in 2009 in
> http://tools.ietf.org/html/draft-ietf-tcpm-1323bis-01
> > I agree, things will usually not go terribly wrong if the assumptions
> don't hold, but the implementations seen behave as described in the latest
> text. I think it would be save to clearly state, that this is how to
> actually use TSopt...
> >
> > Richard
> >