Subject: Re: [tcpm] feedback from group to draft-scheffenegger-tcpm-timestamp-negotiation-02.txt

From: "Scheffenegger, Richard" <[email-masked]>
Date: Wed, 6 Jul 2011 16:58:27 +0100

Hi Joe, group,

it took me a while to get hold of a good volume of trace data (Using the
Waikato set, Jan-Aug 2005), to investigate the prevalence and type of
misbehavior around TCP timestamps in [SYN].

(Unfortunately, I don't have access to more recent CAIDA data sets :( ).

The data I evaluated totals to slightly over 1 billion [SYN].

45% of these [SYN]s carried the Timestamp option.
 8*10^-6 of these SYNs were sent with TSecr not zero (this includes
duplicate/retransmitted [SYN]s; approximately 45% of such [SYN]s are
retransmissions/duplicated - a much higher fraction, than in the normal
population of [SYN]s

I'm currently verifying if these TSecr!=0 events were all preceded by a
TCP session, where the sender got to know a timestamp from the receiver.

Nevertheless, the distribution of the TSecr timestamp values is quite
skewed towards lower values of the timestamps:

3 MSB bits;  number of unique [SYNs]
000          1311
001          341
010          136
011          51
100          55
101          36
110          26
111          20

Based on the current proposal, only the last 3 bins could inadvertently
trigger the changed timestamp handling on the receiver side (82 out of
1947 - 4.2%), but 6 hosts don't negotiate also for SACK (76/1947 =
3.9%).

The hosts in the "100" bin could potentially negotiate successfully for
both the changed timestamp handling (again, 6 Hosts don't negotiate for
SACK), and signal some timestamp clock rate. However, using the REServed
bits (2nd octet), which must be zero according to the draft, none of
these sessions would have successfully negotiated for timestamp
capabilities.

At a high bound, this leaves some 150 sessions out of 1 billion sessions
(1.5*10^-7) to inadvertently have timestamp signaling changed (with a
heavy skew towards getting immediate TS echos, instead of the algorithm
in RFC1323; with legacy hosts, this may lead to an increase in spurious
retransmits - and thereby TCP throughput reductions). 

The upper bound to erroneously negotiate timestamp capabilities, and
also coax the receiver into enabling advances heuristics seems to be
around 10^-10 of all sessions.

Again, this is based on trace data from 2005, and addresses issues with
misbehaving TCP senders, that have broken timestamp implementations.
It's likely, that these fractions reduced over the last 6 years, as
these misbehaving implementations got fixed over time.

I would like to hear from the group, if this rate of false positives is
at an acceptable level, or if further mitigations beyond what is in the
draft are required. (Or if I need to try to get hold of more recent
data). 

Richard Scheffenegger

> -----Original Message-----
> From: Joe Touch [mailto:[email-masked]]
> Sent: Mittwoch, 01. Juni 2011 22:54
> To: Scheffenegger, Richard
> Cc: [email-masked]
> Subject: Re: [tcpm] feedback from group to draft-scheffenegger-tcpm-
> timestamp-negotiation-02.txt
> 
> 
> 
> On 6/1/2011 1:02 PM, Scheffenegger, Richard wrote:
> >
> > Hi Joe,
> >
> > I am sorry, but you lost me...
> >
> > I fully agree, the PRNG assumption may not hold. But I don't quite
> > follow where you set a fixed date (or was your choice of the signed
> unix
> > epoch rollover random?)...
> 
> I picked dates that had the bits set as you indicated ;-)
>

---

From: Joe Touch <[email-masked]>
Date: Wed, 06 Jul 2011 09:09:10 -0700

Hi, Richard,

Well, IMO, the whole point behind TCP is that it works. It never works 
optimally, but if it *can* work it *does* work.

Anything with *any* false positives, excepting those traced to 
implementation bugs, is unacceptable to me.

TCP connections should *never* have to "try again" due to such a 
protocol error.

Others can weigh their views.

Joe

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Wed, 6 Jul 2011 17:27:54 +0100

Thanks Joe,

as all these TSecr!=0 are due to faulty implementation (RFC1323
stipulates that a [SYN] has to have TSecr=0), I take your comment that
the expected false positive rate of 1.5*10^-7 for immediate TS echos /
10^-10 for wrong enhanced receiver side heuristics (putting these broken
implementations at a further disadvantage) is acceptable. 

The outcome of false positives, at the simplest, will be a higher level
of spurious retransmits for those sessions. Also standard TCP CC
reaction and loss recovery instead of enhanced (optimized) reactions
(TCP Chirp; lost retransmission detection...). 

Thus these broken implementations are disadvantaged w.r.t. legacy
RFC1323 implementations, while new implementations can benefit from
improved TCP CC reaction and improved loss recovery...

Richard Scheffenegger

> Sent: Mittwoch, 06. Juli 2011 18:09
> > (Unfortunately, I don't have access to more recent CAIDA data sets
:(
> ).
> > I'm currently verifying if these TSecr!=0 events were all preceded
by
> a

---

From: Joe Touch <[email-masked]>
Date: Wed, 06 Jul 2011 10:08:53 -0700

> (TCP Chirp; lost retransmission detection...).

The part that bothers me is that the retransmits had a much higher 
(nearly 50%) repeat rate of errors, so your logic doesn't hold. I.e., 
this takes a broken implementation that had no impact and makes it break 
a connection.

It'd be useful to track that down to *confirm* that it's a broken 
implementation, and what OS/version it tracks to.

> improved TCP CC reaction and improved loss recovery...

The concern right now is that 'broken' used to mean either silently 
working or at worst slightly slower, and you're breaking it harder. 
Although 1323 says MUST, 'be generous in what you receive' means 
generally ignoring errors in that field, since it wasn't spec'd for 
other uses.

I.e., making TCP more brittle at the expense of a performance win is a 
bad thing, IMO.

If these errors can be tracked and confirmed as a bug (and hopefully a 
patch issued), then that's another story...

Joe

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Thu, 7 Jul 2011 10:47:12 +0100

Hi Joe,

> this takes a broken implementation that had no impact and makes it

I was being imprecise. In the traces, there are series of two or more
SYNs (actually, the few instances I have looked at in detail so far, a
series of very short (<10 pkts) tcp sessions) that are in close
succession, between the same hosts, but different source ports, where
the TSecr is identical and non-zero. 

I still need to investigate, if the later sender has always a preceding
TCP session, by which it learned some TSval from the later receiver.
(I.e. to differentiate between genuinely broken implementations, that
send some non-zeroed memory fragment, vs. implementations that re-use
some prior TCPCB under some circumstances, without clearing it properly
- see below.

When looking at TSecr values and their distribution, I called these
duplicates, as they are duplicating the TSecr in close temporal
proximity. The receivers timestamp clock was still identical between
these sessions - also indicating, that the receiver is not using
timestamp randomization.

> implementation, and what OS/version it tracks to.

One misbehaving stack appears to be a version of linux, from the passive
fingerprint (order of TCP options, initial values used in the tcp header
field of the [SYN]). Again, I need to verify this for all ~2000
occurances and not just a handful.

> I.e., making TCP more brittle at the expense of a performance win is a

IMHO, the draft doesn't make TCP itself more brittle; it puts clients
which are not fully compliant at a slight disadvantage - possibly
motivating the upgrade of these clients.

If the misbehavior only appears when there was a preceding tcp session
within 2 MSL, there may be some mitigations:

A possible workaround would be to have a TCP receiver send TSval=0 with
[FIN] control packets (if the receiver used timestamp randomization), or
check if the TSecr received in a [SYN] is close to its current timestamp
clock. That may indicate  the sender is echoing the timestamp value of a
preceeding session. Obviously, that latter approach fails, if timestamp
randomization is used.

RFC1323 already declares, that TSval=0 has a special meaning (no valid
timestamp available), and with the observed misbehaving clients, this
would effectively clear out the stored timestamp value in the re-used
TCPCB. Afterwards, the next [SYN] would look clean again to the
receiver...

BTW, Linux was long time broken in this respect as well, as TSval=0 in a
[SYN] (i.e. from Win95/98/2000 clients) would disable timestamps for the
session - the [SYN,ACK] from a linux box would not contain the timestamp
option.

> patch issued), then that's another story...

The TCP/IP header options investigated so far indicate, that this
behavior shows with Linux Kernels - see also
http://kerneltrap.org/mailarchive/linux-netdev/2009/4/10/5446374

Seems to be considered a "feature" of linux (in violation of RFC1323),
not a bug... But it's non-default, enabled by tweaking some sysctls.

However, I don't like to cater for specific "features" (bugs) of one
particular TCP implementation (Linux) in a potential standards document,
especially when these are due to non-default tunables or already fixed
bugs (TSval=0 [SYN]).

Best regards,
   Richard

---

From: Joe Touch <[email-masked]>
Date: Thu, 07 Jul 2011 10:36:13 -0700

Hi, Richard,

On 7/7/2011 2:47 AM, Scheffenegger, Richard wrote:
...
> not a bug... But it's non-default, enabled by tweaking some sysctls.

Ick. Pity this isn't more of a surprise.

> especially when these are due to non-default tunables or already fixed

One of our challenges is how to deal with this sort of issue, which 
includes things like ignoring the current IW (and IETF process for that 
one), using a nonstandard congestion alg, etc.

I'm glad to have the WG put its foot down on this sort of thing, but it 
ought to do so uniformly, IMO.

Joe