Subject: [tcpm] Questions on draft-scheffenegger-tcpm-timestamp-negotiation

From: "SCHARF, Michael" <[email-masked]>
Date: Thu, 3 Nov 2011 11:15:58 +0100

Richard, Mirja,

I'd like to trigger a discussion of some aspects of
draft-scheffenegger-tcpm-timestamp-negotiation, given that it proposes a
negotiation feature that we currently don't have in TCP.

Some questions that come into my mind (as individual) regarding the
current draft test are:

- Is there a (prototype) implementation and corresponding lessons
learnt?

- Are there some numbers on the benefit (regarding better OWD
estimation, early spurious retransmit detection, early lost
retransmission detection)?

- What would be the pros and cons of only standardizing a part of what
the draft currently proposes. For instance, what is the cost of not
violating RFC 1323?

Any insight would be helpful.

Thanks

Michael

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Thu, 3 Nov 2011 10:33:04 +0000

Hi Michael,

> As individual: While it is clear that we have some empty
> bits that we could use for this negotiation, I am a little 
> wondering about why we really need that negotiation. 
> Some questions that come into my mind:
> 
> - Is there a (prototype) implementation and corresponding
> lessons learnt?

Not in a kernel stack. The last few months I spent the time 
understanding why some stacks (really mostly Linux) already violate 
RFC1323 during the 3WHS.
http://kerneltrap.org/mailarchive/linux-netdev/2009/4/10/5446374
http://permalink.gmane.org/gmane.linux.network/168864
(Basically, Linux has "TSval==0" is invalid, while RFC1323 states 
without valid TS, send TSval=0 (0 may mean the sender has no valid 
timestamp available, or uses a genuine value of
zero) . There are more messages floating around this issue in the 
discussion around the "bug fix").

> retransmission detection)?

The qualitative aspects are discussed in the draft. There are a few 
(experimental) tcp improvements which use the value contained in TSval 
for timing purposes on the receiver side.
These fall into two categories: Assuming one particular timestamp 
clock interval without training phase, or assume one interval from a 
limited set after a short training phase.
Both will fail when the sender uses some simple form of packet 
counting or reserves a few low-order bits for sanity checking (i.e. 
some experimental Cubic implementation). To quantify how much TCP 
congestion control may go out of whack then, is really hard.

For spurious retransmit detection (ie. Eifel) you need two different 
timestamp values between original segment and it's retransmission; as 
soon as RTT drops below the tcp timer (to be exact: timestamp clock) 
granularity, these feature simply fails to work. So the performance 
evaluation of Eifel can be used as quantitative impact analysis for 
low latency (LAN) environments.

The early lost retransmission aspect would be a protocol in it's own 
right, building upon what Linux is been doing for about a decade now 
(and which is not specified in RFCs).
Qualitatively, Early LRD and Linux LRD relate to each other in a 
similar way as Eifel and F-RTO do - the latter requires the sending 
(and ACK) of a new segment, while the former work implicitly with 
information contained in the timestamps.

But frankly, we are definitely talking about the 99,9-something 
percentile of segment delivery latency.

I received an analysis from Google at one time of their RTOs at some 
time - it was private only after the discussion around the "recovery" 
retransmission to be included in the 3517bis:

(all counters are in thousand)
		baseline sack-reno-recov
Retransmits: 	104187 		104940
Fast: 		16935 (16.3%) 	17984 (17.1%) [+0,8%]
Forward: 	4178 ( 4.0%) 	4162 ( 4.0%)
After-RTO: 	17395 (16.7%) 	17238 (16.4%) [-0,3%]
Timeouts: 	65677 (63.0%) 	65554 (62.5%) [-0,5%]
1st: 		45096 43.3% 	44992 42.9%
Open 		40688 39.1% 	40718 38.8%
Disorder 	2581 2.5% 	2607 2.5%
Recovery 	864 0.8% 	715 0.7%
2nd+: 		21542 20.7% 	21513 20.5%
LostRetrans: 	1250 ( 1.2%) 	1267 ( 1.2%)

Note that the early LDR would address the row "LostRetrans", dealing 
with perhaps 1% of the issues leading to RTOs in a public web server.

I plan to collect similar data from NFS sessions, once we have the 
appropriate instruments in our stack. (The NFS/iSCSI traffic is 
qualitatively different from HTTP-ish traffic - not only because it's 
traffic most likely encountered in private data centers instead of 
across the public internet).

> - What would be the pros and cons of only standardizing a
> part of what the draft currently proposes. For instance, what
> is the cost of not violating RFC 1323?

Without the semantic change of RFC1323, OWD always included the 
processing and delivery delay of a delayed acknowledged data segment. 
So, OWD variance not only captures network related timing variance, 
but also processing variance in the host. Agreed, some processing 
variance of the host would always be part of the measurement, but with 
the current semantics, this is artificially bloated so that OWD 
variance is nearly meaningless for sufficiently low latency 
environments (< ~20ms or so).

Also, the improvements around early lost retransmission require a 
unique timestamp signal also during SACK loss recovery. The current 
RFC1323 semantics render the timestamp value inert during loss, so 
that they effectively contribute no information around the current 
network / receiver state to the sender (despite using up 12 bytes).

This problem was mentioned by Luigi Rizzo already during 
standardization of RFC1323 (see his TSACK idea:
http://info.iet.unipi.it/~luigi/selack.ps ) 15 years ago, but declined 
at the time (SACK was standardized only later, and as you know, there 
is no interaction between TS and SACK, despite both enhancements 
originally being part of a single proposal.

So, without changing the RFC1323 (receiver) semantics, you get:
    Reduced fidelity on the OWD signal (but at least proper knowledge
      of the timestamp timer granularity)
    No improvement during loss episodes
    Simple RTO calculation as the input is already a worst-case estimate 
      RTT (including delayed ACK + processing delays, not just network 
      delay).

Best regards,
   Richard