Subject: Re: [tcpm] WGLC for draft-ietf-tcpm-1323bis

From: Pasi Sarolahti <[email-masked]>
Date: Wed, 15 May 2013 13:29:38 +0300

On May 15, 2013, at 4:42 AM, Mark Allman <[email-masked]> wrote:
> 
>    So, I don't see why 1323bis shouldn't say "if you use this RTTM we
>    codify then you make the following adjustments to TCP's standard RTO
>    procedure [...insert step (5) here...].  To me that is well
>    understood and straightforward.

When you update the RTO RFC the next time, you should remember to include your step (5) :-) RTO calculation is orthogonal to the method of collecting RTT samples, so in my mind it would be better to keep the algorithm in one place. Multiple RTT samples per window can also be taken without timestamps.

The modification seems straightforward enough, but some practical experience might be useful before writing it down in a standards track document. Timestamps have been around for quite a while, so what do implementations currently do about the RTO? Linux has its own tweaks to address this, but how about other implementations? Are they just fine with the current algorithm (maybe because RTO is often clamped to 1 sec)?

- Pasi

---

From: Mark Allman <[email-masked]>
Date: Wed, 15 May 2013 07:49:51 -0400

Pasi-

> When you update the RTO RFC the next time, you should remember to
> include your step (5) :-) RTO calculation is orthogonal to the method
> of collecting RTT samples, so in my mind it would be better to keep
> the algorithm in one place. Multiple RTT samples per window can also
> be taken without timestamps. 

Well, maybe ... 

  - The above is certainly fair pushback.

  - Given that taking multiple RTT samples per RTT doesn't buy anything
    for the RTO estimator codified in RFC6298 I am not sure why we'd
    specify what to do in that case (beyond some "address it" statement,
    which is already there) in a bis of that document.

    But, OK, OK.... we could argue about that for a long time over
    beers.

  - It seems to me that if you are saying take multiple RTT samples per
    RTT then it should be incumbent upon you to address this point and
    not to simply kick it off to somewhere else.  And, the RTTM
    algorithm is saying exactly this: take more RTT samples per RTT.

> The modification seems straightforward enough, but some practical
> experience might be useful before writing it down in a standards track
> document.

That sort of generic pushback doesn't hold a lot of water with me.

  - What "practical experience" do you want?  We have a very long
    history with the alpha and beta gains specified in RFC6298 (and
    previous versions and implementations before that).  All I did was
    to apply those same gains to get the same history window for the
    case when multiple RTT samples are collected per RTT.  Its just
    straightforward math to get to roughly the standardized behavior
    from the estimator.  And, we have experience with that behavior.
    So, I am not sure what you are looking for here.

  - What do you suggest we do?  You say that because we haven't used
    this gain adjustment we can't write it down in a standards track
    document.  But, then what can we write down?  Any scheme we write
    down will have the same issue (e.g., the average the RTTs and feed
    the average to the estimator notion David sketched).  And, writing
    down nothing seems at least dubious when it is so straightforward to
    adjust the gains to make the estimator's behavior consistent with
    long-standing practice and the standard.

We have a standard and long-standing TCP RTO estimator.  All I am
suggesting is that we apply it here.  I have a hard time understanding
why that seems like such a big deal.

allman

---

From: Michael Welzl <[email-masked]>
Date: Wed, 15 May 2013 15:26:46 +0200

>    So, I am not sure what you are looking for here.

I may be wrong, but I think that the behavior with more samples + changed alpha and beta as you suggested can be significantly different.

By averaging over a larger amount of samples, the significance of outliers becomes diminished, even if we average over an accordingly longer time period. I played with Excel a bit to confirm this, and have calculated the following example: R=200ms, and alpha, beta as you suggest / from RFC6298 (1/8 / number_of_samples, 1/4 / number_of_samples). Let's assume that the connection has just begun, i.e. RTTVAR is 100ms. Say that, after 10 RTTs, a delay spike happens, seen by only 1 packet, giving it an RTT of 300ms.

Case 1, 1 sample per RTT: in the next RTT (RTT#11), the RTO is at 329ms (followed by 311 and 296 for the next RTTs).

Case 2, 2 samples per RTT: to have the same situation (but still assuming only one packet is affected), 300ms sample has to be packet #20. Then, the samples of the next packets (one taken every half RTT) are 280, 274, 268, 263, 258, 253. That is, the RTOs in RTT #11, are 280 / 274, in RTT #12 they are 268/263, in RTT #13 they are 258/253.

Using the average of the two samples in case 2, this means that you have 52ms (about 1/4 base RTT) difference in the 1st RTT after the delay spike, followed by 46ms in the 2nd and 41 in the 3rd. Using these specific values, this difference may or may not seem truly significant, but the effect does get amplified when using a larger number of samples...

I have a tendency to always make mistakes in such emails, so chances are, I'm wrong for some reason... in this case, very sorry for the noise.

Cheers,
Michael

---

From: Pasi Sarolahti <[email-masked]>
Date: Wed, 15 May 2013 16:58:38 +0300

>    So, I am not sure what you are looking for here.

Well, if I'm not mistaken, we have long experience using RFC1323 together with RFC6298 as it is currently specified (the default FreeBSD behavior, I suppose, but I haven't followed BSD closely) and nothing has collapsed. But I fully agree with your earlier mail that rfc1323bis should refer to Reiner's paper and Sally's mail about the problems of RTTM and RTO calculation. On the other hand, I think Reiner's paper also proposed a slightly different fix.

I don't think the adjusted RTO algorithm would be "such a big deal", but I'm a bit hesitant adding new RTO algorithm specifications in the context of rfc1323bis, even if it was a minor adjustment to follow the original intent of RFC6298.

(Just a voice of one individual. If people support including the fixed RTO calculation, so be it.)

- Pasi

---

From: David Borman <[email-masked]>
Date: Wed, 15 May 2013 14:16:45 +0000

> (Just a voice of one individual. If people support including the fixed RTO calculation, so be it.)

Ok, given the recent discussion, I propose the following:

1) Add an appendix based on Mark's message about modifying the RFC6298 algorithm.
2) Replace the examples in section 3.4 with a reference to the new appendix.

As long as 3.4 points out there is a problem of feeding more RTTM samples per RTT into an unmodified RTO estimator, we've achieved the primary objective.  Examples of how to address that has always been secondary, and moving it to an appendix explicitly makes it not part of the spec.

			-David
 
> 
> - Pasi
> 
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

----------------------------------------------------------------------
The information contained in this transmission may be confidential. Any disclosure, copying, or further distribution of confidential information is not permitted unless such privilege is explicitly granted in writing by Quantum. Quantum reserves the right to have electronic communications, including email and attachments, sent across its networks filtered through anti virus and spam software programs and retain such messages in order to comply with applicable data security and retention requirements. Quantum is not responsible for the proper and complete transmission of the substance of this communication or for any delay in its receipt.

---

From: Joe Touch <[email-masked]>
Date: Wed, 15 May 2013 07:28:48 -0700

> 1) Add an appendix based on Mark's message about modifying the RFC6298 algorithm.

On the issue, maybe, but on the new code Mark provides, definitely no.

As a current standards-track doc, changes should reflect current 
experience and deployment experience, not intended future practice.

> As long as 3.4 points out there is a problem of feeding more RTTM
> samples per RTT into an unmodified RTO estimator, we've achieved the
> primary objective.

That's useful to point out because it's current experience.

> Examples of how to address that has always been
> secondary, and moving it to an appendix explicitly makes it not part of
> the spec.

Those should be part of either another document or a later update, once 
there is WG consensus on a specific solution.

Joe

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Wed, 15 May 2013 15:23:23 +0000

Hi,

I'd like to point to RFC6298, section 3:

   Traditionally, TCP implementations have taken one RTT measurement at
   a time (typically, once per RTT).  However, when using the timestamp
   option, each ACK can be used as an RTT sample.  RFC 1323 [JBB92]
   suggests that TCP connections utilizing large congestion windows
   should take many RTT samples per window of data to avoid aliasing
   effects in the estimated RTT.  A TCP implementation MUST take at
   least one RTT measurement per RTT (unless that is not possible per
   Karn's algorithm).

   For fairly modest congestion window sizes, research suggests that
   timing each segment does not lead to a better RTT estimator [AP99].
   Additionally, when multiple samples are taken per RTT, the alpha and
   beta defined in Section 2 may keep an inadequate RTT history.  A
   method for changing these constants is currently an open research
   question.

So, it is in fact pointed out in the proper document (that is concerned with calculating the RTO).

However, I looked up recent BSD and Linux:

FreeBSD 9.1 does *NOT* modify alpha/beta when RTT samples are performed on every ACK.

Linux 3.2.4 has apparently removed the earlier variable rttvar update stuff, and is updating sRTT once per window, and RTTvar with the maximum variance observed in the last window (instead of an average/mean/last variance).

Perhaps Kevin or others can comment on other widely used stacks...

To get back to 1323bis: As I have stated, that document concerns itself with obtaining more frequent RTT samples; there used to be an implicit assumption, that the only consumer of that signal will be the RTO calculation (that much has been true when 1323 was published), but nowadays there are many more (potential) uses for timely RTT measurements (cubic - the lack of a security mechanism in 1323TS turned out to be a big problem; chirp, ledbat). Also, there are other uses for timestamps themselves (eifel, passive measurements).

As there is no clear guidance in RFC6298 as to how to deal with more samples per window, it might be prudent to include some text in 1323bis now. My personal preference would be to put that into an appendix though...

One last observation: 

RTO calculations are often performed using fixed-comma (integer) arithmetics.

High timestamp clock granularities and simple adjustments of the alpha/beta factors as suggested might not work properly, when the window sizes grow too large (ie. >16 segments), and sRTT would no longer be updated at all... Based on the papers Mark referred to, that might not be too much of a problem, further masking this issue, but nevertheless a simple "fix" might prove to be worse than the suggestion by David early on, to use only a single RTT sample per window (even though that was commented very strongly by Mark).

Regards,

Richard Scheffenegger

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Joe Touch
> Sent: Mittwoch, 15. Mai 2013 16:29
> To: David Borman
> Cc: tcpm ([email-masked]); <[email-masked]>
> Subject: Re: [tcpm] WGLC for draft-ietf-tcpm-1323bis
> 
> 
> 
> > Examples of how to address that has always been secondary, and moving
> > it to an appendix explicitly makes it not part of the spec.
>

---

From: Mark Allman <[email-masked]>
Date: Wed, 15 May 2013 12:32:20 -0400

Michael-

> I may be wrong, but I think that the behavior with more samples +
> changed alpha and beta as you suggested can be significantly
> different. 

Your comparison is not sound because you are conflating two things: (1)
adjusted gains and (2) multiple samples.

So, yes, you find the RTOs that pop out are different.  E.g., in the
case of one sample per RTT does that get the max sample or the min
sample during the given RTT?  It matters to the result.  Clearly taking
1 and taking N are going to be different.  The argument behind RTTM is
that this difference is important, i.e., more samples gives you a more
accurate assessment of the RTT process by taking into account the
fluctuations that can be observed within an RTT.  Our paper / data
suggests this is wrong.

All I am saying is that the length of the history kept in the EWMA
should be roughly the same regardless of the number of samples taken.

If you do the same test and keep a constant RTT for each RTT you'll see
that the RTOs that pop out are roughly the same.

I.e., pick a series of RTT measurements for each round of the transfer.
I dunno, say 0.1, 0.11, 0.09, 0.85, 0.13 for the first five RTTs.  Now,
use the RFC6298 gains and run each of those samples through the RTO
calculation once.  Call that X.  Now, pick some number of samples N per
RTT and run each of the above numbers through the RTO calculation N
times, but with alpha/N and beta/N as the gains (i.e., you are
simulating multiple samples that are all the same; that isn't realistic,
but shows the math).  Call the last RTO Y.  What you'll find is that the
X and Y are pretty similar.  I.e., we have adjusted the gains to keep
roughly the same amount of history in the overall RTO.  And, that is the
point.

allman

---

From: Joe Touch <[email-masked]>
Date: Wed, 15 May 2013 09:32:42 -0700

On 5/15/2013 8:23 AM, Scheffenegger, Richard wrote:
> As there is no clear guidance in RFC6298 as to how to deal with more
> samples per window, it might be prudent to include some text in 1323bis
> now. My personal preference would be to put that into an appendix though...

I agree that it's important to point out that this is an important known 
issue, but I don't think we should include suggested solutions that we 
come up with on-the-fly right now.

Joe

---

From: Yuchung Cheng <[email-masked]>
Date: Wed, 15 May 2013 09:38:21 -0700

On Wed, May 15, 2013 at 8:23 AM, Scheffenegger, Richard <[email-masked]>wrote:

> Linux 3.2.4 has apparently removed the earlier variable rttvar update
> stuff, and is updating sRTT once per window, and RTTvar with the maximum
> variance observed in the last window (instead of an average/mean/last
> variance).
>
That's simply wrong. I have been working on that piece of code in Linux
everyday since this year. Linux RTO formula is different than the RFC but
it's not what you described. I am happy to point out the details but don't
want to derail the thread.

> To get back to 1323bis: As I have stated, that document concerns itself
> with obtaining more frequent RTT samples; there used to be an implicit
> assumption, that the only consumer of that signal will be the RTO
> calculation (that much has been true when 1323 was published), but nowadays
> there are many more (potential) uses for timely RTT measurements (cubic -
> the lack of a security mechanism in 1323TS turned out to be a big problem;
> chirp, ledbat). Also, there are other uses for timestamps themselves
> (eifel, passive measurements).
>
> High timestamp clock granularities and simple adjustments of the
> alpha/beta factors as suggested might not work properly, when the window
> sizes grow too large (ie. >16 segments), and sRTT would no longer be
> updated at all... Based on the papers Mark referred to, that might not be
> too much of a problem, further masking this issue, but nevertheless a
> simple "fix" might prove to be worse than the suggestion by David early on,
> to use only a single RTT sample per window (even though that was commented
> very strongly by Mark).
>
>
>
> Regards,
>
> Richard Scheffenegger
>
>

---

From: Yuchung Cheng <[email-masked]>
Date: Wed, 15 May 2013 09:45:55 -0700

> I don't think the adjusted RTO algorithm would be "such a big deal", but I'm a bit hesitant adding new RTO algorithm specifications in the context of rfc1323bis, even if it was a minor adjustment t…

+1

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Wed, 15 May 2013 17:18:41 +0000

Mark, 

do you know of any stack that is deployed and uses the suggested adjustable gain? You mentioned that there are stacks doing this...

I'm not questioning the math (and that the numbers aren't exactly the same is fine), but as Joe mentioned, the bar for a standards track document needs to be demonstrated deployment...

If there isn't a widely deployed stack using the adjusted gains, we are probably left with "use only one sample per RTT to update RTO" as that is widely deployed in the linux stack...

Also, that would not change the calculation of 6298 a bit, which seems to be the other requirement...

Richard Scheffenegger

---

From: Mark Allman <[email-masked]>
Date: Wed, 15 May 2013 13:29:19 -0400

[answering sporadically between meetings ... I see a large number of
 messages ... ugh]

> do you know of any stack that is deployed and uses the suggested
> adjustable gain? You mentioned that there are stacks doing this... 

I do not.  But, I have lost the bubble on this.

> I'm not questioning the math (and that the numbers aren't exactly the
> same is fine), but as Joe mentioned, the bar for a standards track
> document needs to be demonstrated deployment... 

Well, that is clearly not correct or we'd never publish anything new on
the standard track.

> If there isn't a widely deployed stack using the adjusted gains, we
> are probably left with "use only one sample per RTT to update RTO" as
> that is widely deployed in the linux stack... 

Well, the problem is "use only one RTT to update the RTO" is isomorphic
with "SHOULD NOT use RTTM".  I am personally fine with deprecating RTTM
completely (note: RTTM is not the same as the TS option and it is not
the same as PAWS).  In my view RTTM does not add anything and so we can
just say that and move on.  But, my sense of the WG is that making it a
MAY is probably more likely to obtain consensus.  So, if you MAY use
RTTM then we're left with this question of how to actually update the
RTO with the multiple samples.  There seem to be two viable choices:

(1) Make no comment and let all samples flow through the RFC6298
    estimator.  We know this produces an aggressive RTO that keeps less
    history and ends up waiting a bit less time for needed RTOs and also
    tripped a whole lot more spurious RTOs.

(2) We recommend adjusting the gains to adapt the history to the spirit
    of TCP's standard RTO algorithm.

allman

---

From: Joe Touch <[email-masked]>
Date: Wed, 15 May 2013 10:39:08 -0700

>> >document needs to be demonstrated deployment...

FWIW, I actually said "As a current standards-track doc, changes should 
reflect current experience and deployment experience, not intended 
future practice."

> Well, that is clearly not correct or we'd never publish anything new on

I agree, but we shouldn't be introducing new mechanism during WGLC either.

Further, this isn't a new standards-track doc; it's an update to an 
existing one. The bar for that ought to be a little higher, and ought to 
include some experience.

Joe

---

From: Mark Allman <[email-masked]>
Date: Wed, 15 May 2013 14:05:39 -0400

> I agree, but we shouldn't be introducing new mechanism during WGLC

We shouldn't be havin' an effin' WGLC.

allman

---

From: Yuchung Cheng <[email-masked]>
Date: Wed, 15 May 2013 11:14:58 -0700

On Wed, May 15, 2013 at 10:29 AM, Mark Allman <[email-masked]> wrote:
[snip]
+1

+1

---

From: Mark Allman <[email-masked]>
Date: Wed, 15 May 2013 14:19:55 -0400

> primary objective.  Examples of how to address that has always been
> secondary, and moving it to an appendix explicitly makes it not part
> of the spec. 

I can abide.

And, shut up. :-)

allman

---

From: Michael Welzl <[email-masked]>
Date: Thu, 16 May 2013 13:32:25 +0200

Hi,

Sorry everyone for reacting to this one so late, I think it's a bit off track of the main discussion now - I'm answering anyway:

> adjusted gains and (2) multiple samples.

No, that was in fact the point I was trying to make...

> fluctuations that can be observed within an RTT.  Our paper / data

I should look at this paper again, I'm sure I read it long ago. Anyway, it seems logical that using more data cannot be a bad thing - I do agree with the hand-wavery in the original RFC1323 in that respect. If that turns out to lead to poor behavior, it tells us that the process behind it (using EWMA, and EWMA+4*EWMA(Var)) is probably not so great.

But that's a different story altogether, I don't want to suggest updating the RTO calculation here... just saying that this notion of "using less information is better" is kinda odd.

> should be roughly the same regardless of the number of samples taken.

Yeah, I got the idea.

> roughly the same amount of history in the overall RTO.  And, that is the

I see what you're trying to achieve, but I don't think the method works out that well. With init. R=0.1 and assuming that the first measurement was already taken, i.e. SRTT=0.1, RTTVAR=0.05, RTO=0.3 N = 1, after 5 RTTs using the RTT values you give above, with N=1 the RTO is 0,87, and with N=3 it is 0,81. I did another calculation where I assumed that these values appear after the RTO has stabilized, i.e. initial RTTVAR=0. Then, with N=1, you get 0,822438965 and with N=3 you get 0,752924008. I don't know if that would qualify to be "roughly the same". I think the problem is that a EWMA with halved alpha isn't quite the same as doubling an SMA, and the error is probably magnified by the 4*RTTVAR.

So this was only the favorable case of equal values in every RTT and just some random numbers you provided for RTT samples... who knows how far off things could get in reality. Essentially, the one-sample-per-RTT estimation assumes that it doesn't matter whatever sample you pick within an RTT. Due to the abs() function in RTO calculation, any deviation from SRTT will make the RTO grow, and if you pick one sample in an RTT, chances are that this gives you more variation than the average of all the measurements you could take in an RTT, and hence a more conservative -- "better working" -- RTO. Duh! :-)  Averaging over that is going to give you something else.

So I don't have the impression that it's valid to say that this is roughly the same, but by all means, let's do it! It's going to flatten out some outliers, and hence make the RTO more aggressive, great! Maybe then it's even going to fire spuriously more often - all the better, then we have a reason to fix it   :-)))

Cheers,
Michael

PS: seriously, I do believe that the problem with the RTO is in the trade-off that it entails: too short = horrible things happen. too long = inefficient. What we should have is a more aggressive timer for retransmitting with less serious consequences, and then we can make the RTO as conservative as we wish. TLP has this timer, I guess I'm arguing for applying it in general, not just to the last 2 packets of a window or so.

PPS: just out of historical curiosity, whatever was the rationale for the number 4 in "+4*RTTAVR"? Was it the number the dice showed?

---

From: Mark Allman <[email-masked]>
Date: Thu, 16 May 2013 08:12:02 -0400

> > Your comparison is not sound because you are conflating two things:
> > (1) adjusted gains and (2) multiple samples.
> 

What I am saying is that there are two questions---one of whether
running multiple samples per RTT helps and one of how long the history
should be---here.  You are trying to answer both of them at the same
time and I am focused in this discussion only on one of the
questions---keeping the history similar irrespective of the number of
RTTs.  So, we're talking a bit of apples and oranges here.

> just saying that this notion of "using less information is better" is
> kinda odd. 

Re: multiple samples per RTT: it isn't that you come up with a better
estimator with one sample, it is that you do not come up with a better
estimator with multiple samples.  They are similar estimators.  (And,
one takes more work, has more on-the-wire overhead, etc.  So, the
"better" is not in terms of the result, but result-for-the-effort.)

> > I.e., pick a series of RTT measurements for each round of the
> > transfer.  I dunno, say 0.1, 0.11, 0.09, 0.85, 0.13 for the first
> > five RTTs.  Now, use the RFC6298 gains and run each of those samples
> > through the RTO calculation once.  Call that X.  Now, pick some
> > number of samples N per RTT and run each of the above numbers
> > through the RTO calculation N times, but with alpha/N and beta/N as
> > the gains (i.e., you are simulating multiple samples that are all
> > the same; that isn't realistic, but shows the math).  Call the last
> > RTO Y.  What you'll find is that the X and Y are pretty similar.
> > I.e., we have adjusted the gains to keep roughly the same amount of
> > history in the overall RTO.  And, that is the point.
> 
> I see what you're trying to achieve, but I don't think the method
> works out that well. 

It does.  I have done the calculations.  I didn't check your work, but
two things ...

  - The history applies to the EWMAs.  So, doing the entire RTO
    calculation makes things look a little worse because the discrepancy
    in the RTTVAR term is multiplied (as you noted).

  - I think that 0.87 & 0.81 (your numbers) are in fact pretty close.
    That is 60msec difference on a number that exceeds 800msec.  I never
    claimed these things were *exactly* equivalent.  But, just chopping
    the gain by N is *approximately* right.  I am sure if you wanted to
    have the samples from the present RTT count the *exact* same in
    terms of history irrespective of the number of samples taken then
    you could sit down with a pencil and figure out how to adjust the
    gain to make that happen.  And, hey, maybe that is a good exercise.
    But, to me the simple division of the gain by N looks pretty close
    and so it doesn't seem terribly important to make things more
    difficult by making it more complex.  But, eh, if you want to do it,
    we can have the discussion.

> So this was only the favorable case of equal values in every RTT and
> just some random numbers you provided for RTT samples... who knows how
> far off things could get in reality.

Again, the "favorable case" is meant to illustrate that the history is
roughly the same the regardless of the number of RTT samples if you make
a straightforward tweak to the gains.  The illustration was meant to
show you that the math works as I have characterized it.

The illustration is absolutely far too simplistic to comment on the
efficacy of the resulting RTOs.  We have published an empirical study to
deal with that question.

> PS: seriously, I do believe that the problem with the RTO is in the
> trade-off that it entails: too short = horrible things happen. too
> long = inefficient.

Right.  And, the tradeoff is fundamental.  We could not find a sweet
spot that optimized for both less waiting time and fewer spurious
retransmits.

> PPS: just out of historical curiosity, whatever was the rationale for
> the number 4 in "+4*RTTAVR"? Was it the number the dice showed? 

I think this is discussed in Jacobson's 1988 paper.  But, I cannot at
the moment conjure the rationale.

allman

---

From: Michael Welzl <[email-masked]>
Date: Thu, 16 May 2013 14:51:10 +0200

Hi,

> RTTs.  So, we're talking a bit of apples and oranges here.

well, my argument is that your focus leaves out an important aspect of reality - the possibility of e.g. two packets in one RTT experiencing a different RTT.
It's like saying "if we assume that this doesn't happen, the values are roughly the same". Fine, as a model, where we can make all kinds of assumptions all the time, but you started out saying that this gives you roughly the same numbers, and I'm saying that only in your perfect model world, it would (plus there's my less relevant (because less significant) criticism of the actual process below).

> "better" is not in terms of the result, but result-for-the-effort.)

ok, I can see that

>    in the RTTVAR term is multiplied (as you noted).

Indeed! I also looked at the pure EWMA values, which indeed (as suspected) look better, but that's just not what it is.

>    That is 60msec difference on a number that exceeds 800msec.  I never

Pretty close for N=3, which was a random number I picked, and for the RTT values you've given.

>    we can have the discussion.

Nah, let's go with what you suggest and see what happens. I don't buy the "roughly similar" bit mainly because of the sampling issue above, but who cares. Small is beautiful.

> spot that optimized for both less waiting time and fewer spurious

... and you left out my suggestion to incorporate a second, more aggressive timer, in TLP style. What do you think, wouldn't that be good?
(yes I know - it's research... do the work first... but well, the Google folks have started, with TLP)

Cheers,
Michael

---

From: Mark Allman <[email-masked]>
Date: Thu, 16 May 2013 09:12:47 -0400

> well, my argument is that your focus leaves out an important aspect of
> reality - the possibility of e.g. two packets in one RTT experiencing
> a different RTT. 

No.

Not at all.

You are completely missing the point.  

Or, trying to cloud it.

We use EWMAs to incorporate both the new sample at some weight and also
some sense of history of what the samples have been in the past.  

Consider three cases ... 

(1) In RTT#N using one sample per RTT the RTT sample will contribute 1/8
    to the SRTT EWMA (alpha in RFC6298).  And, the SRTT history from
    previous rounds will contribute 7/8.

(2) Now consider taking two samples in RTT#N.  Each sample will
    contribute 1/8 to the SRTT EWMA.  So, the total contribution from
    RTT#N is roughly 1/4.  And, the historical contribution is 3/4.

(3) Now consider taking two samples in RTT#N and also adjusting the
    gains to take this into account.  So, we use alpha'=(1/8)/2=1/16.
    So, each of the samples contributes 1/16 to the SRTT.  Together the
    two samples contribute about 1/8 to the SRTT.  And, the historical
    SRTT contributes about 7/8.

All I am saying is that case (2) ages the history out more rapidly and
that (1) and (3) provide roughly the same history in the EWMA.

I am not claiming that the math dictates the RTO (or the SRTT or the
RTTVAR) is the same if you take multiple real-world RTT samples per
RTT.  What I am claiming is that with the gain adjustment we retain
roughly the same amount of history (in terms of time span) in the
estimator regardless of the number of samples we take per RTT because
each round is weighted roughly the same.

None of that says the RTO will be the same with multiple samples.  None
of that means I am living in some fantasy world.  None of that means I
don't appreciate that multiple samples might change the RTO.  All it
says is that a simple adjustment to the gains means that the
contribution to the EWMA is roughly the same per RTT irrespective of the
number of samples.

allman

---

From: Michael Welzl <[email-masked]>
Date: Thu, 16 May 2013 15:31:44 +0200

> You are completely missing the point.  

Missing, perhaps, trying to cloud it, definitely not. Why in the world would I?

>    SRTT contributes about 7/8.

All of that is correct, and I didn't question any of that;

> that (1) and (3) provide roughly the same history in the EWMA.

but my interpretation of what you said (see below) is that the result (which is the RTO) is roughly the same. What you say here is more limiting. Sorry, I didn't want to put words in your mouth. Anyway, I'm concerned about the case where the two samples in (3) are different, which does not at all seem unlikely to me. Then an RTO derived from (1) and (3) can become quite different too.

> contribution to the EWMA is roughly the same per RTT irrespective of the

This is what you now say  :-)   in your original email, you said:

  That isn't **exactly** the same as RFC6298, but it is absolutely
   **almost** the same.  And, it is absolutely within the spirit of
   TCP's standardized RTO.  I.e., we keep about the same amount of
   history and we weight the recent RTT samples with about the same
   weight.

Granted, "That" may be various things, but I thought it refers to the overall calculation and, with it, the result you get from it, which is the RTO. From this text and your recommendation for 1323bis, I took it as "the resulting RTO is roughly the same". Now you limit that statement, but that also weakens the argument for changing 1323bis in this way, I'd say.

Anyway, as I said, you have my vote for changing it, because it makes the RTO less aggressive by smoothing out stuff. Within the limits of avoiding to change the actual calculation method, I do see it as a reasonable approach.

Cheers,
Michael

---

From: Mark Allman <[email-masked]>
Date: Thu, 16 May 2013 10:04:37 -0400

> but my interpretation of what you said (see below) is that the result
> (which is the RTO) is roughly the same. 

No- I did not mean to say the RTO would be the same.  I don't think I
said that---even in re-reading what you pasted in.  If what I wrote was
unclear to you, sorry.  All I am getting at is we should include roughly
the same amount of history in the SRTT and RTTVAR (and hence the RTO).

> Then an RTO derived from
> (1) and (3) can become quite different too. 

Sure.  And, that is the original point of RTTM.  I.e., the claim was
that by taking multiple samples the resulting RTO would be better.  Our
data suggests this hypothesis is not true.

(And, note, that even two RTOs that are "different" might work
similarly.  E.g., say one is X and one is 2X.  If neither ever triggers
a retransmission then we don't really see a difference in the
performance even if the numbers are in fact different.  But, when we do
see retransmissions---needed or not---then we don't see much difference
in the estimators.)

allman

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Thu, 16 May 2013 15:06:39 +0000

Hi,

> 
> Sure.  And, that is the original point of RTTM.  I.e., the claim was that
> by taking multiple samples the resulting RTO would be better.  Our data
> suggests this hypothesis is not true.

Within the confines of the specified linear EWMA calculation given in RFC6298. (Which is also the implicit assumption in this discussion, correct?).

> "better" is not in terms of the result, but result-for-the-effort.)

I looked at a couple of papers about TCP/RTO in the past days, to refresh my memory. 

I think the relevance for 1323bis is limited to stating the fact, that RFC6298 RTO constants work for 1 sample per RTT, and when taking more samples per RTT, either the RTO calculation method needs to be adjusted (huge volume of papers on that, none at the IETF), or the RFC6298 gains need to fixed to keep approximately similar history (current text in the Appendix of 1323bis), or throw out most of the additional information gained by the RTT samples, and run the legacy RTO calculation (e.g. what Linux does).

Taking a step back, I think since the turn of the century, additional mechanisms that unwind the "catastrophic" effects of a spurious RTO (F-RTO, Eifel Response, DSACK) are widely deployed, thus a RTO on the more aggressive side (such as not adjusting the gains and sampling nearly every segment) no longer have as dire consequences as they used to have.

This is also, why I would suggest to disentangle in 1323bis the RTT sampling, from the specific RTO calculation method, to allow a higher frequency RTTM signal become available for other mechanisms as well.

Best regards,
  Richard