Subject: Re: [tcpm] WGLC for draft-ietf-tcpm-1323bis

From: Mark Allman <[email-masked]>
Date: Fri, 10 May 2013 11:37:00 -0400

I adamantly do not support 1323bis being published in its current
state.

  - My overall notion is that bis documents should take into account
    what has been learned since the initial publication.  Sometimes
    we learn the documents are unclear.  Sometimes we learn about
    refinements to techniques that can make things better in some
    fashion.  Sometimes we learn that additional mechanisms would be
    handy.  Sometimes we learn that what we previously wrote is just
    wrong.  We should take all this into account when we update
    documents.

  - 1323bis does not take into account much of what has been learned
    since the publication of 1323.  Sure, it fixes a few small
    things, but it just completely ignores the big things.  There
    are forests.  And, there are trees.  And, 1323bis is concerned
    with bark.

  - 1.1, item (1): This does not say *why* the window is a
    "fundamental performance problem".  I mean, I know it is, but it
    seems the document should at least sketch and cite some notion
    that the maximum window and the RTT constrain the performance.

  - 1.1, item (2): Citing 6675 is fine.  That is how to use SACKs.
    But, seemingly if you want to cite *SACK* you should also cite
    2018 and 2883 as these define SACK.

  - 1.1, item (3): This is not a "fundamental performance problem".
    This should be removed from the list (see below).

  - 1.3 states "it is not expected that most TCP implementations
    will properly handle unknown options".  Why do we have to frame
    this as an expectation when there is empirical evidence that
    this is the case?  Again, bis documents should be open to what
    we have *learned*.

    At least these two papers speak directly to the issue:

      Alberto Medina, Mark Allman, Sally Floyd.  Measuring
      Interactions Between Transport Protocols and Middleboxes.  ACM
      SIGCOMM/USENIX Internet Measurement Conference.  October 2004.

      Alberto Medina, Mark Allman, Sally Floyd.  Measuring the
      Evolution of Transport Protocols in the Internet.  ACM
      Computer Communication Review, 35(2), April 2005.

    There are probably others.  (It is not my intention to flog my
    papers, I just readily know of them.)

  - 1.3: "We recognize there is a tradeoff between the bandwidth
    saved by reducing unnecessary retransmission timeouts, and the
    extra header bandwidth used by this option.  It is required that
    this TCP option will be sent on non-<SYN> segments only after an
    exchange of options on the <SYN> segments has indicated that
    both sides understand this extension."

    Two issues:

      - First, you **never** establish that RTTM "reduc[es]
        unnecessary retransmission timeouts".  You do not cite
        anything that says that.  You do not present data that says
        that.  You are simply waving your hands and hoping that more
        RTT samples means a better RTO and that leads to fewer
        spurious retransmissions.

      - Second, I find the above ambiguous.  When I read it I
        wondered if this was wiggle room to let TCPs not send TS on
        all segments after the 3WHS.  Later in the document you
        unequivocally say that TS must be sent on all segments after
        negotiated.  I think you want "will be sent on ALL
        non-<SYN>" in the above to leave no doubt what you mean.

  - 2.1--2.3: The whole discussion of the WS option is a bit hard to
    follow.  

    The first sentence of the section says: "The window scale
    extension expands the definition of the TCP window to 32 bits
    and then uses a scale factor to carry this 32-bit value in the
    16-bit Window field of the TCP header".

    That is not true because at most the option dictates you can
    scale the window by at most 15 bits---or to an overall window
    size of 31 bits.

    As much is admitted later.

    And, then it is further refined such that the window must
    actually be *less* than 2^31 and so the max WS factor should be
    14 to yield a max window of 2^30.

    The discussion of why this is and all is fine and I have no
    problem with it.  But, don't say the window can be 32 bits or 31
    bits when it can't be.  It just leads to confusion.

  - 2.2: It is confusing that you introduce a "scale factor" that is
    different from the encoded "window scale".  I.e., a "scale
    factor" of 1 seemingly means the window size is not scaled at
    all.  But, this requires a "window scale" of zero.  The
    discussion could just be cleaned up to get rid of the "scale
    factor" notion.  Or, **at least** you could define the scale
    factor as one plus the window scale.

  - 2.2: "A Window Scale option in a segment without a SYN bit
    SHOULD be ignored."  Why is this?  Why not "MUST be ignored"?
    Is there some case where the TCP should really pay attention to
    it?  I can't think of one as things are presently defined.

  - In 2.3 you note (in a somewhat tortured way) that if the WS
    arrives as 15 then it should be assumed to be 14.  You might
    sketch why this is safe.  I.e., even if the other side is
    offering a window of 2^31 bytes you'll only use half that and so
    all will be fine.

  - 2.3: You should cite something for congestion avoidance and slow
    start.  I usually cite both Jac88 and the RFC.  But, either is
    fine.

  - 3.1: "Many TCP implementations base their RTT measurements upon
    a sample of one segment per window or less.  While this yields
    an adequate approximation to the RTT for small windows, it
    results in an unacceptably poor RTT estimate for a LFN."

    Do you have evidence of this?  We have evidence you're wrong:

      Mark Allman, Vern Paxson.  On Estimating End-to-End Network
      Path Properties.  Proceedings of the ACM SIGCOMM Technical
      Symposium, Cambridge, MA, September 1999.

    That shows that the number of samples per RTT is pretty
    immaterial to the effectiveness of the RTO.

  - 3.1: "If we look at RTT estimation as a signal processing
    problem (which it is), a data signal at some frequency, the
    packet rate, is being sampled at a lower frequency, the window
    rate.  This lower sampling frequency violates Nyquist's criteria
    and may therefore introduce "aliasing" artifacts into the
    estimated RTT [Hamming77]."

    This is hand waving.  At best.

    I buy that the RTT is a signal.  The mistake above is trying to
    tightly couple the frequency of that signal to the "packet
    rate".  If that was true then the conclusion that *any* RTT
    measurement strategy that relied only on TCP packets themselves
    would in fact violate Nyquist.  But, why should I think its
    true?  It might be true for one flow over a dumbbell where every
    packet directly and materially influences the RTT.  But, over an
    network with even a little statmux it quickly becomes clear that
    the frequency of the RTT process is not dependent on the packet
    rate of any particular source and so the above reasoning is just
    not sound.

    Or, to put it a different way: when the math and the world
    disagree, the world is right.  And the world says something
    different from what the document says:

      Mark Allman, Vern Paxson.  On Estimating End-to-End Network
      Path Properties.  Proceedings of the ACM SIGCOMM Technical
      Symposium, Cambridge, MA, September 1999.

  - 3.1: "RTT estimator".  Note, TCP does not have an "RTT
    estimator".  Scrub this from the document.  We have an "RTO
    estimator".  These are different things.  Confusing them is a
    fundamental mistake.

  - 3.1: "it becomes effectively impossible to obtain a valid RTT
    measurement".  This is FUD.  The RTO backoff [RFC6298] stays in
    place until you can take an unambiguous RTT sample.  So, this
    statement is not true unless you get to the point when you
    cannot get a single packet through the network per maximum RTO.
    And, in that case, RTTM isn't going to save you.  Please remove
    this notion.

  - 3.1: "It is vitally important to use the RTTM mechanism with big
    windows; otherwise, the door is opened to some dangerous
    instabilities due to aliasing."

    Show me.  Where is the evidence?  Cite something.  This document
    makes a whole lot of statements that seem to come from thin air.
    In contrast to when 1323 was published, we have studied this
    stuff for quite a long time now and we have a better idea how
    these things work.  So, you should be able to readily establish
    such statements or you should not make them.  (Or, **at least**
    explain how you think this "dangerous instability" is going to
    come to pass.)

    This is a whole additional level of ridiculous.  Its one thing
    to say you get a more accurate RTO with more RTT samples.  You'd
    be wrong, but at least all we're talking about is a little more
    or a little less time waiting on retransmissions.  But, to say
    this leads to *instability* is not right.  The backoff in the
    RTO backstops busted RTO (congestion) decisions.  Even if the
    RTO is quite wrong the backoff will kick in to prevent
    "dangerous instabilities".

  - End of 3.3 on RTT sample weighting factors.

    (1) The problem with the history being truncated when using RTTM
        was independently highlighted by Ludwig and Floyd.  We
        should at least have the common courtesy to cite Sally's
        note to e2e and Reiner's paper.

    (2) Instead of some nebulous reference to RTO algorithms you
        could point to the standard one, anyway.  I fully understand
        that some implementations don't use it, but at least as a
        concrete example of an algorithm that has this problem.

    (3) ARE YOU KIDDING ME?  

        This document goes through a tortured and wrong analysis
        telling me how "vitally important" RTTM is to address a
        "fundamental performance problem" of TCP over LFNs and how
        my performance is going to be "unacceptably poor" without
        it. 

        And, then we have this pesky problem that the history in the
        RTO depends on the window size when you actually do the
        thing you suggest we do (use RTTM).

        So, the first way to address this problem with the RTO is:

          "an implementation could choose to just use one sample per
          RTT to update the RTO estimator"

        Thanks for the whiplash!  Holy shit.  Really?! 

        Which one is it?  If I take this advice then aren't I in
        massive violation of Nyquist and causing dangerous
        instability to the network?

        This negates whole tracts of the damn draft.  Be wrong if
        you want, but at least have the decency to stay wrong.

  - Again, bis documents should reflect our understanding of the
    world.  RTTM does not *hurt* anything.  It just doesn't *help*
    anything either.  We should be honest enough to take this into
    account in our documents.

    RTTM should not be deprecated.  It should be a MAY.

    RTTM should not be discussed with breathless bullshit about hand
    wavy math and un-demonstrated stability issues and whatnot.  

    We should say that RTTM is absolutely within compliance of the
    spec and that it will not hurt your RTO.

    We should also say that RTTM is unlikely to help your RTO.

    We should leave it to implementers to decide if RTTM is useful
    for their purposes.

    We should specify a way to vary the gains in the standard RTO
    algorithm based on the current cwnd.

    And, we should absolutely state that there are other uses for
    the timestamp option (like Eifel, like PAWS) and there is
    nothing wrong with the *option* for that purpose.

  - 3.4, (A): Why are we discussing this in terms of the "Kth"
    segment?  Delayed ACKs per the standard is "2nd".  Why do we
    have to make the discussion in terms of some theory rather than
    in terms of what we have specified?

  - 3.2 insinuates that you should not include a timestamp on an
    RST: "TSopt MUST be sent in every non-<RST> segment", implying
    it should not be sent on an <RST> (or you'd have just said
    "every segment").  But, then 4.2 goes on to (rightly IMO)
    develop why we should include it on <RST> segments.  This
    inconsistency needs fixed.

Sorry ... there is just no way this is close to ready, IMO.

allman

---

From: Yuchung Cheng <[email-masked]>
Date: Fri, 10 May 2013 09:30:39 -0700

My feeling about the new bis is it just tries to change/add too many
things, like discussing the weighting factor in RTO formula,
middle-boxes, etc.

As a developer what I am looking for in a -bis is critical changes in
the protocol or implementations for well-identified problems. Like the
sequence check Dave identified. I don't know any existing real big
problem in the RTTM area other than some minutia corner cases. It's
always a MAY to do RTTM with TS to me. It's a why-not if regular
sample is not available due to Karn's check. In the end of the day, we
implement what makes sense on REAL networks, and listen to IETF if
they make good suggestions.

A clear and to the point -bis is more appreciated. If there is any
caveat or caution that's not intuitively clear, please back up with
scientific data.

So after these complaints, maybe we can start with a draft copy of the
minimal changes that address the absolutely critical problems?

On Fri, May 10, 2013 at 8:37 AM, Mark Allman <[email-masked]> wrote:
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm
>

---

From: Mark Allman <[email-masked]>
Date: Fri, 10 May 2013 12:59:29 -0400

My pushback ...

> always a MAY to do RTTM with TS to me.

Of course *everything* is a MAY at the end of the day.  But, when the
document says:

   It is recommended that a modern TCP stack implements and make use of
   the extensions described in this document.

I think it is saying more than "eh, if ya want".  It is saying "we
really think there is a good reason to use these things [including
RTTM]".  And, that isn't so.  Or, at least it isn't demonstrated. 

If the document is staked around a bunch of hand waves and we have
actual data that speaks to the point we should use the data.

And, we should insist that AT LEAST the document be internally
consistent and not say "we had problem P and so we advocate solution X,
but X has a little problem Y and so to get around Y just don't do X".
Just say "we don't know how to soundly solve P" and leave it at that.

allman

---

From: Joe Touch <[email-masked]>
Date: Fri, 10 May 2013 10:36:29 -0700

I'm still digging through the entire doc, but I disagree on the 
following point:

On 5/10/2013 8:37 AM, Mark Allman wrote:
>      "every segment").

"MUST in non-RST" != "MUST NOT" or even "SHOULD NOT" in RST

At best, it might imply "NOT 'MUST' in RST".

The idea with RST is that you can include it or not, and there's nothing 
that tips the balance (i.e., no SHOULD).

 > But, then 4.2 goes on to (rightly IMO)

It might be arguable that:
	- senders SHOULD include TSopt in RST

but, in order to prevent reboots from leaving stale connections around:
	- receivers MUST accept RST with or without TSopt equivalently

Joe

---

From: Mark Allman <[email-masked]>
Date: Fri, 10 May 2013 13:41:31 -0400

> At best, it might imply "NOT 'MUST' in RST".

That is why I used the words "insinuates" and "implying".  

Irrespective of whether timestamps ought to be included on RST packets,
the document should be clear on the point and consistent throughout.

allman

---

From: Joe Touch <[email-masked]>
Date: Fri, 10 May 2013 10:44:57 -0700

> the document should be clear on the point and consistent throughout.

Agreed; IMO, they cannot be required by the receiver. Doing so prevents 
cleanup or old state, and because this isn't a security mechanism, it is 
inappropriate to prevent that cleanup.

Joe

---

From: Yoshifumi Nishida <[email-masked]>
Date: Fri, 10 May 2013 11:03:39 -0700

Hello,

I've been thinking that we might want to have a "RTTM considertation" draft.
I've seen lots of valuable comments during 1323bis discussions. I'll
be disappointed if they are just put in the ML archives.

For me, it's fine to say It is recommended that a modern TCP stack
implements and make use of the extensions" in 1323bis.
Because AFAIK, it is the only standardized way to solve PAWS.

So, my personal preference is something like:
"we had problem P and so we advocate solution X, but we know it's not
a perfect solution. Please check Y for more detail"
This way, we don't have to put many stuff in the bis draft.

Thanks,
--
Yoshifumi

---

From: Mark Allman <[email-masked]>
Date: Fri, 10 May 2013 14:27:32 -0400

I think you're mashing things together that should not be mashed
together. 

> implements and make use of the extensions" in 1323bis.  Because AFAIK,
> it is the only standardized way to solve PAWS.

This doesn't exactly parse for me.  But, don't confuse timestamps and
RTTM.  Small windows (without WS) are certainly a problem.  WS solves
that.  We know that we need some sort of PAWS when using larger windows
and we have a perfectly reasonable PAWS that depends on the timestamp
option.  That is all well and fine.

But, none of that says we need recommend RTTM---which is using the
timestamp option for a whole different reason.

And, it might even be OK with me to continue to recommend RTTM if that
was even remotely well argued and took into account the things we have
learned since the publication of 1323.  But, 1323bis just parrots out
what 1323 says without any indication that we as a community have worked
on these issues in any way in the intervening 20 years.  That is
ridiculous. 

allman

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Mon, 13 May 2013 15:04:45 +0000

Hi Mark,

>     wrong.  We should take all this into account when we update

Agreed.

>     things, but it just completely ignores the big things. 

The bulk of the text was indeed carried over, without updating, 
from RFC1323 (and the stalled attempts to publish updates). I've 
started to edit these text sections to be more 
in line with actual findings since the conception of the text.

>     There are forests.  And, there are trees.  And, 1323bis is 
>     concerned with bark.

To remain in that metaphor, a tree cann't live without bark, thus bark 
is an important aspect to a forest as well :)

>     that the maximum window and the RTT constrain the performance.

Adressed.

>     2018 and 2883 as these define SACK.

Added.

>     This should be removed from the list (see below).

Done; Added a reference to the timestamp option at the mentioning of PAWS in the reliability section.

>     papers, I just readily know of them.)

Fixed the text.

>         spurious retransmissions.

Updated the text to reflect the findings since it's first inception.

>         non-<SYN>" in the above to leave no doubt what you mean.

Fixed.

>     bits when it can't be.  It just leads to confusion.

Updated all these references to 32/31 bits to 30 bits... (with the exception of the implementation hint, where a 32 bit value is still mentioned).

>     factor as one plus the window scale.

Well, that apparently hasn't led to wrong implementations; the window scale is the (integer) exponent, base-two, of the scale factor; and the scale factor is limited to powers of two (1, 2, 4, 8, .. 2^14). As a non-native speaker, I'm struggling re-writing this text only with referenced to "scale exponets", as they are much less natural to speak about than factors. But the factor simply isn't signaled...

Unless there is additional feedback, ideally with a suggestion for an updated text segment, I'd retain this part...

>     it?  I can't think of one as things are presently defined.

Fixed.

>     offering a window of 2^31 bytes you'll only use half that and so

I don't know how to remove the tortured way of describing things; I added "This extends the effective bandwidth * delay product, over which TCP can operate effectively, to 1 GiB. Paths with even higher bandwidths * delay products will not run at maximum possible performance using this mechanism." I hope that clarifies what to expect when one want'S to signal a TCP receive window > 1GiB.

>     start.  I usually cite both Jac88 and the RFC.  But, either is

Done.

Not going into Section 3 for now.

> allman

Thanks for the valuable feedback!

Incorporating your valid critic of RTTM and re-wording large sections of text will take longer than sections 1+2.

Best regards,
   Richard

---

From: Mark Allman <[email-masked]>
Date: Mon, 13 May 2013 11:38:47 -0400

Just one quick clarification ...

> I don't know how to remove the tortured way of describing things; 

The text is in terms of WS values > 14.  Just say "15" because that is
the only such value.  There are just more words that lead to more
thinking than are necessary here.

> I added "This extends the effective bandwidth * delay product, over
> which TCP can operate effectively, to 1 GiB. Paths with even higher
> bandwidths * delay products will not run at maximum possible
> performance using this mechanism." I hope that clarifies what to
> expect when one want'S to signal a TCP receive window > 1GiB.

I think I was unclear or you missed the point because that does not
address my comment.

The text says, effectively, if you receive a window scale of 15 then you
should instead use a window scale of 14 (because you previously worked
out that window scales of 15 are problematic).  I am saying that you
should note that this silent downgrading from 15 to 14 is safe even if
the endpoints are a little out-of-sync.  I.e., if the receiver thinks
the window is 2^31 and the sender thinks it is 2^30 then all is well
because half the buffer will just never be used because the sender will
not exceed a window of 2^30 bytes.  It took me a minute to work this out
and figure out that this silent reduction is OK and so it just seems
like a nice place to throw a sentence to say that this is indeed fine.

allman

---

From: John Leslie <[email-masked]>
Date: Mon, 13 May 2013 20:20:19 -0400

Mark Allman <[email-masked]> wrote:
> out that window scales of 15 are problematic).

   I am fine with forbidding the use of scale 15.

   I am uncomfortable with silently changing it to 14.

> I am saying that you should note that this silent downgrading from
> 15 to 14 is safe even if the endpoints are a little out-of-sync.

   I don't understand why we need to allow the silent change to 14;
but if we do so, we really should give an argument about why it is
safe!

--
John Leslie <[email-masked]>

---

From: Mark Allman <[email-masked]>
Date: Mon, 13 May 2013 21:22:36 -0400

> but if we do so, we really should give an argument about why it is

John-

It doesn't hurt anything.  If you tell me you have 2^31 bytes of buffer
and I only ever use 2^30 bytes of buffer then all is well.  Even without
WS you could give me a window of X and I could use X/2 for whatever
reason. 

I mean, I suppose you could make the case that if someone does something
out of spec like advertise 15 then maybe it comes from some bug and
maybe there isn't near that much buffer because the whole thing comes
From being screwed up.  Well, OK.  I mean it doesn't seem that this has
caused big issues to date (20 years down the line) and so I would think
with an explanation of it is fine.

allman

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Tue, 14 May 2013 10:53:01 +0000

Hi John,

The only potential bad effect would be a reduction in bandwidth, in the unlikely case that the delay * bandwidth product really exceeds 1 GiB (laser link to a moon station, 100 Gbit/s transatlantic link)?

So, how about a sentence like 

     "This is safe as a sender can always
      choose to use any signaled receive window only partially."

Richard Scheffenegger

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]]
> Sent: Dienstag, 14. Mai 2013 03:23
> To: John Leslie
> Cc: Scheffenegger, Richard; tcpm ([email-masked])
> Subject: Re: [tcpm] WGLC for draft-ietf-tcpm-1323bis
> 
> 
> > > The text says, effectively, if you receive a window scale of 15 then
> > > you should instead use a window scale of 14 (because you previously
> > > worked out that window scales of 15 are problematic).
> >
> out of spec like advertise 15 then maybe it comes from some bug and maybe
> there isn't near that much buffer because the whole thing comes From being
> screwed up.  Well, OK.  I mean it doesn't seem that this has caused big
> issues to date (20 years down the line) and so I would think with an
> explanation of it is fine.
> 
> allman
> 
>

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Tue, 14 May 2013 13:58:53 +0000

Hi Mark,

>     immaterial to the effectiveness of the RTO.

I'd like to point out, that the whole section 3 does not make a clear distinction between *timely measuring* RTT, and using that measured RTT for further purposes, such as arriving at a useful value for RTO.

We can also argue about the specific wording in the 2nd sentence; But it appears to me, that you read "poor RTT estimate == poor RTO estimate"?

>     estimator".  These are different things.  Confusing them is a

Correct. And IMHO 1323 is looking at RTT, but, agreed, only with the intended purpose of refining RTO (where these improvements haven't been as beneficial as originally envisioned).

I rewrote the section 3.1 entirely:

3.  TCP Timestamp Option

3.1.  Introduction

   TCP measures the round trip time (RTT), primarily for the purpose of
   arriving at a decent value for the Retransmission Timeout (RTO) timer
   interval.  Accurate and current RTT estimates are necessary to adapt
   to changing traffic conditions, while a conservative estimate of the
   RTO inveral is necessary to minimize spurious RTOs.

   When [RFC1323] was originally written, it was perceived that more
   timely and accurate RTT measurements would contribute to reducing
   spurious RTOs, while maintaining their timeliness.  At the time, RTO
   was also the only mechanism to make use of the measured RTT.  It has
   been shown, that taking more RTT samples has only a very limited
   effect to optimize RTOs [Allman99].

   This document makes a clear distinction between the round trip time
   measurement (RTTM) mechanism, and subsequent mechanisms using the RTT
   signal as input, such as RTO (see Section 3.4).

   It is important to use the timestamp option with big windows, to
   allow the use of the PAWS mechanism (see Section 4).  Furthermore,
   the option is useful for all TCP's, since it simplifies the sender
   and allows the use of additional optimizations such as Eifel
   ([RFC3522], [RFC4015]) and others.

3.2.  Timestamp Option

And made the last paragraph of 3.3 a dedicated section, for the separation between RTTM and RTO:

3.4.  Updating the RTO value

   [KL04] has highlighted the problem that an unmodified RTO
   calculation, which is updated with per-packet RTT samples, will
   truncate the path history too soon.  This can lead to an increase in
   spurious retransmissions, when the path properties vary in the order
   of a few RTTs, but a high number of RTT samples are taken on a much
   shorter timescale.

   Implementers should note that with timestamps multiple RTTMs can be
   taken per RTT.  The [RFC6298] RTO estimator has weighting factors,
   alpha and beta, based on an implicit assumption that at most one RTTM
   will be sampled per RTT.  When using multiple RTTMs per RTT to update
   the RTO estimator, the weighting factor SHOULD be decreased to take
   into account the more frequent RTTMs.

   For example, an implementation could choose to

   o  just use one sample per RTT to update the RTO estimator, or

   o  vary the gain based on the congestion window, or

   o  take an average of all the RTT measurements (and the maximum of
      the variance) received over one RTT,

   and then use that value to update the RTO estimator.  This document
   does not prescribe any particular method for modifying the RTO
   estimator.

>         note to e2e and Reiner's paper.

I tried to find Sally's comment (also trying to find the reference given in your "Using Spurious Retransmissios to Adapt the Retransmission Timeout" paper), but was unsuccessful... Do you have a link?

>     in terms of what we have specified?

fixed

>     inconsistency needs fixed.

fixed

Best regards,
   Richard

---

From: David Borman <[email-masked]>
Date: Tue, 14 May 2013 14:02:36 +0000

> Updated all these references to 32/31 bits to 30 bits... (with the exception of the implementation hint, where a 32 bit value is still mentioned).

I'd like to see what you changed to 30 bits.  Section 2.1, ok.  Section 2.3,
first bullet, ok.  I didn't see anything else that should be changed to 30 bits,
did I miss something?

>>    all.  But, this requires a "window scale" of zero.  The

No, a scale factor of 1 means a shift.cnt of zero.  In general,
a scale factor of 2^N means a shift.cnt of N.  Now that I'm thinking
about this distinction between scale factor vs. shift count:

   Please change Snd.Wind.Scale and Rcv.Wind.Scale to Snd.Wind.Shift
   and Rcv.Wind.Shift throughout the document, as these variables
   contain the shift count, not the scale factor.

>>    factor as one plus the window scale.

That's not right, the scale factor is 2^N, not 1+N.  We should make
this distinction at first mention in section 2.1, rewrite it as:

   The window scale extension expands the definition of the TCP window
   to 32 bits and then uses a scale factor to carry this 32-bit value in
   the 16-bit Window field of the TCP header (SEG.WND in RFC 793).  The
   scale factor, encoded as a shift count, is carried in a TCP option,
   Window Scale.  This option is sent only in a <SYN> segment (a segment
   with the SYN bit on), hence the window scale is fixed in each direction
   when a connection is opened.

The change is the insertion of ", encoded as a shift count,"

> Unless there is additional feedback, ideally with a suggestion for an updated text segment, I'd retain this part…

I've looked through the 1323bis-11 to see if there are places where it
uses just "window scale", but aside from Rcv.Wind.Scale and Snd.Wind.Scale,
it seems to be pretty consistent using the the term "scale factor".
Changing Wind.Scale to Wind.Shift should clean things up.

...
> I don't know how to remove the tortured way of describing things; I added "This extends the effective bandwidth * delay product, over which TCP can operate effectively, to 1 GiB. Paths with even hi…

I think the current text is fine.  Keep in mind that the
clamping of the WS value to 14 applies not only to 15, but also to
16, 17 ... 255.  For any value > 14, log the error but continue the
connection using WS=14.  What else is there to do?  Drop the connection?
Generate an ICMP Parameter Problem message?  The goal of clamping the
WS to 14 is to allow the connection to continue, even if in a crippled
state.

My other comment is that the WS is *not* the advertised window, just
the scaling of the advertised window.  If they are only advertising
a window of 32K (WS=15, window=1), the other side will calculate the
window as 16K (WS=14, window=1).  So for small window values it could
cut the throughput in half.

But the reality, as has been noted, is that WS>14 has not been a
problem for the past 20 years, so implementors have figured it out,
and I see no reason to change from the 1323 clamping of all WS>14.
The text is there to cover the bases from a standards definition
point, as opposed to leaving the behavior undefined and up to the
implementor to decide how to handle WS>14.

At most, a one-line comment after the line that clamps the WS to 14:

   This should allow TCP connections to continue, even if
   operating in a degraded state due to the incorrect WS value.

But I'm not convinced we need that, this hasn't been an area of
problems or confusion over the past 20 years, so perhaps its best
to just leave it alone without trying to explain it.

			-David Borman

----------------------------------------------------------------------
The information contained in this transmission may be confidential. Any disclosure, copying, or further distribution of confidential information is not permitted unless such privilege is explicitly granted in writing by Quantum. Quantum reserves the right to have electronic communications, including email and attachments, sent across its networks filtered through anti virus and spam software programs and retain such messages in order to comply with applicable data security and retention requirements. Quantum is not responsible for the proper and complete transmission of the substance of this communication or for any delay in its receipt.

---

From: David Borman <[email-masked]>
Date: Tue, 14 May 2013 14:21:24 +0000

Richard,

Some minor cleanup for Appendix F:

> Appendix F.  Window Retraction Example
> 
>   Consider a established TCP connection with WSCALE=7 (128 byte
>   receiver window quantization), that is running with a very small

Along with my previous message of changing to Wind.Scale to Wind.Shift
throughout the document, change the previous 2 lines to:

Consider an established TCP connection using a scale factor of
128 bytes, Snd.Wind.Shift=7 and Rcv.Wind.Shift=7, that is running
with very small

>   windows because the receiver is bottlenecked and both ends are doing
>   small reads and writes.
> 
>   Consider the ACKs coming back:
> 
>   SEG.ACK  SEG.WIN computed SND.WIN   receiver's actual window
>   1000     2       1256               1300
> 
>   The sender writes 40 bytes and receiver ACKs:
> 
>   1040     2       1296               1300
> 
>   The sender writes 5 additional bytes and the receiver has a problem.
>   Two choices:
> 
>   1045     2       1301               1300   - BEYOND BUFFER
> 
>   1045     1       1173               1300   - RETRACTED WINDOW
> 
>   This problems is completely general and can in principle happen any
>   time the sender does a write which is smaller than the window scale
>   quanta.

Change the previous 3 lines to:

This is a general problem and can happen any time the sender
does a write which is smaller than the scale factor.

> 
>   In most stacks it is at least partially obscured when the window size
>   is larger than some small number of segments because the stacks
>   prefer to announce windows that are integral numbers of segments
>   (rounded up to the next window quanta).  This plus silly window

Change the previous 2 lines to:

 prefer to announce windows that are an integral numbers of segments,
 rounded to the next scale factor.  This plus silly window

>   suppression tends to cause less frequent, larger window updates.  If
>   the window was rounded down to a segment size there is more
>   opportunity to advance it ("beyond buffer" case above) rather than

>   retracting it.

Change the previous 2 lines to:

 opportunity to advance the window, the BEYOND BUFFER case above, rather
 than retracting the window.

			-David Borman

---

From: Mark Allman <[email-masked]>
Date: Tue, 14 May 2013 10:33:07 -0400

> I'd like to point out, that the whole section 3 does not make a clear
> distinction between *timely measuring* RTT, and using that measured
> RTT for further purposes, such as arriving at a useful value for RTO. 

Correct.  1323 and 1323bis are both very sloppy in this regard.

But, standard TCP only uses RTT samples for one thing: computing an
RTO.  And, there are enough words about the RTO in 1323bis that it is
clear that the intent of RTTM is to feed a whole lot more RTT samples
into the RTO estimator.  So, trying to somehow say RTTM isn't about the
RTO is at best splitting hairs. 

> We can also argue about the specific wording in the 2nd sentence; But
> it appears to me, that you read "poor RTT estimate == poor RTO
> estimate"? 

Yes.  See above.  There is no "RTT estimate" in TCP.  I think this
equivalence is the clear intent of the document.

(Not that it shouldn't be cleaned up ... it should be ...)

Also, note, that nothing I have said precludes someone from needing many
RTT samples per RTT for something else.  E.g., it may well be needed for
delay-based congestion control.  And, if so, the timestamp option can
provide a steady stream of RTT samples.  But, it seems clear to me that
a steady stream of RTT samples is not what the aim of RTTM in 1323bis.

> I rewrote the section 3.1 entirely:

I am not going to try to piece together what you have done from email.
This is not a small change.  When you're ready, issue another draft and
we can all take a look.

> I tried to find Sally's comment (also trying to find the reference
> given in your "Using Spurious Retransmissios to Adapt the
> Retransmission Timeout" paper), but was unsuccessful... Do you have a
> link? 

I figured that'd be easy to track down, but I couldn't find it quickly.
I was wrong, it was not from end2end it was from the tcp-lw list.  But,
even with that bit of information I can't find it.  I guess you can just
forget that Sally worked this out since the Internet has! :)

allman

---

From: Mark Allman <[email-masked]>
Date: Tue, 14 May 2013 10:37:03 -0400

> a scale factor of 2^N means a shift.cnt of N.  

Yeah- OK.  I am not good at math.  But, the point stands: make the
distinction concrete in the text.  Especially for people who are bad at
math. :-)

> Keep in mind that the
> 16, 17 ... 255.

Yeah- I am bad at math.  Somehow I was mapping the WS to 4 bits and not
to 8.

> and I see no reason to change from the 1323 clamping of all WS>14.

Me either.

I just wanted a better explanation that it was indeed safe
(conservative) to clamp.

allman

---

From: Yuchung Cheng <[email-masked]>
Date: Tue, 14 May 2013 09:44:20 -0700

>    taken per RTT.  The [RFC6298] RTO estimator has weighting factors,
So is RTTM without timestamps. It occurs to me this issue belongs to
the RTO RFC 6298.

>       the variance) received over one RTT,
have the latter two designs or examples been tested in the minimal
scale. can we stop proposing designs that don't really belong to this
RFC?

IMO the editor has tendency to keep adding new things or logic that
are either untested (e.g., sack idea) or not strongly relevant.

>    does not prescribe any particular method for modifying the RTO
it's suffice to me to kill the text of three examples above and have
the last sentence.

---

From: Mark Allman <[email-masked]>
Date: Tue, 14 May 2013 14:52:28 -0400

> >    When using multiple RTTMs per RTT to update
> scale.

I think the second is fine.  The idea here is that if you take N samples
per RTT then you weight each sample such that the overall gain adds up
to the gain you would have used if you only took one sample (e.g., use
gain / N).  So, overall the EWMA history moves about like it does for
one sample / RTT.  Not exactly, but close.  So, this is just a little
math and I don't think it is a big departure from what is specified now.

I completely agree with you about the third option.  That seems pulled
out of thin air and without any sort of analysis I don't know why we'd
suggest it.

(I will note there is a contradiction in the above text ... "When using
multiple RTTMs per RTT to update the RTO estimator" [...] "just use one
sample per RTT".  I.e., the first bullet doesn't really address the
situation that was teed up.)

allman

---

From: David Borman <[email-masked]>
Date: Tue, 14 May 2013 19:50:59 +0000

> situation that was teed up.)

Ok, I wrote the original text of this section, along with the examples.
The primary issue is addressed by the text just before the above snippit,
which now reads:

  will be sampled per RTT.

Rather than leaving it with just that statement, it seemed prudent to
have at least one example to give implementors some guidance.  But just
one example didn't seem wise, as that would then become the default, and
we just wanted these to be examples, subject to change over time based
on experience.  The primary goal is to make sure that implementors don't
just blindly start injecting multiple RTTM per RTT into an RTO estimator
that has an implicit assumption of at most one RTTM per RTT.

The third example was to get a single RTTM value based on all the
received RTTM within one RTT, as opposed to just picking one and
ignoring the rest, and inject that averaged information into the
RTO estimator, which assumed only one RTTM per RTT.  It was also
the thought that it would help with getting a different number of
RTTM samples in each RTT.  You don't know a priori how many RTTM
samples there will be in any given RTT, hence you don't really know
how to properly weight them as they arrive, but it's easy to keep
a running average as they arrive.

So, for this section, the primary goal is to make sure that implementors
don't just blindly start injecting multiple RTTM samples per RTT into
the RTO estimator without accounting for the increased sample rate.
The examples are secondary to this.

If there are examples of how to address this issue, there should be
more than one example, unless there now exists one preferred method
for dealing with this issue.  Otherwise, eliminate the examples
altogether, and the implementor will have to get guidance for this
issue from other sources.

			-David Borman

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Tue, 14 May 2013 21:34:38 +0000

Hi Mark,

> forget that Sally worked this out since the Internet has! :)

I think I found it: :)

http://www.ietf.org/mail-archive/web/tcpm/current/msg02508.html

Here is the link to the diff from revision 11:

http://tools.ietf.org//rfcdiff?url1=http://tools.ietf.org/id/draft-ietf-tcpm-1323bis-11.txt&url2=http://tools.ietf.org/id/draft-ietf-tcpm-1323bis-12.txt

Richard Scheffenegger

> Sent: Dienstag, 14. Mai 2013 16:33
> To: Scheffenegger, Richard
> Cc: Pasi Sarolahti; tcpm ([email-masked])
> But, standard TCP only uses RTT samples for one thing: computing an RTO.
> And, there are enough words about the RTO in 1323bis that it is clear that
> the intent of RTTM is to feed a whole lot more RTT samples into the RTO
> estimator.  So, trying to somehow say RTTM isn't about the RTO is at best
> splitting hairs.
> 
> provide a steady stream of RTT samples.  But, it seems clear to me that a
> steady stream of RTT samples is not what the aim of RTTM in 1323bis.
> 
> This is not a small change.  When you're ready, issue another draft and we
> can all take a look.
>

---

From: Mark Allman <[email-masked]>
Date: Tue, 14 May 2013 21:42:45 -0400

> for dealing with this issue.

I think there is a preferred method: the method we have been using for
25 years.  I think you are making this too difficult.  I don't think
this is somehow a deeply mysterious problem.  Let me pop up and walk
through the situation.

(0) We have a standard RTO for TCP.  It is Van's RTO algorithm.  It was
    crudely specified in RFC1122.  We wrote a better version of it in
    RFC2988.  It was updated in RFC6298.

(1) TCP's RTO spec uses EWMAs for both SRTT and RTTVAR.  These then make
    up the RTO.  The goal of the EWMA is to lend some weight to the most
    recent RTT sample while also keeping some amount of history of
    previous samples.  The EWMAs use gains of alpha=1/8 for SRTT and
    beta=1/4 for RTTVAR.

(2) Finally, RFC6298 is for a TCP that takes one RTT sample per RTT.  If
    the gains are held constant and more samples are taken per RTT then
    the recent samples force out the history more rapidly.

(3) The RTTM algorithm suggests taking more than one RTT sample per
    RTT.  So, we end up with an RTO estimator that remembers relatively
    less and less history as the congestion window grows.

(4) If we want to keep roughly the same amount of history (in terms of
    a time span) regardless of the number of samples taken per RTT we
    need to adjust the gains.  E.g., if we were to take two samples per
    RTT then we'd use an alpha of 1/8*1/2 for each sample and that would
    approximate the case of one sample with a gain of 1/8.

(5) We know roughly how many RTT samples a congestion window worth of
    data will yield.  So, something like this will approximate RFC6298
    regardless of how many samples are pushed through the EWMAs per RTT:

    ExpectedSamples = ceiling (FlightSize / SMSS / 2)
    alpha' = alpha / ExpectedSamples
    beta' = beta / ExpectedSamples

    Now just run the algorithm in RFC6298 with alpha' and beta' instead
    of alpha and beta.  I.e.,

        RTTVAR <- (1 - beta') * RTTVAR + beta' * |SRTT - R'|
        SRTT <- (1 - alpha') * SRTT + alpha' * R'

        (for each RTT sample R')

(6) That isn't **exactly** the same as RFC6298, but it is absolutely
    **almost** the same.  And, it is absolutely within the spirit of
    TCP's standardized RTO.  I.e., we keep about the same amount of
    history and we weight the recent RTT samples with about the same
    weight.

    So, I don't see why 1323bis shouldn't say "if you use this RTTM we
    codify then you make the following adjustments to TCP's standard RTO
    procedure [...insert step (5) here...].  To me that is well
    understood and straightforward.  This notion of taking the average
    and using the max variance or something is not as well understood.
    Nor would anything else you cook up.  E.g., why the mean?  Why not
    the median?  Why not the mode?  I don't understand why we would not
    simply take a very small step to say "continue to use TCP's standard
    RTO" in favor of something unknown.  And, I also don't understand
    the argument that we need to give implementers a bunch of choices
    here.  I mean, I am fine with folks going off and playing with
    things in their research.  But, what is the argument that says we
    for sure need to deviate from the standard here?

allman

---

From: David Borman <[email-masked]>
Date: Wed, 15 May 2013 14:46:45 +0000

On May 14, 2013, at 8:42 PM, Mark Allman <[email-masked]> wrote:
...
>    ... I don't understand why we would not

If we have one preferred recommendation, then giving that as the only
example is just fine.  At the time the examples went into the earlier
versions of 1323bis, we wanted to avoid codifying only one way to
address the issue. It appears things have changed.

Here's an excerpt from the 2008 Philly IETF TCPM meeting minutes,
back when Mark was one of the TCPM co-chairs. :-)

>   Joe Touch: If you supply one example in section 3.3 on RTTM, this
>    will become the default. David, we could choose an alternative
>    RTTM implementation style. 
>   Mark: A few people have solved this by changing the weights on the
>    sample. 
>   David said that he expected that this should then illustrate that
>    there were multiple approaches. ... We should be careful that we
>    don't suggest something that is too extreme, we need to support
>    this with research. 
>   Mark said there were published papers that talk about weight
>    changes. 
>   Joe said we should recommend alternatives that we know are safe -
>    but not too much importance to one unless we know how to
>    choose). 
>   David said that he should tell people what to do, and not
>    prescribe an algorithm. 
>   Mark said we needed to keep the variance and adjust the gain based
>    on the cwnd.

So, I'll continue to argue that the primary goal is to tell people they
need to adapt their RTO estimator if they start feeding in multiple RTTM
samples per RTT.  We shouldn't get hung up on the examples, and should
just remove them or move them to an appendix if they are a stumbling block.

Ironically, the next sentence in the 2008 minutes is:
>   Hope to make the next version ready for a WGLC. Lars said we
>   should add milestones for the two documents (Ted to do).

			-David

---

From: David Borman <[email-masked]>
Date: Wed, 15 May 2013 20:34:28 +0000

On Apr 29, 2013, at 4:05 AM, Pasi Sarolahti <[email-masked]> wrote:

> Hi,
> 
> WG chairs believe that the earlier outstanding issues have been addressed in the latest version of 1323bis, and the document starts to be ready to be submitted for publication. Therefore, this mail starts a working group last call for draft-ietf-tcpm-1323bis-11 ("TCP Extensions for High Performance").
> 
> The WG last call runs for two weeks, until Monday, May 13th. Please send any comments on the mailing list by then. Also notifications of approving the current version (even without other comments) are helpful.
> 
> The draft is available at http://tools.ietf.org/html/draft-ietf-tcpm-1323bis-11

I've been reviewing the latest version, -12.

First, I missed this earlier, but between -07 and -08, the "Timestamps" option was renamed to the "Timestamp" option,  did I miss the discussion on making that change?  I'd like to see that put back to the original 1323 name.  It's plural because the option holds two timestamps.  Additionally, there are many instances of "timestamp option" that should all be capitalized as "Timestamps option."

Here are some other nits that I found:

Page 10:

  o  The window field (SEG.WND) of every outgoing segment, with the
     exception of <SYN> segments, is right-shifted by Rcv.Wind.Shift
     bits:

                   SND.WND = RCV.WND >> Rcv.Wind.Shift

That should be "SEG.WND = ..."

Page 14:
	... clock value provides a RTT measurement.
should be:
	... clock value provides an RTT measurement.

Page 13, 19:
Multiple instances of "a <RST>" should be "an <RST>"

Page 27:
The second bullet, the last 2 sentences:
        These middle boxes must also support the Window Scale option
        and apply the scaling when processing segments.  If the window
        scale cannot be determined, it must not do window based
        processing.
should be changed to to use "scale factor":
        These middle boxes must also support the Window Scale option
        and apply the scaling factor when processing segments.  If the scale
        factor cannot be determined, it must not do window based
        processing.

Page 31:
	"The following layouts are recommended for..."
should be:
	"The following layout is recommended for..."
(we only have one layout)

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Wed, 15 May 2013 21:00:37 +0000

Hi David,

> First, I missed this earlier, but between -07 and -08, the "Timestamps"
> option was renamed to the "Timestamp" option,  did I miss the discussion
> on making that change?  I'd like to see that put back to the original 1323
> name.  It's plural because the option holds two timestamps.  Additionally,
> there are many instances of "timestamp option" that should all be
> capitalized as "Timestamps option."

It wasn't consistent (but the majority was Timestamps); 

I have reverted that change, but the text surrounding every instance should be checked by a native speaker if the plual/singular form (will they apply in plural to "timestamps" or singular to the "option"?) are consistent...

> Multiple instances of "a <RST>" should be "an <RST>"

Ok, after my simplistic approach of vowel vs. consonant doesn't work (apparently, in American English, it's the initial sound rather than letter of a proper word, that counts), and the use of a vs an for acronyms seems to be a discussed topic (some count acronyms as proper words, others use the initial word) - is there a guideline from the WG/IETF/RFC-Editor here?

Fixed all the other nits!

Thanks a lot!

Richard Scheffenegger

> From: David Borman [mailto:[email-masked]]
> Sent: Mittwoch, 15. Mai 2013 22:34
> Cc: tcpm ([email-masked])
> > WG chairs believe that the earlier outstanding issues have been
> addressed in the latest version of 1323bis, and the document starts to be
> ready to be submitted for publication. Therefore, this mail starts a
> working group last call for draft-ietf-tcpm-1323bis-11 ("TCP Extensions
> for High Performance").
> >
> > The WG last call runs for two weeks, until Monday, May 13th. Please send
> any comments on the mailing list by then. Also notifications of approving
> the current version (even without other comments) are helpful.
> >
> The information contained in this transmission may be confidential. Any
> disclosure, copying, or further distribution of confidential information
> is not permitted unless such privilege is explicitly granted in writing by
> Quantum. Quantum reserves the right to have electronic communications,
> including email and attachments, sent across its networks filtered through
> anti virus and spam software programs and retain such messages in order to
> comply with applicable data security and retention requirements. Quantum
> is not responsible for the proper and complete transmission of the
> substance of this communication or for any delay in its receipt.