Subject: [tcpm] Comments on draft-blanton-tcpm-3517bis-01

From: Ethan Blanton <[email-masked]>
Date: Wed, 13 Apr 2011 14:24:49 -0400

Hi all,

The 3517bis -01 is out, and available here:

    http://www.ietf.org/id/draft-blanton-tcpm-3517bis-01.txt

We believe that we have addressed all of the concerns with -00 in this
document, and that it is ready for last call.  A quick rundown of the
concerns which came up in -00 and their related changes (or the
reasoning behind not changing anything) follows.

* Matt Mathis brought up a concern with the quoted requirement from
  RFC 2018 that SACK information MUST be discarded following an RTO.
  There is an Errata on 2018 (also by Matt) addressing this.  We were
  not comfortable updating 2018 "on the side" as it were, so rather
  than change the quoted recommendation, we added some text explaining
  that this issue has been reconsidered, that there is an Errata on
  the subject, and that implementors should find out what the state of
  affair is when implementing.  Section 5.1 paragraph 1 now reads:

   In order to avoid memory deadlocks, the TCP receiver is allowed to
   discard data that has already been selectively acknowledged.  As a
   result, [RFC2018] suggests that a TCP sender SHOULD expunge the SACK
   information gathered from a receiver upon a retransmission timeout
   "since the timeout might indicate that the data receiver has
   reneged."  Additionally, a TCP sender MUST "ignore prior SACK
   information in determining which data to retransmit."  However, since
   the publication of [RFC2018] this has come to be viewed by some as
   too strong.  It has been suggested that, as long as robust tests for
   reneging are present, an implementation can retain and use SACK
   information across a timeout event [Errata1610].  While this document
   does not change the specification in [RFC2018], we note that
   implementers should consult any updates to [RFC2018] on this subject.
   Further, a SACK TCP sender SHOULD utilize all SACK information made
   available during the slow start phase of loss recovery following an
   RTO.

* Richard Scheffenegger brought up an end-of-window loss corner case
  which RFC 3517 loss recovery does not handle as aggressively as
  NewReno, potentially leading to RTO.  While the problem itself is
  well-understood, our read of the consensus is that the suggestions
  put forth to mitigate it are not.  Since 3517 is standards track and
  mature, no changes were made to the document regarding end-of-window
  losses.

* There has been ongoing discussion regarding TCP security in relation
  to Fernando's draft.  One of the bullets in that draft involves a
  blind out-of-window attack which can cause a sender to mistakenly
  infer loss in the absence of SACK.  A note was added to security
  considerations to hilight this robustness when DupACKs are properly
  accounted:

   Similarly, [CPNI309] sketches a variant of a blind attack [RFC5961]
   whereby an attacker can spoof out-of-window data to a TCP endpoint,
   causing it to respond to the legitimate peer with a duplicate
   cumulative ACK, per [RFC793].  Adding a SACK-based requirement to
   trigger loss recovery effectively mitigates this attack, as the
   duplicate ACKs caused by out-of-window segments will not contain SACK
   information indicating reception of previously un-SACKED in-window
   data.

* The "Changes Relative to RFC 3517" section has been clarified
  editorially.

* Despite the IETF/IESG joint commitment to making submission of I-Ds
  as painful as possible, we negotiated the boilerplate minefield and
  updated the text and formatting of the draft *just so* to convince
  the automated tools to accept it.

Thanks,
Ethan

---

From: Yuchung Cheng <[email-masked]>
Date: Wed, 13 Apr 2011 12:48:14 -0700

Hi Ethan,

I have two questions:

1) IsLost (SeqNum):

      This routine returns whether the given sequence number is
      considered to be lost.  The routine returns true when either
      DupThresh discontiguous SACKed sequences have arrived above
      'SeqNum' or more than (DupThresh - 1) * SMSS bytes with sequence
      numbers greater than 'SeqNum' have been SACKed.  Otherwise, the
      routine returns false.

Does RFC require DupThresh discontinuous sack blocks? e.g.,
For example, if a sender sends 4 data packets
send pkt1: 1-100
send pkt2: 101-200
send pkt3: 201-300
send pkt4: 301-400
receive ack: ack1 with one sack block 101-400.
does IsLost(1) returns true?

2)
If the incoming ACK is a duplicate acknowledgment and the TCP is

   not currently in loss recovery, the TCP MUST increase DupAcks by
   one and take the following steps:

   (1) If DupAcks >= DupThresh, go to step (4).

   (2) If DupAcks < DupThresh but IsLost (SND.UNA) returns
       true---indicating at least three segments have arrived above
       the current cumulative acknowledgment point, which is taken
       to indicate loss---go to step (4).

can we combine (1) and (2)? go to steps (4) if IsLost(SND.UNA) is
true. I guess it depends on your answer of my first question.

Thanks,

Yuchung

---

From: Mark Allman <[email-masked]>
Date: Wed, 13 Apr 2011 15:59:17 -0400

> does IsLost(1) returns true?

Yes.

If **EITHER**:

  - You have DupThresh distinct blocks above 1 (so, e.g., if you
    continue your example a few more and you collect up SACKs for
    101-200, 301-400, 501-500 you'd declare sequence number 1 to be
    lost), 

    **OR**,

  - You have (DupThresh-1) * SMSS bytes SACKed in the scoreboard (so,
    (400-101) > (2 * 100) in the case you present)

then you declare the given sequence number to be lost.

> true. I guess it depends on your answer of my first question.

My first hit here is that this could indeed be tightened up by just
getting rid of DupAcks and consulting IsLost (SND.UNA).  If that returns
true we enter loss recovery, if not we don't.  Do I understand that to
be what you're saying?

(I do need to go back and look at the context around this bit of the
document to make sure that is plausible, I guess.)

allman

---

From: "Ilpo Järvinen" <[email-masked]>
Date: Wed, 13 Apr 2011 23:11:03 +0300 (EEST)

> true we enter loss recovery, if not we don't.  Do I understand that to

...I see, now we're back in discussing what this DupAcks counter is 
for, right? It was added in sack-recovery-entry ID and had proper 
explation of its purpose there in place. The DupAcks counter is to handle 
case where the sender is sending smaller than SMSS sized segments. 
...Perhaps it would be useful to explain it like the sack-recovery-entry 
did because otherwise this will just keep confusing people?

-- 
 i.

---

From: Ethan Blanton <[email-masked]>
Date: Wed, 13 Apr 2011 16:13:15 -0400

> does IsLost(1) returns true?

Let me clerify this a bit.  Are you positing this when SMSS >> 100
bytes?  If so, then the answer is "maybe".  The draft is written in
the context of a TCP which does not keep track of segments.   If, in
this case, SMSS >> 100 bytes, then IsLost(1) will return false for a
pure byte-counting TCP.  Note from Section 3:

   Finally, note that the algorithm in this document assumes a
   sender that is not keeping track of segment boundaries after
   transmitting a segment.  It is possible that a sender that did
   keep this extra state may be able to use a more refined and
   precise algorithm than the one presented herein, however, we
   leave this as future work.

If SMSS == 100 bytes, then yes, IsLost(1) will return true.

> true. I guess it depends on your answer of my first question.

No, not unless you assume a TCP which tracks segments, you assume that
all segments are SMSS-sized, or you *always* use Nagle's algorithm.
If a TCP is sending segments of size < SMSS, then counting incoming
DupAcks allows you to trigger loss recovery when 3 incoming ACKs with
new SACK information arrive, regardless of the size of the segments
triggering those ACKs.

Hope that helps.

Ethan

---

From: Mark Allman <[email-masked]>
Date: Wed, 13 Apr 2011 16:42:30 -0400

> did because otherwise this will just keep confusing people?

Thanks for the reminder... Ethan also noted that this is for the small
packet case.  I forgot about that.  Why don't we just add a quick note
to the document after the (1) test.  I.e., something like ...

       Note: This check covers the case when a TCP is not sending
       full-sized packets and therefore IsLost() (next step) may be
       hard-pressed to declare a segment as lost.

allman

---

From: "Ilpo Järvinen" <[email-masked]>
Date: Thu, 14 Apr 2011 00:52:02 +0300 (EEST)

>        full-sized packets and therefore IsLost() (next step) may be

packets => segments

>        hard-pressed to declare a segment as lost.

Something simple like that would certainly be useful. 

-- 
 i.

---

From: Yuchung Cheng <[email-masked]>
Date: Wed, 13 Apr 2011 16:56:01 -0700

Thanks for the clarification. My example assumes SMSS=1460 (sorry for
not being clear).

AFAIK, Linux (even with fack disabled) triggers the fast-recovery when
any 3 packets (of any size) are sacked beyond SND.UNA.

On Wed, Apr 13, 2011 at 1:13 PM, Ethan Blanton <[email-masked]> wrote:
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.10 (GNU/Linux)
>
> iQEVAwUBTaYD2/8fixZ3H8crAQgazQf/W0CjlbO5bnIEFJq9syULb/R+bQrF7F+1
> 8g0VtV23PGoifylU+o6k6hxfyL1k4Q4uCpm/DgZEW6aYwo1CBOhTKYhhdAEqDwMK
> m5MRDpt2ZBmULZ8A8NaflrO0mSfLmsRuFcEpXrJm0mQNbIvK65ls2koNcxJ3Zyjl
> 2LsjT5iEzyi7/fBQNPe/AhwInR54ft4kFqZ+rp8AeAffts95xSRm6w4mXQbM18dQ
> urughzVo+dDM+NLO2TxsoeiUomEjRTfRmCfVQ4gx07iMYrLR3aThfniSIMHb0RlB
> 62E/4wLV+mQ/t97D2CgIFkmzp5VAeRCTFnEMg264xXl3hsOGtA+asQ==
> =g9zx
> -----END PGP SIGNATURE-----
>
>

---

From: Ethan Blanton <[email-masked]>
Date: Wed, 13 Apr 2011 20:17:09 -0400

> any 3 packets (of any size) are sacked beyond SND.UNA.

And this is correct behavior, and what 3517/3517bis will both do.

Ethan

---

From: Yuchung Cheng <[email-masked]>
Date: Wed, 13 Apr 2011 17:57:56 -0700

> And this is correct behavior, and what 3517/3517bis will both do.
Ah that's right. The subtle difference is that Linux IsLost(sn)
returns true if any 3 segments beyond sn are sacked, i.e., the seq# of
these segments need not be discontinuous. Therefore the loss marking
is more aggressive after F-R starts (even with FACK disabled).

---

From: Alexander Zimmermann <[email-masked]>
Date: Thu, 14 Apr 2011 07:12:03 +0200

Hi,

> did because otherwise this will just keep confusing people?

Yes, please. I really recommend to add section 3.1 of 
draft-ietf-tcpm-sack-recovery-entry-01.txt to the draft.

Alex

//
// Dipl.-Inform. Alexander Zimmermann
// Department of Computer Science, Informatik 4
// RWTH Aachen University
// Ahornstr. 55, 52056 Aachen, Germany
// phone: (49-241) 80-21422, fax: (49-241) 80-22222
// email: [email-masked]
// web: http://www.umic-mesh.net
//

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Thu, 14 Apr 2011 09:46:26 +0100

I'm also in favor of having this note included - a full section like in draft-ietf-tcpm-sack-recovery-entry may be too much though...

Richard Scheffenegger

> -----Original Message-----
> From: Mark Allman [mailto:[email-masked]]
> Sent: Mittwoch, 13. April 2011 22:43
> To: Ilpo Järvinen
> Cc: [email-masked] Extensions
> Subject: Re: [tcpm] Comments on draft-blanton-tcpm-3517bis-01
> 
> 
> > explation of its purpose there in place. The DupAcks counter is to
> > handle case where the sender is sending smaller than SMSS sized
> segments.
> > ...Perhaps it would be useful to explain it like the
> > sack-recovery-entry did because otherwise this will just keep
> confusing people?
>

---

From: Markku Kojo <[email-masked]>
Date: Thu, 14 Apr 2011 12:21:09 +0300 (EEST)

> is more aggressive after F-R starts (even with FACK disabled).

Right. The difference is that the Linux TCP sender tracks segment 
boundaries and is therefore able to determine that 3 small segments 
have made it to the receiver even though these segments are continuous.

As Ethan noted the draft is written such that it does not require
a TCP sender to keep track of segment boundaries after a segment
has been transmitted (Note in Section 3).

/Markku

---

From: Alexander Zimmermann <[email-masked]>
Date: Thu, 14 Apr 2011 12:54:26 +0200

> has been transmitted (Note in Section 3).

AFAIK you and Ilpo pointed out in your draft that in a case of a
segment-based stack the dupthresh counter is not necessary anymore and
we can handle everything with IsLost(). So, I recommend that this should
be included in 3517bis (as a big side note or in an appendix section)

Alex

> 
> /Markku
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm

//

---

From: Mark Allman <[email-masked]>
Date: Thu, 14 Apr 2011 08:33:39 -0400

> we can handle everything with IsLost(). So, I recommend that this
> should be included in 3517bis (as a big side note or in an appendix
> section)

We do note in the document ...

... which seems reasonable to me.  The document **does not** seek to
offer both byte- and packet-based variants.  It is perfectly fine for a
stack to use a packet-based variant and this document could still form
the basis of such a beast.  But, such an implementation could likely
hone a number of things that are approximations in the current document,
or get by without particular counters or whatnot.  Tracking packets may
in fact be a good tradeoff in some cases.  But, it just isn't what this
document strives to specify.  Some other document can puzzle through a
packet-based variant if folks want such a specification.

allman

---

From: Alexander Zimmermann <[email-masked]>
Date: Thu, 14 Apr 2011 15:45:28 +0200

Hi Mark,

Am 14.04.2011 um 14:33 schrieb Mark Allman:

> ... which seems reasonable to me.

For me too. The only thing that I try to say is, that in the current
version of the draft, it is hard to understand why we need the dupthresh
counter (and can not only rely on IsLost()). If we can add some text
from Ilpo back to the main doc, I'm satisfied...

Alex

//

---

From: "Ilpo Järvinen" <[email-masked]>
Date: Thu, 14 Apr 2011 17:10:51 +0300 (EEST)

On Thu, 14 Apr 2011, Mark Allman wrote:

> in fact be a good tradeoff in some cases. 

I think there are multiple shades of gray there. 

...In the one end there's fully packet based algorithm and in the other 
fully seqno based one.  The first one is certainly a problem that isn't 
even specified outside of recovery fully afaik, and shouldn't be tried 
here (in 3517bis).  But on the middle ground there's such a version where 
the only the heuristics to detect that dupThresh segments above another 
are SACKed is replaced with an accurate version (it actually doesn't even 
have to be a "packet based" implementation although this information is  
naturally always available in case of packet based implementaion which is 
why it gets mentioned usually as "if TCP is packet based blah blah..."). 

Such refined IsLost just does more accurately what the heuristics 
version of IsLost tried to do. I find it unlikely that there's a can of 
worms in such refining because then the goal of the original heuristics is 
just met more often than in the original. Unless there's something flawed 
in the not-always-met goal of the original heuristics?

-- 
 i.

---

From: "Anantha Ramaiah (ananth)" <[email-masked]>
Date: Thu, 14 Apr 2011 07:20:30 -0700

<snip>

> hone a number of things that are approximations in the current
> document, or get by without particular counters or whatnot.  Tracking
> packets may in fact be a good tradeoff in some cases.  But, it just
> isn't what this document strives to specify.  Some other document can
> puzzle through a packet-based variant if folks want such a
> specification.

I have been passively watching this thread. Just wanted to note that our
implementation(s) uses packet based buffering scheme and it comes across
handy in many situations including the current case. So, I think that
adding text that benefits implementations using packet variants would be
really helpful. Coming back to this thread, Between having a separate
document that caters to packet based variants versus folding the
information into the existing document itself, I would prefer the
latter, FWIW.

Thanks,
-Anantha

---

From: Mark Allman <[email-masked]>
Date: Thu, 14 Apr 2011 10:41:58 -0400

> I have been passively watching this thread. Just wanted to note that
> our implementation(s) uses packet based buffering scheme and it comes
> across handy in many situations including the current case.

I do not doubt that.

> So, I think that adding text that benefits implementations using
> packet variants would be really helpful. Coming back to this thread,
> Between having a separate document that caters to packet based
> variants versus folding the information into the existing document
> itself, I would prefer the latter, FWIW.

Let me stress that RFC3517 (and the bis) do not somehow strive to be
*the* SACK-based recovery algorithm.  It is *a* reasonable way to do
loss recovery with SACK.  We worked very diligently to understand how
this algorithm works (and where it doesn't).  The bis adds a small piece
that makes the scheme more robust.  I think it would be a pity to not
publish that quite soon (and there have been no objections as far as I
can tell to the basic technical merits and/or scheme, but only
refinements to the text to make it more understandable).

Holding the (as best as I can tell) agreed upon bis up so that we can go
back and massage in a variant of the algorithm that leverages
understanding of segment boundaries seems imprudent to me given that (a)
it will take great care (if our experience with RFC3517 is any
indication), (b) we have no idea how hard it will be to get right (we
thought the scheme in RFC3517 would be easy, too, and it took us 2.5
years and 14 revisions) and (c) nobody has signed up to do it.

So, I would *strongly* prefer we push what we agree to out.  And, I
*encourage* anyone who has better ideas to pursue those (e.g., as Matt
and the Google gang are doing).  There needn't be only one.

allman

---

From: Matt Mathis <[email-masked]>
Date: Fri, 15 Apr 2011 11:37:30 -0400

I would like to suggest an alternate wording for Section 5.1 paragraph 1:

To make SACK robust in the presence of unknown implementation bugs and
reneging, RFC 2018 indicates that the sender MUST ignore prior SACK
information a on a timeout.   However, since preserving SACK
information on timeouts greatly reduces wasted network capacity under
overload conditions, this requirement is probably too strong.
Errata1610 amends the MUST to a SHOULD and suggests that as long as
there are robust tests for inconsistent SACK information, an
implementation might retain and use SACK  information across a
timeout.  The SACK consistency checks might include snd.una advancing
to just before a byte that has been reported in a SACK block, and
repeated timeouts or DSACKS.   This document does not formally update
the specification in [RFC2018], however implementers should consult
any updates to [RFC2018] on this subject.  Furthermore, a SACK TCP
sender SHOULD utilize all SACK information made available during the
slow start phase of loss recovery following an  RTO.

(Feel free to make additional adjustments)

Thanks,
--MM--
The best way to predict the future is to create it.  - Alan Kay

---

From: Matt Mathis <[email-masked]>
Date: Fri, 15 Apr 2011 13:25:29 -0400

(back on tcpm)
Well, ok, keep your words.  However please at least remove or change
the first sentence.

Reneging is not really about memory, we only used that as an excuse.
We allowed the receivers to renege to force implementers think about
and actually test their code with inconsistent SACK updates.

At the time there was a strong track record for the following oft
repeated ugly scenario: the first implementer has a bug that does not
effect self-interoperability, and goes on to capture a significant
market share.  The second and subsequent implementer do not have the
bug and get punished by having interoperability problems.  Everybody
switches to "implemented but off by default".  Several years go by
waiting for the deployed bugs to vanish (w/o any market pressure)
before the new feature gets turned on....

We wanted to guarantee that SACK could not cause interoperability
problems and that the worst case impact of SACK bugs would be reduced
performance.  The last resort was to use Tahoe style go-back-n timeout
behavior.   Considering how may truly weird traces that we all have
looked at, this worked: as far as I know SACK bugs have never caused
connection hangs or aborts.

At the time 2018 was in draft there was a lot of pressure to take
reneging out.  We knew that it implicitly forbid many useful
optimizations.   It stayed in only because of the robustness argument.

Note that if you ever free queued data because it has been SACKed,
SACK bugs become fatal.

</soapbox>
Now that SACK is well established, it might be a feature to have bugs
cause interoperability problems.

Thanks,
--MM--

On Fri, Apr 15, 2011 at 11:57 AM, Mark Allman <[email-masked]> wrote:
>
> Hi Matt!
>
> Question: do erratas formally update a spec?  I have forgotten.
>
> In any case, I like our words a lot better than your words.  I (1) think
> we're more precise in laying out the current state, (2) don't think it
> is this document's place to get into reneging detection in any sort of
> detail and (3) the detail you do provide in your text is fuzzy and
> non-actionable anyway ("eh, you could use DSACKs, perhaps!").
>
> What we tried to do is to say 3517bis is not normative on this subject
> and so please go look at the current normative stuff and of course use
> all SACK information available at all times.  That seems enough to me.
> Its all a minor point in this document.
>
> (I hate working .bis documents.  Everyone wants something.  That said,
> you should update 2018.)
>
> allman
>
>
>
>