Subject: [tcpm] draft-kuehlewind-tcpm-accurate-ecn-02: Normative language needed

From: Bob Briscoe <[email-masked]>
Date: Thu, 10 Apr 2014 14:37:59 +0100

Richard, Mirja,

There's a general need to make the text describing normative 
requirements clearer. However, we can deal with wordsmithing later. 
First we need to agree what normative statements we are trying to 
make. Here's my suggestions:

S.2.3 More Accurate ECN TCP Receiver

OLD:
    With each ACK the receiver will calculate CI
    modulo 5 and set the respective codepoint in the ACE field (see Table
    2).  In addition, the receiver calculates CI divided by 5 and may set
    the "Top ACE" field to this value, provided the URG flag is not set
    in the segment.
ALTERNATIVE 1:
    With each ACK the receiver will calculate CI
    modulo 5 and MUST set the respective codepoint in the ACE field (see Table
    2).  In addition, if the receiver does not need to set the URG flag, it
    calculates CI divided by 5 and it MUST set the "Top ACE" field to 
this value.
REASONING:
- 1st MUST (setting ACE) is required for basic protocol compliance.
- 2nd MUST (conditionally setting Top ACE) avoids having another 
mechanism for the receiver to say whether it has set Top ACE or not. 
And we don't want a malicious receiver
to be able to choose not to feed back the nonce, while still getting 
the benefit of AccECN.

ALTERNATIVE 2 (if testing finds some middleboxes drop a non-zero URG 
ptr when URG flag=0):
    With each ACK the receiver will calculate CI
    modulo 5 and MUST set the respective codepoint in the ACE field (see Table
    2).  In addition, if the receiver does not need to set the URG flag, it
    calculates CI divided by 5 and it SHOULD set the "Top ACE" field 
to this value.
    A valid reason for not setting Top ACE to CI/5 would be if the 
receiver has
    discovered that such packets are not being forwarded. If, when 
URG flag = 0,
    the receiver does not set Top ACE to CI/5, it MUST set Top ACE to 
zero instead,
    and continue setting it to zero for the remainder of the connection.
REASONING:
We don't want to lose one value of Top ACE by defining URG_flag=0 and 
TopACE=0 as a special 'undefined' value not to be used when cycling 
the counters. Instead, the sender only needs to test for a wrap to 
zero in Top ACE of more than one, then it MUST turn off processing 
Top ACE for the rest of the connection.

Nits in the draft:
_________________

S.2.2.2
s/ETC(1)/ECT(1)/

S.2.2.3
Need to say explicitly that Top ACE is defined as
- the Top of CI if the value of ACE is one of the CI codepoints
- the Top of E1 if the value of ACE is one of the E1 codepoints

S.2.4
s/exclude/decode/

Bob

________________________________________________________________
Bob Briscoe,                                                  BT

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Thu, 10 Apr 2014 16:18:10 +0000

Hi Bob,
>     the "Top ACE" field to this value, provided the URG flag is not set

I'm in favor of

> ALTERNATIVE 2 (if testing finds some middleboxes drop a non-zero URG ptr
> when URG flag=0):

I see more middleboxes sanitizing the URGPTR; but I haven't yet looked specifically at TCP session drops due to a set URGPTR, but that is on my todo list for the next measurement run.

Actually, I found an interesting data point in a book [1] that I was reading - a number of versions of windows inadvertedly leaked data on uninitialized ACK and URGPTR fields for years. Obviously, that issue never caused TCP session setups to fail, thus its save to assume that this worldwide test conducted by Microsoft "fixed" any misbehaving stacks and middleboxes (which would not properly work with URGPRT != 0) long ago.

>     calculates CI divided by 5 and it SHOULD set the "Top ACE" field to
> this value.
>     discovered that such packets are not being forwarded. If, when URG
> flag = 0,
>     the receiver does not set Top ACE to CI/5, it MUST set Top ACE to zero
> instead,
> TopACE=0 as a special 'undefined' value not to be used when cycling the
> counters. Instead, the sender only needs to test for a wrap to zero in Top
> ACE of more than one, then it MUST turn off processing Top ACE for the
> rest of the connection.

I think we should have this even more strict, and have a URGPTR negotiation during 3WHS:

a) set URGPTR = 0x???F in SYN
b) set URGPRT = 0x???1 if ???F was received or 0x???0 if ???0 was received in SYN/ACK

-> negotiate to both ends, if URGPTR may be used.

Then, if URGPTR can NOT be used, have AccECN signal the most conservative congestion indication, whenever an overflow may have occurred (multiple ACKs being lost, even when ECI not changed). This should give a good incentive for sanitizing middleboxes to clean up their act :) Especially when these are close to the server and in the same admin domain as the poor performing service...

And yes, if URGPTR cannot be used, the TCP MUST set TopACE to 0 at all times...

[1] Silence on the Wire: A Field Guide to Passive Reconnaissance and Indirect Attacks, Michal Zalewski, No Starch Press, 2005 - 281 pages

Best regards,
  Richard

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 10 Apr 2014 19:08:35 +0100

Richard,

At 17:18 10/04/2014, Scheffenegger, Richard wrote:
>I see more middleboxes sanitizing the URGPTR; but I haven't yet 
>looked specifically at TCP session drops due to a set URGPTR, but 
>that is on my todo list for the next measurement run.
>
>Actually, I found an interesting data point in a book [1] that I was 
>reading - a number of versions of windows inadvertedly leaked data 
>on uninitialized ACK and URGPTR fields for years. Obviously, that 
>issue never caused TCP session setups to fail, thus its save to 
>assume that this worldwide test conducted by Microsoft "fixed" any 
>misbehaving stacks and middleboxes (which would not properly work 
>with URGPRT != 0) long ago.

That' cool. Obviously your bedtime reading is more interesting than mine ;)

>-> negotiate to both ends, if URGPTR may be used.

Don't agree here.

The point of my suggested text was that, for short connections, 
you'll never need to have a value higher than 0000 in the URG PTR 
anyway. So no point risking timeouts by negotiating it in SYN.

Instead, the first time that the ACE field wraps, the receiver sets 
Top ACE to 0001 anyway. Then whenever the sender code detects a wrap 
in the ACE value, if there's no corresponding increment in the Top 
ACE value it knows a middlebox has normalised it. Then the sender 
must stop reading Top ACE for the rest of the connection (in case the 
normalisation isn't consistent (abnormal normalisation!)).

The same procedure can happen independently on the other half connection.

Ie the sender can detect the problem by inference rather than 
negotiation, because of the way ACE and Top ACE are defined to work together.

If your experiments continue to find no middleboxes actually 
discarding Top ACE != 0 packets, so the problem is only 
'normalisation', then I don't see the need for negotiation about 
using a field that is for optional accuracy anyway.

>Then, if URGPTR can NOT be used, have AccECN signal the most 
>conservative congestion indication, whenever an overflow may have 
>occurred (multiple ACKs being lost, even when ECI not changed). This 
>should give a good incentive for sanitizing middleboxes to clean up 
>their act :) Especially when these are close to the server and in 
>the same admin domain as the poor performing service...

Can a receiver do anything more conservative that what it's doing 
anyway? It's just a dumb reflector. I suppose it could turn off 
delayed ACKs. But is that a good idea to recommend more ACKs whenever 
multiple pure ACKs seem to be getting lost? And how does it know that?

My proposal was on the basis that the receiver doesn't need to know 
that its Top ACE is being normalised, because it can't do anything 
about it anyway. Can it?

>[1] Silence on the Wire: A Field Guide to Passive Reconnaissance and 
>Indirect Attacks, Michal Zalewski, No Starch Press, 2005 - 281 pages

Cheers

Bob

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Fri, 11 Apr 2014 08:17:53 +0000

Hi Bob,

> That' cool. Obviously your bedtime reading is more interesting than mine

Well, it at least means that I'm a complete nerd ;)

> The point of my suggested text was that, for short connections, you'll
> never need to have a value higher than 0000 in the URG PTR anyway. So no
> point risking timeouts by negotiating it in SYN.

Ok, point taken. 

> Instead, the first time that the ACE field wraps, the receiver sets Top
> ACE to 0001 anyway. Then whenever the sender code detects a wrap in the
> ACE value, if there's no corresponding increment in the Top ACE value it
> knows a middlebox has normalised it. Then the sender must stop reading Top
> ACE for the rest of the connection (in case the normalisation isn't
> consistent (abnormal normalisation!)).
> 
> Ie the sender can detect the problem by inference rather than negotiation,
> because of the way ACE and Top ACE are defined to work together.
> 
> If your experiments continue to find no middleboxes actually discarding
> Top ACE != 0 packets, so the problem is only 'normalisation', then I don't
> see the need for negotiation about using a field that is for optional
> accuracy anyway.

So, in the unlikely event when a middlebox doesn't normalize but discard...

Those ACKs wouldn't make it through. On the forward path, everything would look ok (until a TopACE rollover happens). So the sender would perform RTO retransmissions until session timeout, with the receiver ACKing every segment properly. Do we need to care about this pathology in the receiver code?

Or simply leave that unspecified, so that this particular brokenness is fixed either by disableing AccECN or removal of the broken box?

The problem would be, that this effect would happen only a couple (dozen) segments into the session, and non-deterministically (due to CE / ECT(1) marks received on a session), making troubleshooting a nightmare.

> >their act :) Especially when these are close to the server and in the
> >same admin domain as the poor performing service...
> 
> Can a receiver do anything more conservative that what it's doing anyway?
> It's just a dumb reflector. I suppose it could turn off delayed ACKs. But
> is that a good idea to recommend more ACKs whenever multiple pure ACKs
> seem to be getting lost? And how does it know that?

See above; if the negotiation for TopACE fails (SYN with URGPTR==0; SYNACK with URGPTR==0, or SYN / SYNACK need to be retransmitted, eventually with TopACE set to 0), the receiver would never set TopACE. Path changes may still cause problems when the new path suddenly drops TopACE packets...and that we can not address anyhow.

For the sender, when it detects that TopACE is unusable, it can (should) switch into most conservative mode, IMHO.

> My proposal was on the basis that the receiver doesn't need to know that
> its Top ACE is being normalised, because it can't do anything about it
> anyway. Can it?

Silent discards would be the more pronounced issue, normalization can be dealt with rather easily.

But as mentioned, it appears that discard is very uncommon, normalization is rare but exists on numerous paths.

I need to run my next set of tests (SMTP servers this time :), with code to check for discarded SYNs).

Give me a couple days.

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 11 Apr 2014 16:58:08 +0100

Richard,

At 09:17 11/04/2014, Scheffenegger, Richard wrote:

>Those ACKs wouldn't make it through. On the forward path, everything 
>would look ok (until a TopACE rollover happens). So the sender would 
>perform RTO retransmissions until session timeout, with the receiver 
>ACKing every segment properly. Do we need to care about this 
>pathology in the receiver code?
>
>Or simply leave that unspecified, so that this particular brokenness 
>is fixed either by disableing AccECN or removal of the broken box?
>
>The problem would be, that this effect would happen only a couple 
>(dozen) segments into the session, and non-deterministically (due to 
>CE / ECT(1) marks received on a session), making troubleshooting a nightmare.

Yes, nightmare. This is why your experiments are important - to test 
whether we ever see a non-zero URG-PTR dropped.

If the only visible pathology is normalisation (which is all you've 
found so far, and the windows bugs you mentioned help explain why), 
then I would agree with your approach:
actively test for normalising middleboxes in the 3WHS. Ie. we don't 
need to care about the risk of timeouts from a pathology that testing 
cannot detect.

However, if your tests find even a very few middleboxes discarding 
non-zero URG-PTR, then the balance of risks alters. Then we should 
probably take the risk of /not/ actively testing for normalisation in 
the 3WHS, to avoid the risk of a timeout due to a discarding middlebox.

Instead, we consider adopting my original proposal: wait until the 
algo would actually start using a non-zero URG-PTR anyway then, if 
there is a normalising middlebox, there is a small risk that the 
sender won't immediately detect that ACKs are being normalised - but 
only in the unlikely case of long sequences of pure ACK loss. But 
this is worth the small risk, because the consequences of this 
corner-case become less than the timeout corner-case.

In summary, the best design to pick depends on your experiments.

>See above; if the negotiation for TopACE fails (SYN with URGPTR==0; 
>SYNACK with URGPTR==0, or SYN / SYNACK need to be retransmitted, 
>eventually with TopACE set to 0), the receiver would never set 
>TopACE. Path changes may still cause problems when the new path 
>suddenly drops TopACE packets...and that we can not address anyhow.

No, I meant if the receiver knows TopACE is useless and never sets 
it, can its feedback pattern by any more conservative than we have 
specified anyway? (If it /can't/ change its behaviour, it doesn't 
need to know whether it /should/ change its behaviour.)

>For the sender, when it detects that TopACE is unusable, it can 
>(should) switch into most conservative mode, IMHO.
>
>But as mentioned, it appears that discard is very uncommon, 
>normalization is rare but exists on numerous paths.
>

I'm v grateful you're doing these tests.

Cheers

Bob