Subject: Re: [tcpm] ECN++

From: Bob Briscoe <[email-masked]>
Date: Thu, 13 Feb 2020 18:54:12 +0000

Richard,

On 10/01/2020 00:08, Scheffenegger, Richard wrote:
> Marcelo, Bob,
>
> I just noted that there is a slight oversight in FreeBSD currently,
> which results in all session that are simultaneously ECN-enabled and
> SACK-permitted to effectively send out retransmissions with the ECT0
> codepoint.
>
> Strictly speaking, this is in violation of RFC3168, but might also be a
> good (nearly a decade long) validation of the performance of ECN++ for
> all types of data segments (new and retransmitted ones), although at a
> low and implicit exposure...
[BB] Let's hope someone has captured some relevant data.

>
>
> On that note, since I think ECN++ is quite valuable (with a number of
> published research finding this change to be crucial), perhaps you can
> summarize the outstanding issues (other than more reviewers required; I
> admit I still haven't gone through all the delta between -04 and -05).
[BB] I don't think there are any outstanding issues.

I have some word-smithing in my ToDo list, in the light of the most 
recently agreed change regarding ECT on SYN.
But no new technical controversy, AFAICT.

Bob
>
> Best regards,
>   Richard
>
> _______________________________________________
> tcpm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpm
>

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 13 Feb 2020 18:55:24 +0000

Richard,

On 15/01/2020 20:42, Scheffenegger, Richard wrote:
>
> Hi,
>
> Yet another interesting observation – as fbsd currently doesn’t refrain
> from marking SACK-retransmission to be not-ECT, you can actually end up
> getting a CE mark on a retransmission across a ECN-enabled congestion 
> point.
>
> Obviously this is better than loss…
>
> What happens next is, that fbsd "honors" that ECE mark, since it is in
> loss-recovery, not congestion-recovery. It adjusts the recovery_point to
> the current snd_max (rightmost sent segment), and adjusts ssthresh and
> cwnd by multiplicative decrease factor...
>
> Furthermore, it appears that it also resets the traversal of the SACK
> scoreboard (incidential a "good" thing, as a few earlier retransmissions
> also got dropped, not marked, and are being resent without an RTO).
>
> But in the context of ECN++, what would be the expected response here?
[BB] Quoting 3.2.7.  Retransmissions (Send)
https://tools.ietf.org/html/draft-ietf-tcpm-generalized-ecn-05#section-3.2.7

    If the TCP sender receives feedback that a retransmitted packet was
    CE-marked, it will react as it would to any feedback of CE-marking on
    a data packet.

In the ECN++ draft, we tried not to be too prescriptive or inventive 
about congestion behaviour. The draft is intended to focus on the 
sender's wire protocol behaviour, and refer to the default congestion 
response but not require it, so that other congestion control schemes 
can specify different responses without having to update the ECN++ draft.

Bob
>
> I assume, that with the exception of the fresh traversal of the SACK
> scoreboard, the above steps seem sensible.
>
> Any thoughts on this interesting interaction between ECE (during SACK
> loss recovery)?
>
> Best regards,
>    Richard
>
>
>
> Am 10.01.2020 um 01:08 schrieb Scheffenegger, Richard:
>> Marcelo, Bob,
>>

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 13 Feb 2020 18:58:25 +0000

Richard,

Interesting.
See inline tagged [BB]...

On 25/01/2020 10:41, Scheffenegger, Richard wrote:
> Hi Group, Marcelo, Bob,
>
>
> Another update in this context, which IMHO may be discussed as an actual
> change of mechanism with ECN++.
>
> I was looking into the very poor interaction of ECN between a Linux
> client and a BSD server, with request-response workload. That is, where
> each side sends out less than MSS data, before the application waits for
> the other side to respond to this.
>
> Neal pointed out this statement in RFC3168:
>
>    ...the TCP sender sets the CWR flag in
>    the TCP header of the first new data packet sent after the window
>    reduction.  ...
>    When the TCP data sender is ready to set the CWR bit after reducing
>    the congestion window, it SHOULD set the CWR bit only on the first
>    new data packet that it transmits.
>
> However, BSD is sending out the CWR as soon as possible - while Linux
> interprets the SHOULD overly strictly (IMHO) and ignores CWR unless it
> is received with (new) data.
[BB] Assuming the word '(new)' is optional, I think you're implying that 
BSD would set CWR on a pure ACK if that were its first packet after it 
received ECE feedback. Does BSD also set CWR on the first data packet 
after that (if any)?

I think RFC3168 expects CWR only on data packets - so that the sender of 
the CWR can distinguish between ECE that the receiver sends before vs 
after the receiver got the CWR (by whether the ackno of the ECE covers 
the CWR data packet or not).

Consider 4 data packet exchanges of A>B, B>A, B>A, A>B with CWR on pure 
ACKs.

     A>>>B Data#1 <CE in transit>
     A<<<B ACK#1 ECE
                             ...potentially quiet for a time...
     A<<<B Data#101 ECE ACK#1
     A>>>B ACK#101 *CWR*
                             ...potentially quiet for a time...
     A<<<B Data#102 ECE ACK#1
     A>>>B ACK#102
                             ...potentially quiet for a time...
     A>>>B Data#2 *CWR* ACK#102
     A<<<B ACK#2

'A' doesn't know whether the ECE on Data#102 was sent before or after B 
received the CWR on A's pure ACK, so 'A' doesn't know whether to reduce 
its window again or not.

I can't find anything in RFC3168 that explicitly spells out when the 
sender considers ECE to be in a new round. It jumps straight to 
describing the exceptional case of a CWR packet being dropped, and omits 
the 'normal' case of it being delivered.

This seems to be missing - perhaps it ought to be added by an erratum.

>
> But binding the CWR flag to a new data segment delays the ECN signaling
> loop artificially (for long runs of unidirectional transmitted data),
> and it is not clear what the benefit there would be, as the CWR flag is
> not retransmitted anyway (thus not bound to a point in the sequence
> number space).
[BB] Surely long runs of unidirectional transmitted data don't exhibit 
this problem, 'cos there's plenty of new data to carry the CWR. Or have 
I misunderstood?

In fact, the problem I see with RFC3168 is the opposite case. It seems 
there was an assumption that a data sender would be continually sending 
data, so that, once ECE feedback appeared at the sender, it would 
conveniently always have some data to send, on which CWR could be carried.

For instance, in the sequence above, host A might not send Data#2 for 
ages or perhaps never (a typical case if 'A' is a client requesting a 
large object). In the intervening time, B might send far more than the 
two packets shown. If 'A' does not set CWR on pure ACKs, all B's data 
packets would have to carry ECE, perhaps for many hours, until A has 
some more data to send (if ever).

Nontheless, I think ECE continuing for hours is fine within the logic of 
RFC3168. While A isn't sending anything, it only reduces cwnd once, and 
it's not measuring any round trips, so it's not increasing cwnd either.

Can you describe your case more precisely, so I can understand what 
caused the performance hit?

>
> I therefore propose a change in the Generalized ECN draft, to lift the
> above restriction (while it is "only" a SHOULD, this is one more example
> of an overly strict receiving-side implementation), and no longer
> artificially delay the CWR signal - to become also more useful for
> passive measurements.
[BB] I'm not yet convinced that this CWR behaviour is anything to do 
with the ECN++ draft. But that might be because I've misunderstood your 
description. As I said above, it might be possible to rectify omissions 
with an erratum to RFC3168.

more...

>
>
> Richard
>
> For those interested: The effect of ignoring the CWR on non-new-data
> segments by Linux is, that the ECE flag is left latched. Thus BSD
> continues window-after-window with cwnd reductions, 
[BB] If it's not sending new data, how does the BSD host consider that 
windows are starting or completing?

Bob

> and due to another
> bug where the ECN-induced reduction has no lower bound, eventually cwnd
> ends up at 0 Byte and is only increased to 1 Byte by a Timer - until by
> pure chance, the CWR is sent together with 1 new byte of data. But in
> the preceeding minutes, the session only saw progress by less than 1
> byte / RTT...
>
>
> Am 15.01.2020 um 21:42 schrieb Scheffenegger, Richard:
>>
>> Hi,
>>

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Fri, 14 Feb 2020 19:06:27 +0000

Hi Bob,

>> Another update in this context, which IMHO may be discussed as an 
>> actual change of mechanism with ECN++.
>>
>> client and a BSD server, with request-response workload. That is, 
>> where each side sends out less than MSS data, before the application 
>> waits for the other side to respond to this.
>>

Currently, FBSD is sending CWR with the next packet (pure ACK, Window 
Probe, Retransmission or New Data). But this should be fixed soon(ish).

I used the bracket because RFC3168 is very clear, that CWR should *only* 
be sent with new data, but Linux would also accept it on any packet with 
data (this includes retransmissions and window probes).

> I think RFC3168 expects CWR only on data packets - so that the sender 
> of the CWR can distinguish between ECE that the receiver sends before 
> vs after the receiver got the CWR (by whether the ackno of the ECE 
> covers the CWR data packet or not).

Yes, effectively, CWR is bound to snd_max+1 in the sequence space on 
the first transmission with RFC3168. Unfortunately, it's not stated that 
clearly.

> Consider 4 data packet exchanges of A>B, B>A, B>A, A>B with CWR 
> on pure ACKs.
>
> 'A' doesn't know whether the ECE on Data#102 was sent before or 
> after B received the CWR on A's pure ACK, so 'A' doesn't know 
> whether to reduce its window again or not.
>
> I can't find anything in RFC3168 that explicitly spells out when the sender 
> considers ECE to be in a new round. It jumps straight to describing the 
> exceptional case of a CWR packet being dropped, and omits the 'normal' 
> case of it being delivered.

Well, if the ECE do not stop with the ACK covering the (former) snd_max+1,
the CWR may have been lost (or delivered with a CE mark). In either case, 
another congestion response is appropriate.

>> But binding the CWR flag to a new data segment delays the ECN 
>> signaling loop artificially (for long runs of unidirectional 
>> transmitted data), and it is not clear what the benefit there would 
>> be, as the CWR flag is not retransmitted anyway (thus not bound to a 
>> point in the sequence number space).
>
> this problem, 'cos there's plenty of new data to carry the CWR. Or 
> have I misunderstood?

Transactional data has frequent changes in data direction. But that behavior
(delaying CWR to only send it with new data) IMHO makes ECN unusable for Ack
Moderation (AckCC). Just wondering, if a different way of tracking the packets
In flight during a window would allow the immediate transmission of CWR 
with ECN++, to make ECN useful for Ack Moderation or other future purposes.

> data, so that, once ECE feedback appeared at the sender, it would conveniently
> always have some data to send, on which CWR could be carried.

Correct. With transactional data, this assumption breaks and long flights of 
Packets will continuously have ECE set.

> For instance, in the sequence above, host A might not send Data#2 for ages 
> or perhaps never (a typical case if 'A' is a client requesting a large object). In
> the intervening time, B might send far more than the two packets shown. If 'A'
> does not set CWR on pure ACKs, all B's data packets would have to carry ECE, 
> perhaps for many hours, until A has some more data to send (if ever).
>
> Nontheless, I think ECE continuing for hours is fine within the logic of RFC3168. 
> While A isn't sending anything, it only reduces cwnd once, and it's not 
> measuring any round trips, so it's not increasing cwnd either.
>
> Can you describe your case more precisely, so I can understand what caused 
> the performance hit?

The problem comes from FBSD sending CWR (typically) on pure ACKs - where
Linux ignores the CWR and keeps ECE set; Thus FBSD observes "new" ECE in the 
Following window, resulting in another congestion response - while no further
CE mark was actually present.

This leads to a race to the bottom for FBSD, where cwnd contionously shrinks 
(due to another bug, down to a size of 0 [zero]). Eventually, FBSD clocks out
1 byte every 4 seconds with a timer normally used for Window Probes (but
Unlike Window Probes, that one byte is actually new data).

Eventually CWR may be sent with a data packet (that 1-byte probe), accepted
By Linux, ECE unlatched, and cwnd can start growing again. But that event may
Happen only many minutes into entering this situation (got a trace, where this 
effect lasted nearly 10 minutes).

Obviously, the reverse data direction doesn’t suffer the same problem, nor does
a typical FBSD-FBSD ECN session suffer that fate (even though CWR are sent mostly
on pure ACKs)

>> above restriction (while it is "only" a SHOULD, this is one more 
>> example of an overly strict receiving-side implementation), and no 
>> longer artificially delay the CWR signal - to become also more useful 
>> for passive measurements.
>
> [BB] I'm not yet convinced that this CWR behaviour is anything to do with 
> the ECN++ draft. But that might be because I've misunderstood your 

In RFC3168, both ECT-codepoints and the CWR flag are defined to only be set on
new data segments.

Thus setting both can be simplified into a single branch; Getting the CWR indication
Slightly faster (not binding it to snd_max+1), that is, even with the next pure ack, probe
Or retransmission, would possibly provide a faster feedback (passive measurement
Of the window), and could enable the use of ECN signals for ACK moderation.

The drawback would be additional congestion responses, if the packet carrying the CWR
Is dropped (congestion on the path carrying the pure ACKs). But IIRC, reacting to congestion
On the (relatively) low bandwidth reverse direction by moderating the transmission speed 
(or asking for fewer ACKs) may be not too inappropriate...

> windows are starting or completing?

It still tracks snd_recover (snd_max when the first ECE is received). I have not
Looked into this in detail yet, though. It may even reduce cwnd while being
The passive receiver...

---

From: Bob Briscoe <[email-masked]>
Date: Sat, 15 Feb 2020 09:18:32 +0000

Richard, (pls cc tsvwg as the WG equally responsible for TCP ECN 
maintenance, if you think appropriate)

On 14/02/2020 19:06, Scheffenegger, Richard wrote:
> the CWR may have been lost (or delivered with a CE mark).
[BB] Which end are you talking about? As sender or as receiver? Which 
stage in the sequence are you talking about? Can you be more specific 
relative to the above example, where there are two transactions in a row 
initiated from B without any data from A in between. I deliberately 
constructed this so that A will not be able to tell whether B sent the 
ECE on its second volley (Data#102) before or after B received the CWR 
on A's pure ACK of B's first volley (Data#101).

My purpose was to prove that, if A sends CWR on a pure ACK, it makes 
itself unable to tell whether to reduce its window in response to any 
ECE it receives subsequently, because it cannot tell whether they are 
new or repeats. Therefore, RFC3168 must have meant 'the TCP sender MUST 
set the CWR flag in the TCP header of the first new data packet...' when 
it said:

    "the TCP sender sets the CWR flag in
    the TCP header of the first new data packet sent after the window
    reduction."

I suspect no-one thought to preclude the opposite, i.e. to say 'The 
sender MUST NOT set CWR on packets without new data'. Instead, it went 
straight to clarifying whether '/first/ new data' needed to be mandatory.

    "it SHOULD set the CWR bit only on the first
    new data packet that it transmits"

I reviewed RFC3168 at the time, as did many others, but no-one noticed 
these omissions.

> In either case,
[BB] Surely not, if the CWR was on a pure ACK?

> with ECN++, to make ECN useful for Ack Moderation or other future purposes.
[BB] CWR is only used if RFC3168 feedback has been negotiated.
The WG consensus was that we shouldn't put ECT on pure ACKs if RFC3168 
feedback has been negotiated; only if AccECN had been negotiated.

So, once you have AccECN and ECN++, you can do what you are wanting, 
'cos AccECN allows both the data receiver to set ECT on pure ACKs and 
the data sender to count CEs on arriving pure ACKs. The data sender then 
includes this count in the ACE counter it feeds back on subsequent 
packets it sends (probably data packets, but could be on pure ACKs as 
well when data is bi-directional).

See
https://tools.ietf.org/html/draft-ietf-tcpm-generalized-ecn-05#section-3.2.3.2.1 
(normative)
which refers to
https://tools.ietf.org/html/draft-ietf-tcpm-generalized-ecn-05#section-4.4.1 
(informative)

> Packets will continuously have ECE set.
[BB] I have a terminology blockage here. I interpret 'long flights of 
packets' to be a long-running flow - the opposite of 'transactional 
data'. But you seem to be using 'long flights of data' to describe 
'transactional data'. Not sure what the disconnect is.

> CE mark was actually present.
[BB] I think you are solely talking about the case where sending of each 
transactional data volley strictly alternates between A and B.

I am trying to get you to think about a case where one end sends more 
than one volley in succession, in order to show that setting CWR on a 
pure ACK creates problems. That's why I asked, how does an FBSD receiver 
(or Linux receiver for that matter) know what a 'following window' is if 
it's only sending pure ACKs? They all have the same seqno and they don't 
get ACK'd. So the pure receiver can't know what an RTT is.

I also asked whether, if the FBSD host (that has been receiving data) 
does send some data between its pure ACKs, does it repeat the CWR on the 
data even tho it already set it on an earlier pure ACK?

> In RFC3168, both ECT-codepoints and the CWR flag are defined to only be set on
[BB] Not strictly true for the CWR flag, as discussed earlier, this part 
of RFC3168 seems not to have been fully tied up.

> (or asking for fewer ACKs) may be not too inappropriate...
[BB] I don't believe the CWR mechanism is suitable for faster congestion 
notification or for Ack CC. The way it's designed is tied in so much to 
reliable delivery that it actually breaks if you send it unreliably on 
pure ACKs (as I've tried to prove, and as you have discovered).

> Looked into this in detail yet, though. It may even reduce cwnd while being
[BB] (repeating what I've said already) I don't see how it can know what 
an RTT is when its continually sending pure ACKs with the same seqno.

Cheers

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 19 Feb 2020 09:56:48 +0000

Richard,
(pls cc tsvwg as the WG equally responsible for TCP ECN maintenance, if 
you think appropriate,
I've cut everyone except lists off the distro, 'cos posts were getting 
rejected )

Inline...

See

Cheers

Bob