Subject: Re: [tcpm] Fwd: Fwd: New Version Notification for draft-kuehlewind-tcpm-accurate-ecn-04.txt

From: Bob Briscoe <[email-masked]>
Date: Wed, 14 Oct 2015 01:22:05 +0100

Praveen,

Sorry I've taken so long to reply.
Inline...

On 28/09/15 18:35, Praveen Balasubramanian wrote:
>
> Hi all,
>
> I reviewed this draft and have a few comments below. Overall the draft 
> is well written and covers all the corner cases.
>
> Thanks
>
Thankyou. We tried v hard to keep it clear, and to ensure that the 
logical flow was not interrupted by the corner cases.
>
> The section 2.3 summary immediately brings about a question around 
> stretch ACKs. I see that you explain the receiver requirement in 
> section 3.2.2 (the delayed ACK frequency limit of 6 for CE marked 
> packets) but it might be worth suggesting this up front.
>
That's a good point - as you know, section 2 is meant to be a discursive 
summary of the detailed rules in section 3. However, in this case, I 
think we rewrote 3.2.2 after we had written 2.3. So, you're right that 
the summary ought to give more of a hint of how this will be solved. 
Will do.
>
> Is there any model to suggest what would be the accuracy of the scheme 
> described in Appendix A.3 to estimate the number of marked bytes from 
> the ACE field? Using a new TCP option will have deployment issues.
>
Yes, it's important to check how functional the protocol will be for 
those cases where the TCP option won't get through.
But no, we haven't yet modelled the accuracy of estimating marked bytes 
from marked packets.

I don't want to spend too much time on that until someone has attempted 
to implement AccECN (Mirja has made a good start). Because, at least in 
Linux, it might be relatively easy for the sender to record the sizes of 
all the segments in flight, so that the sender can calculate, rather 
than estimate, CE marked bytes.

Estimating, rather than calculating, might still be necessary for Linux, 
and certainly some other OSs such as FreeBSD. But I would rather 
understand what is possible in at least one OS before modelling this 
problem.

At present DCTCP measures the extent of congestion in packets not bytes. 
But we envisaged that an algorithm like Relentless TCP might be used 
where,  every time new CE feedback arrives, the sender just decrements 
cwnd by (CE bytes)/2.

> With AccECN if SYN can now be safely ECN-capable then what are the 
> drawbacks of making it a requirement?
>
FIrst I need to correct the premise of your question. Just because an 
AccECN receiver /can/ feed back CE on a SYN, does not mean it is safe 
for a sender to set ECN-capable on a SYN. A feedback protocol has to 
feed back stuff it receives, whether or not that stuff is valid (so that 
the sender can know what stuff appeared at the other end in order to 
detect middlebox mangling as well as valid protocol transitions).

Quoting section 2.5  entitled "Generic (Dumb) Reflector":

    Providing feedback of CE marking on the SYN also supports future
    scenarios in which SYNs might be ECN-enabled (without prejudging
    whether they ought to be).

Indeed, an ECN-capable SYN is outside the scope of AccECN, which solely 
covers the feedback behaviour of a receiver. An ECN-capable SYN would 
have to be defined in an RFC that covered TCP sender behaviour. E.g. 
perhaps, once your DCTCP draft becomes RFC, we might work on an 
evolution of DCTCP for both data centres and the public Internet, as 
discussed in Prague.

Nonetheless, to answer your question, IMO making a SYN ECN-capable has 
no drawbacks, /as long as/ it is accompanied by the following safety 
measures:

a) At the SYN stage, the TCP client doesn't yet know whether the server 
even understands ECN. So if a SYN were ECN-capable and then it was 
marked CE in the network, a non-ECN capable receiver would just ignore 
the marking. Therefore, if the sender received a SYN-ACK that did not 
complete the ECN negotiation, it would have to conservatively assume 
that the SYN might have been CE-marked, and reduce its initial window as 
if it was.

b) There is some concern that ECN-capable SYN floods could starve 
non-ECN traffic. However, all AQMs are required to switch off ECN 
marking if the queue starts to overload. So, before I believe an 
ECN-capable SYN could be a DoS risk, I would like to see someone 
demonstrate an ECN-capable SYN flood that does more harm than a non-ECN 
SYN flood. I doubt it would be possible, because before the attack could 
have any effect, the cause of the effect (ECN) would be turned off.

Interestingly, in the recent (2015) study on ECN traversal (Trammell et 
al) they found ECT or CE on SYN was dropped in only 0.82% and 0.61% of 
cases (v4 & v6 resp.), which was (to me) surprisingly low, given RFC3168 
says (without evidence) that this could be a DoS attack vector.

> Regarding the requirement that "a TCP server that confirms its support 
> for AccECN (as described in Section 3.1) MUST also include an AccECN 
> TCP Option in the SYN/ACK", it seems onerous to make the supplementary 
> part a MUST. The server may have prior information that packets in 
> 3WHS are dropped on a particular path. The server may also have a knob 
> which is set in cases where the middleware is known to discard packets 
> with unknown TCP option.
>
Good point. We need to allow an exception where the host has cached 
knowledge that AccECN option black-holes (but it should test 
occasionally in case the black hole has been plugged).
Not only for the SYN/ACK from the server, but also for the first ACK and 
first data segment from the client.

> From practical deployment point of view section 3.3 needs to be 
> expanded upon to include how LSO and LRO would work in presence of the 
> new option.
>
Yes, it might be useful to write another appendix with examples of how 
segmentation offload might work. If you could help write that, it would 
be v useful (I know Richard has expertise in this area too).
>
> For the LSO case the host can choose to generate an extra ACK with the 
> option rather than piggybacking it on a large send. However even if 
> the option is set on an LSO send, the NIC can repeat the option in 
> every segment without any adverse affects.
>
Care! We said the following for a reason:

    Nonetheless, such hardware MUST attempt to preserve the timing of
    each ACK (for example, if it coalesced ACKs it would not be AccECN-
    compliant).

If the large send coalesces the ACKs (whether data ACKs or pure ACKs) 
for a number of received segments, the other end would see possibly 
multiple counters incremented, representing different markings on 
different arriving packets.

...Ideally we wanted the feedback to make it possible for a sender to 
extract the order and the timing of each transition between one ECN 
marking and another, as they arrived at the receiver. For instance, in 
2010, Mirja and I wrote a paper about how to implement rapid detection 
of available capacity using packet chirps 
<http://www.bobbriscoe.net/pubs.html#chirp_impl>. It would be really 
useful to be able for AccECN to support any senders trying to do 
something similar, but with a shallow ECN threshold like that used in 
DCTCP, particularly during slow-start.

I'm not saying all senders would have to work like that, just that it 
would be nice if a generic receiver behaviour would allow some senders to.

Examples like this justify what we called "change-triggered ACKs". We 
(the co-authors) had long internal arguments about whether to make 
change-triggered ACKs a MUST or a SHOULD. We decided that the draft 
would say 'SHOULD', but the WG might want to continue our argument.

The problem: if a sender wants to rely on change-triggered ACKs to get 
rapid info early in a new flow, it needs to be sure that the receiver 
has implemented change-triggered ACKs. We wanted to keep it simple, so 
we didn't want this to be a negotiable option. Therefore, such a sender 
would want change-triggered ACKs to be a MUST. However, we didn't want 
to write a spec that might be impossible to implement with today's 
hardware. So we settled on a SHOULD. But then our chirping sender cannot 
be sure whether the feedback is doing what it SHOULD or not, so it has 
to detect what the receiver is probably doing, which wastes time - 
precisely what the sender doesn't want to do.

So, in the end, we decided to write the SHOULD with an extra-strong plea 
to say you really ought to think really hard before not implementing 
this SHOULD - what one might call a "MUST-SHOULD"!

> LRO will be complicated if data segments contain the new option. At 
> least in Windows, LRO does not coalesce pure ACKs.
>
My personal view: I don't worry if acceleration hardware cannot handle 
newly defined capabilities, as long as software implementations are fast 
enough for the new protocol to be usable and to become successful. IMO, 
acceleration hardware is designed to accelerate today's common 
operations, and no-one would expect it to always be able to accelerate a 
newly invented protocol. As long as AccECN is good enough to be 
successful, tomorrow's hardware will accelerate it.

This is definitely an area where we are looking for input from people 
like you and others in the WG.

Thanks again for your comments.
We've got a few minor edits, which we will fold in with the changes 
promised above, assuming you, my co-authors and the WG are all happy 
with my responses.

Cheers

Bob

> *From:*tcpm [mailto:[email-masked]] *On Behalf Of *Bob Briscoe
> *Sent:* Sunday, September 20, 2015 2:46 PM
> *To:* Mirja Kühlewind <[email-masked]>; 
> [email-masked]
> *Cc:* tcpm IETF list <[email-masked]>; Richard Scheffenegger 
> <[email-masked]>
> *Subject:* Re: [tcpm] Fwd: Fwd: New Version Notification for 
> draft-kuehlewind-tcpm-accurate-ecn-04.txt
>
> tcpPrague list and Mirja,
>
> For those of you who don't follow the IETF so closely, the reason for 
> not using DCTCP's original feedback scheme was because it was not 
> designed to cope with ACK loss.
>
> This is explained in the requirements for more accurate ECN feedback 
> that were recently agreed and published as RFC 7560 
> <https://tools.ietf.org/html/rfc7560>. The appendix in that RFC gives 
> some examples where the original DCTCP would get highly confused by a 
> few ACK losses.
>
> In that RFC it was admitted that no-one expected that all the 
> requirements could be satisfied at once.
> * The previous version <draft-kuehlewind-tcpm-accurate-ecn-03> 
> satisfied them all except 'simple', but there was push-back on that.
> * So this time <draft-kuehlewind-tcpm-accurate-ecn-04 
> <https://tools.ietf.org/html/draft-kuehlewind-tcpm-accurate-ecn-04>> 
> has gone for simple, but compromised a bit on 'low to zero overhead'.
>
> As Mirja said, pls read it and tell the IETF tcpm list (in cc) whether 
> you support this new approach, and if not, why not.
> The IETF doesn't charter work if no-one is talking about it.
>
> Cheers
>
>
> Bob
>
> On 09/09/15 17:24, Mirja Kühlewind wrote:
>
>     Hi all,
>
>     find below the mail that I've just sent to the tcpm list. You
>     might have seen this already but I ed explicitly point this out to
>     the tcpprague list as we partly discussed this activity already at
>     the meeting in Prague.
>
>     In short AccECN can be used with DCTCP or any other (DCTCP-like)
>     scheme that needs more accurate information on how many ECN
>     markings have been received. If that's interesting for you, please
>     read the draft and provide feedback. Please make sure that you
>     also cc the tcpm list regarding all feedback that is directly on
>     AccECN and the draft!
>
>     If you would like to discuss future usage of AccECN, this can
>     happen on this list only, or you may cc tcpm if e.g. your use case
>     would require changes to AccECN as currently proposed.
>
>     Thanks!
>     Mirja
>
>
>     -------- Forwarded Message --------
>     Subject: [tcpm] Fwd: New Version Notification for
>     draft-kuehlewind-tcpm-accurate-ecn-04.txt
>     Date: Wed, 9 Sep 2015 17:52:13 +0200
>     From: Mirja Kühlewind <[email-masked]>
>     <mailto:[email-masked]>
>     To: [email-masked] <mailto:[email-masked]> Extensions
>     <[email-masked]> <mailto:[email-masked]>, Bob Briscoe
>     <[email-masked]> <mailto:[email-masked]>, Richard
>     Scheffenegger <[email-masked]> <mailto:[email-masked]>
>
>     Hi all,
>
>     we submitted a new draft for AccECN (see below). We significantly
>     simplified the
>     draft because we believe it is more important to have a solution
>     that is easy to
>     understand and therefore more easy to correctly implement, than
>     having a
>     solution that tries to fulfill all requirements but is overly
>     complex.
>
>     In short, we now only overwrite the 3 TCP header bits (no
>     additional bits in the
>     TCP header are used) and use them simply for a CE packet counter,
>     while
>     bytes-wise information on all other markings is provided by a new
>     TCP option. In
>     case the option is not available, e.g. because it's blocked, this
>     will still
>     provide the needed information to react to a CE-based congestion
>     feedback.
>     However, any advanced mechanism that needs further information on
>     the other
>     markings received will only work if the option is available.
>
>     Further, we also tried to keep the draft as short as possible.
>     That means the
>     actual normative part only has 9 pages. However, we still provide
>     an overview
>     section with some reasoning as well as some discussion on
>     interaction with other
>     mechanisms which leads in total to 25 pages (without appendix).
>
>     I'm currently also working on an implementation of this proposed
>     solution. I
>     will announce it as soon as I'm ready!
>
>     In any case we would like to discuss this at the next meeting. We,
>     the author,
>     think that this proposal is now the right way forward and hope
>     that this
>     simplified proposal will allows us to quickly proceed on AccECN as
>     the need for
>     it is increasing.
>
>     Please let us know if you have any feedback on the draft!
>
>     Mirja
>
>
>     -------- Forwarded Message --------
>     Subject: New Version Notification for
>     draft-kuehlewind-tcpm-accurate-ecn-04.txt
>     Date: Sun, 06 Sep 2015 16:19:47 -0700
>     From: [email-masked] <mailto:[email-masked]>
>     To: Richard Scheffenegger <[email-masked]> <mailto:[email-masked]>,
>     "Mirja Kühlewind"
>     <[email-masked]>
>     <mailto:[email-masked]>, Mirja Kuehlewind
>     <[email-masked]>
>     <mailto:[email-masked]>, Richard Scheffenegger
>     <[email-masked]> <mailto:[email-masked]>, Bob
>     Briscoe <[email-masked]> <mailto:[email-masked]>, Bob
>     Briscoe <[email-masked]> <mailto:[email-masked]>
>
>
>     A new version of I-D, draft-kuehlewind-tcpm-accurate-ecn-04.txt
>     has been successfully submitted by Bob Briscoe and posted to the
>     IETF repository.
>
>     Name:        draft-kuehlewind-tcpm-accurate-ecn
>     Revision:    04
>     Title:        More Accurate ECN Feedback in TCP
>     Document date:    2015-09-06
>     Group:        Individual Submission
>     Pages:        36
>     URL:
>     https://www.ietf.org/internet-drafts/draft-kuehlewind-tcpm-accurate-ecn-04.txt
>
>     Status:
>     https://datatracker.ietf.org/doc/draft-kuehlewind-tcpm-accurate-ecn/
>     Htmlized:
>     https://tools.ietf.org/html/draft-kuehlewind-tcpm-accurate-ecn-04
>     Diff:
>     https://www.ietf.org/rfcdiff?url2=draft-kuehlewind-tcpm-accurate-ecn-04
>
>
>     Abstract:
>         Explicit Congestion Notification (ECN) is a mechanism where
>     network
>         nodes can mark IP packets instead of dropping them to indicate
>         incipient congestion to the end-points.  Receivers with an ECN-
>         capable transport protocol feed back this information to the
>     sender.
>         ECN is specified for TCP in such a way that only one feedback
>     signal
>         can be transmitted per Round-Trip Time (RTT).  Recently, new TCP
>         mechanisms like Congestion Exposure (ConEx) or Data Center TCP
>         (DCTCP) need more accurate ECN feedback information whenever more
>         than one marking is received in one RTT.  This document
>     specifies an
>         experimental scheme to provide more than one feedback signal
>     per RTT
>         in the TCP header.  Given TCP header space is scarce, it
>     overloads
>         the three existing ECN-related flags in the TCP header and
>     provides
>         additional information in a new TCP option.
>
>
>
>
>     Please note that it may take a couple of minutes from the time of
>     submission
>     until the htmlized version and diff are available at tools.ietf.org.
>
>     The IETF Secretariat
>
>
>     _______________________________________________
>     tcpm mailing list
>     [email-masked] <mailto:[email-masked]>
>     https://www.ietf.org/mailman/listinfo/tcpm
>
>
>
> -- 
> ________________________________________________________________
> Bob Briscoehttp://bobbriscoe.net/

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: "De Schepper, Koen (Koen)" <[email-masked]>
Date: Wed, 14 Oct 2015 17:37:14 +0000

Hi,

Just finished reading/reviewing this draft. I think it is really well written, simple and it has gone clearly through a lot of thoughts already. 

Below some questions and remarks:

- Is how this protocol is intended to be used by classic-ECN-TCP and DCTCP documented somewhere? I think it useful to describe (not sure if you want to add it to this doc?)

- Why the quite strong restrictions in what offload engines are allowed to do. Is it useful to clearly specify what an offload engine must do? For example I can imagine is a large list of packets are received in a NIC, the engine can combine all acks and data packets as long as only one counter is incremented, and only report the last value of the counter in the option. Also as long as it can guarantee that the ACE counter is not interpreted as being wrapped, it also can offload by combining up to 7 packets or acks...

- If L4S (DCTCP/TCP-Prague) should use ECT1, then it needs always 11 and sometimes 8 bytes options... Why not putting ECEB first, and allow a more relaxed requirement on signaling EE0B and EE1B? If option space is limited, at least ECEB will be able to get through enough to estimate the congestion.

- If I am interested in the probability of marks, just echoing the last received IP mark would perfectly fit this requirement (assuming random marking). Currently this is not supported, and thinking about this, your 'safe' algorithms will give potentially quite deviating results, taking all possible ACK losses/misses into account. Are these future-use requirements worth the overhead/complexity?

- L4S (DCTCP/Prague/...) can have often (random) marking probabilities around 50%, causing the change triggered acks to ack almost every packet (this is what the current DCTCP is doing too, but on a step-RED AQM you get large on-off series, with DualQ you get a random distribution with none or small series). Is order so important? Otherwise, the acks could just carry EE0B (or even immediately ECEB as above), the sender knows how much bytes it has sent, assuming all the rest is ECEB (EE0B/EE1B as appropriate).

Some typo's and confusing text:

- The term classic ECN confused me a few times. I suppose you mean classic ECN echoing protocol, not the response to congestion in the TCP sender. Maybe consistent use of 'classic ECN echoing' is clearer.

- The draft is structured very good, as I could read sequencially though it, except for one case where I had to look further to understand what was exactly meant: The third paragraph of section 3.1 was "... TCP server MUST set NS=1 on the SYN/ACK.". Changing it to "... TCP server MUST set additionally NS=1 on the SYN/ACK.", and optionally add that this is only when in accECN mode, would have clarified all doubts.

- The sixth paragraph of section 3.1 contains an If (with capital I) in the middle of the fist sentence.

- Section 3.2.1 has "thy" instead of "they".

- Section 3.2.2 has "(n)" instead of "n", later it also appears as "n" without the brackets.

- Section 5 has triiggered instead of triggered

Regards,
Koen.

> -----Original Message-----
> From: tcpm [mailto:[email-masked]] On Behalf Of Mirja Kühlewind
> Sent: woensdag 9 september 2015 17:52
> To: [email-masked] Extensions; Bob Briscoe; Richard Scheffenegger
> Subject: [tcpm] Fwd: New Version Notification for draft-kuehlewind-
> tcpm-accurate-ecn-04.txt
> 
> Hi all,
> 
> draft because we believe it is more important to have a solution that
> is easy to
> understand and therefore more easy to correctly implement, than having
> a
> solution that tries to fulfill all requirements but is overly complex.
> 
> In short, we now only overwrite the 3 TCP header bits (no additional
> bits in the
> TCP header are used) and use them simply for a CE packet counter, while
> bytes-wise information on all other markings is provided by a new TCP
> option. In
> case the option is not available, e.g. because it's blocked, this will
> still
> However, any advanced mechanism that needs further information on the
> other
> Further, we also tried to keep the draft as short as possible. That
> means the
> actual normative part only has 9 pages. However, we still provide an
> overview
> section with some reasoning as well as some discussion on interaction
> with other
> In any case we would like to discuss this at the next meeting. We, the
> author,
> think that this proposal is now the right way forward and hope that
> this
> simplified proposal will allows us to quickly proceed on AccECN as the
> need for
> it is increasing.
> 
> Subject: New Version Notification for draft-kuehlewind-tcpm-accurate-
> ecn-04.txt
> From: [email-masked]
> To: Richard Scheffenegger <[email-masked]>, "Mirja Kühlewind"
> <[email-masked]>, Mirja Kuehlewind
> <[email-masked]>, Richard Scheffenegger
> <[email-masked]>, Bob
> Briscoe <[email-masked]>, Bob Briscoe <[email-masked]>
> 
> 
> Status:         https://datatracker.ietf.org/doc/draft-kuehlewind-tcpm-
> accurate-ecn/
> Htmlized:       https://tools.ietf.org/html/draft-kuehlewind-tcpm-
> accurate-ecn-04
> Diff:
>     Explicit Congestion Notification (ECN) is a mechanism where network
>     than one marking is received in one RTT.  This document specifies
> an
>     experimental scheme to provide more than one feedback signal per
> RTT
>     in the TCP header.  Given TCP header space is scarce, it overloads
>     the three existing ECN-related flags in the TCP header and provides
> [email-masked]

---

From: "De Schepper, Koen (Koen)" <[email-masked]>
Date: Wed, 14 Oct 2015 17:46:30 +0000

An additional thought, also useful for the TCP-Prague list:

- If the goal is to replace classic ECN echoing on the Internet, isn't it better NOT to fall back to classic ECN, but better (as a MUST?) always to drop? It will be an incentive to upgrade on the Internet (and avoid classic ECN echoing behavior). In the datacenter, it should probably fall back assuming there is a large deployment base of 'classic ECN echoing' DCTCPs.

Regards,
Koen.

> From: De Schepper, Koen (Koen)
> Sent: woensdag 14 oktober 2015 19:37
> To: 'Mirja Kühlewind'; [email-masked] Extensions; Bob Briscoe; Richard
> Scheffenegger
> Subject: RE: [tcpm] Fwd: New Version Notification for draft-kuehlewind-
> tcpm-accurate-ecn-04.txt
> 
> Hi,
> 
> Just finished reading/reviewing this draft. I think it is really well
> written, simple and it has gone clearly through a lot of thoughts
> already.
> 
> - Is how this protocol is intended to be used by classic-ECN-TCP and
> DCTCP documented somewhere? I think it useful to describe (not sure if
> you want to add it to this doc?)
> 
> - Why the quite strong restrictions in what offload engines are allowed
> to do. Is it useful to clearly specify what an offload engine must do?
> For example I can imagine is a large list of packets are received in a
> NIC, the engine can combine all acks and data packets as long as only
> one counter is incremented, and only report the last value of the
> counter in the option. Also as long as it can guarantee that the ACE
> counter is not interpreted as being wrapped, it also can offload by
> combining up to 7 packets or acks...
> 
> - If L4S (DCTCP/TCP-Prague) should use ECT1, then it needs always 11
> and sometimes 8 bytes options... Why not putting ECEB first, and allow
> a more relaxed requirement on signaling EE0B and EE1B? If option space
> is limited, at least ECEB will be able to get through enough to
> estimate the congestion.
> 
> - If I am interested in the probability of marks, just echoing the last
> received IP mark would perfectly fit this requirement (assuming random
> marking). Currently this is not supported, and thinking about this,
> your 'safe' algorithms will give potentially quite deviating results,
> taking all possible ACK losses/misses into account. Are these future-
> use requirements worth the overhead/complexity?
> 
> - L4S (DCTCP/Prague/...) can have often (random) marking probabilities
> around 50%, causing the change triggered acks to ack almost every
> packet (this is what the current DCTCP is doing too, but on a step-RED
> AQM you get large on-off series, with DualQ you get a random
> distribution with none or small series). Is order so important?
> Otherwise, the acks could just carry EE0B (or even immediately ECEB as
> above), the sender knows how much bytes it has sent, assuming all the
> rest is ECEB (EE0B/EE1B as appropriate).
> 
> - The term classic ECN confused me a few times. I suppose you mean
> classic ECN echoing protocol, not the response to congestion in the TCP
> sender. Maybe consistent use of 'classic ECN echoing' is clearer.
> 
> - The draft is structured very good, as I could read sequencially
> though it, except for one case where I had to look further to
> understand what was exactly meant: The third paragraph of section 3.1
> was "... TCP server MUST set NS=1 on the SYN/ACK.". Changing it to "...
> TCP server MUST set additionally NS=1 on the SYN/ACK.", and optionally
> add that this is only when in accECN mode, would have clarified all
> doubts.
> 
> having
> will
> the
> the

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 15 Oct 2015 23:05:40 +0100

Koen,

> Just finished reading/reviewing this draft. I think it is really well written, simple and it has gone clearly through a lot of thoughts already.
Thx. Praise is always nice.

Here's a question back to you, especially if you read the previous 
version (-03): If you had to try to meet all the conflicting 
requirements for accurate ECN feedback in RFC 7560, would you have 
chosen the set of compromises we chose here? This time we haven't 
included as many tricks to traverse middlebox meddling, to keep it 
simpler. And this time we've not tried so hard to minimise header 
overhead, again to keep it simple.
> - Is how this protocol is intended to be used by classic-ECN-TCP and DCTCP documented somewhere? I think it useful to describe (not sure if you want to add it to this doc?)
That isn't how to interpret an IETF RFC. RFCs do not tell anyone that 
they have to do anything. An RFC describes the behaviours that an 
implementation MUST/SHOULD/MAY exhibit in order to be able to claim that 
it complies with that RFC.Then, voluntarily, implementers can choose to 
comply with the RFCs in order to be able to claim that they will 
interoperate with others in respect of the relevant scope.

Unlike the ITU or ETSI, the IETF is not a standards body with any legal 
standing. It has a much more practical voluntary model, which means it 
is still populated by some engineers not just political officers.

So, once AccECN is an experimental RFC, an implementer of an 
RFC3168-compliant stack could choose to write a variant that complies 
with AccECN. And similarly, for an implementer of a DCTCP stack.

Have I understood the question correctly?
>
> - Why the quite strong restrictions in what offload engines are allowed to do. Is it useful to clearly specify what an offload engine must do?
See also my response to the same point made by Praveen: any help more 
clearly specifying offload behaviour is welcome.
> For example I can imagine is a large list of packets are received in a NIC, the engine can combine all acks and data packets as long as only one counter is incremented, and only report the last value of the counter in the option. Also as long as it can guarantee that the ACE counter is not interpreted as being wrapped, it also can offload by combining up to 7 packets or acks...
Yes it will be necessary to carefully write this down, because any 
coalescing has to meet constraints on both half-connections - not hiding 
transitions in the ECN field in one direction, and not coalescing ACKs 
that change more than one counter in the other. It's quite tricky to 
find all these constraints in different parts of the doc right now, so 
it would be good to re-write what it means for offload hardware.

I assume you are mostly referring to the sentence:

Timing information is part of the "Accurate Feedback" of AccECN. Timing 
could be written as bits in a packet, or it can be conveyed through the 
actual time at which separate ACKs arrive. We decided not to write 
timing as bits in ACKs, and not to require the timing when every ECN 
marking arrives to be reflected in the timing of separate ACKs. But we 
did want {Note 1} the timing of /transitions/ in the marking to be 
reflected in the timing of an ACK. That is why that statement is there.

{Note 1} I should say "I did want", because Mirja & Richard gave me an 
action to write-up a use-case where this timing info could be useful. My 
argument was based on the work of Mirja & I (and others) on faster start 
up, by the sender probing with different spacing between packets 
(chirps). This can be necessary whenever another flow leaves, not just 
when a new flow starts.
> - If L4S (DCTCP/TCP-Prague) should use ECT1, then it needs always 11 and sometimes 8 bytes options... Why not putting ECEB first, and allow a more relaxed requirement on signaling EE0B and EE1B? If…
If L4S were to use ECT1, we would have to change the way the option is 
arranged.

We had already had a number of different ideas on how to be efficient 
for different possible future patterns of ECT(0) and ECT(1), but we 
wrote only the simplest one for now. it is more important for now to 
keep this draft short and simple. Then, dependent on how the L4S 
identifier debate goes, and whether the WG wants something more generic, 
we can change this aspect.

If you're interested, you can pull the XML from the IETF datatracker 
server. In the XML of Appendix B, there is a commented out alternative 
arrangement for the option - but for a different reason.

>
> - If I am interested in the probability of marks, just echoing the last received IP mark would perfectly fit this requirement (assuming random marking). Currently this is not supported,
Surely you mean that extracting the probability of a mark /is/ currently 
supported, but the simpler feedback that you propose is not supported, 
even though it would be /sufficient/ for your need.

We wanted to have a feedback protocol without any sub-negotiation of the 
behaviour. So that a single receiver behaviour suffices for a wide range 
of possible sender requirements. We could have gone down the route of 
negotiating different behaviours, but that would lead to both complex 
code (e.g. if each end implements something different) and interminable 
arguments about which ones are standardised, and which ones are not.
> and thinking about this, your 'safe' algorithms will give potentially quite deviating results, taking all possible ACK losses/misses into account.
Pls expand on this. It sounds like you're saying we're wrong when we claim:

    Resilience against Bias:  Because feedback is based on repetition of
       counters, random losses do not remove any information, they only
       delay it.  Therefore, even though some ACKs are change-triiggered,
       random losses will not alter the proportions of the different ECN
       markings in the feedback.

> Are these future-use requirements worth the overhead/complexity?
The TCP wire protocol can only be changed once every decade or so, so we 
have to get it right for multiple uses, not just one. If this draft gets 
adopted, it becomes WG property, then everyone (including the authors) 
will be entitled to argue their particular choice of compromise on this 
point.
> - L4S (DCTCP/Prague/...) can have often (random) marking probabilities around 50%, causing the change triggered acks to ack almost every packet (this is what the current DCTCP is doing too, but on…
Agreed that there could be a lot of change-triggered ACKs for some 
marking patterns. But, as the marking probability increases, the strings 
of CE will probably get longer as well - I think it would be quite 
unusual (tho possible) to have constant pkt-by-pkt alternation between 
CE and ECT.

Also, to mitigate the extra change-triggered ACKs, the receiver could 
increase the delayed ACK factor (above 2), so that when there is a 
string of constant marking (whether CE or ECT), it sends less ACKs, 
which could lead to the same or even less ACKs overall (at the expense 
of more burstiness from the sender if it is not pacing).

Again, your question depends on the premise that this feedback will only 
be used for the one thing you want to do today.
> - The term classic ECN confused me a few times. I suppose you mean classic ECN echoing protocol, not the response to congestion in the TCP sender. Maybe consistent use of 'classic ECN echoing' is c…
Good point. We need to define classic ECN echoing for this doc as "the 
TCP feedback aspect of RFC 3168 ECN".

The scope outlined in the introduction is only the layer 4 feedback 
protocol on the wire, not the sender behaviour in response. This was the 
source of a lot of robust discussion between the co-authors.

[I've just realised we've made the abbreviation L4 for layer 4 a bit 
confusable with 'low loss low latency' now!]
> - Section 5 has triiggered instead of triggered
All accepted - I'll try to do an update before Monday's deadline, 
including those I promised Praveen, a few I've noticed myself, and Mirja 
has some changes to the algos from having better understood through 
implementation.

Bob

---

From: Mirja Kühlewind <[email-masked]>
Date: Fri, 16 Oct 2015 09:22:00 +0200

Hi,

> Am 14.10.2015 um 02:22 schrieb Bob Briscoe <[email-masked]>:
> 
> At present DCTCP measures the extent of congestion in packets not bytes. But we envisaged that an algorithm like Relentless TCP might be used where,  every time new CE feedback arrives, the sender just decrements cwnd by (CE bytes)/2.

As a side note, the current DCTCP implementation in Linux uses number of bytes...

---

From: Praveen Balasubramanian <[email-masked]>
Date: Fri, 16 Oct 2015 16:32:04 +0000

Even the Windows implementation uses the number of bytes in CE marked packets. Bob, I am not sure why you think extent of congestion is based on packets and not bytes, please check section 3.3 in the draft. 

-----Original Message-----
From: Mirja Kühlewind [mailto:[email-masked]] 
Sent: Friday, October 16, 2015 12:22 AM
To: Bob Briscoe <[email-masked]>
Cc: Praveen Balasubramanian <[email-masked]>; tcpm IETF list <[email-masked]>; Richard Scheffenegger <[email-masked]>
Subject: Re: [tcpm] New Version Notification for draft-kuehlewind-tcpm-accurate-ecn-04.txt

Hi,

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 18 Oct 2015 18:02:20 +0100

Praveen,

Sorry, there's ambiguity in these emails between the units of feedback 
and the units of calculation of the congestion response. Current DCTCP 
feedback (both Windows and Linux) is in packets, then the sender 
calculates/estimates what this means in bytes which it applies in the 
window reduction algo. This calculation/estimation is accurate for 
long-running flows, but hard for app-limited flows where there are a lot 
of sub-MSS segments.

The example congestion response I gave (Relentless) uses bytes for 
feedback and bytes for the response calculation. This works because 
Relentless is designed for loss and SACK gives loss feedback in bytes 
(Relentless didn't include any change to ECN feedback).

The AccECN draft-04 proposes essential feedback in packets using the 
main TCP header and supplementary feedback in bytes using a new AccECN 
TCP option (middleboxes permitting). If the AccECN option gets through 
OK, it means the sender has bytes directly, so it doesn't have to 
estimate or calculate bytes from packet feedback.

I should add that providing byte counters wasn't the main reason for the 
TCP option. It was primarily to improve protocol resilience in the face 
of heavy ACK loss (when congestion avoidance could be most critical), 
because the 3-bit field available for the packet counter in the main 
header is rather small (it wraps every 8 packets, which is at least 
better than the 1-bit feedback in current DCTCP).

So, once we added the TCP option for resilience, we could use it for 
full byte feedback instead of packet, and we could also provide enough 
info to detect bleaching of the ECN field and to support potential 
future use of ECT(1) as well as ECT(0).

Hope this clarifies the confusion.

Bob

---

From: Mirja Kühlewind <[email-masked]>
Date: Wed, 21 Oct 2015 17:45:26 +0200

Hi Bob,

no, there is no estimation in the current DCTCP implementation because the 
feedback mechanism they use indicates that either all bytes that a certain 
ACK acknowledges have been marked or not; so you directly have the number of 
bytes.

That means effectively AccECN without the Option would provide you less 
information than you currently have with DCTCP's feedback mechanism. However, 
in DCTCP there is a risk of loosing information/obtaining wrong information 
if ACKs get lost. This might be a rare case in data centers, however, I'd 
still argue for safety here.

Mirja

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 28 Oct 2015 13:46:05 +0000

Mirja,

Surely, even if the TCP option is stripped, our proposed AccECN 
mechanism provides the same info as DCTCP currently does, with the 
addition of robustness against ACK loss (because we use a 3-bit counter, 
whereas DCTCP uses a 1-bit flag). At least I believe that's the case if 
the receiver complies with our change-triggered feedback requirement, 
which we have stated as a 'SHOULD' and emphasised that it means "you 
really really SHOULD unless you cannot".

Just like DCTCP, the sender can then calculate the CE bytes from the 
number of bytes the ACK acknowledges.

Bob

On 21/10/15 16:45, Mirja Kühlewind wrote:
> Hi Bob,
>
> no, there is no estimation in the current DCTCP implementation because 
> the feedback mechanism they use indicates that either all bytes that a 
> certain ACK acknowledges have been marked or not; so you directly have 
> the number of bytes.
>
> That means effectively AccECN without the Option would provide you 
> less information than you currently have with DCTCP's feedback 
> mechanism. However, in DCTCP there is a risk of loosing 
> information/obtaining wrong information if ACKs get lost. This might 
> be a rare case in data centers, however, I'd still argue for safety here.
>
> Mirja
>
>
>>> Even the Windows implementation uses the number of bytes in CE 
>>> marked packets. Bob, I am not sure why you think extent of 
>>> congestion is based on packets and not bytes, please check section 
>>> 3.3 in the draft.
>>>
>>> Cc: Praveen Balasubramanian <[email-masked]>; tcpm IETF list 
>>> <[email-masked]>; Richard Scheffenegger <[email-masked]>
>>> Subject: Re: [tcpm] New Version Notification for 
>>>> At present DCTCP measures the extent of congestion in packets not 
>>>> bytes. But we envisaged that an algorithm like Relentless TCP might 
>>>> be used where,  every time new CE feedback arrives, the sender just 
>>>> decrements cwnd by (CE bytes)/2.