Subject: Re: [tcpm] I-D Action: draft-ietf-tcpm-dctcp-01.txt

From: Andrew Shewmaker <[email-masked]>
Date: Mon, 25 Jan 2016 14:02:55 -0700

Hi,

Does DCTCP's exit from slow start need to be addressed in the
Internet-Draft?

I did see item 8 from
https://www.ietf.org/mail-archive/web/iccrg/current/msg01215.html

"[Not discussed in the meeting, but added by Bob B for the record]: Less
drastic exit from slow-start, to match less drastic rate reduction per mark.
Currently, because marking threshold is shallow, slow start exits earlier
than with drop, unnecessarily increasing completion time."

But I haven't seen any later discussion archived. I didn't see anything in
sections 3.3 Processing Congestion Indications on the Sender or 4
Implementation Issues. Perhaps I missed it?

In experiments with Linux DCTCP I've noticed interesting slow start
behavior. In a five flow convergence test in Mininet I'm seeing similar
overall behavior to that shown in the paper. However, when a flow enters, I
see its CWND overshoot (by double what it should be) and results in large
SRTTs, until the buffer is switch bottleneck buffer has drained.

Even if the first ECE flag seen by Linux DCTCP was at the beginning of its
observation window, causing it to not update ALPHA until the end of the
window, I would have thought SSTHRESH would have been immediately set to at
most CWND (or halved, considering that the initial ALPHA is zero). But I
don't think that is what I'm seeing.

I added the following logic into the update alpha code, and it did force
DCTCP to exit immediately:

if (flags & CA_ACK_ECE) {
    if (ca->acked_bytes_ecn == 0 && tcp_in_slow_start(tp))
        tp->snd_ssthresh = tp->snd_cwnd;
...

So CWND did not overshoot and tail latencies were dramatically reduced.
Unfortunately, it also reduces aggregate goodput somewhat. In my Mininet
experiments, the unmodified DCTCP's normalized utilization ranged ~0.78 to
~0.83, whereas the early exit modification resulted in ~0.73 to ~0.81.

I can provide more details and graphs, if necessary.

Is there better logic for slow start exit than the above? I also tried
checking just tcp_in_slow_start(tp) and that hurt goodput much more.

Regardless of whether someone wants to optimize for goodput or latency, I
think that it would be good to address the subject.

Thanks,

Andrew Shewmaker

---

From: Michael Welzl <[email-masked]>
Date: Tue, 26 Jan 2016 00:35:52 +0100

Hi,

> On 25. jan. 2016, at 22.02, Andrew Shewmaker <[email-masked]> wrote:
> 
> Hi,
> 
> Does DCTCP's exit from slow start need to be addressed in the Internet-Draft?
> 
> I did see item 8 from https://www.ietf.org/mail-archive/web/iccrg/current/msg01215.html <https://www.ietf.org/mail-archive/web/iccrg/current/msg01215.html> 
> 
> "[Not discussed in the meeting, but added by Bob B for the record]: Less drastic exit from slow-start, to match less drastic rate reduction per mark.
> Currently, because marking threshold is shallow, slow start exits earlier than with drop, unnecessarily increasing completion time."
> 
> But I haven't seen any later discussion archived. I didn't see anything in sections 3.3 Processing Congestion Indications on the Sender or 4 Implementation Issues. Perhaps I missed it?
> 
> In experiments with Linux DCTCP I've noticed interesting slow start behavior. In a five flow convergence test in Mininet I'm seeing similar overall behavior to that shown in the paper. However, when a flow enters, I see its CWND overshoot (by double what it should be) and results in large SRTTs, until the buffer is switch bottleneck buffer has drained.
> 
> Even if the first ECE flag seen by Linux DCTCP was at the beginning of its observation window, causing it to not update ALPHA until the end of the window, I would have thought SSTHRESH would have been immediately set to at most CWND (or halved, considering that the initial ALPHA is zero). But I don't think that is what I'm seeing.
> 
> I added the following logic into the update alpha code, and it did force DCTCP to exit immediately:
> 
> So CWND did not overshoot and tail latencies were dramatically reduced. Unfortunately, it also reduces aggregate goodput somewhat. In my Mininet experiments, the unmodified DCTCP's normalized utilization ranged ~0.78 to ~0.83, whereas the early exit modification resulted in ~0.73 to ~0.81.
> 
> Is there better logic for slow start exit than the above? I also tried checking just tcp_in_slow_start(tp) and that hurt goodput much more.
> 
> Regardless of whether someone wants to optimize for goodput or latency, I think that it would be good to address the subject.

I don’t know the DCTCP code well enough to fully understand your change, but you do seem to suggest that, upon an ECN mark in SS, CWND should be halved, and I think that’s the root of the problem.
In SS, when getting an ECN mark from a shallow queue, whether you have overshot by a little bit or by almost a BDP is a matter of luck - e.g., for only one flow, it depends on how its cwnd aligns with the path BDP.
We have seen this problem when we developed our ABE proposal ( draft-khademi-alternativebackoff-ecn ) and addressed it by applying a less dramatic back-off (in ABE, a fixed value) also at the end of slow start.
This is explained in the subsection “ABE in Slow Start” on page 5 of http://heim.ifi.uio.no/michawe/research/publications/CAIA-TR-150710A.pdf and, in particular, Figure 6.

According to our understanding, “apply a less dramatic back-off at the end of slow start” doesn’t need to be said: the cwnd reaction upon the first loss or mark should already be CA’s, right?

So, to me, the worrying part is this note in your text above: "(or halved, considering that the initial ALPHA is zero)”.  Maybe the initial ALPHA in CA should then be changed.

Cheers,
Michael

---

From: Andrew Shewmaker <[email-masked]>
Date: Mon, 25 Jan 2016 17:00:27 -0700

On Mon, Jan 25, 2016 at 4:35 PM, Michael Welzl <[email-masked]> wrote:

> I don’t know the DCTCP code well enough to fully understand your change,
> but you do seem to suggest that, upon an ECN mark in SS, CWND should be
> halved, and I think that’s the root of the problem.
> In SS, when getting an ECN mark from a shallow queue, whether you have
> overshot by a little bit or by almost a BDP is a matter of luck - e.g., for
> only one flow, it depends on how its cwnd aligns with the path BDP.
> We have seen this problem when we developed our ABE proposal (
> draft-khademi-alternativebackoff-ecn ) and addressed it by applying a less
> dramatic back-off (in ABE, a fixed value) also at the end of slow start.
> This is explained in the subsection “ABE in Slow Start” on page 5 of
> http://heim.ifi.uio.no/michawe/research/publications/CAIA-TR-150710A.pdf
> and, in particular, Figure 6.
>
> According to our understanding, “apply a less dramatic back-off at the end
> of slow start” doesn’t need to be said: the cwnd reaction upon the first
> loss or mark should already be CA’s, right?
>
> So, to me, the worrying part is this note in your text above: "(or
> halved, considering that the initial ALPHA is zero)”.  Maybe the initial
> ALPHA in CA should then be changed.

​Hi Michael,

Thanks for that link. I'll have to read it more carefully, but that section
seems like a nice writeup of the topic.

I didn't mean to suggest that the CWND should be halved---I think something
like DCTCP's proportional backoff is the way to go. I was trying to say
that DCTCP doesn't have to continue doubling its window once it sees an ECE
flag.

I'm apologize for my lack of clarity. I also made several mistakes when I
was describing Linux DCTCP's initial ALPHA. By default it is ALPHA_MAX
(although it is a module parameter and can be overridden). If a new flow
doesn't see an ECE for several RTTs, then the ALPHA will, of course, have
decayed a bit. So, when DCTCP exists slow start it will back off some
fraction less than half.

The code I added only exits slow start, so DCTCP won't continue doubling
its CWND for the rest of the RTT. Then DCTCP will behave as normal, backing
off according to congestion ratio ALPHA.

Thanks,

Andrew

---

From: John Leslie <[email-masked]>
Date: Tue, 26 Jan 2016 06:59:51 -0500

Michael Welzl <[email-masked]> wrote:
> overshot by a little bit or by almost a BDP is a matter of luck -
> e.g., for only one flow, it depends on how its cwnd aligns with the path
> BDP.

   I believe it's more deterministic than that (given the usual ceteris
parabus disclaimer ;).

   The combination of sender&receiver can know RTT quite accurately. The
sender should know at what rate it was sending one RTT ago.

   Thus, at the receipt of the first ECE, the sender knows the "safe"
sending rate one RTT ago; and, ceteris parabus, that is the sending rate
that deserves to be considered "safe" going forward.

> We have seen this problem when we developed our ABE proposal
> ( draft-khademi-alternativebackoff-ecn ) and addressed it by applying
> a less dramatic back-off (in ABE, a fixed value) also at the end of
> slow start.

   The important point is to stop (or at least greatly reduce) probing
for more bandwidth until at least one RTT later, since by definition the
sender cannot know whether a packet it sends encounters congestion until
one full RTT after sending it.

--
John Leslie <[email-masked]>

---

From: Michael Welzl <[email-masked]>
Date: Tue, 26 Jan 2016 13:11:05 +0100

> On 26 Jan 2016, at 12:59, John Leslie <[email-masked]> wrote:
> 

I talked about the amount of overshoot, not what the previous "safe" rate was.

First, about overshoot: Note, I mean the capacity limit, not capacity + queue.
Say, you ECN-mark using immediate marking, as soon as the queue grows. Then your mark occurs just after you've exceeded the capacity limit.
If you start with 10 packets, and the BDP of the path equals e.g. 39 packets, then you've only overshot by 1 packet.
If however the BDP of the path is e.g. 40 packets, the next round you've sent 80, so overshot by 40, and you'd get a mark for the first of them.
That's what I meant.

About the "safe" rate, yes, halving is the only 100% "safe" way, but it's often way too little whenever you receive a mark or experience a loss at a queue that's less than a BDP.
When it wasn't loss but ECN, it may be more reasonable to back off by less - note that, in case you really didn't back off enough, you immediately get another ECN-mark anyway, so within a few rounds you'll definitely empty the queue.

(with DCTCP, this shouldn't really be a few rounds - as is the case with our ABE proposal - but proportionally reducing should in fact put you right down to the right value)

> one full RTT after sending it.

Sure.
This means that once you saw the first ECN mark you've already sent too much, and - depending on where in the burst the ECN-mark appeared - possibly *way* too much. Thus stopping to increase immediately upon getting an ECN mark (which, if I got this right, is what Andrew's code now does) sounds exactly right.

Cheers,
Michael

---

From: Michael Welzl <[email-masked]>
Date: Tue, 26 Jan 2016 13:13:15 +0100

> On 26 Jan 2016, at 01:00, Andrew Shewmaker <[email-masked]> wrote:
> 
> 
> Thanks for that link. I'll have to read it more carefully, but that section seems like a nice writeup of the topic.
> 
> I didn't mean to suggest that the CWND should be halved---I think something like DCTCP's proportional backoff is the way to go.

Agreed; it didn't sound like you suggested it should be, but it sounded like that's what's currently happening because of the initial ALPHA value

> I was trying to say that DCTCP doesn't have to continue doubling its window once it sees an ECE flag.
> 
> I'm apologize for my lack of clarity. I also made several mistakes when I was describing Linux DCTCP's initial ALPHA. By default it is ALPHA_MAX (although it is a module parameter and can be overridden). If a new flow doesn't see an ECE for several RTTs, then the ALPHA will, of course, have decayed a bit. So, when DCTCP exists slow start it will back off some fraction less than half.

That seems better then  (but what, then, produces the loss in throughput that you described?)

> The code I added only exits slow start, so DCTCP won't continue doubling its CWND for the rest of the RTT. Then DCTCP will behave as normal, backing off according to congestion ratio ALPHA.

and that seems good too

Cheers,
Michael