Subject: [ledbat] 2 week last call for draft-ietf-ledbat-congestion-06

From: Rolf Winter <[email-masked]>
Date: Wed, 1 Jun 2011 07:17:24 +0000

Working Group,

this is to start a two week working group last call of draft-ietf-ledbat-congestion-06.

Please send all comments you have to [email-masked].

This WG last call ends on the 15th of June.

The chairs.

NEC Europe Limited | Registered Office: NEC House, 1 Victoria Road, London W3 6BL | Registered in England 2832014

---

From: Wesley Eddy <[email-masked]>
Date: Mon, 13 Jun 2011 14:14:08 -0400

> This WG last call ends on the 15th of June.

This is a combined WGLC response and AD review.

I think the technical description is pretty good modulo some of the
comments from Lisong Xu.  I only have a few editorial suggestions:

- The document should probably note somewhere that ECN is not handled by
   LEDBAT and making LEDBAT flows ECN capable would be potential future
   work (maybe related to the AQM effects mentioned in the second
   paragraph of section 2.2)

- In the security considerations, it might be worth noting that LEDBAT
   is not known to introduce any new concerns with privacy, integrity,
   or other security issues for flows that use it.  It should be
   compatible with use of IPsec and TLS/DTLS.

- In the abstract, LEDBAT yields not just to TCP but to any other flows
   once latency builds.  This is an important design feature in order for
   delay-sensitive real-time flows to coexist with bulk transfers.  It
   works whether or not the other flows use TCP.

- Same comment as above in second paragraph of the Introduction.

- Same comment as above in section 2.1

- For Vegas, I suggest also citing "Understanding TCP Vegas: A Duality
   Model" by Low, Peterson, and Wang from JACM 49 (2), March 2002.  This
   provides additional analysis that's useful to understand in the
   context of minimizing network queues

- Section 3.1, first sentence, suggest adding "or ECN mark is received"

-- 
Wes Eddy
MTI Systems

---

From: Rolf Winter <[email-masked]>
Date: Tue, 14 Jun 2011 08:07:18 +0000

Hi,

There is only one day left, but the chairs would seriously like a few more reviews. Also just a two thumbs up or similar is valuable feedback.

Thanks,

Rolf

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On
> Behalf Of Rolf Winter
> Sent: Mittwoch, 1. Juni 2011 09:17
> To: [email-masked]
> Subject: [ledbat] 2 week last call for draft-ietf-ledbat-congestion-06
> 
> Working Group,
> 
> NEC Europe Limited | Registered Office: NEC House, 1 Victoria Road,
> London W3 6BL | Registered in England 2832014
> 
> _______________________________________________
> ledbat mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ledbat

---

From: John Leslie <[email-masked]>
Date: Tue, 14 Jun 2011 12:34:48 -0400

Rolf Winter <[email-masked]> wrote:
> 
> There is only one day left, but the chairs would seriously like a few
> more reviews. Also just a two thumbs up or similar is valuable feedback.

   My goodness, it's changed a lot over 20 months!

   It looks ready for publication to me... However:
] 
] 2.2.  Applicability
]...
] LEDBAT seeks to operate in networks with FIFO queues and with tail-
] drop queue management.  Further study is required to understand the
] implications of Active Queue Management (AQM) schemes on LEDBAT
] mechanisms.

   This could be read as "may break under AQM" -- which I believe to
be quite false.

   IMHO, we believe that ECN marking per AQM would be no problem at all,
and packet-drop under AQM would merely cause backoff sooner than needed;
and we have consensus to just accept that consequence.

   I think the language here should be more like "to fully understand..."
and we should record that problems due to AQM are considered manageable.

--
John Leslie <[email-masked]>

---

From: Nicholas Weaver <[email-masked]>
Date: Tue, 14 Jun 2011 11:13:31 -0700

> and we should record that problems due to AQM are considered manageable.

Perhaps something like:

It is unclear how LEDBAT will operate in an environment with AQM at the bottleneck, but as LEDBAT still includes a TCP-equivelent loss-based secondary rate control, it will never acquire more bandwidth than a competing TCP flow would and would be likely to acquire less bandwidth because the packet-drop will cause an earlier backoff.  There is a consensus that, although the exact details are not yet fully understood, the behavior under AQM should be safe and any performance degradation under AQM is acceptable and compatible with LEDBAT's objectives.

Otherwise, thumbsup for me.

---

From: Wesley Eddy <[email-masked]>
Date: Tue, 14 Jun 2011 15:58:34 -0400

> and we have consensus to just accept that consequence.

Since AQMs will generally only mark packets when there is a queue, and
LEDBAT strives to minimize its contribution to queues, it may be a lot
less likely that LEDBAT flows would be marked compared to other flows.
They would have adapted their rates as they received RTT estimates
indicating the buildup of the queue, and potentially already reacted
prior to losses being detected which takes longer without ECN.

-- 
Wes Eddy
MTI Systems

---

From: Mirja Kühlewind <[email-masked]>
Date: Wed, 15 Jun 2011 15:29:39 +0200

Hi Wesley,

thanks for you review. Please see comments inline.

Mirja

>    paragraph of section 2.2)
See other emails...

>    compatible with use of IPsec and TLS/DTLS.
Done.

>    works whether or not the other flows use TCP.
s/any competing TCP flows/any competing flows when latency builds/
Okay? Or should we even add more text here?

> - Same comment as above in second paragraph of the Introduction.
Same here
s/performance of competing TCP flows/performance of competing flows/
s/interference with competing TCP flows/interference with competing flows when 
latency builds/
> - Same comment as above in section 2.1
s/concurrent TCP flows/concurrent flows/
>    context of minimizing network queues
I've been adding the reference. Should I also add a sentence here?

> - Section 3.1, first sentence, suggest adding "or ECN mark is received"
Done.

---

From: Mirja Kühlewind <[email-masked]>
Date: Wed, 15 Jun 2011 15:46:51 +0200

Hi, 

> prior to losses being detected which takes longer without ECN.

The question is, should we add somewhere that LEDBAT should halve its window 
when receiving a ECN mark? Actually in RFC3168 its already mentioned that 
(every) congestion control should react in the same way to ECN marks as it 
would to loss. 
But, as for loss, this would only be a fallback for LEDBAT as it should not 
build up a large queue. This can happen e.g. when the marking threshold is 
quite low and the TARGET value quiet high. By reacting on ECN marks with 
halving, LEDBAT would fallback to normal TCP behavior (with maybe increasing 
the window a little slower). 
And that's also the general issue about AQM. Depending on the parametrization 
of the AQM and LEDBAT, LEDBAT might fallback to standard TCP behavior. 
Implementors could try to recognize this and adapt there parametrization but 
that's not part of the LEDBAT algo itself.  

Mirja

---

From: Rolf Winter <[email-masked]>
Date: Wed, 15 Jun 2011 15:37:47 +0000

Hi,

(chair's hat off. Please treat these comments as any other last call comments received.)

A fine document. Some points below.

"update_current_delay() maintains a list of one-way delay measurements, of which the minimum is used as an estimate of the current end-to-end delay". This is not mentioned when you talk about the implementation of FILTER() which actually gives a very different algorithm as an example.

"In limited experiments with Bittorrent nodes, this controller seems to work well." Refs would be good.

"In the worse case, if the first flow yields at the same rate as the new flow increases its sending rate, the new flow will see constant end-to-end delay, which it assumes is the base delay, until the first flow backs off completely. As a result, by the time the second flow stops increasing its cwnd, it would have added twice the target queueing delay to the network." Maybe I am misreading this but I believe this is not entirely and necessarily correct. The base delay estimation is the minimum of all measured delay samples. Assume the first flow backs off completely, the second flow should be able to get quite close to the absolute base delay with one of its packets and it'll add TARGET on top.

s/In other words, A LEDBAT/In other words, a LEDBAT/
s/senders's/sender's/

Best,

Rolf

---

From: Mirja Kuehlewind <[email-masked]>
Date: Wed, 15 Jun 2011 18:21:15 +0200

Hi,

thanks for the review. Please see my comments inline.

Mirja

On Wednesday 15 June 2011 17:37:47 Rolf Winter wrote:
> "update_current_delay() maintains a list of one-way delay measurements, of
> which the minimum is used as an estimate of the current end-to-end delay".
> This is not mentioned when you talk about the implementation of FILTER()
> which actually gives a very different algorithm as an example.
Good catch! We changed MIN() to FILTER() and we forgot to adapt the text here 
(there are two more places we have to change).

> work well." Refs would be good.
That's the problem, there are no refs...

>
> "In the worse case, if the first flow yields at the same rate as the new
> flow increases its sending rate, the new flow will see constant end-to-end
> delay, which it assumes is the base delay, until the first flow backs off
> completely. As a result, by the time the second flow stops increasing its
> cwnd, it would have added twice the target queueing delay to the network."
> Maybe I am misreading this but I believe this is not entirely and
> necessarily correct. The base delay estimation is the minimum of all
> measured delay samples. Assume the first flow backs off completely, the
> second flow should be able to get quite close to the absolute base delay
> with one of its packets and it'll add TARGET on top.
No, its correct. By the time the first flow backs off completely, the second 
flow has already increase its sending rate such that TARGET extra delay is on 
the link. So every packet that is send out after the back off will still 
measure the base_delay + TARGET and there has never been and will never be a 
lower sample. So during the time period the first one needed to back off, the 
delay was constant (all delay samples have been the same).

cwnd
^
|                        __________________________________
|                      /
|                    /
|                 /
|              /
|  \        /
|     \  / 
|     /  \
|  /        \
-------------------------------------------------------------------> time
              t1

delay
^
|                         __________________________________
|                        /
|                      /
|                   /
|_________ /
| 
|
|
|
-------------------------------------------------------------------> time
                t1

Done.

-- 
-------------------------------------------------------------------
Dipl.-Ing. Mirja Kühlewind
Institute of Communication Networks and Computer Engineering (IKR)
University of Stuttgart, Germany
Pfaffenwaldring 47, D-70569 Stuttgart

tel: +49(0)711/685-67973
email: [email-masked]
web: www.ikr.uni-stuttgart.de
-------------------------------------------------------------------

---

From: Wesley Eddy <[email-masked]>
Date: Wed, 15 Jun 2011 15:06:13 -0400

On 6/15/2011 9:29 AM, Mirja Kühlewind wrote:
> ...

Sounds good.

> s/concurrent TCP flows/concurrent flows/

Sounds good.

> I've been adding the reference. Should I also add a sentence here?

It doesn't matter to me; I'm happy just with the reference that a reader
could follow if they want to.

-- 
Wes Eddy
MTI Systems

---

From: Wesley Eddy <[email-masked]>
Date: Wed, 15 Jun 2011 15:13:59 -0400

> (every) congestion control should react in the same way to ECN marks as it

It doesn't seem that many LEDBAT flows will be ECN-Capable Transports,
but it's probably worth saying that *if* an implementation is
ECN-capable, then it should react to ECN marks as losses.

-- 
Wes Eddy
MTI Systems

---

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 16 Jun 2011 18:44:33 +0100

> ECN-capable, then it should react to ECN marks as losses.
Mirja seems to raise a valid point...

I think it would be extremely good to encourage the use of ECN for such 
applications, by saying explicitly that ECN could offer benefit when the 
infrastructure supports this. To allow this, a LEDBAT session needs to 
send packets with ECT(x), and then MUST include appropriate control 
functions.

If ECN were supported, then an ECN mark (CE) indicates actual 
congestion, which could (?) be taken as more significant than a loss, 
especially since the goal of LEDBAT was to keep the queue low, and 
therefore I suggest it could be reasonable to regard actual reported 
congestion as significant (or take RFC3168's view that at least ECN is 
equivalent to loss).

- whatever, I think reception of an ECN mark MUST cause the 
corresponding sender to perform a congestion response - i.e. can not be 
ignored.

Gorry

---

From: Dario Rossi <[email-masked]>
Date: Fri, 17 Jun 2011 11:45:34 +0200

Hi Mirja, all,

1st of all, sorry for not having been able to cope/help with the 
numerous updates on the draft, and for the after-deadline comment!  
please see inline

> That's the problem, there are no refs...

Our testbed experiments with the BitTorrent libUTP implementation shown 
that latecomer unfairness arises also in the real world, in case of 
/backlogged/ flows ( see Fig1, pag2 in http://arxiv.org/abs/1010.5623 )

Latecomer unfairness may instead vanish in case of /non-backlogged/ 
connections (see Fig6 pag 9 in http://arxiv.org/abs/1010.5623 ; only ns2 
results here though)

While I understand (and agree) that fairness may not be the main aspect 
in ledbat, given the above observations and [1], it seems to me that  
sec 5.3 of the current version of the draft wrongly states that fairness 
is reinstated simply by

	randomness in the inter-packet transmission times exist,

as [1] shows this is not the case  (and though there is real-world noise 
in the testbed of Fig1, pag2 in http://arxiv.org/abs/1010.5623 , this is 
apparently not enough either...)

In other words, real-world noise is not enough to break latecomer 
unfairness, but the use of non-backlogged traffic sources can: notice 
that non backlogged sources assist in emptying the queues, while a 
simple jittering cannot (hence, the explanation in sec 5.3 is misleading?).

[1] G. Carofiglio, L. Muscariello, D. Rossi and S. Valenti, *The quest 
for LEDBAT fairness* 
<http://www.enst.fr/%7Edrossi/paper/rossi10globecom-a.pdf> . In /IEEE 
Globecom'10/, Miami, FL, USA, December 6-10 2010.

Best,
D.

-- 
  Oo    Associate Professor
   >     TELECOM ParisTech
  ~     (formerly ENST)

mail:  [email-masked]
phone: +33.1.4581.7563
fax:   +33.1.4581.7158
web:   http://www.enst.fr/~drossi

---

From: Mirja Kuehlewind <[email-masked]>
Date: Tue, 28 Jun 2011 20:07:40 +0200

Hi Dario,

sorry for the late response.  First of all thanks for your feedback. I'm not 
sure if I exactly got your point. I believe our statement about randomness is 
still not wrong because it's very general. Of course the idea is that the 
queue has to get empty from time to time (to measure the real base delay). 
Maybe we can state that more explicit?

But back to your result. Can you define what you exactly mean by 'backlogged' 
flows? I'm not sure if I fully got you right here...

Mirja

---

From: Mirja Kuehlewind <[email-masked]>
Date: Tue, 28 Jun 2011 20:25:54 +0200

Hi Gorry.

> send packets with ECT(x), and then MUST include appropriate control
I don't think we should recommend for LEBDAT to be ECN-capable because the 
selection of a congestion control mechanism is, from my point of view, not 
related to the question to support ECN or not.
Moreover, ECN is an TCP/IP mechanism and LEDBAT is specified independent of 
any framing protocol (e.g. BitTorrent uses UDP).
And when LEDBAT is working correctly, there shouldn't occur any ECN marking 
(and of course no losses neither) as LEDBAT will reduce it's sending rate to 
basically zero before any queues overflow.

> corresponding sender to perform a congestion response - i.e. can not be
My only question was: Should we mention _explicit_ that LEDBAT has to halve 
its window on an ECN mark or is it enough if RCF3168 is defining this 
_implicit_ by saying (every) congestion control should react in the same way 
to ECN marks as it would to loss.
In both cases a LEDBAT flow has to halve this window on an ECN mark (as on 
loss). 

So in general, I'm not sure if we should talk about an TCP/IP mechanism (ECN) 
in this draft when we only specify the algorithm and not talking about 
anything TCP specific...?

Mirja

---

From: Arjuna Sathiaseelan <[email-masked]>
Date: Tue, 28 Jun 2011 20:28:12 +0100

Dear Mirja,
  Sorry- I am a newcomer to LEDBAT - and I am trying to understand
this. So when you say LEDBAT reduces the sending rate to zero - does
that mean LEDBAT doesnt have the notion of timers/timeouts? So if
LEDBAT knows that the queues are being utilised could it wait for any
amount of time without sending data?

> basically zero before any queues overflow.

Regards
Arjuna

---

From: Dario Rossi <[email-masked]>
Date: Wed, 29 Jun 2011 10:20:07 +0200

Hi Mirja, all,

both questions are deeply related,  I reformulate the explanation with 
more details -- so, sorry for the long mail,

imo, a /random jitter between two consecutive packets/  is not enough to 
trigger changes that avoid latecomer issue (which is what the draft 
suggests)

/backlogged/ means that application have /always/ data to send, so that 
the congestion control  happens on an infinite stream of packets that 
queue at application level (this is the usual case of FTP application). 
In case you have backlogged transfers, latecomer happens in practice  
(eg even with  libUTP implementation [ARXIV] ), and irrespectively of 
random jittering [1]

now, if the source is non-backlogged but /talks in spurts/ (e.g., 
BitTorrent chunks, short HTTP transfers) with some random idle time in 
between the spurts, this amount of randomness can assist flows in 
reaching a (more) fair share of the link.

more generally, I believe that /non-backlogged/ data arrival at the 
congestion control is likely the reason why  BitTorrent never observed 
the latecomer stuff , because now the link is not fully utilised. If the 
link utilization is on average <100% , then the queue seldom empties, in 
agreement with the original motivation for having a non-null target in 
ledbat for efficiency matters

in other words,  the  BitTorrent statement (sorry, rephrased with my own 
words, so do not quoted exactly)  that in practice randomness in real 
world happens due to e.g. "hiccups reading from disks" actually 
translate imo into /non-backlogged/ data transmission. Reason for non 
backlogged transmission (ie. <100% link utilization) was also observed 
in real swarm experiments** by Legout et al. [2,3] **due to /P2P 
protocol dynamics/, more than to disk bottlenecks

now, in *real BitTorrent swarms*, due to non-backlogged transmission, 
latecomer may not happen in practice and LEDBAT can actually translate 
into an advantage (see one of my last mails on the list about the impact 
of LEBDAT on BitTorrent completion time [4]).     However, from a 
*protocol perspective* the issue with latecomer still remains in 
LEDBAT,  as the first of two FTP-like backlogged transfers would face 
the latecomer starvation problem.

hope this clarifies my thoughts,
D.

[1]
G. Carofiglio, L. Muscariello, D. Rossi and S. Valenti, *The quest for 
LEDBAT fairness* 
<http://www.enst.fr/%7Edrossi/paper/rossi10globecom-a.pdf> . In /IEEE 
Globecom'10/, Miami, FL, USA, December 6-10 2010.

[2] A. Legout, N. Liogkas, E. Kohler, and L. Zhang. *Clustering and 
Sharing Incentives in BitTorrent Systems*. In /Proceedings of ACM 
SIGMETRICS'2007/, June 12--16, 2007, San Diego, CA, USA. [.ps 
<http://hal.inria.fr/inria-00137444/en/>, .pdf 
<http://hal.inria.fr/inria-00137444/en/>, .tex 
<http://hal.inria.fr/inria-00137444/en/>].

[3] A. Legout, G. Urvoy-Keller, and P. Michiardi. *Rarest First and 
Choke Algorithms Are Enough*. In /Proceedings of ACM SIGCOMM/USENIX 
IMC'2006/, October 25--27, 2006, Rio de Janeiro, Brazil. [.ps 
<http://hal.inria.fr/inria-00091678/en/>, .pdf 
<http://hal.inria.fr/inria-00091678/en/>, .tex 
<http://hal.inria.fr/inria-00091678/en/>].

[4]
C. Testa and D. Rossi, *The impact of uTP on BitTorrent completion time* 
<http://www.enst.fr/%7Edrossi/paper/rossi11p2p.pdf> . In /IEEE Peer to 
Peer (P2P'11)/, Kyoto, Japan, September 2011.

[ARXIV] G. Carofiglio, L. Muscariello, D. Rossi and S. Valenti, 
"Rethinking low extra delay background transport protocols" 
http://arxiv.org/abs/1010.5623

-- 
  Oo    Associate Professor
  ~     (formerly ENST)

---

From: Mirja Kuehlewind <[email-masked]>
Date: Wed, 29 Jun 2011 13:39:03 +0200

Hi Arjuna,

the minimum cwnd windows is 2 (MIN_CWND). That means LEDBAT will send at least 
two 2 packets per RTT (which is 'basically' zero). LEDBAT will not wait for 
any timers here but send continuously data with a very, very slow rate.

Mirja

On Tuesday 28 June 2011 21:28:12 Arjuna Sathiaseelan wrote:
> > And when LEDBAT is working correctly, there shouldn't occur any ECN
> > marking (and of course no losses neither) as LEDBAT will reduce it's
> > sending rate to basically zero before any queues overflow.
>
> Regards
> Arjuna

tel: +49(0)711/685-67973

---

From: Gorry Fairhurst <[email-masked]>
Date: Wed, 29 Jun 2011 17:09:16 +0100

I think Arjuna has a point, this is MUCH more aggressive than TCP under 
some conditions. Also if this is implemented as UDP, it does not adhere 
to the "worst case" rate described in the BCP for UDP usage, RFC 5405 
(see section 3.1.2).

Why is LEDBAT more aggressive than TCP or that recommended for UDP-based 
applications over severely constrained paths?

Gorry

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 29 Jun 2011 17:44:32 +0100

Mirja,

Going back to your original question, I hope the thoughts below will 
help. They argue against some of the things you said earlier, and 
justify what Gorry concluded.

ECN was never intended to be TCP-only. The IETF debated whether to 
define ECN-IP in a separate RFC to ECN-TCP and decided the TCP part 
should go in the same RFC as the IP part. But RFC3168 encourages 
addition of ECN to other transports, which is happening (SCTP, DCCP, RTP).

However, we shouldn't just follow the RFCs blindly on this issue, 
because I don't think responding to ECN was ever considered in the 
context of a delay-sensing transport like LEDBAT. We have to think 
from scratch...

For those of you on the list who have given up thinking from scratch, 
because it hurts, you don't need to read through the logic in points 
1-5 below. The summary is:

* Sending ECN-capable packets ought to be recommended for LEDBAT, so 
that if ever ECN is deployed in the network, LEDBAT can take advantage of it.
* If LEDBAT sends ECN-capable packets, it should respond to ECN marks 
just like it responds to losses - reverting to TCP-like behaviour.
* However, LEDBAT will probably need the same sort of code as RTP 
needs to detect black-holing of ECN-capable UDP packets.

<STEP_BY_STEP_LOGIC>

1/ LEDBAT is designed to fall-back to TCP behaviour on loss, so that 
it doesn't lose out to other long-running TCPs, which are assumed to 
be a more likely cause of a loss.

2/ Whether an AQM (or tail-drop) signals loss before or after the 
queue reaches the LEDBAT target (100ms) depends on the speed of the 
line. If a fast line is the bottleneck in the path, it will signal 
loss to LEDBAT before it reaches 100ms delay. Therefore LEDBAT will 
always respond to congestion of this fast queue like TCP, not LEDBAT.

I believe using a fixed target delay was a deliberate design choice 
in LEDBAT so that it is more likely to yield to congestion in the 
(slow) upstream queue of your own home gateway, and less likely to 
yield to other people's traffic in the fast queues of a carrier or a LAN.

Put another way, LEDBAT does not assume loss always occurs after 
100ms. But LEDBAT protects itself first, then tries to be 
super-friendly second. Therefore LEDBAT works to the rule "trying to 
be super-friendly in response to a loss is more likely to receive a 
slap for your troubles, so don't bother".

3/ RFC3168 very strongly requires that an AQM signals ECN to 
ECN-capable transports only when it would signal loss to 
non-ECN-capable transports. Beware, there be monsters if you break 
this rule on the public Internet! If an operator places the ECN 
threshold shallower than drop in a public Internet queue, then 
non-ECN-capable traffic would always starve out ECN-capable traffic.

BTW, DCTCP could only have a shallower ECN threshold than drop 
because it could be deployed in a data centre where ECN was turned on 
in every host, and there were no significant non-ECN transports (open 
loop datagram transactions still work fine too).

4/ Combining points 2 & 3, if LEDBAT sends ECN-capable packets, an 
ECN queue would signal congestion at exactly the same queue length as 
it would signal drop for non-ECN-capable packets. Just as with drop, 
this could be before or after the queue reaches LEDBATs 100ms target delay.

So, on the public Internet, LEDBAT should assume "trying to be 
super-friendly in response to an ECN mark is more likely to receive a 
slap for your troubles, so don't bother".

Therefore, on the public Internet, LEDBAT should respond to ECN just 
as it would respond to a loss, ie. revert to TCP-like behaviour. 
Therefore adding ECN to LEDBAT should involve the same code changes 
at user-level as adding ECN to TCP in the kernel.

5/ The only reason not to add ECN to LEDBAT would be if some 
middlebox black-holes ECN-capable UDP packets (or worse we've seen 
ECN-TCP negotiation crash a broken home gateway with widespread 
deployment a few years ago). Otherwise, adding ECN can either do 
nothing or improve matters.

LEDBAT is over UDP, so it won't trigger the particular crash that 
prevented ECN deployment in TCP for so many years (it happened while 
the buggy gateway was looking at the ECN negotiation bits in the TCP 
header of the SYN).

However, more recent buggy routers have been found to discard UDP 
datagrams that are ECN-capable at the IP layer (discovered when 
adding ECN to RTP). These buggy routers are being fixed. It is 
possible to detect this bug and fall-back to non-ECN. But that needs 
a new thread to discuss it.

</STEP_BY_STEP_LOGIC>

Bob

At 12:39 29/06/2011, Mirja Kuehlewind wrote:

________________________________________________________________
Bob Briscoe,                                BT Innovate & Design

---

From: Gorry Fairhurst <[email-masked]>
Date: Wed, 29 Jun 2011 17:44:50 +0100

> related to the question to support ECN or not.
I'm not sure I understand.

> any framing protocol (e.g. BitTorrent uses UDP).
ECN is an IP mechanism that relies on a congestion-responsive transport. 
There is defined support for it in TCP, DCCP and soon  emerging in RTP 
over UDP (and perhaps SCTP).

There is no reason why a congestion-responsive LEDBAT flow over UDP can 
not use ECN, if it provides the functions.

> basically zero before any queues overflow.
I am not easily persuaded the function will always reduced LEDBAT 
throughput below all sharing TCP flows before queue overflow. Where I 
work, I see many types of dynamic layer 2 system in use, not all of 
which are easily mapped to layer 3 buffer - wide-area wireless access 
for instance, which can impose significant variation in RTT.

> anything TCP specific...?
I'm not saying the group should define the mechanism, but I am saying 
that from my point of view, it would be wise that ECN is ignored, and 
that what matters is proper response to "congestion indications", not to 
loss.

Gorry

---

From: Arjuna Sathiaseelan <[email-masked]>
Date: Wed, 29 Jun 2011 19:36:42 +0100

Dear Mirja,
  I am not sure whether cwnd of 2 packets/RTT during periods of heavy
congestion is a good response - because other methods support lower
rates for e.g TFRC would end up at a rate of 1 packet every 64 seconds
or so. So I am wondering whether its possible to reduce the cwnd to a
more lower rate as a response to sustained congestion.

I agree with Bob and Gorry that allowing LEDBAT to support ECN is not
going to hurt.

Regards
Arjuna

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 29 Jun 2011 20:02:11 +0100

Gorry,

At 17:44 29/06/2011, Gorry Fairhurst wrote:
>I'm not saying the group should define the mechanism, but I am 
>saying that from my point of view, it would be wise that ECN is 
>ignored, and that what matters is proper response to "congestion 
>indications", not to loss.

Did you miss a 'not' in the sentence?
"it would be wise that ECN is NOT ignored"

Bob

---

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 30 Jun 2011 09:40:21 +0100

Indeed I missed a "NOT".

To be clear, I think LEDBAT should indicate that ECN is a useful 
mechanism for congestion control, and that instantiations of the LEDBAT 
method in transports should be encouraged to support ECN functionality.

Gorry

On 29/06/2011 20:02, Bob Briscoe wrote:
>> that what matters is proper response to "congestion indications", not
>> to loss.
>

---

From: Mirja Kuehlewind <[email-masked]>
Date: Thu, 30 Jun 2011 13:17:35 +0200

Hi Arjuna, hi Gorry,

I was not aware that there is any recommendation to send only one packet per 
(up to) 64 sec. That seems a very high value for me. It actually means you 
are not sending any data for 64sec no matter what is happening in this 64 sec 
(the link might be empty already after one second)...?

The recommendation of 2 packet per RTT was because we need to monitor changes 
in one-way delay with LEDBAT. That's why we though there should be at least  
two samples per RTT. 

Thinking about it now, you can even get measurement samples by sending packets 
without any data. But that makes the description more complicated.

One more remark: The draft says "MIN_CWND SHOULD be 2". That means if there is 
a good reason (in certain scenarios or with a certain framing protocol), 
another value can be used. In case of allowing a CWND of 0, there has to some 
mechanism to restart the transmission. We did not regard such as mechanism as 
part of the LEDBAT algorithm (so far).

How should be proceed here now?

Mirja 

On Wednesday 29 June 2011 20:36:42 Arjuna Sathiaseelan wrote:
> > the minimum cwnd windows is 2 (MIN_CWND). That means LEDBAT will send at
> > least two 2 packets per RTT (which is 'basically' zero). LEDBAT will not
> > wait for any timers here but send continuously data with a very, very
> > slow rate.
> >
> > Mirja
> >

tel: +49(0)711/685-67973

---

From: Mirja Kuehlewind <[email-masked]>
Date: Thu, 30 Jun 2011 13:41:29 +0200

Hi everybody,

I try to recap now on the ECN issue to see which action is actually needed for 
the draft. (Btw. I believe we DO have the same opinion):

There are two question:

1. Should LEDBAT react on ECN marks as it would on loss?

A very clear YES here! Nobody wants to ignore ECN marks. The reaction to ECN 
is already specified in RFC3168 to be the same than to loss (for every 
congestion control). This was not mention explicit in the current draft 
version. I've been adding the following sentence for the next version (in 
2.2. Applicability):

"If LEDBAT is used with a framing protocol that is
	 ECN-capable, it should react to ECN marks as it would to losses 
	 as specified in RFC3168."

2. Should we recommend that transports using LEDBAT should use/enable ECN
as well?

Here are the responses of everybody:

Bob:
"Sending ECN-capable packets ought to be recommended for LEDBAT, so 
that if ever ECN is deployed in the network, LEDBAT can take advantage of it."

Gorry:
"To be clear, I think LEDBAT should indicate that ECN is a useful 
mechanism for congestion control, and that instantiations of the LEDBAT 
method in transports should be encouraged to support ECN functionality."

Arjuna:
"I agree with Bob and Gorry that allowing LEDBAT to support ECN is not
going to hurt."
Agree, ECN does not hurt...

I (Mirja):
-> I was saying that this question does not need to be answered in this draft, 
as this draft is not talking about ANY framing protocol issues (beside some 
hard requirement on the framing protocol e.g. any kind of feedback is need). 
It only documents the LEDBAT algorithm for informational purpose (its not a 
protocol spec nor an implementation guide). 

How should we continue here? Should we add some text? If yes, where should we 
add the text? Somebody wants to make a proposal for text?

Mirja

tel: +49(0)711/685-67973

---

From: Janardhan Iyengar <[email-masked]>
Date: Thu, 30 Jun 2011 11:43:05 -0400

Dear all,

There is no question that LEDBAT flows will experience loss in the wild, especially, as Bob so eloquently pointed out, when congestion is deeper in the core.

It seems to me that we are all in agreement that ECN marks must be respected.  I don't think that the LEDBAT draft can do more than state what a LEDBAT flow does in the presence of an ECN mark, as Mirja has already included in the Applicability Section.  After all, LEDBAT does not get into protocol mechanisms per se, and cannot really _specify_ anything more with ECN.

For LEDBAT to actively "encourage" ECN adoption, there should be some explicit benefit to the ECN-capable LEDBAT flows over the non-ECN capable ones. And within the scope of the draft, I don't see that we can point to any such benefits.  However, perhaps we could say something in the applicability section about how ECN and AQM can benefit the applications that we expect to use LEDBAT.  Any suggestions?

- jana

On 6/30/11 7:41 AM, Mirja Kuehlewind wrote:
>>> Bob Briscoe, BT Innovate&  Design
>
>
>

-- 
Janardhan Iyengar
Assistant Professor, Computer Science
Franklin & Marshall College
http://www.fandm.edu/jiyengar

---

From: Arjuna Sathiaseelan <[email-masked]>
Date: Thu, 30 Jun 2011 18:23:59 +0100

Dear Mirja,
  We need to think about this..The current nature of transport
protocols seem to be overly conservative to me - for e.g. in the
presence of a RTO, the ssthresh can reduce to a very low value and it
can take ages for a cwnd to grow - so it would be better to stop and
start a new TCP connection! So based on such a theory, we could say
why not use two packets/RTT during severe congestion - because this is
no worse than a LEDBAT starting a new connection with two packets/RTT.
However, the problem here is the lower than best effort nature of
these methods and I dont think we can go in the direction of ICCRG. So
probably we need to stick to conservative methods.

IMHO - I think the best option for LEDBAT during severe congestion is
not to send any data (thats the reason why I asked you the question
about timers and timeouts in my earlier email). However I also agree
that if we dont that we have no way of figuring out the necessary
metrics for LEDTBAT without sending any data.

So howabout a restart mechanism - when the cwnd becomes 0 - can LEDBAT
be allowed to try transmitting one or two packets every second  - and
if this is successful (noting the feedback indicates there is no
congestion - based on the observed delay samples) then it can follow
the normal slow-start procedure? If it still notices congestion, it
stops sending for another second and restarts again..and so on..

Just thoughts..

Regards
Arjuna

---

From: Janardhan Iyengar <[email-masked]>
Date: Thu, 30 Jun 2011 18:32:14 -0400

Arjuna,

Thank you for raising this question --- it is an important one that we missed but, IMHO, need to resolve.  LEDBAT does not currently account for rate reduction in the case of extreme congestion, as both TFRC and TCP do.  A couple of thoughts offhand:

-  Under extreme congestion where TCP would have timed-out and TFRC would have backed-down significantly, LEDBAT will only halve its sending rate.  If LEDBAT gets a good delay measurement, then the sender _could_ set its congestion window to 0, but what do we do when the congestion is extreme and detection is loss-based?

-  There is no notion of a timer in LEDBAT so far.  If we decide to introduce a timer, we could use the current RTT estimate to seed it, or go with a fixed 1 second as the initial seed time, with a backoff of the timer on a subsequent loss.  We could then use the timer to decide how long to go idle before sending out the next packet.

I'll try to hash this out some more and come up with something useful tonight.
- jana

---

From: Lachlan Andrew <[email-masked]>
Date: Fri, 1 Jul 2011 08:44:21 +1000

On 1 July 2011 01:43, Janardhan Iyengar <[email-masked]> wrote:
>
> It seems to me that we are all in agreement that ECN marks must be
> respected.
>
> For LEDBAT to actively "encourage" ECN adoption, there should be some
> explicit benefit to the ECN-capable LEDBAT flows over the non-ECN capable
> ones. And within the scope of the draft, I don't see that we can point to
> any such benefits.

Greetings Jana / all,

I haven't been following Letbat, and this proposal may be heretical,
but it is easy to give a Letbat flow an incentive to be ECN capable,
by specifying that it should back off more when it sees loss than when
it sees ECN.

As long as it backs off at least as much as Reno does on loss, then
this does not violate the spirit of RFC 3168.  It does violate the
requirement that response to ECN be the same as loss, but that was
written at a time when loss was considered synonymous with congestion;
really ECN should be treated more like extra delay for a delay-based
algorithm.

$0.02,
Lachlan

-- 
Lachlan Andrew  Centre for Advanced Internet Architectures (CAIA)
Swinburne University of Technology, Melbourne, Australia
<http://caia.swin.edu.au/cv/landrew>
Ph +61 3 9214 4837

---

From: Lachlan Andrew <[email-masked]>
Date: Fri, 1 Jul 2011 08:54:09 +1000

Greetings Arjuna,

I agree with your main point, that Ledbat must sensibly and
conservatively handle high congestion.

On 1 July 2011 03:23, Arjuna Sathiaseelan <[email-masked]> wrote:
> The current nature of transport
> start a new TCP connection!

No.  This confuses "better" with "gives a higher throughput to this
user".  The whole point of congestion control is to do what is good
for the network rather than maximizing the flow's own throughput.
Throwing away information that the network is currently congested is
not a *better* response.

Cheers,
Lachlan

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 01 Jul 2011 01:08:30 +0100

Janar,

I take Mirja's point that this isn't the right doc to recommend that 
the LEDBAT transport uses ECN, it's solely appropriate to say how 
LEDBAT should respond to ECN CE.

Earlier revs of this draft said a design goal was to use ECN, but 
that got removed. The applicability section currently says:
"
    LEDBAT seeks to operate in networks with FIFO queues and with tail-
    drop queue management.  Further study is required to understand the
    implications of Active Queue Management (AQM) schemes on LEDBAT
    mechanisms.
"
Thinking about this, it's rather oddly worded. LEDBAT doesn't know 
what sort of queues there will be where it "seeks to operate".

How about saying something like the following:

"LEDBAT has been designed so that long-running flows should not build 
a standing queue that would persistently delay TCP flows that share a 
relatively slow FIFO queue using tail drop.

If active queue management (AQM) is configured to start dropping or 
ECN-marking packets before the averaged queue has reached LEDBAT's 
target delay, it will make LEDBAT revert to TCP behaviour, rather 
than yield to other TCP flows. However the AQM will keep the queuing 
delay low, which achieves an outcome similar to that LEDBAT was 
aiming for. Additionally, if the LEDBAT transport supports ECN, it 
will enjoy the same minor advantages that an ECN-TCP enjoys over a 
non-ECN-TCP; avoiding losses and the consequent timeouts.

Fair queuing will prevent LEDBAT yielding to other TCP flows. 
Nonetheless, fair queueing isolates brief TCP flows from any standing 
queue caused by concurrent long-running flows, which again achieves 
an outcome that is not much worse than that LEDBAT was aiming for 
[Schneider10].

Further study is required to better fully understand the behaviour of 
LEDBAT with non-drop-tail non-FIFO queues.
"

[Schneider10] Joscha Schneider. Joerg Wagner, Rolf Winter , 
Hans-Joerg Kolbe, "Out of my Way -- Evaluating Low Extra Delay 
Background Transport in an ADSL Access Network" . In Proceedings of 
the 22nd International Teletraffic Congress (ITC22), Amsterdam, The 
Netherlands, September 7 - 9 2010.
<http://www.i-teletraffic.org/fileadmin/ITCBibDatabase/2010/schneider10.pdf>

At 16:43 30/06/2011, Janardhan Iyengar wrote:
>Dear all,
>
>There is no question that LEDBAT flows will experience loss in the 
>wild, especially, as Bob so eloquently pointed out, when congestion 
>is deeper in the core.
>
>respected.  I don't think that the LEDBAT draft can do more than 
>state what a LEDBAT flow does in the presence of an ECN mark, as 
>Mirja has already included in the Applicability Section.  After all, 
>LEDBAT does not get into protocol mechanisms per se, and cannot 
>really _specify_ anything more with ECN.
>
>For LEDBAT to actively "encourage" ECN adoption, there should be 
>some explicit benefit to the ECN-capable LEDBAT flows over the 
>non-ECN capable ones. And within the scope of the draft, I don't see 
>that we can point to any such benefits.  However, perhaps we could 
>say something in the applicability section about how ECN and AQM can 
>benefit the applications that we expect to use LEDBAT.  Any suggestions?
>
>- jana
>

---

From: Janardhan Iyengar <[email-masked]>
Date: Fri, 01 Jul 2011 00:00:23 -0400

Hi Lachlan,

> by specifying that it should back off more when it sees loss than when

That seems like an artificial incentive.  Why should a LEDBAT flow reduce its rate more than what TCP does on loss?

> really ECN should be treated more like extra delay for a delay-based

So, if I understand you correctly, your suggestion is not so much to _reduce more_ on loss, but to _reduce lesser_ on CE.  While the idea seems intriguing, what you are suggesting seems to be a rewrite of RFC 3168.  The receipt of CE signal is an event that may not be accompanied by any clear delay increase;  it is not clear to me how a CE received in the absence of a clear delay signal can be interpreted as extra delay and not loss.  In other words, how do you translate a CE signal to delay?

Isn't the equivalent argument for TCP-like flows to reduce their windows by lesser than than 1/2 on CE marks and by 1/2 on loss  (or am I missing something)?

- jana

---

From: Lachlan Andrew <[email-masked]>
Date: Fri, 1 Jul 2011 14:47:42 +1000

Greetings

On 1 July 2011 14:00, Janardhan Iyengar <[email-masked]> wrote:
> That seems like an artificial incentive.  Why should a LEDBAT flow reduce
> its rate more than what TCP does on loss?

No -- I suggested there could/should be a difference between how we
respond to loss and to ECN.  I didn't mention TCP.

> So, if I understand you correctly, your suggestion is not so much to _reduce
> more_ on loss, but to _reduce lesser_ on CE.

Yes, they're equivalent.

> While the idea seems
> intriguing, what you are suggesting seems to be a rewrite of RFC 3168.

Not at all.  Most of 3168 is about the mechanism.  That doesn't need
to change.  I'd only challenge the "congestion is equivalent to loss"
prescription.

> The
> receipt of CE signal is an event that may not be accompanied by any clear
> delay increase;  it is not clear to me how a CE received in the absence of a
> clear delay signal can be interpreted as extra delay and not loss.
> In other words, how do you translate a CE signal to delay?

You don't need to "translate a CE signal to delay".  My point was just
that you can treat it as another congestion signal whose severity is
less than loss.

> Isn't the equivalent argument for TCP-like flows to reduce their windows by
> lesser than than 1/2 on CE marks and by 1/2 on loss  (or am I missing
> something)?

No.  It is more equivalent to the argument that it should be *OK* for
TCP-like flows to reduce their windows by 1/2 on CE marks, and by
*more* than 1/2 on loss (if they choose to).

The requirement that CE marks be treated as loss was (presumably) to
prevent flows from using it as a way to be more aggressive than
standard TCP / TFRC.  Ledbat flows are already being much less
aggressive than Reno.  There is no logical reason that something that
backs of more timidly on loss should also back off more timidly on CE
(although there is a logical reason that something should back off on
CE by at least as much as "the universal congestion control when ECN
was introduced" backed off on "the universal congestion signal before
ECN was introduced").

The primary argument in RFC 3168 for making the response to ECN
essentially the same as that to a loss was to allow incremental
deployment of 3168.  If making the response identical actually
*stifles* the deployment of ECN by prohibiting the correct incentives,
then perhaps the wording of 3168 should be fixed...

Cheers,
Lachlan