Subject: Re: [conex] Potentially interesting ConEx problem

From: Bob Briscoe <[email-masked]>
Date: Tue, 5 Mar 2013 09:53:12 +0000

Matt,

The solution to the access link part of this problem is explained in:
<http://tools.ietf.org/html/draft-briscoe-conex-initial-deploy-03#section-4.1.1>

It can only distinguish access link congestion from shared 
congestion. Distinguishing home network congestion is harder, but 
possible with a tunnel to the home router, which already exists in 
some DSL arrangements (e.g. in BT we use a tunnel for services that 
share a line).

Bob

At 22:18 04/03/2013, Matt Mathis wrote:
>I have an interesting problem that I wish ConEx could solve, but I
>don't see how.   I want to distinguish between congestion on the path
>between a server and a subscriber aggregation point (e.g. DSLAM etc)
>and congestion beyond the aggregation point (e.g. the access links
>themselves, or the user's home network).
>
>There are some ways to infer this, but as far as I can tell there is
>not a reasonable way to measure it directly.   This is a problem that
>we have touched on in the past, but I don't recall discussing any
>solutions.
>
>Thanks,
>--MM--
>The best way to predict the future is to create it.  - Alan Kay
>
>Privacy matters!  We know from recent events that people are using our
>services to speak in defiance of unjust governments.   We treat
>privacy and security as matters of life and death, because for some
>users, they are.

________________________________________________________________
Bob Briscoe,                                                  BT

---

From: Matt Mathis <[email-masked]>
Date: Tue, 5 Mar 2013 09:36:34 -0800

Bob, I do know how to do it from the ISP's vantage point.

It is a bit harder from the content provider's vantage point.   A
large content provider might, for example, want to be able to
distinguish between appropriately filling subscriber links and
aggravating some capacity failure elsewhere along the path.
Traditional serving optimizations actually make this problem far
worse: serving from as many places as possible, as close to the users
as possible makes the system even more brittle to ISP failures.

There is not an easy way to do this today.  There are inference
signals (e.g. correlated losses across flows) but the signals are
expensive and slow.

Thanks,
--MM--

On Tue, Mar 5, 2013 at 1:53 AM, Bob Briscoe <[email-masked]> wrote:
> It can only distinguish access link congestion from shared congestion.
> Distinguishing home network congestion is harder, but possible with a tunnel
> to the home router, which already exists in some DSL arrangements (e.g. in
> BT we use a tunnel for services that share a line).
>
>
> Bob
>
>

---

From: John Leslie <[email-masked]>
Date: Tue, 5 Mar 2013 13:12:28 -0500

Matt Mathis <[email-masked]> wrote:
> 
> A large content provider might, for example, want to be able to

   It sounds as if Matt could make use of a hop-count recorded by an
ECN-aware node experiencing a loss in the ConEx Destination Option.
(I'd rather hear from Matt whether that could work _before_ hearing
from others why it's a ReallyBadIdea...)

--
John Leslie <[email-masked]>

---

From: "Andrew McLachlan (amclachl)" <[email-masked]>
Date: Tue, 5 Mar 2013 18:13:23 +0000

Is this problem not mostly negated at the application layer in most video clients? They dynamically adjust to the realtime end to end bandwidth on an end user by end user basis. 

For a metro or backbone topology failure this method once again adapts per end point affected, over any aggregate path these clients are running over.

Moreover the expression of congestion by an ISP to an off net provider is not really an attribute you would like to advertise and used to berate the SP with.

QoS is your friend here if you wish to actually protect valued traffic, rather the bringing whole bunches of users to the lowest common bandwidth for an app.

Andrew

On 5 Mar 2013, at 17:36, "Matt Mathis" <[email-masked]> wrote:

> _______________________________________________
> conex mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/conex

---

From: Matt Mathis <[email-masked]>
Date: Tue, 5 Mar 2013 14:37:17 -0800

> Is this problem not mostly negated at the application layer in most video clients? They dynamically adjust to the realtime end to end bandwidth on an end user by end user basis.

The clients are not immutable, nor does the server have to tell the
clients about every available format.

> For a metro or backbone topology failure this method once again adapts per endpoint affected, over any aggregate path these clients are running over.

Right, but if I am serving from 2 mS away and the data rate is only
128kb/s, my expected TCP equilibrium loss rate might be well above 10%
(it depends on the queue size).  It makes a huge operational
difference to know if the bottleneck is one subscriber's access link
or 100k busy users trying to share a very saturated 10Gb/s link.  If
it is the latter, you really don't want to be one of the other 100k
users.

> Moreover the expression of congestion by an ISP to an off net provider is not really an attribute you would like to advertise and used to berate the SP with.

Perhaps true, but the risks flow in both directions, and the
alternative to sharing information has the potential to have much
worse outcomes for both parties.  Besides, some inference techniques
will always work.

> QoS is your friend here if you wish to actually protect valued traffic, rather the bringing whole bunches of users to the lowest common bandwidth for an app.

Lets think about that idea: unconditionally mark all of my traffic to
be lower priority than my competitors...  I don't think that will pass
muster.   There are additional more incremental solutions in this
space, including weighted congestion control, with the ability to trim
the weights by small amounts in real time.  (Which will indirectly
cause clients to choose lower rate formats).

The question I was asking was can ConEx help the content provider know
which bottleneck prevails?   Bob?

Yes, John L, knowing which hop is key.  but recording it in the ConEx
destination option seems like a ReallyBadIdea.  For one thing all
potential tags are out of scope (I don't want to know hop counts or IP
addresses towards the user, nor if the user has a bottlenecked WiFi
between the client and the access link.)   What I really want to know
is: What are the traffic aggregates/routing prefixes sharing the same
bottleneck?

Thanks,
--MM--

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 6 Mar 2013 06:20:48 +0100 (CET)

> which bottleneck prevails?  Bob?

I tried to get the TCP people look into this, but got shot down. They 
didn't feel TCP would work better by knowing more, or I guess they didn't 
want to handle th complexity.

My idea was that TCP would over time "find out" what the access looked 
like if it indeed was the only thing congesting, and even that the 
connection manager on the machine would inform TCP what the properties of 
the primary connection was (speed, buffering, retransmits, half/full 
duplex etc), and if something substantial (like the link going down or 
coming up, speed changed etc) happened, TCP would be informed of this.

They felt it was better to have TCP find out by itself instead of getting 
fed this information via some API.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Matt Mathis <[email-masked]>
Date: Wed, 6 Mar 2013 10:13:18 -0800

So as an intermittent TCP person, I agree with them: this does not
belong at the TCP layer, but at some slightly higher serving services
layer, because it is has to be coupled to things like load balancers,
etc.  Per server state isn't useful.

Which is not the same as saying the idea isn't valuable, it is just
that TCP itself isn't quite the right place.

Thanks,
--MM--

On Tue, Mar 5, 2013 at 9:20 PM, Mikael Abrahamsson <[email-masked]> wrote:
> I tried to get the TCP people look into this, but got shot down. They didn't
> feel TCP would work better by knowing more, or I guess they didn't want to
> handle th complexity.
>
> My idea was that TCP would over time "find out" what the access looked like
> if it indeed was the only thing congesting, and even that the connection
> manager on the machine would inform TCP what the properties of the primary
> connection was (speed, buffering, retransmits, half/full duplex etc), and if
> something substantial (like the link going down or coming up, speed changed
> etc) happened, TCP would be informed of this.
>

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 6 Mar 2013 19:31:26 +0100 (CET)

On Wed, 6 Mar 2013, Matt Mathis wrote:

> Which is not the same as saying the idea isn't valuable, it is just that 
> TCP itself isn't quite the right place.

Well, I can't personally understand why the end user wouldn't get a better 
experience by TCP knowing that the wifi link is now up again, so it might 
be good to reset the retransmit timers to 1s instead of waiting 60 seconds 
before trying again.

---

From: "Scharf, Michael (Michael)" <[email-masked]>
Date: Wed, 6 Mar 2013 20:46:07 +0100

Mikael,

I do not understand how a congestion manager solves the problem raised here. Also, since you seem to be unhappy with TCPM,  please recall that at least I was interested in seeing empirical data for the timer problem you described (e. g., real OS traces). But that discussion would be better suited for the TCPM list, imho.

Actually, I do think that another recent TCPM discussion is closer related to this question: http://www.ietf.org/mail-archive/web/tcpm/current/msg06829.html (But Matt is certainly aware of this already, and it doesn't solve the problem).

Another random idea that comes into my mind is the Quick-Start nonce mechanism. In Quick-Start one can ensure that certain actions can only be performed by downstream routers. Sure, this is not what is needed here, but maybe another interesting pointer?

Michael

________________________________________
Von: [email-masked] [[email-masked]] im Auftrag von Mikael Abrahamsson [[email-masked]]
Gesendet: Mittwoch, 6. März 2013 06:20
An: ConEx IETF list
Betreff: Re: [conex] Potentially interesting ConEx problem

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 7 Mar 2013 05:34:23 +0100 (CET)

On Wed, 6 Mar 2013, Scharf, Michael (Michael) wrote:

> I do not understand how a congestion manager solves the problem raised 
> here.

Are you referring to TCP or Conex here?

> Also, since you seem to be unhappy with TCPM, please recall that at 
> least I was interested in seeing empirical data for the timer problem 
> you described (e. g., real OS traces). But that discussion would be 
> better suited for the TCPM list, imho.

Well, I'm not going back to the TCPM list with my idea (I felt there was 
consensus there this wasn't anything for TCP), I'll try to pitch my idea 
to other parties that might actually have an understanding for my the 
problem I'm experiencing and get running code into operating systems 
first.

I've moved over to "mosh" (http://mosh.mit.edu/) which solves my problem 
nicely by avoiding TCP and using UDP instead.

---

From: Mirja Kuehlewind <[email-masked]>
Date: Thu, 7 Mar 2013 22:19:18 +0100

Hi Matt,

> which bottleneck prevails?   Bob?

(sorry I'm not Bob but will give it a try anyway)

I don't think ConEx can help here. I believe you assume a scenario where the 
content provider is in most cases the sender. With ConEx information will be 
re-insert form the sender into the network. Thus ECN or loss information thus 
will be sufficient at the sender. 

Anyway as I've been recently looking in loss measurement/estimations methods, 
I actually came over the same question. Here my current ideas:

- The client/receiver might know its access bandwidth and could provide this 
information to the server (in a higher layer protocol) or send some 
measurement traffic to the server or a measurement server

- distinguish self-congestion (only one flow on bottleneck link) form shared 
congestion by looking at the lost pattern, e.g. a flow using cubic or reno 
has a very regular loss burst with a defined number of losses in a defined 
period (I'm currently looking at this) if there is no cross traffic 
disturbing the flow

- correlate the losses (or loss pattern) of multiple flows (from the same 
server to different clients) to see if losses occur at the same time/in the 
same period (see network tomography in literature)

Does this help?

Mirja

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 7 Mar 2013 23:40:12 +0000

Andrew,

Going back in the thread to your point...

At 18:13 05/03/2013, Andrew McLachlan (amclachl) wrote:
>Is this problem not mostly negated at the application layer in most 
>video clients? They dynamically adjust to the realtime end to end 
>bandwidth on an end user by end user basis.

Yes, by driving load until they detect losses. That's just normal 
regular congestion avoidance & control. So far, that's nothing to do 
with ConEx, which doesn't propose to change what the network tells 
hosts. Rather, it's about hosts telling the network what e2e 
congestion they have detected.

>For a metro or backbone topology failure this method once again 
>adapts per end point affected, over any aggregate path these clients 
>are running over.

Yes. Again, nothing new here.

>Moreover the expression of congestion by an ISP to an off net 
>provider is not really an attribute you would like to advertise and 
>used to berate the SP with.

A link cannot not signal congestion (double negative intended) if 
hosts are overloading it with traffic, even briefly. It has to 
discard as much traffic as it has to discard. It's only choice is in 
whether it does that randomly (ie proportionate to instantaneous 
load) or focused (leading into your QoS point next).

This still isn't related to ConEx, which is about the sender 
signalling the e2e congestion it detects back into the network, in an 
e2e field, which hosts can integrity-protect against networks 
altering it if they choose. ConEx is not about whether the network 
signals congestion (because it has no choice over that).

>QoS is your friend here if you wish to actually protect valued 
>traffic, rather the bringing whole bunches of users to the lowest 
>common bandwidth for an app.

ConEx is about intra-class protection of traffic. The operator can 
use QoS classes (or not), and it could manage traffic within a class 
using ConEx (or not).

ConEx will be useful for focusing policing on particularly heavy 
sources within a class. QoS (Diffserv) focuses discards on all lower 
class traffic indiscriminately. The two could work together, so 
during overload QOS protects valued traffic and even in the lower 
classes ConEx will protect most traffic, while only shedding traffic 
from the heaviest customer(s) in the lower class.

I believe you are dividing traffic up into valued (premium) and not, 
but customers also value and pay for this so-called 'not valued' 
traffic. And operators don't only care about premium traffic - they 
know they can't afford to lose custom for either service type.

Matt's question was about how the network can infer whether 
congestion is shared or self-congestion (policing the major 
contributers would help in the former, but not the latter).

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 8 Mar 2013 00:06:22 +0000

Matt,

> > which bottleneck prevails?   Bob?

Sorry, seeing your repeated question, I had misinterpreted.

As Mirja says below, I can't see how ConEx could help the /content 
provider/ to know anything. The content provider sends the ConEx 
signal in the first place, so it surely cannot learn anything new 
from a signal it sends itself.

I think Mirja's on the right track for the content provider to infer 
the difference between shared and self-congestion. Fixed-line 
self-congestion should lead to a very regular total bandwidth across 
flows, and a regular loss pattern.

That isn't a panacea tho, because there might be tons of devices 
within a home sharing a fixed access link making all the flows to any 
single device see what looks like shared and variable congestion. Or 
the link might have a wireless tail that is shaking about all over 
the bandwidth range.

Does it matter? If there's a link failure so loads of people's 
traffic all gets squeezed into some remaining link, the content 
provider will see the congestion, and adjust down. The congestion 
level will be higher, so it should go slower. Sorted?

However, I don't understand the last part of your question (below). 
Why more brittle?

Bob