Subject: [conex] comments on draft-moncaster-conex-concepts-uses-01.txt

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 12 Aug 2010 13:45:12 +0200 (CEST)

1. "Users are increasingly seeing congestion at peak times"

I disagree with this. In my market this was a problem 5-10 years ago. 
There are testing tools available so that people can test their connection 
and if they don't get ok speed they have a right to cancel their 
subscription. In Sweden the ISPs nowadays advertise speeds with a max and 
lower bound guarantee speed, and people use their right to change 
providers when they don't work well. This basically means very few 
providers have been underprovisioning their networks, thus there is not 
much congestion outside of the customer access port.

Also, calling congestion "unforseen" is a stretch, it very seldom is. It's 
when the engineering side clashes with the marketing side which wants to 
sell high bandwidth service but doesn't want to spend the money on 
upgrading the network to accomodate this increased traffic.

I agree that implementing LEDBAT will make it really hard to get a grip on 
how the network is behaving. Looking at an MRTG graph that is flatlining 5 
hours per day makes it really hard to understand what the user experience 
during those 5 hours is. Metrics like average queue depth and alike might 
be needed, but again we run into the problems of equipment with very small 
buffers and very little L3 features.

"But while flat-rate
    pricing avoids billing uncertainty, it creates performance
    uncertainty: users cannot know whether the performance of their
    connection is being altered or degraded based on how the network
    operator manages congestion."

I'd say that a token based system where you have a monthly cap and then 
you have to actively log in to a system and purchase additional credits to 
get a new "cap" is very obvious and understandable to the user. When the 
user is out of their cap they might be ratelimited to a very low access 
speed, like 64 kilobits/s or alike. This will remove and uncertainty about 
what's going on, otherwise the user might be moved to a service vlan where 
a captive portal requires purchase of additional credits to resume the 
server.

I also feel that the talk about "congestion charges" is dystopian. The 
Internet succeeded because it was basically "bill and keep", and it relies 
on each end of the communication to make their own customer happy. 
Implementing a billing system for congestion would be approximately as 
hard as implementing a QoS system end-to-end and charging extra for high 
priority traffic, so why wouldn't we do that instead? Anyway, I don't 
think this is the way forward, it's much too complex.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: John Leslie <[email-masked]>
Date: Thu, 12 Aug 2010 08:38:14 -0400

> I disagree with this. In my market this was a problem 5-10 years ago. 

   In US markets, it's a continuing problem.

> There are testing tools available so that people can test their
> connection and if they don't get ok speed they have a right to cancel
> their subscription.

   In monopoly-heavy US, this "right" often brings heavy cancellation
charges.

> In Sweden the ISPs nowadays advertise speeds with a max and lower bound
> guarantee speed, and people use their right to change providers when
> they don't work well.

   This is not the forum to argue national policies...

> This basically means very few providers have been underprovisioning
> their networks, thus there is not much congestion outside of the
> customer access port.

   I wish we could avoid arguing to what extent providers "underprovision"
their networks. ConEx could be useful in an all-fibre distribution system
where the bottleneck is the cheap ethernet switch the customer adds
between "modem" and his computers.

> Also, calling congestion "unforseen" is a stretch, it very seldom is.

   Actually, it is often "unforseen" by network engineers, who lack
information of what customers will want tomorrow.

> It's when the engineering side clashes with the marketing side which
> wants to sell high bandwidth service but doesn't want to spend the
> money on upgrading the network to accomodate this increased traffic.

   That's largely true, but rather unfair, because the neither department
makes the upgrade-spending decisions. The department which _does_ make
those decisions rightly wants justification for the costs.

> I agree that implementing LEDBAT will make it really hard to get a
> grip on how the network is behaving.

   Good!

> Looking at an MRTG graph that is flatlining 5 hours per day makes it
> really hard to understand what the user experience during those 5 hours
> is. Metrics like average queue depth and alike might be needed, but
> again we run into the problems of equipment with very small buffers
> and very little L3 features.

   Those metrics are unlikely to help -- you need to know what traffic
the _user_ considers interactive and what the effect of congestion is
on that traffic.

> "But while flat-rate pricing avoids billing uncertainty, it creates
>  performance uncertainty: users cannot know whether the performance of
>  their connection is being altered or degraded based on how the network
> get a new "cap" is very obvious and understandable to the user.

   But most users won't "log in to a system and purchase additional
credits". They'll call support and claim not to understand how this
could happen.

> When the user is out of their cap they might be ratelimited to a very
> low access speed, like 64 kilobits/s or alike. This will remove and
> uncertainty about what's going on, otherwise the user might be moved to
> a service vlan where a captive portal requires purchase of additional
> credits to resume the server.

   You're welcome to try this business model in your own business. But
I wouldn't try it in mine. You're proposing to limit overall rate when
there may be no reason to limit it, and when the user is probably
unaware of any action they took to "deserve" this "punishment". My users
would be on the phone by now...

> I also feel that the talk about "congestion charges" is dystopian.

   That is a widespread belief, which we probably need to discuss.

> The Internet succeeded because it was basically "bill and keep",

   True -- simplicity of billing is central to our success.

> and it relies on each end of the communication to make their own
> customer happy. 

   Unfortunately, there's too little under our control to do so. :^(

> Implementing a billing system for congestion would be approximately
> as hard as implementing a QoS system end-to-end and charging extra
> for high priority traffic, so why wouldn't we do that instead?

   Implementing a "billing system" for congestion isn't hard at all.
To the user, an ISP sets a congestion allowance, different for different
service levels: any service call is an opportunity to sell a higher
service level.

   To its peers, each ISP has a system of settlements: if congestion
volume matches "closely enough" no money changes hands; if you want
to send more congestion volume than you receive, it's a business
decision whether to pay the agreed price. This is simple.

   Implementing a QoS end-to-end system requires agreements at every
interchange point -- to an arbitrarily large number of potential paths.
I wouldn't bother to try. :^( (Of course, QoS end-to-end within a
single provider is simple enough...)

> Anyway, I don't think this is the way forward, it's much too complex.

   Question to clarify: is what I outlined above "too complex" or is
it somehow incomplete?

--
John Leslie <[email-masked]>

---

From: Dirk Kutscher <[email-masked]>
Date: Thu, 12 Aug 2010 12:47:26 +0000

Hi,

Interesting discussion.

I would like to add that congestion exposure is a tool that can be used in different ways, congestion charges not necessarily being the most desirable one, neither assessing and comparing ISP performance. 

Another way to see it to use congestion exposure for incentivizing better application and transport layer responsiveness to congestion indication, i.e., giving users a tool for capacity sharing without giving up
end-to-end congestion control. Users would be able to have certain sessions throttle down, based on their own decision/configuration, leveraging indications about current contribution to path congestion.

Now, in such a setting, the actual incentive is not the congestion indication (otherwise we could just use ECN), but the capability of the network to assess path congestion and to police traffic and/or account for congestion accordingly. Policing and accounting could take different variants -- in fact some form of rate-limiting could apply after a certain congestion budget has been used.

Mikael is saying that congestion is mainly caused by overselling and should not occur in well-managed networks. In an ideal world, this might be true. A specific example where it is not true is best-effort traffic in networks with shared media, e.g., wireless networks. But even without that, I don't think that we can expect all fixed networks to be designed in way that backhaul capacity always matches aggregated access network capacity, even if technically feasible.

Best regards,

Dirk

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Mikael Abrahamsson
> Sent: Thursday, August 12, 2010 1:45 PM
> To: [email-masked]
> Subject: [conex] comments on draft-moncaster-conex-concepts-uses-01.txt
> 
> 
> connection
> It's
> _______________________________________________
> conex mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/conex

---

From: "Toby Moncaster" <[email-masked]>
Date: Thu, 12 Aug 2010 15:34:42 +0100

> Of John Leslie
> Sent: 12 August 2010 13:38
> To: Mikael Abrahamsson
> Cc: [email-masked]
> Subject: Re: [conex] comments on draft-moncaster-conex-concepts-uses-
> 01.txt
> 

And in the UK - indeed in some ways it is getting worse as more and more
users turn to VoD as a primary source of visual entertainment - Here the UK
may be leading the trend with the ever increasing popularity of iPlayer,
Skyplayer, 4oD, Demand Five, etc all of which offer some form of streaming
VoD TV catchup service.

>    In monopoly-heavy US, this "right" often brings heavy cancellation

This seems to be an example of Sweden being a leading light in championing
consumer rights... Anyway, it is not an argument against ConEx...

>    This is not the forum to argue national policies...

Indeed not

> between "modem" and his computers.

The problem is providers can always get caught out. At the moment there is
often a huge spare capacity of unlit dark fibre so some operators can
re=provision on very short timescales (hours even), but that may not last...

> information of what customers will want tomorrow.

Indeed, and it is certainly unforeseen by the users that have caused it...

> the _user_ considers interactive and what the effect of congestion is

That is the crux of the problem - knowing what the user really wants, rather
than guessing... 

> credits". They'll call support and claim not to understand how this

However, this is not unlike the congestion rationing system that we
discussed in the paper I referred to last rely
http://bobbriscoe.net/projects/refb/polfree_rearch08.pdf 

I (speaking personally) think most users can understand the concept of a
variable bill for mobile phone services - they get a "free" allowance but if
they go over then they will get a bill for it...

> would be on the phone by now...

So the answer there is that the ISP has to provide the user with a means to
control this (ideally a simple application that does it all for them!). But
you are right John, there are going to be lots of different business models
out there. ConEx is simply a metric/tool to enable such things.

>    That is a widespread belief, which we probably need to discuss.

It almost deserves a discussion forum of its own as it will swamp any normal
WG discussions...

>    True -- simplicity of billing is central to our success.

Yes... But we (techy types at the IETF) are not specialists in consumer
behaviour and it really is not our place to discuss these sort of things -
they are anyway orthogonal to the issue of whether and how to standardise a
ConEx mechanism...

> > for high priority traffic, so why wouldn't we do that instead?

Two reasons, firstly it is far more flexible in how it is used, secondly it
is more future-proof...

>    Question to clarify: is what I outlined above "too complex" or is

Just to clarify - that is one way to use ConEx to implement a system wide
congestion-based billing system. It is also well beyond the charter as it
stands

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 12 Aug 2010 21:35:05 +0200 (CEST)

On Thu, 12 Aug 2010, John Leslie wrote:

>   I wish we could avoid arguing to what extent providers 
> "underprovision" their networks. ConEx could be useful in an all-fibre 
> distribution system where the bottleneck is the cheap ethernet switch 
> the customer adds between "modem" and his computers.

But is this really a device which will be ECN capable? It's typically L2 
only.

> information of what customers will want tomorrow.

I guess we can agree to disagree then.

> those decisions rightly wants justification for the costs.

So if the marketing department makes decisions in pricing where the income 
can't cover the network buildout but they still want to advertise high bw 
access speeds to customers, you get distribution/core congestion. It's 
still the ISP doing the wrong thing and it's perfectly avoidable.

> would be on the phone by now...

We've been doing it for years on our mobile broadband. It's a bit of 
a special case, but I'd say it's more understandable to users (since it's 
actually actively published and advertised, it's not done covertly) than a 
lot of other schemes proposed. It's also net neutral.

> service levels: any service call is an opportunity to sell a higher

It's hard when it has to be done inter-provider.

> decision whether to pay the agreed price. This is simple.

I don't agree. Most of the time today no money changes hands when doing 
peering, it's done on a mutual benefit basis. Having congestion metrics to 
keep track of means considerable complication compared to what we have 
today.

>   Question to clarify: is what I outlined above "too complex" or is it 
> somehow incomplete?

Well, I don't understand how it would work in real life, so I guess it 
could be both because I'm trying to fill in blanks and there might be 
simpler solutions that I can't think of.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 13 Aug 2010 09:43:58 +0200 (CEST)

On Thu, 12 Aug 2010, Toby Moncaster wrote:

> This seems to be an example of Sweden being a leading light in 
> championing consumer rights... Anyway, it is not an argument against 
> ConEx...

No, it's not an argument against ConEx but it's an argument against the 
current problem description in the draft of with parts I don't agree with.

> The problem is providers can always get caught out. At the moment there 
> is often a huge spare capacity of unlit dark fibre so some operators can 

Well, I guess we can agree to disagree. Most of the time when I've seen 
congestion just "happen" (as some see it), it's been a matter of "I told 
you so" but people took a risk and gambled that something wouldn't happen.

It's not like it came out of nowhere.

> Indeed, and it is certainly unforeseen by the users that have caused it...

That I can agree with, customers don't expect congestion and they expect 
their access speed to be the speed at which they can communicate.

> variable bill for mobile phone services - they get a "free" allowance 
> but if they go over then they will get a bill for it...

It's my opinion that users want predictability. The EU has instated a 
mandatory demand on mobile phone operators to implement a 50EUR/month cap 
on billable traffic, then the user has to be notified and agree to a 
higher cap, and this is in line with the thought of predictability.

A good compromise of this predictability is to start nagging the user and 
degrading performance to almost unusable state until the user makes a 
decision to pay more.

> It almost deserves a discussion forum of its own as it will swamp any normal

Agreed.

> Two reasons, firstly it is far more flexible in how it is used, secondly it

I don't think I agree on this point. QoS is done between ISPs without 
really involving end systems (apart from what the systems mark). What 
we're proposing here needs to end up in the IP stack of the end system, 
and we all know how many people still use Windows 95/98/XP. I'd say there 
definitely is parity in how hard both schemes would be to implement.

---

From: John Leslie <[email-masked]>
Date: Fri, 13 Aug 2010 09:56:01 -0400

Mikael Abrahamsson <[email-masked]> wrote:
>> To the user, an ISP sets a congestion allowance, different for
>> different service levels: any service call is an opportunity to sell
>> a higher service level.
> 

   That part _isn't_ inter-provider: it's strictly ISP-to-customer.

> peering,

   "Close enough" is in the eye of the beholder. For a large fraction
of peering, I'd expect 2:1 ratios to be "close enough".

> it's done on a mutual benefit basis. Having congestion metrics to 

   Tracking congestion volume is certainly simple enough.

   Any complexity arises from what to do when it gets out of balance.

> simpler solutions that I can't think of.

   There certainly could be simpler solutions: I'm not claiming otherwise.

   As for the "blanks" to fill in, basically it's what to charge, and
perhaps there's an implied blank in how quickly charges may change.

   I think we can safely set an upper bound of, say, $1 per gigabyte of
congestion-volume: I doubt anyone will agree to pay more than that. The
lower bound is clearly zero. (Just keeping track is sufficient for many
peering cases now.)

   As for what constitutes "close enough", I'd guess an upper bound of
2:1 and a lower bound of 1.1:1 -- I don't believe anyone's interested in
exceeding those bounds.

   As for how quickly charges may change, I'd say an upper bound of
10-cents per year (increase, per gigabyte) and a lower bound of 1-cent
per year -- beyond that business planning becomes too hard. (But of
course, forbearance of charging will be common.) ISPs will arrange
their customer service levels to cover the potential increases.

   Are there other "blanks" you're concerned with?

---

From: John Leslie <[email-masked]>
Date: Fri, 13 Aug 2010 10:02:59 -0400

Mikael Abrahamsson <[email-masked]> wrote:
>> Two reasons, firstly it is far more flexible in how it is used, secondly
>> it is more future-proof...
> 

   "Parity" is a funny name for this...

   QoS has had plenty of time to deploy; and has shown essentially zero
deployment beyond single-provider uses. I see no reason to believe it
will deploy any faster in dual-provider uses, or at all in three-or-more-
provider uses.

   Operating system software turns over with great regularity; and the
types of changes in end-user systems we're talking about don't even need
OS turnover. Recall that non-ConEx traffic is expected to co-exist with
ConEx traffic for the forseeable future.

---

From: Christopher Morrow <[email-masked]>
Date: Fri, 13 Aug 2010 10:08:21 -0400

>> I don't agree. Most of the time today no money changes hands when doing

it's possible that the difference here is 'peer' vs 'customer' and
that time when someone changes from 'peer' to 'customer' (when their
ratio gets out of whack) There are implications beyond $$ changing
hands though, routing policy changes and expected traffic destinations
will change.

When conex leans toward 'settlement' I tend to look at it askance...
For ISP -> Customer discussions today the 'settlement' already
happens, the bits above (and what I snipped below) seem to be what my
first paragraph outlines: DePeering.

-chris

---

From: John Leslie <[email-masked]>
Date: Fri, 13 Aug 2010 10:30:43 -0400

Christopher Morrow <[email-masked]> wrote:
> ratio gets out of whack)

   I'm not sure we all understand what Christopher is saying...

   In current ISP-ISP relations, "peering" refers to an arrangement
where routes are exchanged, and no traffic is default-routed in either
direction. (There are other limits as well.) Generally, no money changes
hands in a "peering" arrangement (except for "fabric" over which traffic
flows to and from the peering point).

   When "peering" conditions (which may include balance of traffic in
one direction vs. the other) are exceeded, one of the peers refuses to
continue the "peering" arrangement, and offers the other peer a
"customer" arrangment, typically including the right to default-route
customer traffic to the "transit provider".

> There are implications beyond $$ changing hands though, routing policy
> changes and expected traffic destinations will change.

   Exactly! ...if the relationship changes to "customer".

   It is certainly premature to say whether congestion-volume will become
a "condition" of "peering", but I won't claim it necessarily won't.

   If, in fact, an ISP is offered a "peering" arrangement which _limits_
congestion-volume (instead of charging for the imbalance), that ISP
will need to limit the congestion-volume _sent_to_that_peer.

   This suggests that ConEx "congestion-expected" traffic might travel
a different route than non-ConEx traffic, which I agree is possible.
This is probably sub-optimal, but not as bad as it sounds. The ISP that
declines this traffic at one peering point will have to accept it at
another, and any increased latency will disadvantage its customers at
least as much as the sender. I don't expect this case will prevail for
long.

> first paragraph outlines: DePeering.

   We have been through the history of de-peering based on total volume.
It took a while, but saner heads _did_ (mostly) prevail.

   (BTW, I don't blame folks for worrying about the possibility of
changing peering agreements: it tends to be painful!)

---

From: "STUART VENTERS" <[email-masked]>
Date: Fri, 13 Aug 2010 09:31:47 -0500

>their access speed to be the speed at which they can communicate.

It seem to me that an expectation of congestion is a good thing if it lets you usually run at a speed much higher than you could without it.  The keys are an acceptable definition of 'usually' and how well the network shares during congestion.

{Begin thought experiment}
If you dreamed of building an Internet without congestion...

In order to meet an absolutely congestionless expectation you would have to build the Internet backbone without using statistical multiplexing.  That is, the available rate would be limited by the access rate and use severe topology constraints in the backbone mesh.

Checking the economics on the back of an envelope:
   If a port on a backbone router costs 1$/Mbit/sec and on average 10 hops are required, then the backbone switching cost would be $10/Mbit.  $100 backbone switching cost for 10Meg service seems possible assuming the starting numbers are right and you don't put in much path redundancy.  (Perhaps someone here has an idea how the operating and fiber costs would fit in here.)

>From a practicality standpoint in the US, if 100M customers each get 10M of service, that's 10**15 bits/sec or 10,000 100GigE's.  That seems a lot, but maybe possible if backbone switching technology improves.

History shows that if you did build such a backbone, some folks would notice that with 10M access links, it's idle most of the time.  A reasonable response is to switch to 100M access for some new killer app which puts us back where we started.  Which keeps the definition of 'usually' and sharing during congestion on the table.
{End thought experiment}

---

From: Christopher Morrow <[email-masked]>
Date: Fri, 13 Aug 2010 16:19:28 -0400

>   I'm not sure we all understand what Christopher is saying...

apologies, sometimes I forget not everyone uses the same terms, and
peering is certainly overloaded in the networking world :)

>   Exactly! ...if the relationship changes to "customer".

Which I think is a perfectly fair way to look at this discussion...
keep in mind that most 'peering' arrangements (between larger networks
at least) have other factors aside from 'i send you 1bit, you can send
me 2bits' to keep track of, things like:
  o latency improvements
  o cost to ship the bits across transit/pay links
  o other business relationships (telephone call exchange)
  o ceo-golf-timez
(you get the picture)

Adding congestion cause/effect isn't unreasonable

> a different route than non-ConEx traffic, which I agree is possible.

this gets hairy, the current routing protocols don't really have a
method to account for this... so you'd also have to affect
bgp/isis/etc and that strikes me as very dangerous, more state in the
network here is a step in the wrong direction. (plus what does that do
to paths and routes and ... yikes no.)

> It took a while, but saner heads _did_ (mostly) prevail.

we still have these events... about every 9-12 months some large
provider depeers another. It's painful, sometimes it gets reverted,
sometimes not. The 'settlement' portion I was getting at wasn't really
about de-peering as much as deciding if/when 2 networks should decide
to pay each other for 'congestion causing' traffic.

To me that sounds a whole lot like the settlement system (which takes
months/years to come to resolution, apparently) which exists in the
long-distance telephone world. That sort of thing is a non-starter I
believe. Using the congestion contribution data in peering agreements
isn't as far fetched, I suppose.

> changing peering agreements: it tends to be painful!)

and take a very long time, so... unless a peer is constantly causing
problems they tend to not get addressed... 'constantly' means 24/7 for
months or +year. it's very hard to re-arrange things once you set up
peering (settlement-free), and the cost to upgrade a 'peer' port is
likely cheaper than upgrading transit + transit costs would be for the
same traffic flows reverting to transit paths.

-chris

---

From: ken carlberg <[email-masked]>
Date: Fri, 13 Aug 2010 16:36:03 -0400

> to paths and routes and ... yikes no.)

what about the case of more than one equal cost path via ISIS/OSPF, where in the "congested-expected" traffic travels a different path than the one normally selected through a simple hash?  ie, change the tuple for another path (or even LSP if we're thinking mpls) of the ingress router so that perhaps a minor change in the route may help alleviate the congestive condition.

-ken

---

From: John Leslie <[email-masked]>
Date: Fri, 13 Aug 2010 19:54:56 -0400

>> a "condition" of "peering", but I won't claim it necessarily won't.

   To be clear, I agree this is "possible"; but I didn't say it was
"reasonable", or even "likely"...

> method to account for this...

   All the more reason it shouldn't happen!

   Note also, that under current peering arrangements, congestion-volume
won't be counted at all, so the question doesn't even arise.

> so you'd also have to affect bgp/isis/etc and that strikes me as very
> dangerous, more state in the network here is a step in the wrong
> direction.

   Agreed.

   Indeed, faced with this unbalance, and a threat to de-peer, you'd
probably arrange for more congestion on your side.

   This is not quite as silly as it sounds: I've been causing intentional
congestion on incoming port 25 for years -- from IP ranges that send
too much spam. I'd never ask folks to pay me for that "congestion", but
it does give me some wiggle room. (If folks _did_ pay me for it, I'd most
likely reduce the amount of congestion for sources that pay...)

   But the real point is, this is simply a silly proposition: it's far
more likely your peering arrangements won't actually penalize you until
the peers see it as a profit-center and set a settlement rate.

> (plus what does that do to paths and routes and ... yikes no.)

   It's not _really_ that scary... Aggregating the congestion-volume by
CIDR block would give a limited number of routes to that peer that you
can disincent. It probably would even be stable enough to do almost
manually.

   But again, this eventuality is silly-season and shouldn't happen at
all in a one-tenth reasonable universe.

> provider depeers another.

   (Notice, I didn't ask for a "half-way reasonable" universe... :^(

> It's painful, sometimes it gets reverted, sometimes not. The 'settlement'
> portion I was getting at wasn't really about de-peering as much as
> deciding if/when 2 networks should decide to pay each other for
> 'congestion causing' traffic.

   This is all new territory. I have posted my best guesses in a separate
email.

   The ISPs with the greatest congestion towards their users will see
the greatest benefit from settlements, but they'll find their peers less
than anxious to pay -- absent actual evidence that payments will lead
to significant improvements. This will slow down any actual payments.

   To the extent such settlement payments _actually_ lead to improvents
the charges become somewhat self-liquidating...

> isn't as far fetched, I suppose.

   In that market, there is a government mandate to interconnect, which
leads to lawyers arguing rather than engineers upgrading.

   I see no reason for government mandates for ConEx settlements, nor
even for lawyers. Folks who don't want to pay simply don't pay; and
they get regular Best Effort service. Folks who do pay get priority
for their congestion-marked packets.

   Rather than threaten to de-peer, a sane policy would be to tax
incoming ConEx packets (of their remaining congestion allowance) to
restore balance from any peer that chooses not to pay a settlement.

   But frankly, we are rather a ways from that issue even arising; and
it is, after all, not within the scope of ConEx to specify what to do
if there's a perceived imbalance.

> same traffic flows reverting to transit paths.

   Exactly!

---

From: Christopher Morrow <[email-masked]>
Date: Fri, 13 Aug 2010 21:07:19 -0400

> what about the case of more than one equal cost path via ISIS/OSPF, where in the "congested-expected" traffic travels a different path than the one normally selected through a simple hash?  ie, cha…

care to explain what part of the 5tuple I use today for hashing would
be used in this decision process? in today's hardware this isn't
feasible, today's routing protocols essentially select a single
destination (which may be ecmp'd ec here == 'equal cost').

If you had the magic of mpls you could decide to put traffic on
different colored LSP's based on some packet marking (tos/dscp bits).
I bet you can't do that fast enough to matter in a network large
enough to matter. You'd still have to maintain this new/large/fast
state in your network/routing protocols, that cost is not
insubstantial.

-chris

---

From: Christopher Morrow <[email-masked]>
Date: Fri, 13 Aug 2010 21:21:45 -0400

> congestion on incoming port 25 for years -- from IP ranges that send

at the end system level folks are free to do as they please, if the
want to enable 'nice mode' they surely can. Routers in the core of the
network are a completely different story, as are BRAS type devices
(your first L3 hop after your dsl-modem).

> the peers see it as a profit-center and set a settlement rate.

agreed, until they feel they can turn you from a 'peer' to a 'customer'.

> can disincent. It probably would even be stable enough to do almost

on a core router at a large ISP today (a 'tier-1' isp) there are
upwards of 4million paths, maintaining that state isn't trivial,
adding to that a potentially quickly changing (as congestion
arrives/dissipates) new set of state for even the advertised netblocks
which hold 'conex sources' (or destinations, the core router doesn't
actually know which side) is really not going to be good.

I'd actually say that you don't want to use the covering prefix here
either, you really want to use the actual contributors, my home dsl
traffic isn't causing the congestion, the akamai server in the same
/20 as me is....

> all in a one-tenth reasonable universe.

agreed.

> the greatest benefit from settlements, but they'll find their peers less

at some point 'toward the user' means 'last mile' and whatever physics
(and customer money) constraints there are on that last-mile will
rule. Customers may chose to enable 'nice mode' (pay attention to
conex/re-ecn data) if they feel there is benefit to THEM to do that.
I'd say that a default 'on' for this at the CPE/customer-OS sounds
great, people will get a better internet experience with the pipe they
have.

> to significant improvements. This will slow down any actual payments.

in the last-mile case, the ISP really can't do anything here, there
are constraints beyond what money will solve, and their customers are
seemingly happy to pay for the pipe-size they have. no incentive
exists here.

> for their congestion-marked packets.

money == contracts == lawyers, there won't be payments made across the
local bar, there will be 'bills' sent. Or in the case of 'peer' ->
'customer' shifts... lots of fun times, then payments along with
contracts and the usual customer things.

> if there's a perceived imbalance.

agreed.

again, thanks.
-chris

---

From: Kevin Mason <[email-masked]>
Date: Tue, 17 Aug 2010 17:34:35 +1200

A  comment on the draft.

On accouinting approaches to using congtion information

I would challenge the view that another service provider "causes" congestion on another ISP's network. Data is only passed between sender and receiver so it is the ISP's customer that is requesting the data from customers of the other provider, not the provider "sending it".

So if the "sending" provider is not causing the congestion (because it is the receiving provider's customer the requested it) then arguing the "sending" provider should pay for any congestion the resulted might be difficult. I can see endless legal arguments as to why one or the other party is culpable and therefore who should endure any commercial consequences.

On network uses of the information I think there is a general concept that is not being captured well. 

In ISP networks there are, very simply, two parts. Firstly there is the connectivity between each account holders demarcation (UNI) and a IP edge device (BRAS/BNG in Broadband Forum speak). The IP edge device typically facitites AAA functions as well as user based policy enforcement and by necessity is fully aware of what flows below to what UNI's (e.g. because they are all on a single authenticated VLAN or PPP tunnel).

Beyond that point between the IP edge and any peering point, the network does not maintain an specific awareness of individual end points. Routers could  theoretically maintain information about each source/destination pair AND consult some database to relate that to a end user profile, but this is not very scaleable. 

So in the core ISP network, if a forwarding next hop is approaching overload, then the egress config of the router must deal with the aggregate flow and act on information that is in the packet header alone. Maintaining knowledge of which "users" have "caused" the most congestion in recent times is too hard.

So it would appear that a possible scheme might be that a two stage queue management regime might be desirable, whereby at a lower queue size packets begin to be congestion marked if ECN capable and maybe discarded if not, but at a slightly higher queue depth, packets that are ECN capable but with a high positive congestion value get discarded, on the basis that these have a lower chance of reaching their destination than packets not declaring an expectation of congestion on the rest of the path. I don't see a "border monitor" as described in the draft being very practical or useful in this part of the network.

When packets arrive at the BRAS destined for the user, then user based policy could be applied. One such policy might be to discard all packets with a congestion deficit of more than x. This is the safety net against dishonesty by the sender. An additional policy might be to discard packets that have experienced congestion above a threshiold (which may be different for different user profiles) so far AND that are destined to a user that has a recent history of high congestion marked packets. If previous congestion marks have result in the user backing off then this policy would not be invoked, so it would only apply to users that are persistently contributing to congestion somewhere on the path traversed (on the same provider or any preceding providers network).

These policy actions could constitute the "edge monitor" functions referred to in the draft but would actually be part of the policy functions of the edge device itself, not any independent function.

Other may have diffeent views of how the revealed congestion information might be used but I believer it is useful to at least consider the two parts of an ISP network when discussing possible used for the information.

Cheers
Kevin Mason

---

From: "STUART VENTERS" <[email-masked]>
Date: Tue, 17 Aug 2010 09:15:40 -0500

Kevin Said:

>On accounting approaches to using congestion information
>

>So if the "sending" provider is not causing the congestion (because it is the receiving provider's customer the requested it) then arguing the "sending" provider should pay for any congestion the re…

Two cents worth:

On a packet by packet basis, it seems like congestion is directly caused by the customer sending the packets.  On a longer term, perhaps network engineering and marketing have something to do with it as well.  Exposing congestion could provide good, constructive feedback to the customers causing it.  I wonder if the sign of the feedback for the service providers could be in the opposite direction.

With regards to a more indirect cause of congestion, the idea of requester versus sender being responsible for congestion is interesting.  If a customer requests a web page, it doesn't seem to me that he also desires all the ads that come with it.  That choice seems to be coming from the web server.

---

From: "Woundy, Richard" <[email-masked]>
Date: Tue, 17 Aug 2010 15:17:19 -0400

>>I would challenge the view that another service provider "causes"
congestion on another ISP's network. Data is only passed between sender
and receiver so it is the ISP's customer that is requesting the data
from customers of the other provider, not the provider "sending it".

Kevin, you make a good point (customer pulls rather than provider
pushes), but I don't think that is the entire story.

Consider a hypothetical example (an *extreme* case to be sure, but not
totally disconnected from reality) in which a service provider changes
its routing policy in the following manner: from forwarding traffic
somewhat equally over 10 interconnects to a downstream ISP, to
forwarding all traffic over a single interconnect to the downstream ISP
(such as changing policy to "hot potato routing" from a central hosting
center to the downstream ISP). That change in policy is very likely to
cause a lot of congestion over the single interconnect link, even as the
overall consumer behavior doesn't change at all.

There are a lot less extreme, real-world examples of routing policy
changes that would have a similar network impact.

-- Rich

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf
Of Kevin Mason
Sent: Tuesday, August 17, 2010 1:35 AM
To: [email-masked]
Subject: [conex] comments on draft-moncaster-conex-concepts-uses-01.txt

A  comment on the draft.

I would challenge the view that another service provider "causes"
congestion on another ISP's network. Data is only passed between sender
and receiver so it is the ISP's customer that is requesting the data
from customers of the other provider, not the provider "sending it".

So if the "sending" provider is not causing the congestion (because it
is the receiving provider's customer the requested it) then arguing the
"sending" provider should pay for any congestion the resulted might be
difficult. I can see endless legal arguments as to why one or the other
party is culpable and therefore who should endure any commercial
consequences.

On network uses of the information I think there is a general concept
that is not being captured well. 

In ISP networks there are, very simply, two parts. Firstly there is the
connectivity between each account holders demarcation (UNI) and a IP
edge device (BRAS/BNG in Broadband Forum speak). The IP edge device
typically facitites AAA functions as well as user based policy
enforcement and by necessity is fully aware of what flows below to what
UNI's (e.g. because they are all on a single authenticated VLAN or PPP
tunnel).

Beyond that point between the IP edge and any peering point, the network
does not maintain an specific awareness of individual end points.
Routers could  theoretically maintain information about each
source/destination pair AND consult some database to relate that to a
end user profile, but this is not very scaleable. 

So in the core ISP network, if a forwarding next hop is approaching
overload, then the egress config of the router must deal with the
aggregate flow and act on information that is in the packet header
alone. Maintaining knowledge of which "users" have "caused" the most
congestion in recent times is too hard.

So it would appear that a possible scheme might be that a two stage
queue management regime might be desirable, whereby at a lower queue
size packets begin to be congestion marked if ECN capable and maybe
discarded if not, but at a slightly higher queue depth, packets that are
ECN capable but with a high positive congestion value get discarded, on
the basis that these have a lower chance of reaching their destination
than packets not declaring an expectation of congestion on the rest of
the path. I don't see a "border monitor" as described in the draft being
very practical or useful in this part of the network.

When packets arrive at the BRAS destined for the user, then user based
policy could be applied. One such policy might be to discard all packets
with a congestion deficit of more than x. This is the safety net against
dishonesty by the sender. An additional policy might be to discard
packets that have experienced congestion above a threshiold (which may
be different for different user profiles) so far AND that are destined
to a user that has a recent history of high congestion marked packets.
If previous congestion marks have result in the user backing off then
this policy would not be invoked, so it would only apply to users that
are persistently contributing to congestion somewhere on the path
traversed (on the same provider or any preceding providers network).

These policy actions could constitute the "edge monitor" functions
referred to in the draft but would actually be part of the policy
functions of the edge device itself, not any independent function.

Other may have diffeent views of how the revealed congestion information
might be used but I believer it is useful to at least consider the two
parts of an ISP network when discussing possible used for the
information.

Cheers
Kevin Mason

---

From: Kevin Mason <[email-masked]>
Date: Wed, 18 Aug 2010 11:40:45 +1200

Cheers
Kevin Mason

> From: Woundy, Richard [mailto:[email-masked]]
> Sent: Wednesday, 18 August 2010 7:17 a.m.
> To: Kevin Mason; [email-masked]
> Subject: RE: [conex] comments on draft-moncaster-conex-concepts-uses-
> 01.txt
> 

[Kevin Mason] The culpability for congestion on the interconnecting hop is very dependent on who dimensions it and the commercials that underpin it. If the "sending" provider dimensions it then congestion on this hop is solely the accountability of the sending proivder, no need for congestion exposure here as they can directly measure it today (queuing and discards). IF the receiving provider dimensions it then they have no current visibility of the congestion on the preceding link, but congestion exposure will potentially only tell them that congestion has already been experienced, not where. So if there is any SLA around performance of the interconnection hop then the receiving party still has to get info from the sending provider to ascertain that it is the interconnecting hop that is the problem and not a hop deeper in the sending provder's network.

Capturing the information for network management purposes at the interprovider level may well be very useful for overall network planning purposes if practical, but using it to underpin payment between providers is very different.

However I do not think we need to get too hung up on this, the point is that it is debateable who "causes" the congestion in an interprovider context, and therefore getting agreement on who might "pay" for it has the potential only to enrich the legal industry. I am however in favour of using congestion information for accounting purposes at a individual ISP customer account level to recognise and reward cooperative end user behaviour (e.g. congestion caps)

To my mind the power of Conex is to provide a forwarding device a richer set of information to make better decisions on how to manage their queues for the greater good.

---

From: "Toby Moncaster" <[email-masked]>
Date: Wed, 18 Aug 2010 10:34:47 +0100

Inline...

Toby

> Sent: 18 August 2010 00:41
> To: Woundy, Richard
> Cc: [email-masked]
> [Kevin Mason] The culpability for congestion on the interconnecting hop
> is very dependent on who dimensions it and the commercials that
> underpin it. If the "sending" provider dimensions it then congestion on
> this hop is solely the accountability of the sending proivder, no need
> for congestion exposure here as they can directly measure it today
> (queuing and discards). IF the receiving provider dimensions it then
> they have no current visibility of the congestion on the preceding
> link, but congestion exposure will potentially only tell them that
> congestion has already been experienced, not where. So if there is any
> SLA around performance of the interconnection hop then the receiving
> party still has to get info from the sending provider to ascertain that
> it is the interconnecting hop that is the problem and not a hop deeper
> in the sending provder's network.
> 
> Capturing the information for network management purposes at the
> interprovider level may well be very useful for overall network
> planning purposes if practical, but using it to underpin payment
> between providers is very different.

I obviously bow to the greater knowledge of Kevin and Rich in this, but it
seems to me there are scenarios where ConEx information may be a sensible
basis for settlements (perhaps not with money changing hands, but for
instance with shared backbones such as that used by the UK academic
community (ja.net)...

> 
> However I do not think we need to get too hung up on this, the point is
> that it is debateable who "causes" the congestion in an interprovider
> context, and therefore getting agreement on who might "pay" for it has
> the potential only to enrich the legal industry. I am however in favour
> of using congestion information for accounting purposes at a individual
> ISP customer account level to recognise and reward cooperative end user
> behaviour (e.g. congestion caps)

OK, this is in danger of becoming highly philosophical! Let's take 3 common
scenarios:

1) A user accesses video content via a CDN
2) A user uploads photos to facebook
3) A user does a web search and visits a link from google

In all these scenarios it is debatable who CAUSES any congestion this
traffic encounters. In 1) it could be said to be the user (he wanted to
watch the video) or it could be the CDN (they sent the actual traffic) or it
could be the owner of the content (they presumably gain in some manner from
people watching that content). In case 2) the user is clearly responsible,
but it might be argued that facebook also gains, because it depends on
having users doing this sort of thing as its business model... In 3) it
could be the user, it could be google (they get paid for the click), it
could be the final site from which the data came, etc. But in all cases
there is an argument that the upstream forwarding nodes are also to some
extent responsible - they aren't in any way responsible for the content, nor
are they responsible for it arriving at their inbound interface, but they
are responsible for any routing decisions they make which might have a bad
impact on downstream networks. However I accept Kevin's core point that we
should avoid apportioning blame in our descriptions...

Incidentally, it would be interesting to know to what extent routing
decisions by one network have a knock-on effect on the traffic and
congestion in downstream networks... This seems to be a key bit of
information that ConEx can provide that is currently missing.

Oh, and BTW, I personally don't believe ConEx is going to be able to be used
for traffic engineering on fast timescales - the congestion simply varies
too quickly over packet timescales (or it does with window-based
controllers, does anyone know if this is also true for controllers such as
Cubic?)

> 
> 
> To my mind the power of Conex is to provide a forwarding device a
> richer set of information to make better decisions on how to manage
> their queues for the greater good.

YES! Or perhaps less evangelically, ConEx provides information so they know
what impact their decisions are having downstream (after all there may be
operators that want to hinder rather than help).

> > changes that would have a similar network impact.

To say nothing of fast re-routes, etc

---

From: Christopher Morrow <[email-masked]>
Date: Wed, 18 Aug 2010 09:53:34 -0400

> instance with shared backbones such as that used by the UK academic

on the discussion of settlements ... the only place I think that makes
sense is in determining if/when a relationship should change, from
'customer' to 'peer' or vice-versa.

In the case of purely 'customer' relationships (which won't change,
for instance 'dsl customer') today most folks are just charged for the
connection. if you propose to convert them to a form of billing based
on congestion you'll have to find story that doesn't end up just
confusing the customer I think. confused customers ==
call-center-questions :(

I also bet that the congestion here is going to be mostly on the last
mile link, customers may be interested in changing their behavior to
better utilize their bw. It's not clear that there's a cost benefit
for the customer since no amount of money is going to change their
last mile problems.

> should avoid apportioning blame in our descriptions...

yes

> information that ConEx can provide that is currently missing.

this is readily apparent today... there are folks that use these
'tricks' (decisions) in order to affect outcomes of peering agreements
actually. it's really not that hard to figure out. I think rich's
earlier example (10 links with one carrying traffic only) is an
example of a mis-configuration really, not an intentional
configuration.

> controllers, does anyone know if this is also true for controllers such as

I agree with this (no TE from conex).

>> their queues for the greater good.

perhaps, though we do this today with QOS markings, or by
classification of traffic based on 5-tuple info that is in turn
converted to QOS markings and actions. Adding another few bits to
watch for classification marking seems fine.

-Chris

---

From: John Leslie <[email-masked]>
Date: Wed, 18 Aug 2010 10:08:33 -0400

Kevin Mason <[email-masked]> wrote:
> congestion on another ISP's network.

   I expect "causes congestion" to disappear from the next version.

> Data is only passed between sender and receiver so it is the ISP's
> customer that is requesting the data from customers of the other
> provider, not the provider "sending it".

   This model doesn't actually work unless the requesting customer has
a way to know how much data s/he is "requesting", which is not the case
for the typical "click".

   Senders, OTOH, always know how much they're sending.

   But, in practice, neither end knows how fast it can be sent without
congestion being experienced -- until the sender actually tries sending
it. We can imagine a world in which the receiver offers advice on how
fast to send -- and perhaps that "should" be built into TCP, but it's
not in-charter for ConEx to standardize that.

   And, in practice, the receiver doesn't know how fast the sender can
send without experiencing congestion either: for the last-mile may share
fabric with other users, or there may be sufficient unsolicited traffic,
or some anomaly along the path may arise, or...

> party is culpable and therefore who should endure any commercial

   You are imagining a government-mandated process. While ConEx cannot
avoid such an unfortunate outcome, I hope we don't refuse to standardize
a way to pass information, just because it could be misused. :^(

---

From: John Leslie <[email-masked]>
Date: Wed, 18 Aug 2010 10:47:30 -0400

Kevin Mason <[email-masked]> wrote:
> enforcement and by necessity is fully aware of what flows below to
> what UNI's (e.g. because they are all on a single authenticated VLAN or
> PPP tunnel).

   That is one model (and probably deserve consideration as such), but
not the only model.

> does not maintain an specific awareness of individual end points.

   Agreed. (And I don't believe anyone here is proposing that this change.)

> Routers could  theoretically maintain information about each source/
>destination pair AND consult some database to relate that to a end user
> profile, but this is not very scaleable. 

   Exactly!

> alone. Maintaining knowledge of which "users" have "caused" the mosti

   Agreed.

   Note, however, that collecting information which "could" be used for
settlements is easy enough.

> size packets begin to be congestion marked if ECN capable

   There is a technical problem there: ECN marking currenly means that
the receiver should treat it exactly the same as a drop for purposes
of congestion-control. Thus, ECN marking might be thoroughly unpopular
if applied when packet drop is not "imminent"...

> and maybe discarded if not,

   If you're talking Random Early Discard, we're OK here, but from what
follows, I'm not sure...

> but at a slightly higher queue depth, packets that are ECN capable but
> with a high positive congestion value get discarded, on the basis that
> these have a lower chance of reaching their destination than packets
> not declaring an expectation of congestion on the rest of the path.

   This confuses me a bit...

   What I envision is that ConEx packets will contain both an indication
of congestion "predicted" (set by the sender) and congestion "experienced"
(set by ECN or some similar mechanism), and that the difference measures
how likely the packet is to reach its destination without the difference
going negative (meaning that the congestion prediction was low).

   To me, packets with a negative difference (which I call "predicted
congestion exhausted") deserve a lower priority in the queue than non-
ConEx packets, while packets with a sufficiently positive difference
deserve a higher priority in the queue.

   (Implementation of this in a core router is unlikely -- but that does
not change the question of "deserving". Implementation could be done in
a separate box placed between routers at points where congestion is
common, and the queue could be maintained in that box.)

> I don't see a "border monitor" as described in the draft being very
> practical or useful in this part of the network.

   A Border Monitor can provide information useful for settlements or
planning upgrades. The concept I described above could be in a "Border
Policer", but that's probably not the right name for it...

> policy could be applied. One such policy might be to discard all
> packets with a congestion deficit of more than x.

   Exactly!

> This is the safety net against dishonesty by the sender. An additional
> policy might be to discard packets that have experienced congestion
> above a threshiold (which may be different for different user profiles)
> so far AND that are destined to a user that has a recent history of
> high congestion marked packets.

   This is a way to implement a "receiver pays" paradigm; and I don't
believe ConEx should forbid such a practice.

   And, indeed, while I would wish for "sender pays" to be the norm, I
wouldn't consider such a practice "disruptive"... If the sender pays
but experiences no benefit, that's the sender's problem: practical
implementations _will_ have to deal with "over-paying" for the benefit
received.

> traversed (on the same provider or any preceding providers network).

   I'm not sure I follow this...

> functions of the edge device itself, not any independent function.

   I don't believe the authors consider that behavior appropriate for
an "Edge Monitor"; but it is probably appropriate for an "Edge Policer".
(I doubt we're yet anywhere near WG consensus on what functions an
"Edge Policer" should do to fit within a ConEx standard.)

> Other may have different views of how the revealed congestion
> information might be used but I believer it is useful to at least
> consider the two parts of an ISP network when discussing possible
> used for the information.

   Agreed. We should have a way of talking about what "Edge Policers"
(or any other name we may give them) may do.

---

From: Kevin Mason <[email-masked]>
Date: Fri, 20 Aug 2010 18:03:27 +1200

Cheers
Kevin Mason
> From: John Leslie [mailto:[email-masked]]
> Sent: Thursday, 19 August 2010 2:09 a.m.
> To: Kevin Mason
> Cc: [email-masked]

[Kevin Mason] Wherever charging occurs between providers industry watchdogs will take an interest, it is inevitable, but this is a distraction.

If financial penalties are to be imposed then the party that the penalty is imposed upon has to be able to practically avoid the consequences of the penalty (charges) if they chose to. I do not see any dialogue in the draft to indicate how a "sending provider" might practically achieve that.

The issue for me is that the current draft leaves me with a sense that charging at the provider boundary is perceived to be a significant benefit for use of conex. I believe this implication is unhelpful.

I would encourage this to be moderated to just talk about collection of information for a variety of uses e.g. network management, network planning and a possible basis for underpinning bilateral commercial arrangements.

---

From: Kevin Mason <[email-masked]>
Date: Mon, 23 Aug 2010 11:46:15 +1200

> I agree with this (no TE from conex).

My expectation is the traffic engineering is more background activity, monthly/quarterly action not automated
> watch for classification marking seems fine.

QoS marking cannot tell the router the likelihood of any packet getting to its ultimate destination. So for packets with the same classification (marking) if the capacity for that "class" is approaching overload, favouring flows that are not expecting onward congestion has the potential to improve outcomes for all. If this is not a practical use of Conex then I am a loss as to what else it would be useful for.

Cheers
Kevin Mason
> From: [email-masked] [mailto:[email-masked]]
> On Behalf Of Christopher Morrow
> Sent: Thursday, 19 August 2010 1:54 a.m.
> To: Toby Moncaster
> Cc: Kevin Mason; Woundy, Richard; [email-masked]
> >> what
> >> packets

---

From: Christopher Morrow <[email-masked]>
Date: Sun, 22 Aug 2010 23:54:28 -0400

> If financial penalties are to be imposed then the party that the penalty is imposed upon has to be able to practically avoid the consequences of the penalty (charges) if they chose to. I do not see…

keeping in mind also that the 'sending provider' could be more than 1
AS away (like the average as-path length today on the Internet is
3.something).

> The issue for me is that the current draft leaves me with a sense that charging at the provider boundary is perceived to be a significant benefit for use of conex. I believe this implication is unh…

I agree with this, the idea that conex was/is/could-be used as a
settlement regime turns the discussion into something that isn't
helpful. conex, it seems to me, could really just be a method to help
folk better use the bw they have. by 'better use' I really mean 'more
efficiently utilize'.

-Chris

---

From: Christopher Morrow <[email-masked]>
Date: Sun, 22 Aug 2010 23:58:45 -0400

> My expectation is the traffic engineering is more background activity, monthly/quarterly action not automated

uhm... openflow has the promise to make this sort of calculation
happen very much more rapidly than 1/month || 1/quarter. In today's
large internet networks I believe much of this is done daily and often
much more often than that. (without openflow)

> QoS marking cannot tell the router the likelihood of any packet getting to its ultimate destination. So for packets with the same classification (marking) if the capacity for that "class" is approa…

right, qos I was using as method to trigger a different forwarding
decision (forwarding and/or marking really). Adding conex info to what
makes the this happen (as I said) seems fine.

"Traffic to the destination port 77 + protocol 85 + source-address 12
has been marked by the conex mechanism, so potentially will get
dropped downstream, if you have the same profile traffic headed to
port 78 maybe prioritize that higher."

(in short I think we are agreeing here)

-chris

---

From: John Leslie <[email-masked]>
Date: Mon, 23 Aug 2010 00:55:29 -0400

>>>>> their queues for the greater good.

   This is on the vague side...

>> QoS marking cannot tell the router the likelihood of any packet
>> getting to its ultimate destination.

   Agreed.

>> So for packets with the same classification (marking) if the capacity
>> for that "class" is approaching overload, favouring flows that are
>> not expecting onward congestion has the potential to improve outcomes
>> for all.

   I don't follow the logic here...

   If you mean that a "provisioned bit rate" flow that is "predicting
congestion" is in danger of failing, I can agree. But you aren't limiting
it to that, AFAICT.

   At first blush, I can't see why favoring a different flow _could_
improve outcomes for a flow "predicting congestion". And that doesn't
depend on whether we're in a "sender pays" paradigm, a "receiver pays"
paradigm, or a "nobody pays" paradigm.

   In a "provisioned bit rate" case, terminating _any_ flow when the
aggregate bit-rate exceeds capacity _will_ necessarily improve outcomes
for the remaining flows. But I don't follow why a flow predicting
congestion is a "better" candidate for termination.

   Unless we know we're at the final bottleneck (shared by all flows in
that "class", "predicted congestion" in one flow doesn't tell us what
actual congestion other flows may experience. And if we are at the final
bottleneck, expediting the flow that _does_ predict congestion would
seem to improve the speed with which it can respond to the actual
congestion (by receiving a "congestion experienced" marking rather than
waiting for a timeout).

   Possibly, of course, I'm guessing wrong what you mean by "favouring
flows"...

>> If this is not a practical use of Conex then I am a loss as to what
>> else it would be useful for.

   ConEx clearly can be useful for distinguishing "less than best effort"
flows from "interactive" flows, and using that information to target
efforts at bandwidth upgrades. LBE flows don't call for bandwidth
upgrades; interactive flows do.

> makes the this happen (as I said) seems fine.

   Again, on the vague side...

   If you have a queue at a forwarding point, you _could_ adjust
priorities to forward some traffic more quickly or more slowly; or
you could set probability-of-drop higer or lower in a RED algorithm.
I'm not sure what you have in mind here...

> port 78 maybe prioritize that higher."

   Obviously I can't prevent folks from applying such an algorithm,
but it seems ill-advised to me.

   We really don't know what some downstream forwarder will do in the
event of congestion; and I don't expect to try to standardize that in
a ConEx protocol. But were I designing the forwarding algorithms, I'd
try to deliver ConEx traffic which predicts more congestion than I
know to exist _more_quickly_ than non-ConEx traffic, in hopes that
any "congestion experienced" markings will cause backoff more quickly.

   Recall that the ConEx mechanism _will_ (somehow) contain both
"congestion predicted" and "congestion experienced so far".

   Also recall that a newly-starting interactive flow has good reason
to estimate high on "predicted congestion" until path-congestion
information returns to the sender.

   Thus, dropping traffic which estimates high, simply _because_ it
is ConEx aware, seems unlikely to help.

   (YMMV, no doubt...)

---

From: "Toby Moncaster" <[email-masked]>
Date: Mon, 23 Aug 2010 14:52:59 +0100

> From: [email-masked]
> [mailto:[email-masked]] On Behalf Of Christopher Morrow
> Sent: 23 August 2010 04:59
> To: Kevin Mason
> Cc: Toby Moncaster; Woundy, Richard; [email-masked]
> >> > Oh, and BTW, I personally don't believe ConEx is going to be able
> to be
> >> used
> >> > controllers, does anyone know if this is also true for controllers
> such
> >> as
> >> > Cubic?)
> >>
> > My expectation is the traffic engineering is more background
> activity, monthly/quarterly action not automated
> 

Any chance of a quick 101 on OpenFlow? Routing is not my strongest suite!
Can it do these calculations across AS boundaries? Can it see how decisions
made locally impact networks downstream? That is the extra information that
ConEx provides...

Certainly re-configuration of BW shares can (and does) happen on timescales
of hours. I suspect though the monthly/quarterly reference above is more
about provisioning physical infrastructure (not the overlying logical
network structure).

> getting to its ultimate destination. So for packets with the same
> classification (marking) if the capacity for that "class" is
> approaching overload, favouring flows that are not expecting onward
> congestion has the potential to improve outcomes for all. If this is
> not a practical use of Conex then I am a loss as to what else it would
> be useful for.
> 

We are certainly heading towards agreement I think.

At this stage in the life of ConEx, what matters is anything that actually
acts as a technical block to its deployment. The chance are that the way
operators, app writers and vendors end up using ConEx will differ from how
we imagine things at this stage - ConEx is an enabler for many things and
none of us have the ability to see the future... Consequently all the use
cases document can do is capture some of the more likely scenarios where
ConEx can make an impact.

Toby

> >> >> > congestion on another ISP's network. Data is only passed
> between
> >> >> sender
> >> >> > Consider a hypothetical example (an *extreme* case to be sure,
> but
> >> >> not
> >> >> > forwarding all traffic over a single interconnect to the
> downstream
> >> >> ISP
> >> >> > center to the downstream ISP). That change in policy is very
> likely
> >> >> to
> >> >> > cause a lot of congestion over the single interconnect link,
> even as
> >> >> the
> >> > I obviously bow to the greater knowledge of Kevin and Rich in
> this, but
> >> it
> >> > OK, this is in danger of becoming highly philosophical! Let's take
> 3
> >> common
> >> > scenarios:
> >> >
> >> > watch the video) or it could be the CDN (they sent the actual
> traffic)
> >> or it
> >> > could be the owner of the content (they presumably gain in some
> manner
> >> from
> >> > extent responsible - they aren't in any way responsible for the
> content,
> >> nor
> >> > are they responsible for it arriving at their inbound interface,
> but
> >> they
> >> > are responsible for any routing decisions they make which might
> have a
> >> bad
> >> > impact on downstream networks. However I accept Kevin's core point
> that
> >> we
> >> > Oh, and BTW, I personally don't believe ConEx is going to be able
> to be
> >> used
> >> > controllers, does anyone know if this is also true for controllers
> such
> >> as
> >> > Cubic?)
> >>
> >> > YES! Or perhaps less evangelically, ConEx provides information so
> they
> >> know
> >> > what impact their decisions are having downstream (after all there
> may
> >> be
> >> >> > Subject: [conex] comments on draft-moncaster-conex-concepts-
> uses-
> >> >> 01.txt
> >> >> >
> >> >> > A  comment on the draft.
> >> >> >
> >> >> > congestion on another ISP's network. Data is only passed
> between
> >> >> sender
> >> >> > So if the "sending" provider is not causing the congestion
> (because
> >> >> it
> >> >> > is the receiving provider's customer the requested it) then
> arguing
> >> >> the
> >> >> > "sending" provider should pay for any congestion the resulted
> might
> >> >> be
> >> >> > difficult. I can see endless legal arguments as to why one or
> the
> >> >> other
> >> >> > In ISP networks there are, very simply, two parts. Firstly
> there is
> >> >> the
> >> >> > UNI's (e.g. because they are all on a single authenticated VLAN
> or
> >> >> PPP
> >> >> > tunnel).
> >> >> >
> >> >> > Beyond that point between the IP edge and any peering point,
> the
> >> >> network
> >> >> > discarded if not, but at a slightly higher queue depth, packets
> that
> >> >> are
> >> >> > ECN capable but with a high positive congestion value get
> discarded,
> >> >> on
> >> >> > than packets not declaring an expectation of congestion on the
> rest
> >> >> of
> >> >> > the path. I don't see a "border monitor" as described in the
> draft
> >> >> being
> >> >> > When packets arrive at the BRAS destined for the user, then
> user
> >> >> based
> >> >> > with a congestion deficit of more than x. This is the safety
> net
> >> >> against
> >> >> > packets that have experienced congestion above a threshiold
> (which
> >> >> may
> >> >> > this policy would not be invoked, so it would only apply to
> users
> >> >> that
> >> >> > might be used but I believer it is useful to at least consider
> the
> >> >> two

---

From: Kevin Mason <[email-masked]>
Date: Wed, 25 Aug 2010 11:29:35 +1200

Cheers
Kevin Mason
> Sent: Monday, 23 August 2010 4:55 p.m.
> To: Christopher Morrow
> Cc: Kevin Mason; [email-masked]
[Kevin Mason] 
Let me try and explain again

A router in the core backbone is not practically aware of which packets belong to which ultimate end users, what the commercial plan the destination user is (i.e. what they are fairly entitled to) and what the recent activity history of that user has been.

If a core router is making a forwarding decision and the queue for that class is filling, then it makes some queue management decisions, e.g. RED. RED in essence is sending a message to the user host that capacity is approaching overload somewhere on the path and congestion avoidance action should be taken.

With ECN capable endpoints the same overload onset indication can be sent by marking, with hopefully the same or similar end user host response.

Now with CONEX an additional dimension might be added to the queue management logic. The challenge is what could be done at this point to use the conex information to improve overall capacity sharing.

My assumption is that as the queue fills the first RED queue threshold might be to;
- begin random discarding non ECN traffic
- begin random marking ECN traffic not CONEX enabled, 

The mark probability for CONEX traffic might be a function of the predicted congestion, so traffic that is predicting high onward congestion would have a higher mark probability than traffic with a lower forward prediction

The logic for CONEX behaviour is that if a packet is predicting further congestion then invoking a congestion response early may help relieve capacity overload at more than one point in the path. 

When a packet eventually arrives at a user endpoint aware policy point (e.g. a BRAS), packets that have a congestion prediction deficit might be subject to random discard if they are destined for a user that has exceeded a user policy based volume and/or "congestion experienced" threshold. This might discourage persistent under prediction and also try and invoke a rate reduction response for those longer duration flows traversing paths on which congestion is persistent and/or rising.

For user endpoints with no recent history (i.e. are below a "congestion experienced" or volume threshold), no action may be taken. An alternative might be to "unmark" the ECN indication so no congestion response would be invoked and this flow could continue to enjoy wire speed performance. Unmarking may also help flows in start-up initially predicting high forward congestion (that may have been marked in a core router queue) as they would not be penalised (congestion response would not be invoked at the host) if they were from an itinerant user.

> congestion" is in danger of failing, I can agree. But you aren't limiting

No I am not considering guaranteed bit rate flows here.
> upgrades; interactive flows do.

I don't see how this can be done by CONEX, traffic class (priority) is not conveyed by CONEX. 
> know to exist _more_quickly_ 

How does a router know what congestion actually exists beyond its egress interface. CONEX predictions across multiple packets might gives some clues, but the router does not inherently know what distant path any packet will traverse so a lower prediction on one packet does not mean that the current packet being forwarded with a higher prediction value will actually traverse exactly the same path to its destination (and can be assumed to be actually over predicting).

than non-ConEx traffic, in hopes that