Subject: Re: [conex] Discussion of new Use Cases pt 1

From: "Toby Moncaster" <[email-masked]>
Date: Tue, 1 Mar 2011 10:08:15 -0000

3 comments inline

> -----Original Message-----
> From: Mcdysan, David E [mailto:[email-masked]]
> Sent: 01 March 2011 02:07
> To: John Leslie; Toby Moncaster
> Cc: 'Alissa Cooper'; [email-masked]
> Subject: Re: [conex] Discussion of new Use Cases pt 1
> 
> Hi John,
> 
> Some responses in line below.
> 
> Dave
> 
> On 2/26/11 9:56 AM, "John Leslie" <[email-masked]> wrote:
> 
> >Toby Moncaster <[email-masked]> wrote:
> >> To: "'Alissa Cooper'" <[email-masked]>,
> >>
> >>> It seems to me that there are two different approaches being taken
> to
> >>> what is or is not in scope for both the use cases and the mechanism
> >>> definition:
> >>>
> >>> Approach #1:
> >>> a. Assume that the CONEX mechanism will be used exclusively to
> signal
> >>> the amount of congestion encountered by previous packets on the
> same
> >>> flow
> >>> b. Document the different uses to which that signal could be put
> >>> (traffic management, incentivizing scavenger services, etc.) in the
> >>> use cases draft
> >>> c. Document how the signal is set in the mechanism draft
> >>
> >> This is certainly the approach in draft-ietf-conex-concepts-uses and
> the
> >> co-authors of that document assumed this was what the charter was
> >>looking
> >> for.
> >
> >   Agreed.
> >
> >>> Approach #2:
> >>> a. Assume that the metric that CONEX will signal is open for
> >>> discussion, or that the mechanism may signal multiple different
> metrics
> >>> b. Document use cases for a variety of different potential metrics
> in
> >>> the use cases draft (congestion encountered previously on same
> flow,
> >>> long-term user burstiness as you describe above, possibly others)
> >>> c. Document how network nodes could respond to CONEX signals to
> >>> effectuate particular outcomes (dropping more traffic from non-
> bursty
> >>> users as you describe above)
> >>
> >> The potential problem with this approach is that there is no end to
> the
> >> possible metrics that we could explore. In theory we are meant to be
> >> submitting the Use Cases document by March this year! So we do need
> to
> >> be sensible about what is in and out of scope.
> >
> >   While I believe there's still some room for discussion what we mean
> >by "congestion", I don't believe there's any room (left) for
> discussion
> >that we mean to record anything other than congestion expected during
> >transit of the packet in question (with some wiggle room for quantized
> >encoding a-la ECN). There just aren't enough bits.
> >
> >   ConEx-aware boxes along the way are _very_ likely to aggregate our
> >ConEx markings, over time and/or over flows. A use-cases document can
> >discuss such things, whereas a mechanism document is probably more
> >limited.
> 
> Good point. I think this is the general direction that Marcello may
> have
> had in mind when he commented during my presentation in Beijing. Not
> sure
> what Marcello, Rich or Bob had in mind here as discussed in the meeting
> minutes.  I suggest that we need to work toward some proposed text for
> the
> wg use case draft that describes this.

All offers of text gratefully received ;)

> 
> 
> >
> >>> It seems as though you are taking approach #2 whereas others may be
> >>> taking approach #1, and these two approaches probably need to be
> >>> reconciled. I don't see how the same mechanism can signal both
> >>> congestion previously experienced on the path and peak/average
> >>> sending rate (although I'm happy for you or someone else to explain
> >>> how it can be done). If it can't be done, then it seems like we are
> >>> talking about signaling two different metrics.
> >
> >   IMHO, discussion of a use-cases document has some leeway here --
> >not in what the metric will signal, but in what may be done based on
> >aggregations of that metric. I do not believe Dave McDysan has
> exceeded
> >that leeway -- though discussing use-cases in the absence of an agreed
> >metric _does_ tend to lead to confusion...
> >
> >> I certainly favour approach 1, and this seems to better match with
> our
> >> charter. On the other hand there is potential for looking at broader
> >> definitions of congestion within the use cases document. One of the
> >> big issues facing the WG is defining congestion in a sensible
> fashion.
> >
> >   Clearly, we do have to produce a use-cases document, though I would
> >prefer that at least an "abstract mechanism" could be agreed to as a
> >basis for it.
> 
> Not sure I understand what you are proposing. The wg decided to take
> mechanism largely out of the use case draft in IETF 78, and in IETF 79
> the
> exclusion of a use case that did not match the adopted abstract
> mechanism
> was explicitly agreed (see meeting minutes snippet in my response to
> Allison).
> 
> 
> >   I tend to take a long-term view which says whatever we settle on
> for
> >a mechanism should allow for future extensions. (Thus I am very much
> >at home with desires for different ways to measure "congestion".)
> >
> >>> If we are focused on a single metric, then I think we need to
> document
> >>> use cases for that metric. If there are multiple metrics, we should
> be
> >>> clear about which use cases make use of which metric. And in either
> >>> case, I thought the point of the mechanism draft was to explain how
> >>> the chosen metric is measured so that it can be encoded at the IP
> >>> layer.
> >
> >   I am hesitant to go the path of labeling use-cases according to
> what
> >metric best serves them. IMHO, our use-cases document should only
> include
> >things achievable with a particular metric, which I have been assuming
> >to be congestion-expected-during-transit.
> 
> Metric is mentioned once in the use case document in section 3 in the
> context of existing approaches. Are you referring to the measures in
> definitions of use cases, or something in the abstract mechanism draft?
> (where the term metric is not used at all)

I (more or less deliberately) avoided using the word "metric" in the use
cases document. The main reason being that it may confuse people that are
not used to thinking of congestion as something with a measurable value
(after all TCP assumes congestion exists or it doesn't).

In this discussion, metric is being used as short-hand for "the means by
which we establish how much congestion a given packet/flow/user expects to
create"

> 
> >
> >   I do spend time thinking about "concrete" mechanisms, but I'm not
> sure
> >it would help to discuss them now. Certainly such discussion doesn't
> >belong in this thread.
> >
> >> I favour concentrating on a single metric relating as closely as
> >>possible
> >> to "real" congestion. But Dave does have a very valid point - for
> >> whatever reason, many ISPs use a shaper to rate limit traffic
> to/from
> >> their customers. I would argue that at peak times this is probably
> not
> >> the limiting factor, but off-peak it will be the biggest source of
> >> apparent loss/congestion.
> >
> >   Certainly, ConEx will need to operate in an environment where such
> >"shaping" occurs. I am considerably less sure how (if at all) it
> should
> >be discussed in a use-cases document.
> 
> 
> I asked this question in IETF 78 of Bob regarding a non-work conserving
> scheduler (e.g., shaper), which can experiences the congestion measure
> as
> defined in the current use case draft.
> 
> At a minimum I think the statement in the first paragraph of section 4
> that describes what I called "inter-user congestion" as "----
> congestion
> means that your traffic impacts other users, and conversely that their
> traffic impacts you." needs to exclude the use case of what I called
> "self-congestion" needs to address the use case employed by many ISPs
> (as
> Toby points out).

There is an interesting related discussion here - where a user is policed
because of the level of congestion they have caused, do they then have to
"declare" the policing losses as expecting a higher level of congestion?

Going back to your point, you really need to address this with Bob. I think
that there is a difference between what might be deemed "self-congestion"
which has no impact on other users, and "cross congestion" where your
traffic is actually impacting on others, and hence creates a cost for them.
It is this latter form of congestion that allows people like Frank Kelly to
apply micro-economics to this problem

Toby

> 
> >To me, it is "congestion" even if
> >it is not resource-congestion. Congestion-expected markings could
> better
> >inform decisions about "shaping". The issues about whether ConEx
> marking
> >is considered at any point of congestion are much the same...
> 
> I didn't follow this. Could you please elaborate.
> 
> >
> >> One interesting thought experiment - what if, instead of a shaper,
> the
> >> ISP introduces a congestion marking system. Then it can rate limit
> >> traffic by marking it before the traffic needs to be disrupted. The
> >> only problem then is how to deal with non-responsive traffic.
> >
> >   That undoubtedly deserves its own thread!
> >
> >--
> >John Leslie <[email-masked]>

---

From: John Leslie <[email-masked]>
Date: Tue, 1 Mar 2011 11:59:03 -0500

Toby Moncaster <[email-masked]> wrote:
> [<[email-masked]> wrote:]
>> [John Leslie <[email-masked]> wrote:]
>> 
>> >
>>> what metric best serves them. IMHO, our use-cases document should
>>> only include things achievable with a particular metric, which I
>>> have been assuming to be congestion-expected-during-transit.

   (My apologies for hand-waving here: I'll get back to what I meant.)

>> definitions of use cases, or something in the abstract mechanism
>> draft? (where the term metric is not used at all)
> 
> I (more or less deliberately) avoided using the word "metric" in the
> use cases document. The main reason being that it may confuse people
> that are not used to thinking of congestion as something with a
> measurable value (after all TCP assumes congestion exists or it
> doesn't).
> 
> In this discussion, metric is being used as short-hand for "the means
> by which we establish how much congestion a given packet/flow/user
> expects to create"

   (Toby is hand-waving here... ;^)

   I realize we're not conveying the right idea when we use the word
"metric". First, let me explain what I have meant (and I was assuming
the other authors meant).

   The dictionary-definition I try to fit under is:
" 
" 2. a standard of measurement...

by which I mean "a set of rules which allows different observers to
generally report the same number when they set out to measure the
same observable".

   (Whether that "number" is binary, integer, or real should be derivable
from the rules.)

   I have been using the word "metric" because I don't believe we have
any consensus what "rules" apply. Some of us are very strict adherents
to the rules of ECN; others think ECN is all wet; et cetera...

   The hope is that we can (later) agree on one or more actual "metrics"
(sets of rules) for a packet being forwarded from source to destination;
and that we can concentrate our effort (for now) on how to carry the
result of that measurement through the ConEx process: back to the sender
and through the forwarding process for a future packet.

   (Getting back to my earlier hand-waving: the unstated assumption is
that the receiver of a packet will decode a number according to a rule
for "congestion experienced by a packed during transit", report that
number back to the sender, and the sender will encode a corresponding
number into future packet(s) it sends, intending it to convey its
prediction of "congestion-experienced-during-transit".)

   I've taken some time before replying, trying to come up with a term
to substitute for "metric" which will still allow us to gloss over the
issue of how we measure "congestion" -- and (for now, at least) given
up. I'd be delighted to hear suggestions from other participants (or
even lurkers)...

---

From: "Toby Moncaster" <[email-masked]>
Date: Tue, 1 Mar 2011 17:17:31 -0000

(NB I haven't yet properly parsed John's reply).

Surely what it boils down to is we need to define the "congestion" that will
be exposed by "congestion exposure"!

And yes, lurkers are very welcome to join in this discussion - it is
important we reach some sort of consensus here (not on the use of the word
"metric", but on what the word is applied to).

Oh, and a (not serious) suggested alternative word: "quantifier"...

Toby

> From: John Leslie [mailto:[email-masked]]
> Sent: 01 March 2011 16:59
> To: Toby Moncaster
> Cc: 'Mcdysan, David E'; 'Alissa Cooper'; [email-masked]
> Subject: "Metric"
>

---

From: Steve Bauer <[email-masked]>
Date: Tue, 1 Mar 2011 12:29:27 -0500

> "metric", but on what the word is applied to).

Awhile ago we wrote the following paper which may be relevant to
inject at this point:

The Evolution of Internet Congestion
http://mitas.csail.mit.edu/papers/Bauer_Clark_Lehr_2009.pdf

In particular see section 3, "Defining Congestion".

Steve Bauer
MIT

---

From: "Mcdysan, David E" <[email-masked]>
Date: Wed, 2 Mar 2011 10:03:41 -0500

I think the term "measure" used in the use case draft is sufficient.

>From http://en.wikipedia.org/wiki/Measure_(mathematics)

"... a measure on a set is a systematic way to assign to each suitable
subset a number ..."

The use case document defines congestion as "Congestion is a measure of
the probability that a packet
will be marked or dropped as it traverses a queue." The abstract mechanism
draft specifically defines these as LOSS and ECN "signals" passed
(implicitly or explicitly) from the congestion point to the receiver. In
measure theory terms the implicit/explicit "signals" associated with these
packets that experience congestion are a systematically defined subset of
the set of all packets delivered to the receiver.

The abstract mechanism then defines (implicit/explicit) loss/marking
feedback signals to the sender. (In a subset of all reverse direction
packets of the same flow (e.g., TCP ACKs). Finally feed-forward
(re-inserted) information developed in a systematic way from the fed-back
congestion (loss, ECN marking) subset of reverse direction packets is
inserted into a subset of the packets from sender to receiver.

I think that all of these subsets can be assigned to a number.

Similar arguments can be made for Con-ex capable and credit bits from the
abstract mechanism draft.

Therefore, I believe the mathematical meaning of the term "measure" is
sufficient.

This is what I had in mind when I proposed a burstiness "measure." As Toby
mentioned it may be possible to map the conex abstract mechanism to the
burstiness measure I defined in my Beijing draft and an earlier post.

If anyone has another basis for terminology, then please propose it.
Otherwise, I would like to proceed on attempting to doing what Toby
proposed using the concept and terminology of "measure."

Thanks,

Dave

---

From: John Leslie <[email-masked]>
Date: Thu, 3 Mar 2011 15:34:56 -0500

Mcdysan, David E <[email-masked]> wrote:
> Not sure I understand what you are proposing.

   I was stating a preference (not expecting it to be satisfied).

> The wg decided to take mechanism largely out of the use case draft in
> IETF 78, and in IETF 79 the exclusion of a use case that did not match
> the adopted abstract mechanism was explicitly agreed (see meeting
> minutes snippet in my response to Allison).

   My preference was that we might have had sufficient agreement on an
abstract mechanism so that we didn't have to dance-around that issue
(which IMHO tends to confuse readers).

>> I am hesitant to go the path of labeling use-cases according to what
>> metric best serves them. IMHO, our use-cases document should only include

   I have explained elsewhere what I mean by "metric". I may well continue
to use that term (to mean what I said I meant) in preference to "measure"
until we come up with an agreed meaning for "measure". (Alas, the
mathematical usage of "measure" is IMHO a poor fit to what I hope we
mean to talk about.

   (I do not wish to argue semantics -- I merely reserve the right to
speak exactly instead of vaguely, where what I say will mean different
things to different readers.)

>> "shaping" occurs. I am considerably less sure how (if at all) it should
> scheduler (e.g., shaper), which can experiences the congestion measure as

   Now _I'm_ not sure I understand what you're proposing...

   "Shaper" is a vague term -- and AFAICT must remain vague because its
function(s) will be determined by many parties so as to satisfy their
(many different) business models.

   Discussion elsewhere throws light on what some of us are thinking:
that whether or not we accept "shaping" as "congestion", it is not
"contention" -- it doesn't reflect an attempt to take resources away
from other flows.

   IMHO, if we go the ECN route, we have to accept ECN marks as indicating
"congestion": if a "shaper" ECN-marks, that's "congestion" we must expose.
OTOH, if the "shaper" drops, it's no longer a "must"; and if a "shaper"
merely delays, we don't necessarily have to do anything.

   Speaking for myself, I don't believe we can cover all the cases of
"shaper" actions. At some point, we have to "treat it as damage" (and
assume routing-around-it is Somebody Else's Problem). ;^)

> that describes what I called "inter-user congestion" as "---- congestion
> "self-congestion" needs to address the use case employed by many ISPs (as
> Toby points out).

   I agree -- that's not a sufficient definition of "congestion".

>> To me, it is "congestion" even if it is not resource-congestion.
>> Congestion-expected markings could better inform decisions about
>> "shaping". The issues about whether ConEx marking is considered at
>> any point of congestion are much the same...
> 

   So far, I don't see a practical way to distinguish ECN-marking due to
queue-filling from ECN-marking due to shaping. I have little desire to
try; and I suggest that deserves to be an area-for-future-work. I
consider both to be "congestion" that we are charged to "expose".

---

From: John Leslie <[email-masked]>
Date: Thu, 3 Mar 2011 15:42:52 -0500

> In particular see section 3, "Defining Congestion".

   I have read through this carefully.

   Section 3 feels as if you were reading my mind (or I yours)! I most
thoroughly agree that people of differing backgrounds naturally attach
very different meanings to "congestion".

   We've gotten as far as we have in "congestion control" because we've
proposed things that do a decent job of controlling whichever of those
meanings we attach.

   I seriously would like to reach agreement on what we mean (rather
than wordsmith until the poor reader can no longer tell which).

   This is hard!

   I do recommend this paper to anyone willing to put forth the effort.

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 28 Mar 2011 12:43:38 +0100

John,

Ah - I realised I didn't click send on this, some time back...
[I hope people can reload the state from the quoted thread - point to 
is relevant to discussion right now.]

1/ ECN is a signal of congestion. Drop is (ambiguously) a signal of 
congestion or the outcome of policy action (or 13 other things). Just 
because drop shares one meaning with ECN, doesn't mean ECN has to 
share one of the meanings of drop.

A policer needs to impair the traffic. ECN isn't impairment. So ECN 
isn't relevant as an outcome of policing.

There's no room for debate here IMO - a policer adding ECN markings 
is just plain stupid. If someone isn't responding sufficiently to ECN 
markings, adding more ECN markings is never going to be used by a 
sentient network operator to punish them.

2/ Can we stop calling the policer a shaper please?
Shaping is theoretically one action a policer might take, but it's 
one that causes more confusion and arguments because it's so weak.

Shaping is something some operators still do to make a packet network 
look like a phone network in the belief it will improve things, when 
it merely adds delay deterministically that might otherwise not be 
added statistically. Like ECN, it's not really a useful tool in the 
punishment armory. A thing that punishes out of contract behaviour is 
called a policer. Let's stop using confusing terms.

3/ Distinguishing self-congestion from shared congestion is a real issue.

It's actually even more subtle. Self-congestion can be further divided into:
* congestion of a resource you're leasing from your provider for 
(e.g. your access line), which you expect the provider to maintain 
and upgrade,
* congestion of something you bought yourself (e.g. your home network).

The network provider might want to include the former 
(sole-congestion of leased equipment) with shared congestion as part 
of the economic signals to indicate what needs upgrading.

Bob

At 21:34 03/03/2011, John Leslie wrote:
>_______________________________________________
>conex mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/conex

________________________________________________________________
Bob Briscoe,                                BT Innovate & Design

---

From: Mikael Abrahamsson <[email-masked]>
Date: Mon, 28 Mar 2011 13:51:02 +0200 (CEST)

On Mon, 28 Mar 2011, Bob Briscoe wrote:

> There's no room for debate here IMO - a policer adding ECN markings is 
> just plain stupid. If someone isn't responding sufficiently to ECN 

I don't agree with this at all.

Let's take the case with a router/switch with small buffers. I don't see 
the huge difference in having a policer doing ECN markings at 95% of 
wirespeed and then letting the small buffer taildrop at 3-5ms of 
buffering, compared to something that has 50ms of buffering and using RED 
starts doing ECN markings/RED after 5ms of buffers fill.

> Shaping is theoretically one action a policer might take, but it's one that 
> causes more confusion and arguments because it's so weak.

A policer NEVER ever shapes. Shaping is the delaying/buffering of packets. 
A policer is something that has buckets and either drops, marks and/or 
transmits packets, never ever delays them.

> Shaping is something some operators still do to make a packet network look 
> like a phone network in the belief it will improve things, when it merely 
> adds delay deterministically that might otherwise not be added statistically. 
> Like ECN, it's not really a useful tool in the punishment armory. A thing 
> that punishes out of contract behaviour is called a policer. Let's stop using 
> confusing terms.

It seems we need a discussion on the basic terms people use to describe 
behaviour.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: "Mcdysan, David E" <[email-masked]>
Date: Mon, 28 Mar 2011 08:32:18 -0400

I agree with Mikael that we need to agree on what is meant by specific
terms. When I used the term "shaper" I use it as defined in RFC 2475 as
follows:

2.3.3.3  Shapers

   Shapers delay some or all of the packets in a traffic stream in order
   to bring the stream into compliance with a traffic profile.  A shaper
   usually has a finite-size buffer, and packets may be discarded if
   there is not sufficient buffer space to hold the delayed packets.

I propose that a reference to this definition be added to the definitions
section in the conex use case document to resolve this terminology issue.
Does anyone in the wg object?

A few responses in line below.

Thanks,

Dave

>starts doing ECN markings/RED after 5ms of buffers fill.

I think that different cases are assumed here.
>transmits packets, never ever delays them.

See above.
>>look 

Shapers, as defined in RFC 2475, also known as "rate limiters" on a queue
as described in Appendix B of DSL Forum TR-059 are used in many ISP
networks.

[DSL Forum TR-059] Technical Report DSL Forum TR-059, "DSL Evolution -
Architecture Requirements for the Support of QoS-Enabled IP Services,"
September 2003, 
http://www.broadband-forum.org/technical/download/TR-059.pdf

A shaper defines a virtual circuit peak bandwidth and burst duration type
behavior for downstream and upstream customer access. This is easy for
customers and connecion-oriented operators to understand and relatively
easy to measure. 

However, it does have issues since operators don't allocate the peak rate
and instead allocated aggregate capacity across users based upon a
multiplier of the average rate. I posed the 20% of users create 80% of the
load problem and conex information may be helpful in addressing part of
this issue. 

Section 6.3 of the current draft was based upon a concept advanced by
Toby, edited by myself and John to describe a possible way to do this. The
feedback from at least 3 people on the list was that this level of detail
was not needed in the wg use case document and that a briefer description
would be appropriate.

I hope to send out some proposed additional text for section 5.1 where a
sentence or two in an additional paragraph would capture this though.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Mon, 28 Mar 2011 15:00:23 +0200 (CEST)

On Mon, 28 Mar 2011, Mcdysan, David E wrote:

> Shapers, as defined in RFC 2475, also known as "rate limiters" on a 
> queue as described in Appendix B of DSL Forum TR-059 are used in many 
> ISP networks.

For me a "rate limiter" is a policer, not a shaper. I'd imagine most of 
the ISP business would give you the exact same answer.

> A shaper defines a virtual circuit peak bandwidth and burst duration 
> type behavior for downstream and upstream customer access. This is easy 
> for customers and connecion-oriented operators to understand and 
> relatively easy to measure.

A shaper emulates the behaviour of a port that is full and is doing output 
queueing, but at any speed and in any direction. It can do WRED/ECN, just 
like a port doing queueing.

I think someone needs to define the terms in the document itself, 
otherwise you'll get confusion as different organisations define these 
terms differently.

Also, I think anyone who wants to present CONEX to a wider audience needs 
to create me concrete examples about what's going on. I've been doing 
routing/QOS engineering for 10+ years, most of the time I don't get what 
people in this group are talking about. It's like you're living in a 
completely different world and using completely different words to 
describe things.

I understand exactly how ECN works, I have read the CONEX documents fairly 
well, but it's still extremely abstract what the discussions are about and 
this is going to work in the real world (ie where the rubber meets the 
road).

---

From: "Toby Moncaster" <[email-masked]>
Date: Mon, 28 Mar 2011 15:34:00 +0100

> From: [email-masked] [mailto:[email-masked]] On Behalf
> Of Mikael Abrahamsson
> Sent: 28 March 2011 14:00
> To: [email-masked]
> Subject: Re: [conex] Discussion of new Use Cases pt 1 - Shaper
> Definition
> 

I suspect a lot of the confusion comes about because many ISPs use what are
sometimes called DPI boxes or traffic management boxes to impose a host of
controls on a user's traffic. Some of these controls are shaping to a given
rate, some are policing by dropping, some I suspect are something of a mix.
>From my (admittedly fairly limited) observations, ISPs tend to be prosaic
about these things - they don't care what the official terms are for what
they are doing, they simply fiddle until they reach a situation where most
customers are getting a better service with minimal cost to the ISP.

> otherwise you'll get confusion as different organisations define these

I agree 100% - we will make sure we define exactly what we mean within the
document to overcome this confusion over terminology.

> this is going to work in the real world (ie where the rubber meets the

The BIG problem all along for ConEx is that it operates (almost uniquely) at
a complex intersection between network economics, ISP network management,
e2e transport, IP, etc. Experts in all these fields are contributing, but
all of them speak very different languages and all viewing the world with
very different pre-conceptions. It is proving nigh impossible to write text
that is unambiguous enough to get everyone to understand what we are aiming
to do. Almost every time we describe it in a different way we get new people
understanding it, but seem to lose some others that had though they knew
what we were doing!

Part of the problem is that ConEx truly is a revolutionary concept - it
isn't easy to see what it does, or why it is needed. It is currently stuck
in the trap famously described by Thomas Kuhn in "The structure of
scientific revolutions" - we are in challenging some of the paradigms of
networking, and just to make life interesting, we have chosen to challenge
more than one at a time!

BTW, I realise that does nothing at all to explain ConEx to you! What
allowed me to "get it" originally was the belief that knowledge (and by
extension, information) is good - ConEx closes the information gap between
the network and end user and allows the network to be vaguely intelligent
about how it controls the end user (and please don't restart the arguments
about why a network needs to control users at all - it is apparent that a
majority of ISPs do think it necessary to impose controls, so let's try and
help them do it more sensibly).

Toby

---

From: "Toby Moncaster" <[email-masked]>
Date: Mon, 28 Mar 2011 15:40:44 +0100

Borrowing Bob's Devil's Advocate hat...

> Of Bob Briscoe
> Sent: 28 March 2011 12:44
> To: John Leslie
> Cc: [email-masked]

So why can't a "policer" work by ECN marking traffic? Obviously the traffic
could ignore the marks, but the action of marking traffic would force a
compliant (RFC3168) sender to react as if it had seen drop, in other words,
as if the policer had dropped its traffic

> called a policer. Let's stop using confusing terms.

I think we need a standalone discussion on appropriate terms to use for
ConEx. There seems to be too much confusion round "shaper", "policer" and
"rate limiter"

> * congestion of something you bought yourself (e.g. your home network).

And even congestion of a local medium due to current conditions (eg wireless
congestion) which is outside anyone's control

Toby

> draft?

---

From: "Mcdysan, David E" <[email-masked]>
Date: Mon, 28 Mar 2011 11:29:32 -0400

HI Mikael,

Dave

>the ISP business would give you the exact same answer.

The term rate limiter is not used often when referring to routers, and has
the ambiguity of whether it is in fact a policer or a shaper as defined in
RFC 2475.  In L2 ATM DSLAMs, the term used in Appendix B of TR-059 is in
fact rate limiter of a queue and IS NOT a policer (policers are called
policers).

>like a port doing queueing.

In routers this is true, in some L2 devices (e.g., DSLAMs) WRED/ECN is not
done by a rate limiter on a queue. I think it may be helpful to define
this distinction in the document.

>otherwise you'll get confusion as different organisations define these

Agreed. Instead of a reference, the document should quote the relevant
terms, mapping any choices between different names referring to the same
thing to a single agreed name.

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 28 Mar 2011 18:37:07 +0100

Mikael (& David),

Are you in Prague? I'd like to find what you don't like/understand 
about what's written, wihch may be quicker f2f, if you are willing.

I've been talking about these things to various of the audiences that 
Toby lists for a number of years now. I've learned to be careful 
about explaining what it's not, because most of the time the problem 
is that someone makes some jump in their own mind that isn't 
appropriate, but it's hard to stop people assuming things.

Shifting to the subject of the thread, it's a congestion-rate 
limiter. There is absolutely no (= none, zero, nothing, zilch) 
bit-rate limiting needed for ConEx-capable traffic.

The concept of a congestion-rate limiter is not defined in any RFCs, 
because it was impossible before ConEx exposed congestion so that a 
congestion-rate could be limited. Therefore we should only reference 
RFCs about policers in the negative.

But my earlier point was:
* the term shaper generally means smoothing inter-packet gaps, which 
requires adding delay to some packets (because you can't make time go 
backwards for the other packets).
* the general term policer is about keeping traffic within a contract.

So I added the word congestion to the word policer to define a new concept...

The contract is a congestion-rate (and a congestion-volume burst), so 
I thought it's reasonable to call the thing a congestion-rate 
policer, (or congestion policer for short, but perhaps we should 
avoid shortening this?)

Does this help?

Bob

---

From: Mikael Abrahamsson <[email-masked]>
Date: Mon, 28 Mar 2011 20:09:57 +0200 (CEST)

On Mon, 28 Mar 2011, Bob Briscoe wrote:

> Are you in Prague? I'd like to find what you don't like/understand about 
> what's written, wihch may be quicker f2f, if you are willing.

Nope, sorry, wasn't able to go.

> I've been talking about these things to various of the audiences that Toby 
> lists for a number of years now. I've learned to be careful about explaining 
> what it's not, because most of the time the problem is that someone makes 
> some jump in their own mind that isn't appropriate, but it's hard to stop 
> people assuming things.

I think someone needs to make a document that explains thing in a 
practical hands-on kind of way. Graphics would help.

<http://www.cisco.com/en/US/products/hw/switches/ps646/products_tech_note09186a00800feff5.shtml>

This talks about tokens, burst size, out of profile tokens etc. The 
policer acts on all packets according to a simple algorithm. It doesn't 
act on "flows".

> The contract is a congestion-rate (and a congestion-volume burst), so I 
> thought it's reasonable to call the thing a congestion-rate policer, (or 
> congestion policer for short, but perhaps we should avoid shortening this?)
>
> Does this help?

I think of a world in packets, queuing, packets being dropped or marked. 
If they are queued, dropped or in ECN case, marked, there is congestion.

The draft-ietf-conex-concepts-uses talks about "CONEX policers" and 
"congestion volume" and "flows". I guess I just fail to see how this is 
actually implemented in a real world device and what it would do, and how 
this would scale.

---

From: "Mcdysan, David E" <[email-masked]>
Date: Mon, 28 Mar 2011 14:32:32 -0400

Bob,

 In my view "shaper" and "policer"  and "rate limiter" are terms used to
describe what exists and this text would be in Existing Approaches section
of the use cases draft.

What you are describing is a new concept with Conex, and this should
probably be described in mechanisms with some high-level reference in use
cases. The [Policing Freedom] reference is currently given in both drafts
and I recommend that to Mikael and others wanting to better understand
this new approach. The terminology in that paper seemed clear enough to
me, and I suggest using those instead of creating new terms as you have at
the end of your post below. (There are some inconsistencies in the current
drafts use of terminology that would need to be made to do this).

I think that section 5 of the use cases draft would be a good place to
summarize how the new approaches solve problems with the existing
approaches. 

A few questions, responses in line below.

I would like to meet f2f to see if we can agree on how to describe this
better and also better describe the potential benefit of the abstract
mechanism.

Thanks,

Dave

>bit-rate limiting needed for ConEx-capable traffic.

For the case described in [Policing Freedom] this is true. But if an ISP
still wants to sell service based upon bandwidth tiers, then an "Existing
Approach Shaper" may still be needed, and the action of this "Existing
Approach Shaper" could create loss and/or ECN marking, which after
thinking about this more should work.

>RFCs about policers in the negative.

Negative means existing approach? That is, conex eliminates the need for
them?

>* the general term policer is about keeping traffic within a contract.

These describe certain aspects of how these are used in existing
approaches.
>policer, (or congestion policer for short, but perhaps we should

Suggest using terminology from [Policing Freedom].

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 28 Mar 2011 23:10:22 +0100

Mikael,

At 19:09 28/03/2011, Mikael Abrahamsson wrote:
>>I've been talking about these things to various of the audiences 
>>that Toby lists for a number of years now. I've learned to be 
>>careful about explaining what it's not, because most of the time 
>>the problem is that someone makes some jump in their own mind that 
>>isn't appropriate, but it's hard to stop people assuming things.
>

Try this - I have found this paper best describes the idea to someone 
familiar with existing traffic management techniques:

Policing Freedom to Use the Internet Resource Pool
<http://www.bobbriscoe.net/projects/refb/polfree_rearch08.pdf>

However, it was written pre-ConEx. The big difference is that 
document only focused on using ECN, not loss on the first of the 
three passes around the feedback loop. ConEx now aims to work with 
either loss or ECN.

We're going to have to invest the time in doing more ASCII art.

Bob

>policer acts on all packets according to a simple algorithm. It 
>doesn't act on "flows".
>
>>The contract is a congestion-rate (and a congestion-volume burst), 
>>so I thought it's reasonable to call the thing a congestion-rate 
>I think of a world in packets, queuing, packets being dropped or 
>marked. If they are queued, dropped or in ECN case, marked, there is 
>congestion.
>
>"congestion volume" and "flows". I guess I just fail to see how this 
>is actually implemented in a real world device and what it would do, 
>and how this would scale.
>
>--

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 28 Mar 2011 23:12:32 +0100

Mikael,

At 12:51 28/03/2011, Mikael Abrahamsson wrote:
>>is just plain stupid. If someone isn't responding sufficiently to 
>>ECN markings, adding more ECN markings is never going to be used by 
>>a sentient network operator to punish them.
>

See my comment at the end - I believe that the reason you disagree 
with this is because *you* are reading and using the term 'policing' 
inappropriately. Unless you can quote an authoritative source for 
using the term policing in this sense, please re-adjust your dictionary.

>Let's take the case with a router/switch with small buffers. I don't 
>see the huge difference in having a policer doing ECN markings at 
>95% of wirespeed and then letting the small buffer taildrop at 3-5ms 
>of buffering, compared to something that has 50ms of buffering and 
>using RED starts doing ECN markings/RED after 5ms of buffers fill.

This is AQM. There's nothing wrong with what you're saying, it's just 
very little to do with ConEx.

Looking again at the ConEx docs (particularly conex-abstract-mech) I 
can see the problem: We omit to say that one basic idea of ConEx is 
that all the forwarding nodes (L2 & L3) do nothing more than FIFO 
forwarding preferably with AQM and possibly ECN marking. But none of 
that is policing. Each forwarding node alone has insufficient 
information to determine whether a particular site is keeping to 
contract or not. So they don't do any policing (meaning keeping 
traffic to a contract).

Instead, with ConEx, congestion-rate policing is done at one node 
that all a customer's traffic passes through (or in multi-homed cases 
one logically single policing node), preferably near the ingress 
edge. Much like the ingress to the 2475 Diffserv architecture. But 
unlike Diffserv, each interior forwarding node does no policing 
whatsoever - nothing, zilch, zero, null, none.

Already, in the existing Internet architecture, through e2e transport 
feedback the source sees information about congestion (loss or ECN) 
from all the forwarding nodes its traffic is going through, 
everywhere it passes in the whole Internet. The idea of ConEx is for 
all the sources within a site to expose this information back to the 
network, so that the policing node at the ingress where the site 
attaches can limit the congestion this site as a whole contributes to 
the Internet. The site might be a campus, or a home network, or just 
a tenant on one blade in a data centre.

This ingress policing node uses information from ConEx, which has 
come round the feedback loop and back into the front of the network 
via the sender. It doesn't police based on its local congestion, it 
polices based on all the congestion it is told about by the source.

I can see we need to say a lot more about what ConEx is NOT.

>A policer NEVER ever shapes. Shaping is the delaying/buffering of 
>packets. A policer is something that has buckets and either drops, 
>marks and/or transmits packets, never ever delays them.

Exactly - people in this wg keep using the term shaping where it's 
not applicable, and it's just confusing everyone.

>>Shaping is something some operators still do to make a packet 
>>network look like a phone network in the belief it will improve 
>>things, when it merely adds delay deterministically that might 
>>otherwise not be added statistically. Like ECN, it's not really a 
>>useful tool in the punishment armory. A thing that punishes out of 
>>contract behaviour is called a policer. Let's stop using confusing terms.
>
>It seems we need a discussion on the basic terms people use to 
>describe behaviour.

I think actually you are incorrectly stretching the term 'policing' 
above in you ECN marking at 95% wirespeed. No-one else calls that 
policing. When a node polices to a rate, it drops packets above that 
rate (possibly with a burst allowance). If it ECN marks above a rate, 
that is AQM or it might be called metering and marking, but not 
policing. You are describing metering & marking for pre-congestion 
notification (RFC5696).

Does this help?

Bob

---

From: Michael Menth <[email-masked]>
Date: Tue, 29 Mar 2011 00:35:54 +0200

Hi Bob, Mikael, Dave,

Policer: makes flow compliant to some profile by dropping packets.
Shaper: makes flow clompliant to some profile by delaying packets to 
some extent, dropping starts only if the delaying buffer overflows.

Do you agree?

With conex, a flow must be compliant to the "congestion-profile" defined 
by the "congestion allowance".

Using the above definitions suggests that the "conex-policer" may be 
implemented either as policer or as shaper. Why do we always talk only 
about a conex-policer? Why is delaying packets not an option? I think 
the traffic leaving the "conex-policer" must just be compliant to the 
"congestion-profile.

Best wishes,

     Michael

-- 
Prof. Dr. habil. Michael Menth
University of Tuebingen
Faculty of Science
Department of Computer Science
Chair of Communication Networks
Sand 13, 72076 Tuebingen, Germany
phone: (+49)-7071/29-70505
fax: (+49)-7071/29-5220
mailto:[email-masked]
http://www-kn.informatik.uni-tuebingen.de

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 29 Mar 2011 00:29:28 +0100

Michael,

>some extent, dropping starts only if the delaying buffer overflows.

Yes.

>With conex, a flow must be compliant to the "congestion-profile" 
>defined by the "congestion allowance".
>
>implemented either as policer or as shaper. Why do we always talk 
>only about a conex-policer? Why is delaying packets not an option? I 
>think the traffic leaving the "conex-policer" must just be compliant 
>to the "congestion-profile.

It is *theoretically* possible to shape traffic to keep it to a 
congestion profile, but it would require such huge amounts of delay 
that would make it unworkable. This is mostly because congestion is 
signalled as a unary encoding (some packets marked, some not). So the 
congestion rate is only measurable on occasional packets. SO if you 
tried to shape this, you would have to hold back thousands of packets 
waiting for a token to release a marked packet and the other unmarked 
packets stacked behind it.

Bob

---

From: Mikael Abrahamsson <[email-masked]>
Date: Tue, 29 Mar 2011 05:26:06 +0200 (CEST)

On Mon, 28 Mar 2011, Bob Briscoe wrote:

>>> markings, adding more ECN markings is never going to be used by a sentient 
>>> network operator to punish them.
>> 
> See my comment at the end - I believe that the reason you disagree with this 
> is because *you* are reading and using the term 'policing' inappropriately. 
> Unless you can quote an authoritative source for using the term policing in 
> this sense, please re-adjust your dictionary.

See the cisco document I referred to earlier. I don't know if you think 
this is authoritative or not, but they have 40% of the market so they have 
considerable influence on the terms people use.

But I'll wait for next iteration of documents, because from reading so far 
I still don't understand what does what and how, in the proposed CONEX 
world.

---

From: Michael Menth <[email-masked]>
Date: Tue, 29 Mar 2011 08:14:29 +0200

Hi Bob,

> packets stacked behind it.
Yes that's true. Shaping in this case can even create bursty traffic 
which is possibly not a good idea. But still, most important is that a 
conditioner enforces the "congestion profile". If sticking to the mere 
policer, a hint should be added why the conditioner must drop packets 
but cannot delay them. That improves clarity and underlines the behavior 
of the policer.

Best wishes,

     Michael

---

From: John Leslie <[email-masked]>
Date: Tue, 29 Mar 2011 06:24:53 -0400

> the ISP business would give you the exact same answer.

   Well, I wouldn't...

   But then, I don't believe in rate-limiting by delay: many ISPs do.

   Rate-limiting, to me, means some packets must be dropped; to many
ISPs, it means you need bigger buffers.

   Thus, I avoid talking of a "rate-limiter" in such a way as to
either include or exclude shaping by delay.

>> type behavior for downstream and upstream customer access. This is
>> easy for customers and connecion-oriented operators to understand
>> and relatively easy to measure.

   Well, actually it isn't easy to _understand_ -- it's merely easy to
pretend you understand... :^(

> A shaper emulates the behaviour of a port that is full and is doing
> output queueing, but at any speed and in any direction. It can do
> WRED/ECN, just like a port doing queueing.

   I agree with the "emulates" part.

> otherwise you'll get confusion as different organisations define
> these terms differently.

   You're almost certainly right.

   Alas, defining within the document may not be enough: folks tend
to stick with their familiar definitions regardless of definitions
within the document. :^(

> Also, I think anyone who wants to present CONEX to a wider audience
> needs to create me concrete examples about what's going on.

   I'm not sure we're there yet...

> I've been doing routing/QOS engineering for 10+ years, most of the
> time I don't get what people in this group are talking about.

   Welcome to the club!

> It's like you're living in a completely different world and using
> completely different words to describe things.

   Unfortunately, Humpty-Dumpty lives on in all too many of us.

> I understand exactly how ECN works, I have read the CONEX documents
> fairly well, but it's still extremely abstract what the discussions
> are about and this is going to work in the real world (ie where the
> rubber meets the road).

   You're the _ideal_ contributor to this work!

   Some of the abstract-ness is forced upon us by our charter. We
_can't_ do anything to fix that right now -- our Area Director is
retiring. We could tackle that in a few months -- but it's not clear
it would be worth the effort.

   We have two WG documents: a Use-Cases and an Abstract-Mechanism.
They _must_ serve those purposes. There seems to be consensus that
it isn't helpful for Use-Cases to separately state mechanism; and
Abstract-Mechanism _must_ remain abstract.

   datatracker.ietf.org/wg/conex lists six "related documents", all
individual submissions. None of these reflects WG consensus, and
it's not at all clear any of them deserve to become WG documents
at this time. (Several of them clearly don't, being absorbed into
the two WG documents.)

   I could try to outline possibilities for how ConEx may work in
the "real world"; but I tend to avoid it, because opinions are so
strong that even suggesting that policing might be done in two
different ways tends to start two flame-wars about how anyone could
be so stupid as to consider the other way. :^(

   Nonetheless, I'm open to suggestions...

---

From: John Leslie <[email-masked]>
Date: Tue, 29 Mar 2011 06:48:19 -0400

> <http://www.cisco.com/en/US/products/hw/switches/ps646/products_tech_note09186a00800feff5.shtml>

   "Understanding QoS Policing and Marking ond the Catalyst 3550"

> policer acts on all packets according to a simple algorithm. It doesn't 

   The Catalyst 3550 does many things other than QoS policing.

> If they are queued, dropped or in ECN case, marked, there is congestion.

   Ah, our old friend: which meaning of "congestion" are you using?

> and how this would scale.

   How it is _actually_ implemented isn't a protocol issue; and we're
not even talking _protocol_ yet. :^(

   How it would scale is a very good question -- and I'm happy to talk
about that until the WGCs stamp on me. ;^)

   The chief scaling issue is keeping flow state. We don't actually
envision flow state being kept at all except near sender and/or
receiver.

   Even there, we have plenty of ideas how to radically minimize
flow-state -- it's just that we avoid stating them for fear of
flame-wars.

   But I'll state flat-out that there is nothing about the _protocol_
we envision which requires flow-state anywhere except the two
endpoints. There _are_ some policing algorithms that could work
better with some flow-state; but they absolutely are NOT necessary
to the protocol. The protocol can work fine treating all traffic
to/from one customer (or peer) as a single flow.

   If you're happy thinking in terms of dropping packets, go ahead
and think of it that way. The packet marking we think in terms of
is either by the sender or by an ECN-enabled forwarder. (Though I
must admit some of the possibilities we entertain might better be
called "forged" ECN marking, they're fundamentally indistinguishable
from "genuine" ECN marking. ;^)

---

From: Ingemar Johansson S <[email-masked]>
Date: Tue, 29 Mar 2011 13:45:21 +0200

Hi

I have always had a hard time to understand IETF docs, mainly because of the ASCII-flaw. For this reason, to understand ConEx what what it does and why, it is probably better to read some of the re-ECN stuff that contains human readable graphics :-). Bobs page is a good starter
http://bobbriscoe.net/projects/refb/

This doc can be particularly useful
http://bobbriscoe.net/projects/refb/polfree_rearch08.pdf 

I am not that worried whether or not you call the "policer" or "shaper" a "shaper" or "policer", I would rather say that if sufficient time is spent on understanding the concept, it becomes quite clear what is meant in the end. Of course it is good to have a good terminology but please don't waste all the energy on this.

Regards

-----Original Message-----
From: Mikael Abrahamsson [mailto:[email-masked]] 
Sent: den 29 mars 2011 05:26
To: Bob Briscoe
Cc: [email-masked]
Subject: Re: [conex] Discussion of new Use Cases pt 1

On Mon, 28 Mar 2011, Bob Briscoe wrote:

> with this is because *you* are reading and using the term 'policing' inappropriately.
> Unless you can quote an authoritative source for using the term 
> policing in this sense, please re-adjust your dictionary.

See the cisco document I referred to earlier. I don't know if you think this is authoritative or not, but they have 40% of the market so they have considerable influence on the terms people use.

But I'll wait for next iteration of documents, because from reading so far I still don't understand what does what and how, in the proposed CONEX world.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Tue, 29 Mar 2011 13:46:02 +0200 (CEST)

> not even talking _protocol_ yet. :^(

I'm quite baffled by this statement. Could you please elaborate on this 
view?

Whenever I design something, I have in the back of my mind how it would 
actually be implemented and if it actually could be done in the real world 
in the forseeable future at decent cost.

The whole "use-case" of CONEX is to save money by not upgrading 
access/distribution link speed. If it's going to be usable in this 
scenario it needs to be cheap, otherwise there is no money to be saved.

---

From: John Leslie <[email-masked]>
Date: Tue, 29 Mar 2011 08:08:35 -0400

This deserves a brief answer:

> I'm quite baffled by this statement. Could you please elaborate on this 

   It's a simple statement of process: we're considering two WG documents,
neither of which discusses _actual_ protocol; _one_ of them discusses
"abstract mechanism", which it a pre-protocol stage.

> in the forseeable future at decent cost.

   I fully agree. But the WG isn't designing a protocol yet.

> scenario it needs to be cheap, otherwise there is no money to be saved.

   Actually, use-cases covers a lot of other reasons to proceed with
ConEx.

---

From: "Toby Moncaster" <[email-masked]>
Date: Tue, 29 Mar 2011 14:17:52 +0100

> Of John Leslie
> Sent: 29 March 2011 13:09
> To: Mikael Abrahamsson
> Cc: [email-masked]

And that is how the WG was chartered... 

>    I fully agree. But the WG isn't designing a protocol yet.

There are people that have specific protocols in mind for ConEx. I accept
that in the absence of a specific protocol you can reasonably ask "how do we
know this will work?" but that is not the same as judging whether the
concept of ConEx is worthwhile.

>    Actually, use-cases covers a lot of other reasons to proceed with

"saving money" is actually fairly low on the list of aims for ConEx. Much
more significant are: giving users more control over their network
connections, allowing users to prioritise their own traffic (rather than
having priorities foisted on them), allowing operators to track sources of
congestion, allowing operators to make intelligent decisions about traffic
management, allowing operators to target upgrades most effectively, ...

---

From: Mikael Abrahamsson <[email-masked]>
Date: Tue, 29 Mar 2011 16:59:01 +0200 (CEST)

On Tue, 29 Mar 2011, Toby Moncaster wrote:

> "saving money" is actually fairly low on the list of aims for ConEx. 
> Much more significant are: giving users more control over their network 
> having priorities foisted on them), allowing operators to track sources 
> of congestion, allowing operators to make intelligent decisions about 
> traffic management, allowing operators to target upgrades most 
> effectively, ...

For me, most (if not all) of these boil down to "save money".

But I guess we can agree to disagree on this point.

---

From: "Toby Moncaster" <[email-masked]>
Date: Tue, 29 Mar 2011 16:12:02 +0100

I guess actually operators will be wanting to maximise revenue or maximise
return on investment (subtly different from saving money)...

> Sent: 29 March 2011 15:59
> To: Toby Moncaster
> Cc: [email-masked]

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 30 Mar 2011 08:24:56 +0200 (CEST)

On Mon, 28 Mar 2011, Bob Briscoe wrote:

> However, it was written pre-ConEx. The big difference is that document only 
> focused on using ECN, not loss on the first of the three passes around the 
> feedback loop. ConEx now aims to work with either loss or ECN.

Another way would be that instead of inventing something new, to use 
existing tech like DSCP marking. Today the Internet doesn't really support 
it, but that's a choice made by the ISPs. The general consensus in the ISP 
community is that DSCP bits are ok to reset to 0 for users not purchasing 
a "QoS service". If this opinion/implementation was changed, then ISPs 
facing congestion could act on TOS bits to prefer interactive flows 
(they'd have to be marked as such by the user), and they could also bill 
on the amount of volume a user sent that was marked interactive and had no 
ECN markings. If it had ECN marked, then the user would only be billed on 
the traffic that came back with ECN bit set (I guess this is somewhat what 
re-ECN tries to do). It would also mean that interactive flows would get 
ECN bit set whenever there was congestion on a link, but the interactive 
flow was prioritised and didn't really notice the congestion. This also 
means ECN bits would have different behaviour depending on what the TOS 
bits are. For interactive TOS marked packets it would mean "you're being 
billed", for non-interactive it would have the standard ECN behaviour of 
today, and TCP would have to be changed appropriately (if TOS = 
interactive, don't treat ECN bit as indication of need to back off).

The advantage of not inventing new bits is that this scheme would work out 
of the box with IPv6 as well.

Generally, any solution that doesn't require the device to understand 
state and "flow" is beneficial, that's what I'm trying to achieve. 
Increasing counters is easy, understanding if a "flow" (L4 tuple) is 
misbehaving and acting on it is a lot harder. I guess this is why I don't 
really "get" CONEX, because all my interpretations of text I'm reading 
seems to indicate need to keep state and understand flows.

---

From: John Leslie <[email-masked]>
Date: Thu, 31 Mar 2011 00:57:59 -0400

Mikael Abrahamsson <[email-masked]> wrote:
> existing tech like DSCP marking.

   Actually, it's likely ConEx may define one or more DSCP codes.

> Today the Internet doesn't really support it, but that's a choice made
> by the ISPs. The general consensus in the ISP community is that DSCP
> bits are ok to reset to 0 for users not purchasing a "QoS service".

   Very sad...

   But please recall, we're working in IPv6, not IPv4. Zeroing DSCP in
IPv6 is going to be considered "damage".

> If this opinion/implementation was changed, then ISPs facing congestion
> could act on TOS bits to prefer interactive flows (they'd have to be
> marked as such by the user), and they could also bill on the amount of
> volume a user sent that was marked interactive and had no ECN markings.

   That's a rather biggish flag-day...

   Nonetheless, it is something specifically _permissible_ in ConEx
(though not recommended).

> If it had ECN marked, then the user would only be billed on the traffic
> that came back with ECN bit set (I guess this is somewhat what re-ECN
> tries to do).

   Actually, ConEx would be more likely to recommend no charge for
traffic with their DSCP codepoint _unless_ it has a congestion-expected
mark. (Of course, saying what we'll actually recommend is way premature.)

   (I don't know what you mean by "came back with ECN bit set"; and I'm
not going to guess -- but if you mean anything like keeping flow-state,
it sounds like a really-bad idea.)

> It would also mean that interactive flows would get ECN bit set whenever
> there was congestion on a link, but the interactive flow was prioritised
> and didn't really notice the congestion.

   If there is actual charging for congestion-expected marking, that's
obviously reasonable. We tend to expect the contractual arrangement to
be an "allowance" instead of a per-mark charge, and we argue this is
also reasonable.

   (For this to work, pretty much every "interactive" flow would have
to start with what we call "building a ConEx credit"; but that's an
issue between end-user and ISP, not a "protocol" issue.)

> This also means ECN bits would have different behaviour depending on
> what the TOS bits are.

   Not necessarily...

> For interactive TOS marked packets it would mean "you're being billed",
> for non-interactive it would have the standard ECN behaviour of 

   "No need to back off" is a non-starter. "Back-off", of course, is
up to the sender. Already, not all senders _do_ back off; but ConEx isn't
intended to say it's OK to not back off: instead it encourages you to
say, "I expect this much congestion." _One_ possible implementation at
full deployment is that the ISP has a contractual arrangement to accept
a larger number of congestion-expected packets and pay settlements to
ensure preferential delivery. But we're a long way from specifying any
such thing: we're not even looking at a "sufficiently-full" deployment
to do that.

> The advantage of not inventing new bits is that this scheme would work
> out of the box with IPv6 as well.

   Please! We're talking IPv6-only in ConEx. IPv4 is entirely outside
our charter.

> misbehaving and acting on it is a lot harder. I guess this is why I
> don't really "get" CONEX, because all my interpretations of text I'm
> reading seems to indicate need to keep state and understand flows.

   I'd appreciate you pointing out anything in use-cases that leads you
to think that. It is definitely not what the authors intend.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 31 Mar 2011 07:14:55 +0200 (CEST)

> IPv6 is going to be considered "damage".

Please elaborate. This is the first I have heard of this and I would bet 
I'm no the only one in the ISP community.

> it sounds like a really-bad idea.)

How does the receiver signal to the sender that he received an ECN marked 
packet which had the ECN "would have dropped bit" set? Anyhow, it's this 
congestion signal I meant.

>   Please! We're talking IPv6-only in ConEx. IPv4 is entirely outside our 
> charter.

I wasn't aware of this. I thought I saw some IPv4 discussion here, but I 
might have it mixed up with members active in here being active in other 
WGs talking about using new IPv4 header bits.

> to think that. It is definitely not what the authors intend.

I re-read the use-case and it seems I must be misremembering (or got the 
idea from another text), because as it stands right now, it seems fairly 
clear that there is no per-flow action going on.

Perhaps the use-case could explicitly refer to itself and CONEX being 
about IPv6 only btw, I couldn't find any reference to neither IPv4 or 
IPv6.

---

From: John Leslie <[email-masked]>
Date: Thu, 31 Mar 2011 02:19:05 -0400

Mikael Abrahamsson <[email-masked]> wrote:
> Please elaborate.

   Not sure what I can say without starting a flame-war. :^(
Basically, the baggage of actually _doing_ something based on those
bits is no longer there in IPv6; and they're the only thing available
for introducing new features.

> This is the first I have heard of this and I would bet I'm no the only
> one in the ISP community.

   I'm sure you're right about that!

> congestion signal I meant.

   That signalling path is transport-dependent. For TCP I recommend

http://en.wikipedia.org/wiki/Explicit_Congestion_Notification

> WGs talking about using new IPv4 header bits.

   Most of Bob Briscoe's leading up to ConEx was written for IPv4
originally: thus it's easy to think IPv4 when reading it...

> about IPv6 only btw, I couldn't find any reference to neither IPv4 or 

   Good point!

---

From: ken carlberg <[email-masked]>
Date: Thu, 31 Mar 2011 03:23:13 -0400

>> bits are ok to reset to 0 for users not purchasing a "QoS service".

actually, this falls un the category of Your Mileage May Vary.  an intern in my office did a series of pings with different ToS byte values (both ECN and diff-serv fields) and found that the majority of ISPs left the entire byte "as is", ie, didn't zero out the entire byte.  some zero'ed the byte, and a smaller subset just zero'ed out just the old Precedence set of bits.  I'll caveat the statement that the testing wasn't fully exhaustive, so this is just food for thought.

-ken

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 31 Mar 2011 09:26:21 +0200 (CEST)

On Thu, 31 Mar 2011, ken carlberg wrote:

> actually, this falls un the category of Your Mileage May Vary.  an 
> intern in my office did a series of pings with different ToS byte values 
> (both ECN and diff-serv fields) and found that the majority of ISPs left 
> the entire byte "as is", ie, didn't zero out the entire byte.  some 
> zero'ed the byte, and a smaller subset just zero'ed out just the old 
> Precedence set of bits.  I'll caveat the statement that the testing 
> wasn't fully exhaustive, so this is just food for thought.

Yes, that is my experience as well. YMMV.

My point was not "all ISPs zero", but "it's ok to zero if you want to".

I brought this up on nanog-l and I felt the consensus was as I described 
above.