Subject: Re: [conex] byte vs packet counting

From: John Leslie <[email-masked]>
Date: Fri, 2 Dec 2011 18:20:51 -0500

Dirk Kutscher <[email-masked]> wrote:
> [John Leslie <[email-masked]> wrote:]
> 
>> I sincerely hope Bob isn't saying that an audit function has to
>> evaluate _every_ packet which passes it!
> 
> Clearly, it at least has to be able to do that, yes!

   Clearly, _if_ it has to do that, we're extremely limited on _where_
we can put an auditing function. Auditing is not going to get built
into the hardware of 10-gig routers.

> Auditing is really a fundamental piece of the basic Conex mechanism,

   I'd be happy to discuss that. I see no reason to bother auditing
if the entire path is known to be free of congestion. Auditing might
make sense if you expect traffic you send to encounter congestion
later in the path, but only if you have some contractual arrangement
calling for it. (Otherwise, you can't know what the effect of
ConEx marks may be.)

   Auditing every packet is plausible near the sender and receiver,
but not in the backbone. I don't believe we have agreement among
ourselves what a receiver's uplink will do if their auditing shows
"abuse" of ConEx marking: so I don't see what auditing at the
sender's uplink can do and be confident it's the right thing.

   I suppose it's possible the experiments we do will enlighten
that question, but I'm not confident. :^(

> and has nothing to do with policy. Auditing is the basis for accurate
> congestion exposure.

   The Internet works as well as it does precicely _because_ nobody
gets too paranoid about how "accurate" packets are -- instead we
spend our efforts worrying how well the implementations of our
protocols "interoperate". (That keeps us busy enough, IMHO.)

> Only if you have that, you can think about implementing policies.

   As an ISP, I "implement policies" all the time based upon guesswork.
The general principle is that a partly-wrong policy implemented at the
right time is almost always better than a "right" policy implemented
too late.

> Having said that, it is certainly implementation-specific how
> auditing really works -- there may be tradeoffs between efficiency
> and effectiveness -- but for the sake of the discussion here, I'd
> say it's best to assume that, yes, auditing evaluates every packet.

   IMHO, it's _always_ best to assume otherwise.

   Your life will be easier if you don't depend on assumptions which
are likely to be false.

   We'd do better, when discussing this, to get specific about how
auditing might improve the use-cases we're pushing.

   Unless I misread draft-ietf-conex-concepts-uses, our principle
use case is Informing Traffic Management. I have posted before that
the most important feature of ConEx marking (IMHO) is informing
any node along the way that the sender is knowingly sending packets
despite having reason to doubt that congestion has cleared.

   Auditing does nothing to help here. The abusive thing to do is
to _clear_ ConEx marks -- and auditing can't detect that.

   I haven't yet argued a specific benefit to ECN-marking instead
of dropping -- when I do, I'll be arguing for ECN-marking well
before packets need to be dropped. I simply don't believe that any
amount of auditing will convince providers to trust ConEx marking
to mean these packets will never be dropped.

   Auditing, to tell truth, may or may not prove useful after we
have done enough experiments. IMHO, if it proves useful we'll find
that it's useful when done statistically, rather than on every
packet.

--
John Leslie <[email-masked]>

---

From: John Leslie <[email-masked]>
Date: Fri, 2 Dec 2011 19:21:50 -0500

Matt Mathis <[email-masked]> wrote:
> 
> Bob and I prefer to argue in private, because it works better.

   The IETF way, alas, is to argue in public. We need to learn how to
do so without overly harming our ability to work together.

> We always assume that the other person is correct, and make every
> effort to try to understand what they think and why. This is why we
> have often been able to clearly restate each others arguments.

   A decidedly useful practice.

   Alas, I seem to have no luck restating Bob's arguments. He reminds
me a bit of John Campbell of "Analog" fame, of whom it was frequently
said, "if he finds you're agreeing, he changes the subject!"

> What typically happens is that we discover some hidden assumption,
> and that both of our initial positions are over simplifications and
> at some level both are wrong.

   Congratulations!

   I'm more than willing to both be wrong. ;^)

> When we argue in public, the entire discussion get rat holed about
> details that are aren't on the table, either because the rest of us
> already have consensus or they just are not important at this stage.

   Consensus isn't something for "the rest of us" to have.

   In the IETF, we talk of "rough consensus". That doesn't mean that
some of us deserve to be ignored: it means that some of us agree to
"be in the rough" and stop arguing an issue.

   As for "important at this stage", I'm afraid none of us are
omniscient enough to know what's important.

   There is, however, a time when folks are "ready to learn" and a
much larger time when they aren't. :^(

   But IMHO they only deceive themselves when they say "it's not
important" instead of "I'm not ready".

   But I digress...

> My take away from the intended discussion with Bob is that
> -abstract-mech- is flawed in that it is entirely too casual about
> units.

   I'd be happy to work on what the "units" should be.

> The units (primarily bits v bytes v packets of congestion) should
> be an explicitly chosen parameter for each span of the model
> (congested router to receiver, ACKs carrying the signal back to
> the sender, reinserted feedback from the sender to the audit
> and policy functions).

   That sounds difficult...

> While I am inclined to agree with Bob that the congestion signal
> should be in bytes,

   (I'm less inclined...)

> I am less convinced that it is ok for the ACKs to only carry
> segment counts.  I am willing to proceed on the basis of his
> assumptions, however it is critical that we all understand that
> this is a design decision that might be changed in the future if
> there is a compelling reason to do so.

   I understand _that_ Bob believes counting bytes is more correct
than counting packets, but I don't understand why he believes it.
I see folks responding to his repeated claims with, "But it's
not the case for this common situation;" and there the thread dies.

   Elsewhere, I read of concerns that "small packets shouldn't be
penalized for being small," but I don't know if that's Bob's
argument.

   Whether or not it is, I find it unconvincing. If small packets
were _actually_ being significantly punished for being small, the
sender would find a way to enlarge them. I simply don't see that
in the wild.

   And that argument seems to really be, "Look how nice I'm being
by sending smaller packets! Why don't folks reward me more?"

   (I can think of a half-dozen reasons, which I won't list.)

   In practice, the folks who do send lots of small packets want
to see less jitter in RTT. But it's the large packets which
usually cause the jitter. Alas, the medium is shared...

> The abstract mech doc needs to state that proposed signal mappings
> for a given tentative encoding must be explicit about units.

   That sounds like an improvement...

> It is also important to note that for legacy interoperability
> using a sender only implementation, we can not assert different
> algorithms than are present in current network devices and ECN
> enabled receivers.

   I agree these, in practice, are unknowable. :^(

> My example earlier in the thread illustrates a rather extreme
> pathological case where bytes v segments might change sender
> behavior. In the common case and as a natural approximation,
> the sender will probably assume that the data in not lumpy, and
> that reflecting the correct number packets is good enough.

   It strikes me as seriously sub-optimal to say, "It's reasonable
to assume X" and at the same time, "It's more correct to punish
anyone who assumes X".

   I frankly don't see why abstract-mech needs to say either of
these. We should perhaps state that experiments may include
scenarios where some nodes may punish X while other nodes may
punish Y; but I'm very uncomfortable with suggesting that either
of those punishments is "correct".

> Clearly when all of the packets in a given connection have uniform
> sizes, it doesn't matter to any component, except the policy devices.

   I think we have agreement there...

> The places proper treatment of lumpy data might matter are the
> policy and audit devices, which will be the last part of ConEx to
> have fully bound specifications.

   Agreed.

> Some of their details are even explicitly out of our scope,
> although we do have to demonstrate the existence of algorithms that
> provide useful signals.

   Algorithms, plural, yes.

> As I reflect on the broader debate, it is clear to me that we
> really need to understand choosing units of congestion as an
> engineering compromise.

   Perhaps even an engineering compromise where different senders
and receivers make different trade-offs?

> Like all such compromises, it has to balance the costs of
> difficulty of implementation vs opportunity cost of reduced
> precision.

   Agreed.

> Furthermore, extended debate without hard data is pointless.
> It just doesn't matter at this stage. We can accept Bob's assumption
> for the time being, note that it is "provisional" and revisit it
> later, if or when we encounter a problem that would be better
> solved with different assumptions.

   Remember the Tao of Internet: "We reject kings, presidents, and
voting..." I don't remember electing Bob as President; nor do I
see good reason to accept him as King. What other reason is there
to accept Bob's assumption when we have so much reason te believe
it is incomplete or worse?

> John, can we arrange a phone call some time?

   (Matt and I have talked at some length on the phone. I like to
believe I have a better understanding of research in this field
because of this. I just don't read the research the same way Bob
does...)

---

From: Toby Moncaster <[email-masked]>
Date: Mon, 5 Dec 2011 17:55:00 +0000

Not sure if this email ever hit the mailing list…

Begin forwarded message:

> From: "T. Moncaster" <[email-masked]>
> Subject: Re: [conex] byte vs packet counting
> Date: 3 December 2011 10:10:52 GMT
> To: John Leslie <[email-masked]>
> Cc: Dirk Kutscher <[email-masked]>, ConEx IETF list <[email-masked]>
> 
> I have been purposely not contributing to this thread over the past few weeks as I wasn't sure that more voices in the argument would actually help things. However I feel things are now at a point where I can contribute something (hopefully) useful.
> 
> On Dec 2 2011, John Leslie wrote:
> 
> I think there is a subtle difference between stating that auditing is a fundamental part of ConEx and saying that auditing can never be turned off. I would definitely agree that auditing is required in any network where there are potential trust issues and where you can't guarantee the whole-path network conditions. But I do think there are cases where it could be turned off (for instance in private corporate networks using conex as a means to control their monthly network bills).
> 
>> 
> I agree, it seems hard to see how auditing something when you are missing half the information is going to be hard. There are possible arguments along the lines of an ISP might make a decision that all users of a given access node (B-RAS or whatever) are likely to be experiencing roughly similar amounts of congestion. So any customer that declares no congestion is probably not being honest.
> 
>> 
> My view is that we're arguing the wrong point - auditing every packet seems trivial if we are assuming that most routers can mark a packet (e.g. can do ECN). In that case they are already inspecting 2 bits of the IP header, so asking them to inspect a 3rd bit seems reasonable. However, on a per-packet basis audit is meaningless. Audit only has any meaning over a period of time across a given flow (or set of flows). There is a separate accounting mechanism where you can measure bulk flows of congestion, but that has to be based on a belief in the accuracy of the marking of the underlying individual flows.
> 
>> 
> No. ConEx is about a balance of markings. To cheat you fail to send forward into the network sufficient ConEx markings to cover the congestion your flow is causing (or has caused). So the function of auditing is to check that the two numbers roughly match. And yes, this is a very difficult problem to solve if you can't assume ECN is being used.
> 
> 
>> 
> I don't think ConEx is ever meant to say to a provider "these packets are honest so must never be dropped". ConEx is meant to provide a better metric to network users to allow them to see the actual impact of any given set of packets in terms of congestion caused.
> 
>> 
> I think this may point to the heart of the issue. Yes, you can sample to do audit, but only if you assume a relatively high density of markings. If ConEx info is unary encoded into packets then you can only measure it by measuring long streams of the packets. If ConEx is a specific number carried in each packet then sparse sampling may well give you enough to be able to achieve the aims of audit.
> 
> Which brings me on to the aims of audit. To my mind there are two possible aims. 1) to guarantee that every single packet, flow, aggregation of traffic in the network has accurately declared their congestion or 2) to act as a "speed camera" that catches a sufficiently high proportion of infringements to ensure that senders don't risk under-declaring their congestion. My own view is that the second is more feasible in the real world, although the first might well be more desirable.
> 
> Toby
> 
>> 
>> --
>> _______________________________________________
>> conex mailing list
>> [email-masked]
>> https://www.ietf.org/mailman/listinfo/conex
>>

---

From: John Leslie <[email-masked]>
Date: Mon, 5 Dec 2011 18:01:53 -0500

T. Moncaster <[email-masked]> wrote:
> 
> I have been purposely not contributing to this thread over the past few 
> weeks as I wasn't sure that more voices in the argument would actually
> help things.

   IMHO, _any_ additional voices would help. (Of course, I may be biased:
if nobody takes my ideas seriously it's easier to declare me "in the
rough"...)

> However I feel things are now at a point where I can contribute
> something (hopefully) useful.

   Toby has _always_ had something useful to contribute!!

> I think there is a subtle difference between stating that auditing
> is a fundamental part of ConEx and saying that auditing can never
> be turned off. 

   Agreed... but it's not obvious which of these Dirk meant. :^(

> I would definitely agree that auditing is required in any network
> where there are potential trust issues and where you can't guarantee
> the whole-path network conditions.

   In other words, any real-world network...

> But I do think there are cases where it could be turned off (for
> instance in private corporate networks using conex as a means to
> control their monthly network bills).

   Is that really so different from real-world networks?

   (An aside: I talked with Matt today, and we're making progress on
understanding each other on what the WG should mean by "auditing".
Stay tuned...)

> I agree, it seems hard to see how auditing something when you are
> missing half the information is going to be hard.

   (I'm having trouble parsing that sentence.)

> There are possible arguments along the lines of an ISP might make
> a decision that all users of a given access node (B-RAS or whatever)
> are likely to be experiencing roughly similar amounts of congestion.

   (Seems unlikely to me...)

> So any customer that declares no congestion is probably not being
> honest.

   This sort of assumption bothers me. What if that customer is using
a particularly good LBE protocol?

> My view is that we're arguing the wrong point - auditing every packet
> seems trivial if we are assuming that most routers can mark a packet
> (e.g. can do ECN). In that case they are already inspecting 2 bits of
> the IP header, so asking them to inspect a 3rd bit seems reasonable.

   (Talking with Matt today, I am beginning to understand he may be
assuming that "auditing" measures only ConEx marks versus ECN marks
for a particular packet; and that the action taken on that basis
will be simply and only dropping the packet where the imbalance is
detected, keeping no state. That much could be implemented anywhere;
but it falls far short of what we are led to believe is promised by
his "auditing function". I'm hoping the next revision will assign
more intuitive names to whatever his current "auditing function"
proposes...)

> However, on a per-packet basis audit is meaningless.

   I think I agree with Toby.

> Audit only has any meaning over a period of time across a given flow
> (or set of flows).

   There's a pretty big difference between auditing a particular
flow versus auditing the particular set of flows represented by all
traffic an ISP receives from a peer at a peering point... I'm not
sure exactly what Toby refers to here.

> There is a separate accounting mechanism where you can measure bulk
> flows of congestion, but that has to be based on a belief in the
> accuracy of the marking of the underlying individual flows.

   I don't see how that follows.

> No. ConEx is about a balance of markings. To cheat you fail to send
> forward into the network sufficient ConEx markings to cover the
> congestion your flow is causing (or has caused).

   Not quite... an ISP can cheat by clearing marks which one sender
placed in his outgoing traffic -- that is what I meant to refer to.

> So the function of auditing is to check that the two numbers
> roughly match.

   (I note that Toby says "roughly match". Others seem to be assuming
a much stricter match. I'm afraid we don't have a clear understanding
here.)

> And yes, this is a very difficult problem to solve if you can't
> assume ECN is being used.

   I absolutely agree with Toby here.

> I think this may point to the heart of the issue. Yes, you can sample
> to do audit, but only if you assume a relatively high density of
> markings.

   I don't follow why that is true. If auditing is not strict, sampling
should work for a wide range of density; if auditing is strict, it
will fail pretty much regardless of the density...

> If ConEx info is unary encoded into packets then you can only measure
> it by measuring long streams of the packets.

   I agree with what I think Toby intends to say: that the congestion
metric of ConEx is a fraction, not a binary function per packet.

> If ConEx is a specific number carried in each packet then sparse
> sampling may well give you enough to be able to achieve the aims of
> audit.

   Carrying what I call "congestion-expected" as a multi-bit fraction
is an interesting idea, in which I see potential benefits. I'd be
happy to discuss it, but my impression is that we're not heading in
that direction.

> Which brings me on to the aims of audit. To my mind there are two
> possible aims.
> 1) to guarantee that every single packet, flow, aggregation of traffic
>    in the network has accurately declared their congestion; or
> 2) to act as a "speed camera" that catches a sufficiently high
>    proportion of infringements to ensure that senders don't risk
>    under-declaring their congestion.

   I don't see auditing as either of these; but that may be more a
function of my tendency to think of auditing as gathering information,
not enforcing disciplinary measures based on this information.

   IMHO, "guaranteeing" much of anything isn't worth the effort over
organizational boundaries. At significant expense, we could use
cryptography to give arbitrarily high confidence to a particular
"guarantee" having come from a particular entity, but without also
signing the data "guaranteed", the effort is entirely wasted.

   (And processing all that cryptography in backbone routers seems
a non-starter to me.)

> My own view is that the second is more feasible in the real world,
> although the first might well be more desirable.

   I at least partly agree here. However, my approach to thinking
about such problems simply stops descending into details when I reach
a feasibility blockage.

---

From: Toby Moncaster <[email-masked]>
Date: Tue, 6 Dec 2011 09:50:27 +0000

On 5 Dec 2011, at 23:01, John Leslie wrote:

>   In other words, any real-world network…

indeed

>   Is that really so different from real-world networks?

The main difference is that they may have an additional "big stick" that they can beat their users with if they are found to have cheated. In other words you can make the trust  separate issue.

> Stay tuned…)

That sounds like an interesting chat - looking forward to seeing the output of it.

>   (I'm having trouble parsing that sentence.)

I think we are hitting the issue of "what is audit?" If audit is taken to mean accounting for ConEx marks, then that can happen anywhere. However I was taking auditing to mean accounting for the balance of ConEx marks with actual congestion. That function can only work somewhere where you can see both the congestion and the ConEx marks. The only way you can achieve that at the sender side is by inspecting the ACKs on the return path (which may be impossible).

>   (Seems unlikely to me…)

Oh, patently unlikely, but it is the sort of thing I can (sadly) imagine some ISPs doing.

> a particularly good LBE protocol?

Well quite - that is the argument against blanket rate limiting of all P2P, even if it is running on LBE transport.

> proposes…)

That has been sort of my assumption, but with the proviso that you can only really tell the imbalance over a set of packets (see below)

> sure exactly what Toby refers to here.

Audit has two granularities - either you are auditing a given user to ensure his flow (or flows if he is using several) are correctly reporting the expected level of congestion, or you are auditing a whole peer network to ensure they aren't allowing excess congestion to enter your network without it being declared. These require (subtly?) different audit mechanisms

>   I don't see how that follows.

Measuring the bulk is very easy, but if all the flows making up the bulk are understating their congestion then the bulk will be understated as well. Bob always had in mind a clever mechanism to do spot checking at any border to identify persistent under (or over) declaration, but I never entirely understood how it was meant to work.

> placed in his outgoing traffic -- that is what I meant to refer to.

OK, yes. That is a separate requirement for audit. 

> a much stricter match. I'm afraid we don't have a clear understanding

Clearly, given the min 1RTT delay between getting congestion info and responding to it, there can never be a completely accurate match. I did have numerous debates with Bob about this topic. Basically you run the risk of a user just terminating his flow rather than making up his losses. You also have to be cautious that the actual process of dropping packets for audit doesn't simply distort the apparent congestion information (because the sender will respond by retransmitting and hopefully adding more ConEx marks)

> will fail pretty much regardless of the density…

OK, this comes down to the point below

> metric of ConEx is a fraction, not a binary function per packet.

Which follows very naturally from ECN - ECN (RED) randomly decides whether or not to mark a packet, with the aim that the overall number of marks (very roughly) reflects the congestion level experienced at that queue. Because this is a random unary scheme, any ConEx mechanism based on ECN (and that is trying to audit against current network conditions) will need to track this over a few packets to see how many ConEx marks it should expect. OR the sender has to have sent sufficient credit up front.

Even in a scheme that is trying to identify either gaps in sequence space (dangerous - especially in an MPTCP world) or is trying to identify retransmissions (again, dangerous in MPTCP), then you have to have some sort of memory of what went before (a counter of congestion seen against ConEx credit seen for instance).

> happy to discuss it, but my impression is that we're not heading in

That was certainly how I understood a re-ECN like scheme working… Te whole core of this argument (should you account for bytes or packets) has always been a major bone of contention between me and Bob. Not least because it can be impossible for a sender to comply (it is very easy to construct pathological sequences of packets where the sender is forced to either over or under declare if they are doing per-byte auditing). I think this is a case where the IETF view has to diverge from the researcher's view. In the IETF we have to make prosaic engineering decisions. Personally I would like ConEx to account on a per-packet basis, BUT with an allowance for it to be more accurate if an operator so wishes (in other words MUST as a minimum account per packet, but MAY account per byte)

> not enforcing disciplinary measures based on this information.

Perhaps auditing is an unhelpful word to use. To my mind an audit implies an accountancy audit that tries to balance the books. I agree that carries no implication of policing or enforcement, but I think that a lot of this debate has assumed the two concepts are identical

>   (And processing all that cryptography in backbone routers seems

Agreed, this has to be a lightweight mechanism. All the guarantees have to come from the idea that operators have to have some level of trust among one another (or at least, the realisation that an operator found to have cheated at the expense of others runs  risk of getting sat on heavily by the others),

> about such problems simply stops descending into details when I reach

And therein lies the lesson for ConEx. We HAVE to fix on a SIMPLE subset of ConEx that we can all agree on and that seems at least feasible in the real world. Arguments like the recent ones make ConEx appear to be still firmly stuck in the land of research (IRTF). Too many people (on all sides of all arguments) seem to want perfection and that is not sensible in the real world. That is just a recipe for ratholing ourselves to the point where the IESG declares our WG dysfunctional… 

Toby

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 13 Dec 2011 20:12:50 +0000

Toby,

Catching up after been away for a week...
Inline, and I've snipped everything I agree with...

> > T. Moncaster <[email-masked]> wrote:

[snip]

>I think we are hitting the issue of "what is 
>audit?" If audit is taken to mean accounting for 
>ConEx marks, then that can happen anywhere. 
>However I was taking auditing to mean accounting 
>for the balance of ConEx marks with actual 
>congestion. That function can only work 
>somewhere where you can see both the congestion 
>and the ConEx marks. The only way you can 
>achieve that at the sender side is by inspecting 
>the ACKs on the return path (which may be impossible).

[BB]: I want to take issue with 'the only way'. 
There's at least one other very plausible 
scenario. Assume by design there's one major 
bottleneck on the upstream path from a sender 
(e.g. a radio access network). Then a network 
node at the congested point can see all the ConEx 
and most of the loss (or ECN). It cannot know 
what 'most' means, but bear with me...

Imagine a sender is trying to cheat. This can 
even be a really knowledgeable sender - a 
disgruntled ex-employee of the network operator 
who knows that the audit function is badly placed 
- not near the receiver, but near the sender.

Even if there might sometimes be another 
bottleneck nearer the receiver on the path, when 
this dishonest sender sees congestion feedback 
(from loss or ECN), it cannot know which 
bottleneck each congestion signal was from.

So if this sender is trying to cheat, it will 
sometimes understate ConEx markings with respect 
to the first bottleneck alone. The audit at that 
first bottleneck will detect persistent cheating, 
so it knows the sender is suspect. An honest 
sender might accidentally get ConEx marks wrong 
occasionally, but not persistently badly like this.

>Measuring the bulk is very easy, but if all the 
>flows making up the bulk are understating their 
>congestion then the bulk will be understated as 
>well. Bob always had in mind a clever mechanism 
>to do spot checking at any border to identify 
>persistent under (or over) declaration, but I 
>never entirely understood how it was meant to work.

[BB]: The bulk auditing idea is primarily 
designed to remove the motivation for networks to 
launch attacks on /each other/ using ConEx. I 
don't expect many networks would even think of 
doing that, but unless you have a way of 
detecting breaches of trust, you cannot rely on 
that trust. You don't necessarily have to use the 
mechanism; other networks just need to know that 
there is feasible mechanism and you might be using it.

>OK, yes. That is a separate requirement for audit.

[BB]: Just to ensure we make clear which sort of 
abuse we're talking about at any one time, I 
detect three different types of abuse being discussed here:
a) The sender continuing to send despite 
expecting that congestion has not cleared
b) The sender understating ConEx markings relative to path congestion
c) An ISP clearing the sender's ConEx markings

a-type behaviour could describe normal TCP/IP or 
UDP. It's only abuse if in excess. The idea of 
ConEx is to give ISPs the info, so ISPs can draw 
the line between normal and abuse, rather than 
the IETF making this judgement call.

(b) and (c) are about validity of ConEx 
information, whereas (a) is about behaviour 
measured by ConEx info. The idea is to prevent 
b-type & c-type behaviour so that ConEx info is 
reliable enough that ISPs can use it to make judgements about (a).

b-type behaviour is intended to be addressed by audit.

c-type behaviour can be addressed by e2e verification of ConEx info.

We're not doing protocol work on (c) in the w-g, 
but the info is in the protocol if we wanted to. 
I can straight-off think of three possible protocols to address this:
- The source could protect the integrity of the 
DSOPT extension header with IPsec AH, so if an 
ISP altered ConEx markings, AH integrity checks 
would detect that something on-path had altered an immutable header.
- We could add an e2e feedback field in TCP so 
that the receiver can feed back the received 
ConEx markings to the sender (re-re-feedback?). 
Then the sender can check that an ISP hasn't changed what it sent.
- The sender could log the packets it marks with 
ConEx, and the receiver could occasionally send 
back a log of the packets it receives with ConEx 
markings for the sender to check (not sure what 
protocol this would be added to).

>Clearly, given the min 1RTT delay between 
>getting congestion info and responding to it, 
>there can never be a completely accurate match. 
>I did have numerous debates with Bob about this 
>topic. Basically you run the risk of a user just 
>terminating his flow rather than making up his losses.

[BB]: That attack is dealt with. That's why I/we 
decided that the sender has responsibility for 
keeping ConEx info greater than or equal to 
congestion, despite RTT delay. Then if the sender 
runs up a 'loss' and jumps to a new flow, the end 
of its last flow will have been discarded by the 
audit function, so the sender won't have gained by this cheat.

>You also have to be cautious that the actual 
>process of dropping packets for audit doesn't 
>simply distort the apparent congestion 
>information (because the sender will respond by 
>retransmitting and hopefully adding more ConEx marks)

[BB]: We (in BT) established that's also not a 
problem. The sender ought to match these audit 
losses with the ConEx marks it didn't send in the 
first place. If it doesn't, the audit node can 
detect that, because it knows about the losses it introduced.

> >   I absolutely agree with Toby here.

[BB]: I also agree audit is hard without ECN - 
but I believe it's not a lost cause (excuse the 
pun). I started to agree with Matt when I 
realised cases like that discussed earlier ('not 
the only way' above) are common enough for ConEx without ECN to be useful.

>OK, this comes down to the point below

[BB]:  Sampling isn't applicable if anyone who is 
'caught' can just whitewash their reputation by 
adopting a new clean identity. The problem is 
that identity here is just a flow ID and new flow IDs can be created at will.

A dishonest transport can cheat in as many flows 
as it can get away with, until sampling catches 
one of its flows, which it just terminates and 
starts another flow. E.g. a msbehaving receiver 
could understate congestion feedback to a server. 
When the receiver detects high loss, it assumes 
it is being randomly audited, so it just 
terminates that connection and opens a new one.

"Reputation whitewashing" is easy when your 
identity is as emphemeral as a flow ID and you're 
the one who controls the end with the ephemeral ports.

>That was certainly how I understood a re-ECN 
>like scheme working… Te whole core of this 
>argument (should you account for bytes or 
>packets) has always been a major bone of 
>contention between me and Bob. Not least because 
>it can be impossible for a sender to comply (it 
>is very easy to construct pathological sequences 
>of packets where the sender is forced to either 
>over or under declare if they are doing per-byte 
>auditing). I think this is a case where the IETF 
>view has to diverge from the researcher's view. 
>In the IETF we have to make prosaic engineering 
>decisions. Personally I would like ConEx to 
>account on a per-packet basis, BUT with an 
>allowance for it to be more accurate if an operator so wishes

[BB]: that's my position too.
Assuming "more accurate" means per-byte.

I don't think this is disagreeing with the 
wording in abstract-mech, or the proposed wording 
in the tcp mods draft. If it is, pls say how.

>(in other words MUST as a minimum account per 
>packet, but MAY account per byte)

Except I wouldn't state it in terms of what audit 
must or may do, because the argument is about 
what we say the transport should do (to be robust against what audit may do).

>Perhaps auditing is an unhelpful word to use. To 
>my mind an audit implies an accountancy audit 
>that tries to balance the books. I agree that 
>carries no implication of policing or 
>enforcement, but I think that a lot of this 
>debate has assumed the two concepts are identical

The critical issue is whether the audit function 
is possible. Once it is, the action as a result of audit can be added.

When audit is in a remote network from the 
sender, the only action available seems to be to 
'punish' the packets (e.g. drop) because it's too 
complex in general to track down and 'punish' a 
user that might be in another network.

>Agreed, this has to be a lightweight mechanism. 
>All the guarantees have to come from the idea 
>that operators have to have some level of trust 
>among one another (or at least, the realisation 
>that an operator found to have cheated at the 
>expense of others runs  risk of getting sat on heavily by the others),

See point above - in a large distributed system, 
trust can only be relied on if it would at least 
be feasible to detect breaches of trust (in the 
case of trust building, random sampling is highly appropriate).

>And therein lies the lesson for ConEx. We HAVE 
>to fix on a SIMPLE subset of ConEx that we can 
>all agree on and that seems at least feasible in 
>the real world. Arguments like the recent ones 
>make ConEx appear to be still firmly stuck in 
>the land of research (IRTF). Too many people (on 
>all sides of all arguments) seem to want 
>perfection and that is not sensible in the real 
>world. That is just a recipe for ratholing 
>ourselves to the point where the IESG declares our WG dysfunctional…

See above.

Bob

>Toby

________________________________________________________________
Bob Briscoe,                                BT Innovate & Design

---

From: Toby Moncaster <[email-masked]>
Date: Wed, 14 Dec 2011 16:43:17 +0000

Hi Bob,

On 13 Dec 2011, at 20:12, Bob Briscoe wrote:

> Inline, and I've snipped everything I agree with…

OK

> [BB]: I want to take issue with 'the only way'. There's at least one other very plausible scenario. Assume by design there's one major bottleneck on the upstream path from a sender (e.g. a radio access network). Then a network node at the congested point can see all the ConEx and most of the loss (or ECN). It cannot know what 'most' means, but bear with me...
> 
> Imagine a sender is trying to cheat. This can even be a really knowledgeable sender - a disgruntled ex-employee of the network operator who knows that the audit function is badly placed - not near the receiver, but near the sender.
> 
> Even if there might sometimes be another bottleneck nearer the receiver on the path, when this dishonest sender sees congestion feedback (from loss or ECN), it cannot know which bottleneck each congestion signal was from.
> 
> So if this sender is trying to cheat, it will sometimes understate ConEx markings with respect to the first bottleneck alone. The audit at that first bottleneck will detect persistent cheating, so it knows the sender is suspect. An honest sender might accidentally get ConEx marks wrong occasionally, but not persistently badly like this.

It would be interesting to get a feel for how common this scenario is. I think what you are saying is that the audit function doesn't have to be perfect, it just needs to be reasonably likely of catching persistent under-declaration of congestion?

> [BB]: The bulk auditing idea is primarily designed to remove the motivation for networks to launch attacks on /each other/ using ConEx. I don't expect many networks would even think of doing that, but unless you have a way of detecting breaches of trust, you cannot rely on that trust. You don't necessarily have to use the mechanism; other networks just need to know that there is feasible mechanism and you might be using it.

A key point to get across.

> [BB]: Just to ensure we make clear which sort of abuse we're talking about at any one time, I detect three different types of abuse being discussed here:
> a) The sender continuing to send despite expecting that congestion has not cleared
> a-type behaviour could describe normal TCP/IP or UDP. It's only abuse if in excess. The idea of ConEx is to give ISPs the info, so ISPs can draw the line between normal and abuse, rather than the IETF making this judgement call.
> 
> (b) and (c) are about validity of ConEx information, whereas (a) is about behaviour measured by ConEx info. The idea is to prevent b-type & c-type behaviour so that ConEx info is reliable enough that ISPs can use it to make judgements about (a).
> 

I think this is a really good and clear description of the ways to distort ConEx info. I would like to see something like this as a stand-alone informational draft (at least for now) so we have it captured somewhere and can refer back to it as and when needed.

> 
> We're not doing protocol work on (c) in the w-g, but the info is in the protocol if we wanted to. I can straight-off think of three possible protocols to address this:
> - The source could protect the integrity of the DSOPT extension header with IPsec AH, so if an ISP altered ConEx markings, AH integrity checks would detect that something on-path had altered an immutable header.
> - We could add an e2e feedback field in TCP so that the receiver can feed back the received ConEx markings to the sender (re-re-feedback?). Then the sender can check that an ISP hasn't changed what it sent.
> - The sender could log the packets it marks with ConEx, and the receiver could occasionally send back a log of the packets it receives with ConEx markings for the sender to check (not sure what protocol this would be added to).

I'm afraid I'm being lazy for now and not evaluating these suggestions.

>> Clearly, given the min 1RTT delay between getting congestion info and responding to it, there can never be a completely accurate match. I did have numerous debates with Bob about this topic. Basically you run the risk of a user just terminating his flow rather than making up his losses.
> 
> [BB]: That attack is dealt with. That's why I/we decided that the sender has responsibility for keeping ConEx info greater than or equal to congestion, despite RTT delay. Then if the sender runs up a 'loss' and jumps to a new flow, the end of its last flow will have been discarded by the audit function, so the sender won't have gained by this cheat.

I think this may be one of the things people haven't understood. Namely, we need any sender to start by assuming they will cause a little bit of congestion and paying for that up-front.

> 
>> You also have to be cautious that the actual process of dropping packets for audit doesn't simply distort the apparent congestion information (because the sender will respond by retransmitting and hopefully adding more ConEx marks)
> 
> [BB]: We (in BT) established that's also not a problem. The sender ought to match these audit losses with the ConEx marks it didn't send in the first place. If it doesn't, the audit node can detect that, because it knows about the losses it introduced.

The only drawback then is that the aggregate over the path is possibly distorted (as in, some of the stated congestion isn't actual congestion) which may have an impact on some future ConEx use cases.

> [BB]: I also agree audit is hard without ECN - but I believe it's not a lost cause (excuse the pun). I started to agree with Matt when I realised cases like that discussed earlier ('not the only way' above) are common enough for ConEx without ECN to be useful.

I never said it was impossible, it's just that ECN does give you a lot more information

> [BB]:  Sampling isn't applicable if anyone who is 'caught' can just whitewash their reputation by adopting a new clean identity. The problem is that identity here is just a flow ID and new flow IDs can be created at will.
> 
> A dishonest transport can cheat in as many flows as it can get away with, until sampling catches one of its flows, which it just terminates and starts another flow. E.g. a msbehaving receiver could understate congestion feedback to a server. When the receiver detects high loss, it assumes it is being randomly audited, so it just terminates that connection and opens a new one.
> 
> "Reputation whitewashing" is easy when your identity is as emphemeral as a flow ID and you're the one who controls the end with the ephemeral ports.

So what you are arguing is that speed camera-style auditing is not sensible because in the Internet you can change your car registration at will with near zero cost, and there is no external mechanism (licensing authority) that can retrospectively apply a sanction to you? (sorry for the convoluted simile)

>> That was certainly how I understood a re-ECN like scheme working… Te whole core of this argument (should you account for bytes or packets) has always been a major bone of contention between me and Bob. Not least because it can be impossible for a sender to comply (it is very easy to construct pathological sequences of packets where the sender is forced to either over or under declare if they are doing per-byte auditing). I think this is a case where the IETF view has to diverge from the researcher's view. In the IETF we have to make prosaic engineering decisions. Personally I would like ConEx to account on a per-packet basis, BUT with an allowance for it to be more accurate if an operator so wishes
> 

That was my implication, yes ;)

> 
> I don't think this is disagreeing with the wording in abstract-mech, or the proposed wording in the tcp mods draft. If it is, pls say how.
> 
>> (in other words MUST as a minimum account per packet, but MAY account per byte)
> 
> Except I wouldn't state it in terms of what audit must or may do, because the argument is about what we say the transport should do (to be robust against what audit may do).

OK, but we probably need to have some minimal black-box definition of what an audit function needs to do to be "compliant"

> The critical issue is whether the audit function is possible. Once it is, the action as a result of audit can be added.

I agree we clearly have two functions - audit and reaction to audit. Audit in and of itself simply says there is an issue.

> 
> When audit is in a remote network from the sender, the only action available seems to be to 'punish' the packets (e.g. drop) because it's too complex in general to track down and 'punish' a user that might be in another network.

Agreed

> See point above - in a large distributed system, trust can only be relied on if it would at least be feasible to detect breaches of trust (in the case of trust building, random sampling is highly appropriate).

Absolutely. Among operators you can genuinely rely on the speed camera approach to audit - the risk of getting caught may be low (but has to be finite) but the harm to your reputation makes the risk unjustifiable 

>> And therein lies the lesson for ConEx. We HAVE to fix on a SIMPLE subset of ConEx that we can all agree on and that seems at least feasible in the real world. Arguments like the recent ones make C…

I actually think the disagreements among us are quite small and are (generally) semantic. The problem is that they look much bigger to any outsider. 

Toby

---

From: John Leslie <[email-masked]>
Date: Thu, 15 Dec 2011 15:07:01 -0500

Toby and Bob raise a number of points I will follow-up in separate threads,
but one point deserves to be in this thread.

Toby Moncaster <[email-masked]> wrote:
>>> The whole core of this argument (should you account for bytes or
>>> packets) has always been a major bone of contention between me and
>>> Bob. Not least because it can be impossible for a sender to comply
>>> (it is very easy to construct pathological sequences of packets
>>> where the sender is forced to either over or under declare if they
>>> are doing per-byte auditing). I think this is a case where the
>>> IETF view has to diverge from the researcher's view. In the IETF
>>> we have to make prosaic engineering decisions. Personally I would
>>> like ConEx to account on a per-packet basis, BUT with an allowance
>>> for it to be more accurate if an operator so wishes
>> 

   I must dissent on calling per-byte "more accurate" than per-packet
when the number of packets is clearly defined and the number of bytes
is inferred (and in at least some cases, inaccurate).

   Bob seems immovable in his belief that per-byte accounting is
"more correct", despite a number of posts showing cases where the
per-packet overhead dominates.

   I feel just as immovable in claiming that neither will _always_
dominate. I prefer counting packets because it's simpler; I'm willing
to count bytes if a clear benefit is demonstrated during Experimental
phase.

   Thus it's time for a specific proposal in compromise.

   I propose that the ConEx Destination Option carry a 16-bit field
for byte-count, and this field be optional -- both for the sender to
fill or not, and for any node along the path to check or not.

   Thus, during the Experimental phase, any sender could set it and
see what changes (if anything); and any node along the path could
gather data on actual variations in this number and draw conclusions
about potential benefits of its use.

   IMHO, unless senders voluntarily set it and at least some nodes
voluntarily use it, there _can_be_ no benefit. But I'm willing to
leave the field in place against a possible future use. (I hope to
live long enough to see 4k packets become common in the wild; this
could expose a greater benefit from considering bytes per packet.)

   I'm not willing to "go along" with Bob on _inferring_ a byte count
and calling that more accurate; but I'm willing to accept an explicit
byte count with exact details of how the auditing function may use
it left for future work.

---

From: John Leslie <[email-masked]>
Date: Thu, 15 Dec 2011 16:18:01 -0500

This is still rather longish, but I think it holds together enough...

Toby Moncaster <[email-masked]> wrote:
>> Imagine a sender is trying to cheat. This can even be a really
>> knowledgeable sender - a disgruntled ex-employee of the network
>> operator who knows that the audit function is badly placed - not
>> near the receiver, but near the sender.
>>
>> Even if there might sometimes be another bottleneck nearer the
>> receiver on the path, when this dishonest sender sees congestion
>> feedback (from loss or ECN), it cannot know which bottleneck each
>> congestion signal was from.

   The most likely bottleneck near the sender is the uplink from the
customer of the ADSL or Cable modem.

>> So if this sender is trying to cheat, it will sometimes understate
>> ConEx markings with respect to the first bottleneck alone.

   Assuming Bob means the cheater _is_ the sender, it can still have
a pretty good picture of the congestion rate in its upstream link.

>> The audit at that first bottleneck will detect persistent cheating,
>> so it knows the sender is suspect...

   Understand that ConEx credit marks can get lost: the audit function
can't expect to see _every_ ConEx mark.

   Even assuming that the cheating is sufficiently flagrant, I don't
see how we could specify what the audit function must do...

>> [BB]: Just to ensure we make clear which sort of abuse we're talking
>> about at any one time, I detect three different types of abuse being
>> discussed here:
>> a-type behaviour could describe normal TCP/IP or UDP. It's only abuse
>> if in excess. The idea of ConEx is to give ISPs the info, so ISPs
>> can draw the line between normal and abuse, rather than the IETF
>> making this judgement call.

   I agree the IETF shouldn't make that judgment call.

>> (b) and (c) are about validity of ConEx information, whereas (a) is
>> about behaviour measured by ConEx info. The idea is to prevent
>> b-type & c-type behaviour so that ConEx info is reliable enough
>> that ISPs can use it to make judgements about (a).
>>
> I think this is a really good and clear description of the ways to
> distort ConEx info. I would like to see something like this as a
> stand-alone informational draft (at least for now) so we have it
> captured somewhere and can refer back to it as and when needed.

   I'd love to see Toby work on such an I-D. I'd be willing to help.

>> We're not doing protocol work on (c) in the w-g, but the info is
>> in the protocol if we wanted to. I can straight-off think of three
>> possible protocols to address this:
>> - The source could protect the integrity of the DSOPT extension
>>   header with IPsec AH, so if an ISP altered ConEx markings, AH
>>   integrity checks would detect that something on-path had altered
>>   an immutable header.
>> - We could add an e2e feedback field in TCP so that the receiver
>>   can feed back the received ConEx markings to the sender
>>   (re-re-feedback?). Then the sender can check that an ISP hasn't
>>   changed what it sent.
>> - The sender could log the packets it marks with ConEx, and the
>>   receiver could occasionally send back a log of the packets it
>>   receives with ConEx markings for the sender to check (not sure
>>   what protocol this would be added to).

   I don't believe the IPsec case is worth pursuing.

   The second and third might prove worthwhile, but at this point I
don't see what we could specify which would help.

>>> Clearly, given the min 1RTT delay between getting congestion info
>>> and responding to it, there can never be a completely accurate
>>> match. I did have numerous debates with Bob about this topic.
>>> Basically you run the risk of a user just terminating his flow
>>> rather than making up his losses.
>>
>> [BB]: That attack is dealt with. That's why I/we decided that the
>> sender has responsibility for keeping ConEx info greater than or
>> equal to congestion, despite RTT delay. Then if the sender runs up
>> a 'loss' and jumps to a new flow, the end of its last flow will
>> have been discarded by the audit function, so the sender won't have
>> gained by this cheat.
>
> I think this may be one of the things people haven't understood.
> Namely, we need any sender to start by assuming they will cause a
> little bit of congestion and paying for that up-front.

   Worth including in that I-D I hope Toby will write...

   Nonetheless, I don't think we should make any requirement for ConEx
credits for _every_ packet in flight (which I have seen proposed).

   While, indeed, 100% loss is possible, that's not a case worth a lot
of worry -- it's essentially a dead path, and optimizing a dead path
seems silly.

   Nor do I think we should _define_ a likely loss rate for packets
in flight. That deserves to be a judgment call for each sender. The
question is, if the sender guesses low, what should happen?

   It doesn't bother me if an audit policer drops some more packets
in that case; nor does it bother me if it doesn't. It's not worth the
effort to me to keep that much state about connections originating at
some other ISP.

   But I'm happy to see a statement that senders SHOULD estimate a
likely loss rate for packets in flight.

>>> You also have to be cautious that the actual process of dropping
>>> packets for audit doesn't simply distort the apparent congestion
>>> information (because the sender will respond by retransmitting
>>> and hopefully adding more ConEx marks)
>>
>> [BB]: We (in BT) established that's also not a problem. The sender
>> ought to match these audit losses with the ConEx marks it didn't
>> send in the first place. If it doesn't, the audit node can detect
>> that, because it knows about the losses it introduced.

   (Assuming a single audit node...)

> The only drawback then is that the aggregate over the path is possibly
> distorted (as in, some of the stated congestion isn't actual congestion)
> which may have an impact on some future ConEx use cases.

   IMHO, we're going to have to live with cases of overstated congestion;
and it isn't worth the effort of trying to stop them.

>> [BB]:  Sampling isn't applicable if anyone who is 'caught' can just
>> whitewash their reputation by adopting a new clean identity. The
>> problem is that identity here is just a flow ID and new flow IDs
>> can be created at will.

   Actually, I think we still waving hands wildly on what constitutes
a "flow" as we get farther from the sender. The IPv6 "flow ID" field
is almost never used, and proposals I've seen to get it used more seem
a bit buggy...

>> A dishonest transport can cheat in as many flows as it can get away
>> with, until sampling catches one of its flows, which it just
>> terminates and starts another flow. E.g. a misbehaving receiver
>> could understate congestion feedback to a server. When the receiver
>> detects high loss, it assumes it is being randomly audited, so it
>> just terminates that connection and opens a new one.

   In truth, just the escalation of state information to be kept would
be a problem. (And there probably isn't even a need to terminate a
"connection".)

   IMHO, keeping per-flow state at all far from the sender (or the
receiver, if the receiver is being held responsible for congestion) is
a non-starter: I just don't see how we can limit the state information
to a manageable level.

   OTOH, the overall amount of congestion coming into my AS from each
peer is a concern worthy of keeping per-AS state: thus I expect per-AS
state to be kept and acted upon. For purposes of per-AS state, it
doesn't help to argue whether it's honest or dishonest: either way, it
threatens to congest _all_ my traffic to the particular destination.

>> "Reputation whitewashing" is easy when your identity is as emphemeral
>> as a flow ID and you're the one who controls the end with the
>> ephemeral ports.

   Agreed. But reputation of my peer AS's can't be whitewashed.

> So what you are arguing is that speed camera-style auditing is not
> sensible because in the Internet you can change your car registration
> at will with near zero cost, and there is no external mechanism
> (licensing authority) that can retrospectively apply a sanction to
> you? (sorry for the convoluted simile)

   I'll argue that one: speed-camera is not particularly sensible for
blaming individual _flows_ because we have no protection of any of the
available methods to identify the flows to blame.

>> When audit is in a remote network from the sender, the only action
>> available seems to be to 'punish' the packets (e.g. drop) because
>> it's too complex in general to track down and 'punish' a user that
>> might be in another network.
>
> Agreed

   Likewise, agreed -- with the additional note that matching the drops
to the actual flow will not always be possible.

   Nonetheless, I would agree it's reasonable to drop packets that _are_
marked ConEx-aware but appear to be part of a flow which has not built up
sufficient credit -- in those cases where we have reason to believe
_some_ packets to that destination will have to be dropped anyway.

   I don't honestly see much benefit from "punishing" a failure to
estimate enough congestion unless we have evidence that packet drop is
inevitable (somewhere) anyway. (If nothing else, that would be a DoS
invitation.)

>> See point above - in a large distributed system, trust can only be
>> relied on if it would at least be feasible to detect breaches of
>> trust (in the case of trust building, random sampling is highly
>> appropriate).
>
> Absolutely. Among operators you can genuinely rely on the speed camera
> approach to audit - the risk of getting caught may be low (but has to
> be finite) but the harm to your reputation makes the risk unjustifiable

   Exactly!

>>> And therein lies the lesson for ConEx. We HAVE to fix on a SIMPLE
>>> subset of ConEx that we can all agree on and that seems at least
>>> feasible in the real world. Arguments like the recent ones make
>>> ConEx appear to be still firmly stuck in the land of research
>>> (IRTF). Too many people (on all sides of all arguments) seem to
>>> want perfection and that is not sensible in the real world.
>>> That is just a recipe for ratholing ourselves to the point where
>>> the IESG declares our WG dysfunctional?
>>
>> See above.
>
> I actually think the disagreements among us are quite small and are
> (generally) semantic. The problem is that they look much bigger to
> any outsider.

   IMHO our disagreements, whether small or not, are often fundamental;
and our intransigence is what makes the problems appear so big. To
whatever extent we are willing to let go of our insistence upon any
particular solution, I think folks are happy to see this work continue;
but where we hang on too tightly to one particular "solution", we start
to look "dysfunctional". :^(