Subject: Re: [conex] byte vs packet counting

From: Bob Briscoe <[email-masked]>
Date: Sun, 20 Nov 2011 19:56:14 +0000

Matt,

[Adding the ConEx list back in]

I think we're both now agreeing that the /goal/ should be 
byte-balance for ConEx auditing. Correct?

I am also in full agreement that this goal isn't always easy to meet 
if retrofitting ConEx onto an existing transport like TCP. It works 
if the packet sizes are regular, but if they are lumpy, the sender is 
likely to understate or overstate. And I agree that these aren't 
necessarily just corner-cases (well, I guess there is a large set of 
cases with regular packet sizes, but the corner is also quite large).

You ask what I think we should do about this. My view, we should:
a) state what the goal should be for all transports (byte-balance)
b) give advice on how to implement a TCP sender to do as well as it can
c) run experiments and see whether the outcome is good enough, or if 
further protocol mechanism is needed.

Concerning (b) advice on the best that TCP implementations can do:
- the TCP receiver (if optionally supporting ConEx) should suppress 
ACK delay whenever it detects congestion and instead give immediate feedback
- TCP sender can use ack sequence numbers to improve its guess of the 
required size of an re-echo-loss or re-echo-ECN
- If the receiver is not ConEx-aware, a sender that knows it is 
sending lumpy packet sizes can introduce additional credit to cover 
any mistakes.

If a ConEx sender does unintentionally understate bytes of 
congestion, the auditor will discard some packets, then the sender 
will redress the balance with some more re-echo-loss and it should 
all heal itself, albeit having suffered some losses in the process.

Do you think this is sufficient (at least to run experiments to test 
whether it is)?

Bob

At 20:55 18/11/2011, Matt Mathis wrote:
>So here is a pathology: I have an application that sends 15001 bytes 
>(10 segments + 1 byte), followed by 1 byte beacons every 10 second 
>for 100 seconds and then the entire pattern repeats (total of 
>10*1500 + 10*1 segments every 100 seconds).
>
>Now suppose there are ECN marks near the end of the burst on every 
>repeat.   How does this get signaled, and what is the correct sender 
>response?   Due to delayed ACKs it may be ambiguous exactly which 
>segment was marked (e.g. was it 1500 or 1 bytes?)   Even if the 
>sender knows for sure that it was the 1500 byte segment that was 
>marked, it has to wait until the next burst to send the proper feedback.
>
>The problem is the ACKs carry the count of the number of marked 
>segments, not marked bytes.  Also the re-feed itself has no size, 
>except the length of the segments carrying it.
>
>What do you think should happen in this case?
>(BTW modern persistent web protocols, such as SPDY do stuff like 
>this all the time, although perhaps not as extreme as my 
>example.   BGP also has streaming data with very irregular message sizes.)
>
>This would be completely clear if the ACKs and re-feedback were 
>counts of bytes.
>
>Thanks,
>--MM--
>The best way to predict the future is to create it.  - Alan Kay
>
>
>
>On Thu, Nov 17, 2011 at 7:07 PM, Matt Mathis 
><<mailto:[email-masked]>[email-masked]> wrote:
>
>I'm sorry I did not make my self clear. I understand and agree with 
>your argument.
>
>The problem is when the forward path is carrying lumpy data : highly 
>irregular segment sizes, typical of streaming media, p-http, or bgp 
>at edge of the onset of congestion. When there is an ecn mark the 
>ack  only indicates that a segment was marked and not its size. If 
>the sender has a choice of segments to re-feedback it has has no 
>idea which to choose.  Worse it
>may not have a choice.   Furthermore under pathological conditions 
>it will persistently get it wrong.
>
>I can explain better when I have a real. Keyboard.
>On Nov 17, 2011 10:02 PM, "Bob Briscoe" 
><<mailto:[email-masked]>[email-masked]> wrote:
>Matt,
>
>This issue has come up in at least two threads recently: 
>"Congestion" vs. "Congestion Volume" and "byte-counting in conex-destopt"
>
>Consider the buffer into the high speed line you were talking about 
>this morning - it holds packets in equal MTU-sized packet buffers, 
>whatever the packet size.
>
>When it buffers packets, we need to ask what exactly is congested: 
>the line or the buffer? In this case, the line is congested. Packets 
>are temporarily in the buffer because more bits arrived than 
>departed. The queue is merely a symptom of how congested the line 
>is. Certainly, once the buffer gets full, we could say the buffer 
>has become congested too. But the root of the problem is how fast 
>the line can carry away the bits.
>
>If the queue was building up behind forwarding look-ups or a 
>firewall, then the problem would be the number of packets headers, 
>not bits. But if it's a queue into a transmission line, the problem is bits.
>
>The point is, whether to count bits or packets depends on the 
>process /in front/ of the queue (whether its a bit transmission line 
>or processing packet headers). The internals of the buffer itself 
>are irrelevant.
>
>Then the question is how prevalent are per-packet processes as 
>sources of congestion? Answer: There seems to be good reason why 
>per-packet congestion will remain in the minority relative to per-byte...
>
>During the process of writing draft-ietf-byte-pkt-congest, all the 
>machine design folk who were consulted said that machines are 
>generally designed so that any per-packet-processing can cope with a 
>workload at line rate consisting mostly of small packets.
>
>IOW, machine designs tend to use bit-congestion to protect the 
>packet-processor from congestion.
>
>____________________
>For those who prefer an example, assume:
>MTU,       M = 1,500B = 12,000b
>Line rate, X =    48Gbps
>
>[The numbers are chosen to make the maths easy, not to reflect 
>typical scenarios. I'm going to work in bits not bytes from now on.]
>
>Imagine at some time that 260 of these fixed-size packet buffers are 
>full, with 10 large and 250 small packets.
>
>packet size       | S    | 12,000b  |   480b   |
>no. pkts buffered | N    |     10   |   250    |
>buffer space used | NM   |    120kb |     3Mb  |
>pkt bits buffered | NS   |    120kb |   120kb  |
>time to drain all | NS/X |  2,500ns | 2,500ns  |
>
>Although each small packet takes up the same space in the buffer as 
>a large packet, it's faster to get rid of it. 250 small packets take 
>as long to drain as 10 large packets.
>
>
>Bob
>
>
>________________________________________________________________
>Bob Briscoe,                                BT Innovate & Design
>

________________________________________________________________
Bob Briscoe,                                BT Innovate & Design

---

From: John Leslie <[email-masked]>
Date: Sun, 20 Nov 2011 16:40:12 -0500

> Matt,

   (I do understand that Bob is addressing Matt...)
 
> [Adding the ConEx list back in]

   I do not agree that this is a discussion which should gate on the
opinions of Bob and Matt.

   I do understand that both of these folks think they understand the
mathematical issues involved better than the rest of us. They may well
be right.

   But this is an IETF Working Group, not an academic paper review
board. I'm not interested in arguing mathematics -- I believe we need
to discuss how to get a protocol which enough operators will implement.
And I'm interested in arguments of what will harm our chances of
widespread implementation.

> byte-balance for ConEx auditing. Correct?

   Hopefully it's no surpprise to anyone that _I_ don't agree.

> likely to understate or overstate.

   I can agree with that. (It seems to me that that's a potential
barrier to implementation.)

> And I agree that these aren't necessarily just corner-cases (well, I
> guess there is a large set of cases with regular packet sizes, but
> the corner is also quite large).

   Again, I agree.

> further protocol mechanism is needed.

a) I see no benefit in _merely_ stating a goal: if indeed the goal is
   worthwhile (of which I remain unconvinced), we need to justify the
   additional effort which may be required to meet it.

b) I see danger (not benefit) from asking implementors to do much in
   the way of modifications to TCP. I believe our goal is unattainable
   unless we can demonstrate benefit for flows where no TCP modifications
   are needed. (Our charter clearly allows for modification to the
   _receiver_ TCP implementations; but we should promote that, if at
   all, as an _additional_ benefit of ConEx with receiver modifications.)

c) I have _no_ problem with running experiments with sender modifications
   to see if there _is_ a benefit from byte-balance. My problem is with
   jawboning about the abstruse benefits of byte-balance in our _initial_
   documents.

>   sending lumpy packet sizes can introduce additional credit to cover 

- I support anything to encourage a faster feedback loop for reacting
  to congestion signals. I only suggest that asking for any particular
  changes to TCP _at_this_point_ may be counterproductive.

- I believe that what a modified TCP sender might do in reaction to
  congestion should follow from widespread experimental experience.

- I have a problem with asking a sender to _guess_ at what ConEx signal
  is needed in response to a congestion event.

> all heal itself, albeit having suffered some losses in the process.

   IMHO, we're not ready to prescribe any particular _policing_ actions.
"Auditor" to me means a point of collection of information, not a point
of enforcement.

   Enforcement, to me, is premature in our primary documents now being
LastCalled. No doubt Bob feels confident he can prescribe enforcement
actions which are mathematically correct. Perhaps he can.

   But to me, describing _any_ policing at this time is dangerous.
Folks _will_ implement policing according to their _own_ interpretation
of the information available -- not necessarily the way we describe. :^(

> Do you think this is sufficient (at least to run experiments to test 

   I think we would do well to run experiments an that basis.

   (I do not think we should publish initial documents claiming the
"rightness" of what such experiments seek to confirm.)

--
John Leslie <[email-masked]>

---

From: Bob Briscoe <[email-masked]>
Date: Sun, 20 Nov 2011 23:26:11 +0000

John,

Terminology for this email [abstract-mech]:
   i) Policing /based on/ ConEx information and
   ii) Audit /of/ ConEx information
are distinct functions.

I fully agree that policing will be done how operators want to do it, 
and we have no need to standardise anything about it.

But audit is about checking one stream of signals against another. It 
is a factual function, not a policy-related function. Without audit, 
ConEx information is unreliable for policing, monitoring or anything. 
I will now show that, if we don't define what fact the audit function 
is meant to be checking, ConEx cannot work.

Imagine an Internet where it is undefined whether ConEx and 
congestion signals should balance per packet or per byte. Then both 
types of audit and both types of transport will be implemented.

Let's assume some byte-based audit devices will discard traffic that 
understates ConEx bytes. And some packet-based audit devices will 
discard understatement in packets. Then these packet-balanced 
auditors will drop byte-balanced transports. And vice-versa.

ConEx audit will become a machine for dropping large proportions of packets.

Please explain how you propose to prevent this disaster if we leave 
the byte-packet question open.

I am all for allowing flexibility in standards wherever possible, in 
order to encourage diversity and innovation. There can certainly be 
all sorts of implementations of the audit function, but we have to 
minimally define whether audit is per byte or per packet - for 
interoperability.

Can we find some way of resolving this argument, rather than keep 
revisiting it?

Bob

---

From: Fred Baker <[email-masked]>
Date: Mon, 21 Nov 2011 07:41:10 +0800

on the matter of bytes and packets, as I tried (apparently unsuccessfully) to say in the other email, what I am worried about first is time, which both bytes and packets are a readily-measured analog to. When I probed to see what interface someone knew about that sent messages as messages rather than a string of bits, I was told that WiFi measurably operates with an upper bound packet rate and is the poster child for packet-based interfaces. Great, but there is no single device that can perceive or measure it that way - wonderful academic observation, but I have no idea how to use it in an implementation. The point on a WiFi interface is neither that I have too many bytes or packets queued up; the point is that one of those is true in the context of a congested shared medium, and we *also* have to get control of the medium.

I find myself interested in bytes-as-an-analog-for-time on a non-shared interface and time on a shared interface, in both cases because I can measure them in an implementation.

On Nov 21, 2011, at 3:56 AM, Bob Briscoe wrote:

> I think we're both now agreeing that the /goal/ should be byte-balance for ConEx auditing. Correct?
> 
> I am also in full agreement that this goal isn't always easy to meet if retrofitting ConEx onto an existing transport like TCP. It works if the packet sizes are regular, but if they are lumpy, the sender is likely to understate or overstate. And I agree that these aren't necessarily just corner-cases (well, I guess there is a large set of cases with regular packet sizes, but the corner is also quite large).
> 
> c) run experiments and see whether the outcome is good enough, or if further protocol mechanism is needed.
> 
> - the TCP receiver (if optionally supporting ConEx) should suppress ACK delay whenever it detects congestion and instead give immediate feedback 
> - TCP sender can use ack sequence numbers to improve its guess of the required size of an re-echo-loss or re-echo-ECN
> - If the receiver is not ConEx-aware, a sender that knows it is sending lumpy packet sizes can introduce additional credit to cover any mistakes.
> 
> If a ConEx sender does unintentionally understate bytes of congestion, the auditor will discard some packets, then the sender will redress the balance with some more re-echo-loss and it should all heal itself, albeit having suffered some losses in the process.
> 
> Do you think this is sufficient (at least to run experiments to test whether it is)?
> 
> 
> 
> Bob
> 
>> So here is a pathology: I have an application that sends 15001 bytes (10 segments + 1 byte), followed by 1 byte beacons every 10 second for 100 seconds and then the entire pattern repeats (total of 10*1500 + 10*1 segments every 100 seconds).  
>> 
>> Now suppose there are ECN marks near the end of the burst on every repeat.   How does this get signaled, and what is the correct sender response?   Due to delayed ACKs it may be ambiguous exactly which segment was marked (e.g. was it 1500 or 1 bytes?)   Even if the sender knows for sure that it was the 1500 byte segment that was marked, it has to wait until the next burst to send the proper feedback.
>> 
>> The problem is the ACKs carry the count of the number of marked segments, not marked bytes.  Also the re-feed itself has no size, except the length of the segments carrying it.
>> 
>> (BTW modern persistent web protocols, such as SPDY do stuff like this all the time, although perhaps not as extreme as my example.   BGP also has streaming data with very irregular message sizes.)
>> 
>> This would be completely clear if the ACKs and re-feedback were counts of bytes.
>> 
>> Thanks,
>> --MM--
>> On Thu, Nov 17, 2011 at 7:07 PM, Matt Mathis <[email-masked]> wrote:
>> 
>> I'm sorry I did not make my self clear. I understand and agree with your argument.
>> 
>> The problem is when the forward path is carrying lumpy data : highly irregular segment sizes, typical of streaming media, p-http, or bgp at edge of the onset of congestion. When there is an ecn mark the ack  only indicates that a segment was marked and not its size. If the sender has a choice of segments to re-feedback it has has no idea which to choose.  Worse it
>> may not have a choice.   Furthermore under pathological conditions it will persistently get it wrong.  
>> 
>> On Nov 17, 2011 10:02 PM, "Bob Briscoe" <[email-masked]> wrote:
>> Matt,
>> 
>> This issue has come up in at least two threads recently: "Congestion" vs. "Congestion Volume" and "byte-counting in conex-destopt"
>> 
>> Consider the buffer into the high speed line you were talking about this morning - it holds packets in equal MTU-sized packet buffers, whatever the packet size.
>> 
>> When it buffers packets, we need to ask what exactly is congested: the line or the buffer? In this case, the line is congested. Packets are temporarily in the buffer because more bits arrived than departed. The queue is merely a symptom of how congested the line is. Certainly, once the buffer gets full, we could say the buffer has become congested too. But the root of the problem is how fast the line can carry away the bits.
>> 
>> If the queue was building up behind forwarding look-ups or a firewall, then the problem would be the number of packets headers, not bits. But if it's a queue into a transmission line, the problem is bits.
>> 
>> The point is, whether to count bits or packets depends on the process /in front/ of the queue (whether its a bit transmission line or processing packet headers). The internals of the buffer itself are irrelevant.
>> 
>> Then the question is how prevalent are per-packet processes as sources of congestion? Answer: There seems to be good reason why per-packet congestion will remain in the minority relative to per-byte...
>> 
>> During the process of writing draft-ietf-byte-pkt-congest, all the machine design folk who were consulted said that machines are generally designed so that any per-packet-processing can cope with a workload at line rate consisting mostly of small packets.
>> 
>> IOW, machine designs tend to use bit-congestion to protect the packet-processor from congestion.
>> 
>> ____________________
>> [The numbers are chosen to make the maths easy, not to reflect typical scenarios. I'm going to work in bits not bytes from now on.]
>> 
>> Imagine at some time that 260 of these fixed-size packet buffers are full, with 10 large and 250 small packets.
>> 
>> Although each small packet takes up the same space in the buffer as a large packet, it's faster to get rid of it. 250 small packets take as long to drain as 10 large packets.
>> 
>> 
>> Bob
>> 
>> 
> _______________________________________________
> conex mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/conex

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 21 Nov 2011 09:22:12 +0000

Fred,

Thanks for this. Yes time is a useful metric to think of.

When you say the upper bound to the WiFi sending rate is in pkt /sec, 
is this the upper bound of one interface, or of the shared medium (spectrum)?

When we implemented a measure of WiFi congestion (in the RTS/CTS 
style of 802.11), we used the rate of not getting back a CTS as an 
indication of congestion of the shared spectrum. The underlying 
congestion of the spectrum that led to it not being clear-to-send 
came from bits per second. But the interface controls this on a 
per-packet basis. Is this what you mean?

In infrastructure mode, the uplink is congested by interference at 
the receiving antenna, which depends on the intensity of bits in the 
air. The downlink gets congested when the antenna hits its power 
limit, and power depends on bits sent. So in both cases, the shared 
medium gets congested by bits. But the particular protocol used can 
consume the shared medium somewhat with signalling which creates a 
slight per-packet base load.

Bob

At 23:41 20/11/2011, Fred Baker wrote:

>on the matter of bytes and packets, as I tried (apparently 
>unsuccessfully) to say in the other email, what I am worried about 
>first is time, which both bytes and packets are a readily-measured 
>analog to. When I probed to see what interface someone knew about 
>that sent messages as messages rather than a string of bits, I was 
>told that WiFi measurably operates with an upper bound packet rate 
>and is the poster child for packet-based interfaces. Great, but 
>there is no single device that can perceive or measure it that way - 
>wonderful academic observation, but I have no idea how to use it in 
>an implementation. The point on a WiFi interface is neither that I 
>have too many bytes or packets queued up; the point is that one of 
>those is true in the context of a congested shared medium, and we 
>*also* have to get control of the medium.
>
>I find myself interested in bytes-as-an-analog-for-time on a 
>non-shared interface and time on a shared interface, in both cases 
>because I can measure them in an implementation.
>
>>I am also in full agreement that this goal isn't always easy to 
>>meet if retrofitting ConEx onto an existing transport like TCP. It 
>>works if the packet sizes are regular, but if they are lumpy, the 
>>sender is likely to understate or overstate. And I agree that these 
>>aren't necessarily just corner-cases (well, I guess there is a 
>>large set of cases with regular packet sizes, but the corner is 
>>also quite large).
>>
>>- TCP sender can use ack sequence numbers to improve its guess of 
>>the required size of an re-echo-loss or re-echo-ECN
>>>So here is a pathology: I have an application that sends 15001 
>>>bytes (10 segments + 1 byte), followed by 1 byte beacons every 10 
>>>second for 100 seconds and then the entire pattern repeats (total 
>>>of 10*1500 + 10*1 segments every 100 seconds).
>>>
>>>repeat.   How does this get signaled, and what is the correct 
>>>sender response?   Due to delayed ACKs it may be ambiguous exactly 
>>>which segment was marked (e.g. was it 1500 or 1 bytes?)   Even if 
>>>the sender knows for sure that it was the 1500 byte segment that 
>>>was marked, it has to wait until the next burst to send the proper feedback.
>>>
>>>The problem is when the forward path is carrying lumpy data : 
>>>highly irregular segment sizes, typical of streaming media, 
>>>p-http, or bgp at edge of the onset of congestion. When there is 
>>>an ecn mark the ack  only indicates that a segment was marked and 
>>>not its size. If the sender has a choice of segments to 
>>>re-feedback it has has no idea which to choose.  Worse it
>>>Consider the buffer into the high speed line you were talking 
>>>about this morning - it holds packets in equal MTU-sized packet 
>>>buffers, whatever the packet size.
>>>the line or the buffer? In this case, the line is congested. 
>>>Packets are temporarily in the buffer because more bits arrived 
>>>than departed. The queue is merely a symptom of how congested the 
>>>line is. Certainly, once the buffer gets full, we could say the 
>>>buffer has become congested too. But the root of the problem is 
>>>how fast the line can carry away the bits.
>>>process /in front/ of the queue (whether its a bit transmission 
>>>line or processing packet headers). The internals of the buffer 
>>>itself are irrelevant.
>>>generally designed so that any per-packet-processing can cope with 
>>>a workload at line rate consisting mostly of small packets.
>>>Imagine at some time that 260 of these fixed-size packet buffers 
>>>are full, with 10 large and 250 small packets.
>>>Although each small packet takes up the same space in the buffer 
>>>as a large packet, it's faster to get rid of it. 250 small packets 
>>>take as long to drain as 10 large packets.
>>>
>>>Bob
>>>
>><mailto:[email-masked]>[email-masked]

---

From: John Leslie <[email-masked]>
Date: Mon, 21 Nov 2011 15:33:56 -0500

>   ii) Audit /of/ ConEx information

   Agreed.

> and we have no need to standardise anything about it.

   Good!

> But audit is about checking one stream of signals against another.

   "Audit" tends to imply sampling and verifying an approximate match
of things sampled. I'm afraid we've been a bit too vague about the
sampling ratio...

> It is a factual function, not a policy-related function.

   Umm... the sampling is somewhat policy-related...

> Without audit, ConEx information is unreliable for policing,

   Actually, I don't agree with this. It's perfectly reasonable to
design a policing function based on allowances, without reference to
the accuracy of congestion prediction/responses.

> monitoring or anything.

   Oh, hardly: monitoring is the first function of research; it need
not depend on any particular balance of the things monitored.
 
> types of audit and both types of transport will be implemented.

   So stipulated...

> auditors will drop byte-balanced transports. And vice-versa.

   I don't agree. Knowing that some implementations count packets and
others count bytes, it would be somewhat stupid to punish based upon
your personal preference between these two.

> ConEx audit will become a machine for dropping large proportions of

   I don't follow... how could "audit" be a "machine for dropping"
anything?

> the byte-packet question open.

   Actually, that's simplicity personified -- define ConEx marking
with a byte-count field: if left blank the sender is byte-agnostic.
(Though, to tell truth, I believe such a field SHOULD be optional,
and I see no particular reason we need it at all...)

> minimally define whether audit is per byte or per packet - for 

   I'm quite happy to define it "per packet", for now...

> Can we find some way of resolving this argument, rather than keep 

   I believe _we_ could...

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 21 Nov 2011 23:14:31 +0000

John,

>of things sampled. I'm afraid we've been a bit too vague about the

[BB]: You're thinking of random audit. There is no implication of 
randomness or sampling in the word audit alone AFAIK.

>    Umm... the sampling is somewhat policy-related...

[BB]: This debate is over whether we must define /what/ is being 
audited. Sampling is about /how/: we all agree that ConEx docs should 
not constrain the 'how'.

>the accuracy of congestion prediction/responses.

[BB]: This assertion is missing one hugely important piece of 
context. ConEx information is volunteered by the sender.

As soon as the sender knows the info could be used against its 
interests, it has no reason to speak the truth. So the network will 
only use the sender's info for policing if it can verify its correctness.

>not depend on any particular balance of the things monitored.

[BB]: As above. The situation changes drastically, as soon as 
monitored info might be used against the interests of the sender 
volunteering the information.

>your personal preference between these two.

[BB]: Then each type of audit function has to allow so much leeway 
that it opens a security hole the size of a bus. Even if byte-balance 
is the rule, audit security is balanced on a knife edge. Taking away 
this rule makes the security collapse.

I suggest you go away and implement your proposed audit function, or 
at least produce a pseudocode design, then evaluate its security and 
bring it to the list if it passes muster. I can assure you, there is 
absolutely no value in inventing piecemeal in this space. The value 
of the whole ConEx system depends on the integrity of the signals 
that audit assures. It's extremely tricky to get right.

There is value in brainstorming, but there is a time and place for 
such research. This ML is for standardisation. And we're late.

>    I don't follow... how could "audit" be a "machine for dropping"

[BB]: ...by the assumption above that operators use their audit 
devices to discard traffic that does not cross-check. As you yourself 
said, we cannot tell operators what to do or not to do. We have to 
assume many will discard non-compliant traffic.

>with a byte-count field: if left blank the sender is byte-agnostic.

[BB]: OK, now you're switching to a different argument.

The integrity of this flag would then need to be protected, to 
prevent it being changed enroute (and so far ConEx integrity works 
without any crypto, implying no key management complexity).

(Give the list a little more time and I'm sure we will be able to 
think up a number of other security problems. Because the more 
flexibility, the larger the space of possible vulnerabilities.)

Flexibility is fine if it's needed, but the burden is on you to prove 
a choice between packet and byte balance is necessary.

>and I see no particular reason we need it at all...)

So, that little excursion was just a waste of my time.

>    I'm quite happy to define it "per packet", for now...

You have to argue against the reasons given why this is the wrong choice.

The others in this discussion have all agreed that bytes is the 
correct choice. It is not sufficient to dismiss everyone else's 
arguments as irrelevant by calling them mathematical (I didn't see 
any explicit maths in the discussion, so perhaps you are objecting to 
the use of logical reasoning?).

> > Can we find some way of resolving this argument, rather than keep

The only thing we agree on is that deployment must be possible.

The drafts on the table are deployable. They are based on 
implementation experience. What you're really objecting to is stating 
the goal as byte-balance, even if we allow some implementers to do it 
approximately as packet-balance. Instead you want us to say the goal 
should be packet-balance, purely because it's easier to implement (in 
the case of TCP). However, altho your argument is only over what we 
state the goal as, you're not prepared to have that argument, because 
you reject what you call 'mathematical' argument.

You have no implementation, no proposed design and you don't agree 
with arguing scientifically.

I think it's time to ask the chairs to intervene.

Bob

---

From: John Leslie <[email-masked]>
Date: Mon, 21 Nov 2011 19:19:28 -0500

> randomness or sampling in the word audit alone AFAIK.

   (Usually it's easy to goad me into dictionary quotes, but today
I shall resist.)

   I sincerely hope Bob isn't saying that an audit function has to
evaluate _every_ packet which passes it!

> audited. Sampling is about /how/: we all agree that ConEx docs should 

   I'd prefer we didn't cast this discussion as a "debate".

>> design a policing function based on allowances, without reference
>> to the accuracy of congestion prediction/responses.
> 

   I don't follow.

   Whether or not ConEx information is "volunteered" by the sender,
it is unreasonable to allow an unlimited number of ConEx congestion-
expected marks without some cost recovery.

> interests, it has no reason to speak the truth.

   Good lord, Bob: all senders already know that anything they send
"could be used against" them.

   And the truly critical piece of information is whether the sender,
knowing of congestion, chooses to send more before the congestion has
cleared. Standard TCP does. Less-than-best-effort transports try not
to.

   Not all senders are evil.

> So the network will only use the sender's info for policing if it
> can verify its correctness.

   "Verify its correctness" sounds like handwaving to me. IMHO research
is needed into what heuristics would help...

> volunteering the information.

   Research doesn't _need_ to work against the interests of the sender.

> this rule makes the security collapse.

   "Audit security" had better not be "balanced on a knife edge!"
Whether it is, IMHO, depends far more on what you choose to punish
than what it is the sender intends to signal.

   (If we really need to argue "security", I'd advise seeking help
from the Security Directorate.)

> that audit assures. It's extremely tricky to get right.

   That would be foolish.

   Before going into that much detail, we should seek WG agreement
on what we're trying to accomplish. For myself, I'd like to see
folks _want_ to turn on ECN notifications instead of drops. The WG
discussion so far doesn't seem to be considering that a goal. :^(

> such research. This ML is for standardisation. And we're late.

   Yes, we're late.

   No, we're not doing "standardization". We're producing an
Experimental Track specification. Hopefully we'll design experiments
and report on them, in due course.

> assume many will discard non-compliant traffic.

   I don't have to assume that.

   And I don't assume that operators will do much of anything at first.
We will define ConEx signals which will be blithely ignored by the
substantial majority of operators unless they see a _substantial_
problem or are intrigued by a significant perceived benefit.

   There will be a few who go off on a tangent, preferentially dropping
ConEx congestion-expected-marked packets. In our research, hopefully
we will discover these misguided folks; and in an ideal world we'd
convince them to preferentially ECN-mark-and-forward some limited
amount of it. (This will not be trivial!)

> [BB]: OK, now you're switching to a different argument.

   No, I'm merely responding to your request. I didn't even "argue"
how it can work...

> without any crypto, implying no key management complexity).

   That would be counter-productive. No operator can afford to verify
cryptographic signatures for every packet in time of congestion.

> flexibility, the larger the space of possible vulnerabilities.)

   I can think of quite enough security issues already, thank you.

> a choice between packet and byte balance is necessary.

   I think _you're_ the one arguing such a choice is needed now.

> So, that little excursion was just a waste of my time.

   I'm sorry if you think so...

> You have to argue against the reasons given why this is the wrong choice.

   I'm willing to do so. Please state them.

> the use of logical reasoning?).

   I know you're capable of the math -- I gave you the benefit of the
doubt whether you'd actually done it.

> The only thing we agree on is that deployment must be possible.

   That would be a good start.

> implementation experience.

   I respect implementation experience (though I seem to have missed
the reports of the benefits of precise byte-counting that ensued).

> What you're really objecting to is stating the goal as byte-balance,

   That's true, I guess -- though I've been trying to elicit reasons
why counting packets instead of bytes is harmful before "objecting"
on the list.

> even if we allow some implementers to do it approximately as
> packet-balance.

   This strikes me as a poor practice. If byte-counting is so important,
we should set out to convince implementors to do it.

> Instead you want us to say the goal should be packet-balance, purely
> because it's easier to implement (in the case of TCP).

   Ease of implementation is a strong argument when you're looking for
widespread deployment. (I could go into other reasons, but that seems
counter-productive.)

> However, although your argument is only over what we state the goal
> as, you're not prepared to have that argument, because you reject
> what you call 'mathematical' argument.

   No -- I don't reject it, until I see it.

> with arguing scientifically.

   I don't generally approach IETF work by first choosing an implementation,
then arguing for it. I approach IETF work as design, trying to balance
a lot of differing concerns.

> I think it's time to ask the chairs to intervene.

   I don't ask for that -- and I think it premature until we have others
stating that one or the other of us is disruptive.

   IMHO, there's been little enough traffic on this list to classify
_anyone_ as disruptive.

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 22 Nov 2011 03:48:22 +0000

John,

>evaluate _every_ packet which passes it!

I am.

You might be surprised to learn that nearly every aspect of a 
communications system evaluates every packet that passes it.

> > context. ConEx information is volunteered by the sender.

A policy device like a congestion policer using ConEx signals to 
decide which traffic to police. But the ConEx signals it relies on 
are declared by the sender who stands to lose from the policing. 
There is an (obvious) security flaw in that arrangement where the 
sender can just make up ConEx signals that suit its interests, rather 
than re-echoing actual congestion. That is solved by the network 
auditing ConEx signals against the actual congestion signalling that 
is independent of the sender (ie against actual losses or ECN marks).

>expected marks without some cost recovery.

That's true but missing the point about audit. Without audit, if the 
network imposes a limit on ConEx marks or some form of cost recovery, 
the sender could just not set ConEx marks and still cause congestion.

>"could be used against" them.

I am guessing what you are thinking here. You might mean how much 
volume the sender sends could be used against it. You might mean the 
information the sender puts in the payload could be used against it.

But ConEx is different - the sender doesn't have to send ConEx info 
in order to communicate the payload information it wants to send. So 
the operator has to be able to make it in the sender's interest to do 
so. And we (the IETF) have to define the protocol so that operators can.

>is needed into what heuristics would help...

Please don't read this as handwaving. Verifying correctness is what 
audit does. You can look at the code.

If you want, you can imagine the code was written randomly by waving 
a hand at a keyboard. But actually it was thought about for months, 
and every time a vulnerability was discovered, the whole 
protocol/audit system was reworked. Then the code was optimised for 
performance. And it was tested and reworked. Then reworked again. For years.

Perhaps, if you believe these things are the result of hand-waving, 
that says more about how you expect to work.

>    Research doesn't _need_ to work against the interests of the sender.

Eh? Perhaps s/Research/Monitoring/ ?

No it doesn't have to. But it is enough for there to be a risk. Then 
the sender will not want to tell the truth.

>than what it is the sender intends to signal.

Audit security /is/ balanced on a knife-edge.

>from the Security Directorate.)

Had you not appreciated previously that ConEx is /primarily/ about 
security? Astounding.

> > that audit assures. It's extremely tricky to get right.

What!? Are you not aware of the ground rules here? Rough consensus 
and running code.

I don't bring anything to the IETF until it has been implemented, 
tested and independently verified. Especially if it's a change to 
something as fundamental as IP.

If you or anyone else wants to play in a different ball-game, those 
of us who play by the rules are entitled to ask you to go somewhere else.

>on what we're trying to accomplish.

That should largely be done by the end of chartering stage.

I fundamentally disagree with you here. I think everyone else does 
too. Defining an experimental protocol still requires the same 
decisions to be made as a stanrds track protc0l. It has to be 
implementable. Quickly.

>For myself, I'd like to see

This is not in the CoNEx charter.
(But I'm still pursuing this in my own company too)

However, my take is that ConEx deployment should make operators want 
to turn on ECN. I realised early on that ECN doesn't provide enough 
advantage on its own to warrant the deployment upheaval.

>and report on them, in due course.

Sorry. wrong word. But I had the right sentiment. This ML is for 
defining the experimental protocols, not musing about possible 
designs that you haven't thought through.

>    I don't have to assume that.

If an operator is going to rely on the info in ConEx signals to drive 
a policer, it will need an audit device (recapping the earlier argument).

If the audit device detects sender's ConEx info is lying relative to 
actual congestion info, but it does nothing about it, what stops the 
sender lying?

>problem or are intrigued by a significant perceived benefit.

Yup. That's how I see it starting too.

>amount of it. (This will not be trivial!)

I hope 'it' means all ECN-capable traffic, not just the 
congestion-expected-marked packets.

>    I'm willing to do so. Please state them.

All the reasons already given in this thread that you have glossed 
over, while focusing on occasional words and phrases instead of 
people's arguments.

>the reports of the benefits of precise byte-counting that ensued).

Ditto.

>why counting packets instead of bytes is harmful before "objecting"

If you've been trying to elicit these reasons, it's strange you missed them.

>we should set out to convince implementors to do it.

I gave the approach we can offer to do byte-counting early in the 
thread. And this is largely in the TCP modes I-D too.

The above sentence just says we don't need to insist on how 
implementers meet the goal, and they can even do packet-balance if 
they think they will get it through a byte-balancing auditor.

>widespread deployment. (I could go into other reasons, but that seems

How many times do I have to say:
i) it's no good ONLY caring about ease of implementation, without 
caring about whether it meets a sensible goal too.
ii) the strategies in the second mail in this thread are easy to implement

>    No -- I don't reject it, until I see it.

You saw it and rejected it. You have a short memory.

>a lot of differing concerns.

You don't get to choose the rules. THe IETF constitution is "running code".

I'm not arguing /for/ an implementation. I'm arguing for a system 
design. Implementation gives confidence one is speaking from real 
experience. Then one can make compromises with others who made 
different choices in their design, knowing better that the new mix of 
designs will work.

What about the arguing scientifically point?

>stating that one or the other of us is disruptive.

I was asking for the chairs to intervene. I'm tired of arguing with 
someone who is working to different rules than everyone else - rules 
that don't admit to reasoning, so we can never have any way to 
determine an argument.

>    IMHO, there's been little enough traffic on this list to classify

Small precise discussion is far preferable to voluminous imprecise waffle.

Bob

---

From: Dirk Kutscher <[email-masked]>
Date: Tue, 22 Nov 2011 09:04:35 +0000

Hi folks,

> >> "Audit" tends to imply sampling and verifying an approximate match of
> >> things sampled. I'm afraid we've been a bit too vague about the
> >> sampling ratio...
> >
>    (Usually it's easy to goad me into dictionary quotes, but today I shall resist.)
> 
>    I sincerely hope Bob isn't saying that an audit function has to evaluate
> _every_ packet which passes it!

Clearly, it at least has to be able to do that, yes! Auditing is really a fundamental piece of the basic Conex mechanism, and has nothing to do with policy. Auditing is the basis for accurate congestion exposure. Only if you have that, you can think about implementing policies.

Having said that, it is certainly implementation-specific how auditing really works -- there may be tradeoffs between efficiency and effectiveness -- but for the sake of the discussion here, I'd say it's best to assume that, yes, auditing evaluates every packet.

Best regards,

Dirk

> > [BB]: This assertion is missing one hugely important piece of context.
> > ConEx information is volunteered by the sender.
> 
>    I don't follow.
> 
>    Whether or not ConEx information is "volunteered" by the sender, it is
> unreasonable to allow an unlimited number of ConEx congestion- expected
> marks without some cost recovery.
> 
>    Good lord, Bob: all senders already know that anything they send "could be
> used against" them.
> 
>    And the truly critical piece of information is whether the sender, knowing of
> congestion, chooses to send more before the congestion has cleared.
> Standard TCP does. Less-than-best-effort transports try not to.
> 
> > So the network will only use the sender's info for policing if it can
> > verify its correctness.
> 
>    "Verify its correctness" sounds like handwaving to me. IMHO research is
> needed into what heuristics would help...
> 
> >>> monitoring or anything.
> >>
> >> Oh, hardly: monitoring is the first function of research; it need not
> >> depend on any particular balance of the things monitored.
> >
> Whether it is, IMHO, depends far more on what you choose to punish than
> what it is the sender intends to signal.
> 
>    (If we really need to argue "security", I'd advise seeking help from the
> Security Directorate.)
> 
> > absolutely no value in inventing piecemeal in this space. The value of
> > the whole ConEx system depends on the integrity of the signals that
> > audit assures. It's extremely tricky to get right.
> 
>    That would be foolish.
> 
>    Before going into that much detail, we should seek WG agreement on what
> we're trying to accomplish. For myself, I'd like to see folks _want_ to turn on
> ECN notifications instead of drops. The WG discussion so far doesn't seem to
> be considering that a goal. :^(
> 
>    No, we're not doing "standardization". We're producing an Experimental
> Track specification. Hopefully we'll design experiments and report on them,
> in due course.
> 
> We will define ConEx signals which will be blithely ignored by the substantial
> majority of operators unless they see a _substantial_ problem or are
> intrigued by a significant perceived benefit.
> 
>    There will be a few who go off on a tangent, preferentially dropping ConEx
> congestion-expected-marked packets. In our research, hopefully we will
> discover these misguided folks; and in an ideal world we'd convince them to
> preferentially ECN-mark-and-forward some limited amount of it. (This will
> not be trivial!)
> 
> >> Actually, that's simplicity personified -- define ConEx marking with
> >> a byte-count field: if left blank the sender is byte-agnostic.
> >
> > The integrity of this flag would then need to be protected, to prevent
> > it being changed enroute (and so far ConEx integrity works without any
> > crypto, implying no key management complexity).
> 
> > arguments as irrelevant by calling them mathematical (I didn't see any
> > explicit maths in the discussion, so perhaps you are objecting to the
> > use of logical reasoning?).
> 
>    I know you're capable of the math -- I gave you the benefit of the doubt
> whether you'd actually done it.
> 
>    I respect implementation experience (though I seem to have missed the
> reports of the benefits of precise byte-counting that ensued).
> 
>    That's true, I guess -- though I've been trying to elicit reasons why counting
> packets instead of bytes is harmful before "objecting"
> on the list.
> 
>    This strikes me as a poor practice. If byte-counting is so important, we
> should set out to convince implementors to do it.
> 
> > as, you're not prepared to have that argument, because you reject what
> > you call 'mathematical' argument.
> 
> then arguing for it. I approach IETF work as design, trying to balance a lot of
> differing concerns.
> 
>    I don't ask for that -- and I think it premature until we have others stating
> that one or the other of us is disruptive.
> 
>    IMHO, there's been little enough traffic on this list to classify _anyone_ as
> disruptive.
> 
> --

---

From: Matt Mathis <[email-masked]>
Date: Tue, 22 Nov 2011 13:19:29 -0800

Rolling back this whole thread...

> opinions of Bob and Matt.

Bob and I prefer to argue in private, because it works better.   We always
assume that the other person is correct, and make every effort to try to
understand what they think and why.   This is why we have often been able
to clearly restate each others arguments.  In that process
comes enlightenment, deeper understanding and an opportunity to
tease apart our underlying assumptions.  What typically happens is that we
discover some hidden assumption, and that both of our initial positions are
over simplifications and at some level both are wrong.

When we argue in public, the entire discussion get rat holed about details
that are aren't on the table, either because the rest of us already have
consensus or they just are not important at this stage.

My take away from the intended discussion with Bob is that -abstract-mech-
is flawed in that it is entirely too casual about units.  The units
(primarily bits v bytes v packets of congestion) should be an explicitly
chosen parameter for each span of the model (congested router to receiver,
ACKs carrying the signal back to the sender, reinserted feedback from the
sender to the audit and policy functions).

While I am inclined to agree with Bob that the congestion signal should be
in bytes, I am less convinced that it is ok for the ACKs to only carry
segment counts.  I am willing to proceed on the basis of his assumptions,
however it is critical that we all understand that this is a
design decision that might be changed in the future if there is a
compelling reason to do so.  The abstract mech doc needs to state that
proposed signal mappings for a given tentative encoding must be explicit
about units.

It is also important to note that for legacy interoperability using a
sender only implementation, we can not assert different algorithms than are
present in current network devices and ECN enabled receivers.   My example
earlier in the thread illustrates a rather extreme pathological case where
bytes v segments might change sender behavior.  In the common case and as a
natural approximation, the sender will probably assume that the data in not
lumpy, and that reflecting the correct number packets is good enough.
  Clearly when all of the packets in a given connection have uniform sizes,
it doesn't matter to any component, except the policy devices.

The places proper treatment of lumpy data might matter are the policy and
audit devices, which will be the last part of ConEx to have fully bound
specifications.   Some of their details are even explicitly out of our
scope, although we do have to demonstrate the existence of algorithms that
provide useful signals.

As I reflect on the broader debate, it is clear to me the we really need
to understand choosing  units of congestion as an engineering compromise.
 Like all such compromises, it has to balance the costs of difficulty of
implementation vs opportunity cost of reduced precision.   Furthermore,
extended debate without hard data is pointless.   It just doesn't matter at
this stage.   We can accept Bob's assumption for the time being, note that
it is "provisional" and revisit it later, if or when we encounter a problem
that would be better solved with different assumptions.

John, can we arrange a phone call some time?  I want to understand your
vision for ConEx better.  Is the phone number on your 2017 consulting page
correct?  (New Hampshire)

Thanks,
--MM--