Subject: Re: [conex] Fwd: Review: draft-ietf-conex-destopt-06

From: Mirja Kühlewind <[email-masked]>
Date: Tue, 12 Aug 2014 20:43:52 +0200

Hi Bob,

thanks for your review. Please see comments inline.

(Please also note my new email address...)

> ----------  Forwarded Message  ----------
>
> Subject: Review: draft-ietf-conex-destopt-06
> Date: Tuesday 12 August 2014, 01:26:01
> From: Bob Briscoe <[email-masked]>
> To: Suresh KRISHNAN <[email-masked]>, Mirja KUEHLEWIND
> <[email-masked]>, Carlos Ucendo <[email-masked]>
> CC: ConEx IETF list <[email-masked]>
>
> Suresh, Mirja, Carlos,
>
> As promised at the authors mtg in Toronto, I have reviewed
> draft-ietf-conex-destopt-06.
>
> At this late stage, I prefer to suggest text, not just criticise. But
> pls don't take this to mean I expect you to use my suggested text.
>
> I had a lot of nits, so I thought it easiest to deal with these by
> annotating the draft (using MS Word, but also supplied as PDF output):
> <http://www.bobbriscoe.net/projects/conex/draft-ietf-conex-destopt-06-rvw-bb.doc>
> <http://www.bobbriscoe.net/projects/conex/draft-ietf-conex-destopt-06-rvw-bb.pdf>
>
> I have also included all my more substantial concerns in the above
> annotations - highlighted in yellow.
>
> I've listed the major items here, as hooks if mailing list discussion
> is needed on any of them (but see the annotations for detail before
> discussing).
>
> My most controversial disagreement is around IPsec compatibility,
> which we ought to spin into a separate thread if you want to discuss/argue.
>
> Most of the other additions are because conex-abstract-mech says a
> concrete protocol spec has to address specific aspects, which weren't
> addressed.
>
> ==Intro==
> * Added scoping in Intro (solely wire protocol; specific transport
> protocol specs will determine when specifically to set each marking,
> e.g. conex-tcp-modifications)

k. I added something to the previous paragraph because this was already 
intended to say that this document is mainly for people that implement a 
network node and want to do something with ConEx.

>
> * Standards Track -> Experimental
Yes, thanks.

> * Added subsection of intro on experiment goals: criteria for success
> and duration

I believe most of the text actually should go in the tcp mods draft. I 
not sure if there is a common sense in the mean time to have such a 
section in very exp document. But if not I would rather not have it in 
this document because I'm not sure how to define if an experiment was 
successful. If fact the CDO is the only approach at could fulfill our 
requirements, so there is no other  option. And if the coding itself 
with the four bits is useful or not, is not really a question of this 
document, but amybe more of the whole mechanism (incl. auditing and 
policing) or maybe the tcp mods document...?

>
> ==Requirements==
>
> * Referred to abstract-mech for requirements, explained that it would
> be hard to satisfy them all, and explained which one wasn't satisfied
> (visibility in outer), referring to section on fast-path performance.

Took over some of your text at the beginning of this section (and also 
reused some text we already used to have there).

Do you really think that the two requirements you've added are needed? 
Because both basically say that the ConEx coding should encode the Conex 
signal, which is, from my point of view, the whole purpose of this 
document. I do have the feeling that the other requirements listed here 
are on a slightly different level because they are more related to 
deployment issues.

And regarding tunneling: you are right that we need to give more advise 
on tunneling. Shouldn't we just say that one MUST copy the inner ConEx 
Option to the outer header (to solve the visibility problem)?

> ==CDO==

The current version says 'immediate nodes SHOULD only read the CDO 
field' which you changed to 'MUST forward unaltered'. This 'SHOULD' was 
actually a SHOULD because you (!) came up with the idea that a policer 
should be able to remove ConEx markings instead of dropping immediately.
I don't think and never thought that this is a good idea because it 
changes the ConEx information for other nodes later on the path. If you 
think that is definitely not needed anymore, I'm happy to have a MUST there.

>
> * For any text about ignoring invalid fields, explicitly said that
> intermediate nodes MUST NOT normalise. Also, specified to treat
> packets with invalid fields like a non-ConEx-capable packet.

ACK, will add your text here.

>
> * Specified precisely which IP header is included in the byte count.
So you suggest to not include any options? Why? I'd say you either 
include all bits because all of them contribute to congestion, or none 
of the IP header bits because that's just the overhead you can't avoid 
if you what to send anything. Also of course you can generate a larger 
percentage of overhead if you send smaller packets.

>
> * Suggested deleting example of Not-ConEx-capable packets (see
> separate thread to conex-tcp-modifications authors about TCP pure ACKs).
I can remove the example but not sure why you are suggesting this. If 
you actually imply that the X bit should never be zero that we have to 
discuss if the X bit is needed at all.

>
> ==Fast-path==
>
> * CDO as first destination option: changed from MUST to SHOULD (with
> an example of when not to).
I believe this really needs to be a MUST. I know that might restrict the 
use of ConEx with potential other options that might have the same 
requirement (for different reasons). But if you don't put a MUST here, 
you cannot implemented the suggested way in the fast path.

>
> ==Config & Management==
>
> * Added section, mainly to say there is no config & mgmt (required by
> abstract-mech).
I don't thing that the statement about being able to switch ConEx on and 
off belongs in this document (but in tcp mods).
I can add one sentence on warnings or error messages if really needed, 
but I don't think that needs an own section.

>
> ==IPsec compatibility==
>
> * Suggested ConEx counts the AH header, and the outer tunnel mode
> header, with reasoning.
Yes, need to be more precise. Will add.

>
> * Suggested the section is restructured because I believe the
> visibility problem is not related to tunnel mode, but only to ESP in
> tunnel mode.
I agree that tunneling was not addressed well.

>
> * Added a para about the possibility of implementing a ConEx proxy
> (without breaking e2e authentication).
ACK will add.

>
> ==Tunnelling==
>
> * Section added, to generalise from just IPsec to any IP-in-IP
> tunnelling (particularly relevant to mobile scenarios).
If we have a separate section on tunneling. Shouldn't we have tunning 
first and then IPSec...?

>
> * Suggested optional copying of CDO to outer, but also a simpler 'Do
> not copy CDO' alternative.
I don't really get you SHOULD NOT but MAY here...?

>
> ==Security Considerations==
>
> * Added lots, all pointers to where security issues are discussed in
> other places (which is what security directorate reviewers need).
Okay I can add that if you think it's necessary (I would say it's just 
redundant, but you be might right that it just helps the sec dir).

>
> ==IANA==
>
> * I think the act bits need to be 00 not 10 to avoid ConEx packets
> being dropped by non-ConEx nodes (including by non-ConEx receivers)?
> But I'm willing to be corrected.
I agree; Will ask Suresh why he has put a 10 though.

Thanks,
Mirja

>
>
> Regards
>
>
>
>
> Bob
>
>
>
> ________________________________________________________________
> Bob Briscoe,                                                  BT
>
>
> -------------------------------------------------------
>

-- 
------------------------------------------
Dipl.-Ing. Mirja Kühlewind
Communication Systems Group
Institute TIK, ETH Zürich
Gloriastrasse 35, 8092 Zürich, Switzerland

Room ETZ G93
phone: +41 44 63 26932
email: [email-masked]
------------------------------------------

---

From: Bob Briscoe <[email-masked]>
Date: Wed, 13 Aug 2014 20:06:30 +0100

Mirja,

At 19:43 12/08/2014, Mirja Kühlewind wrote:
>k. I added something to the previous paragraph 
>because this was already intended to say that 
>this document is mainly for people that 
>implement a network node and want to do something with ConEx.

Surely this doc is just as important for the 
sender, esp as a generic definition of the ConEx 
protocol for all transport protocols, not just TCP.

>I believe most of the text actually should go in 
>the tcp mods draft. I not sure if there is a 
>common sense in the mean time to have such a 
>section in very exp document. But if not I would 
>rather not have it in this document because I'm 
>not sure how to define if an experiment was 
>successful. If fact the CDO is the only approach 
>at could fulfill our requirements, so there is 
>no other  option. And if the coding itself with 
>the four bits is useful or not, is not really a 
>question of this document, but amybe more of the 
>whole mechanism (incl. auditing and policing) or 
>maybe the tcp mods document...?

The solution to this would be to refer to one doc 
that has the expt goals. However, I believe each 
doc can be seen as a separate piece of a bigger expt.
* The expt that this doc describes is a choice of 
encoding (see below - there are other choices).
* The main expt that the TCP doc describes is how to set credit.

>Took over some of your text at the beginning of 
>this section (and also reused some text we already used to have there).
>
>Do you really think that the two requirements 
>you've added are needed? Because both basically 
>say that the ConEx coding should encode the 
>Conex signal, which is, from my point of view, 
>the whole purpose of this document.

Perhaps you're too close to ConEx. If these reqs 
hadn't been defined in abstract-mech, you 
wouldn't know what "the ConEx signal" was. These 
reqs say what the component parts of the ConEx 
signal are. It could say you need separate 
signals for ECN-credit and loss-credit. It could 
say ConEx nodes must successfully negotiate ECN 
(then re-ECN would be a solution).

The alternative is to refer to abstract-mech for 
all the requirements and not list them here, or 
list the reqs in abstract-mech that are relevant to the network layer encoding.

>I do have the feeling that the other 
>requirements listed here are on a slightly 
>different level because they are more related to deployment issues.

You could say that (but you don't). That's why it 
seemed odd that you listed some requirements, but not the basic ones.

>And regarding tunneling: you are right that we 
>need to give more advise on tunneling. Shouldn't 
>we just say that one MUST copy the inner ConEx 

You can't say MUST, because IPv6 dest opts are 
meant to be e2e only, so no IPv6 tunnel currently 
even understands what a dest opt is, let alone 
copies any dest opt to the outer. A MUST would 
ring huge alarm bells in the vendor community.

I added it as a MAY, purely because it might be 
considered a performance optimisation. I'm still 
worried about the size of the alarm bells that 
will ring. It could prevent this draft 
progressing thru the IESG. Suresh will know best 
how this might be received in the IESG.

>>==CDO==
>
>The current version says 'immediate nodes SHOULD 
>only read the CDO field' which you changed to 
>'MUST forward unaltered'. This 'SHOULD' was 
>actually a SHOULD because you (!) came up with 
>the idea that a policer should be able to remove 
>ConEx markings instead of dropping immediately.
>I don't think and never thought that this is a 
>good idea because it changes the ConEx 
>information for other nodes later on the path. 
>If you think that is definitely not needed 
>anymore, I'm happy to have a MUST there.

OK, you got me.
Let's make this a MUST.

I forgot about what my other personality thought 
when I visited you back in 2010;)
My currently dominant personality isn't convinced 
by those rather tentative ideas.{See Note 1}

>So you suggest to not include any options?

I didn't say that, did I? If the wording I used is ambiguous, pls fix it.

>Why? I'd say you either include all bits because 
>all of them contribute to congestion, or none of 
>the IP header bits because that's just the 
>overhead you can't avoid if you what to send 
>anything. Also of course you can generate a 
>larger percentage of overhead if you send smaller packets.

That's why I think it is reasonable to include 
the IP header (and its options) that immediately 
encapsulates the ConEx dest opt.

>I can remove the example but not sure why you 
>are suggesting this. If you actually imply that 
>the X bit should never be zero that we have to 

I have never thought the X flag was needed. 
There's probably some email on the list somewhere 
in the past from me that says that.

As I put in one of the comment bubbles:
"The only need I can see for the X-flag is if
the Reserved field gets used in future for
something in addition to ConEx. Then there
would be a need to identify packets that
are not ConEx-capable but still carry the
CDO option (for the new reason)."

Can anyone think of a use for the X flag?

>I believe this really needs to be a MUST. I know 
>that might restrict the use of ConEx with 
>potential other options that might have the same 
>requirement (for different reasons). But if you 
>don't put a MUST here, you cannot implemented 
>the suggested way in the fast path.

A SHOULD still means it will be the first option 
in all current implementations. However, I 
suggest a SHOULD, precisely because performance 
reasons are not absolute, so they don't require a 
MUST. If another dest opt cannot work at all 
unless it is first, that would be a valid reason 
for CDO coming second, because it still works, it's /just/ slower.

The IESG will (rightly) be very wary of any draft 
that says an option MUST be the first option.

I suggested the following text after this: "(This is not
stated as a 'MUST', because some future destination option might need to
be placed first for functional rather than just performance reasons.)"

>I don't thing that the statement about being 
>able to switch ConEx on and off belongs in this document (but in tcp mods).

Good point.

>I can add one sentence on warnings or error 
>messages if really needed, but I don't think that needs an own section.

Pls do. I was just making sure all the reqs in abstract-mech were satisfied.

>Yes, need to be more precise. Will add.

This one wasn't just clarity. I've actually 
contradicted what was said, so pls make sure 
there wasn't a good reason for why it was like it was.

I was most concerned about suggesting this 
change, because it was the only one that caused a technical difference.

>If we have a separate section on tunneling. 
>Shouldn't we have tunning first and then IPSec...?

Could do, I guess.

>I don't really get you SHOULD NOT but MAY here...?

See earlier. Tunnels don't normally understand 
dest opts, which is why I said SHOULD NOT. But 
the MAY is a performance optimisation. Am I helping?

>Okay I can add that if you think it's necessary 
>(I would say it's just redundant, but you be 
>might right that it just helps the sec dir).

It's not always obvious which aspects relate to 
security. Especially when the security is 
structural rather than crypto. So I think these 
sentences are useful to sec dir.

>I agree; Will ask Suresh why he has put a 10 though.

Yes, he's the right guy to check with.

Bob

>>Bob

{Note 1}
For anyone watching on the list, the tentative 
idea that Mirja has reminded me of is documented 
in 11.3.1 of my PhD thesis entitled "Covert Markings as a Policer Signal".

The potential problem: A ConEx policer punishes 
punishment. If a congestion policer starts 
dropping packets because the user has contributed 
excessively to congestion, in subsequent rounds 
the user has to re-echo 'L' markings for the 
policer drops as well. This can drive the policer 
further into 'debit'. This might make it 
difficult for the user to get out of trouble once 
she's started getting into trouble.

The basic idea was that when a congestion policer 
drops packets (because the user is causing more 
congestion than her allowance), it will also 
remove ConEx markings. Then (if there is some way 
for the receiver to feed this back), the sender 
knows not to send more ConEx marks because these 
aren't congestion drops, they are policer drops.

We didn't that double punishment made it hard to 
get out of trouble in any policer experiments so 
far, so let's not allow for a possible solution 
to a problem that we probably don't even have. 
The current crop of ConEx drafts are experimental 
anyway. If this problem does surface, then we can reconsider.

---

From: Mirja Kühlewind <[email-masked]>
Date: Thu, 14 Aug 2014 18:41:45 +0200

Hi Bob,

see inline

> The solution to this would be to refer to one doc that has the expt
> goals. However, I believe each doc can be seen as a separate piece of a
> bigger expt.
> * The expt that this doc describes is a choice of encoding (see below -
> there are other choices).
So is it okay if I just add one sentence in the intro (instead of having 
an own section):

"This specification is experimental to allow the IETF to assess whether 
the decision to implement the ConEx signal as a destination option 
fulfills the requirements stated in this document, as well as to 
evaluate the proposed encoding of the ConEx signals."

Does this work for you?

>> Because both basically say that the ConEx coding should encode the
>> Conex signal, which is, from my point of view, the whole purpose of
>> this document.
>
> Perhaps you're too close to ConEx. If these reqs hadn't been defined in
> abstract-mech, you wouldn't know what "the ConEx signal" was. These reqs
> say what the component parts of the ConEx signal are. It could say you
> need separate signals for ECN-credit and loss-credit. It could say ConEx
> nodes must successfully negotiate ECN (then re-ECN would be a solution).
>
> The alternative is to refer to abstract-mech for all the requirements
> and not list them here, or list the reqs in abstract-mech that are
> relevant to the network layer encoding.
>
>> I do have the feeling that the other requirements listed here are on a
>> slightly different level because they are more related to deployment
>> issues.
>
> You could say that (but you don't). That's why it seemed odd that you
> listed some requirements, but not the basic ones.
Okay the intent of these requirements was not to rephrase what is 
already written down in the abstract-mech because I assume that people 
reading this document do not really care why the conex signals look as 
they do but just want to know which conex signals are there (and what 
there meaning is).
The reason to have the requirement listed is simply to justify the 
potentially awkward choice to encode them in a destination option.

I now have the following text:

"A set of requirement for an ideal concrete ConEx wire protocol is given 
in <xref target="I-D.ietf-ConEx-abstract-mech"/>. In the ConEx working 
group is was recognized that it will be difficult to find an encoding in 
IPv6 that satisfies all requirements. The choice in this document to 
implement the ConEx information in a destination option satisfies most 
of those requirements, briefly summarized below:"

+ I added your paragraph on visibility at the end.

Okay?

>> And regarding tunneling: you are right that we need to give more
>> advise on tunneling. Shouldn't we just say that one MUST copy the
>> inner ConEx Option to the outer header (to solve the visibility problem)?
>
> You can't say MUST, because IPv6 dest opts are meant to be e2e only, so
> no IPv6 tunnel currently even understands what a dest opt is, let alone
> copies any dest opt to the outer. A MUST would ring huge alarm bells in
> the vendor community.
>
> I added it as a MAY, purely because it might be considered a performance
> optimisation. I'm still worried about the size of the alarm bells that
> will ring. It could prevent this draft progressing thru the IESG. Suresh
> will know best how this might be received in the IESG.
Okay, got it; see further below...

>> Why? I'd say you either include all bits because all of them
>> contribute to congestion, or none of the IP header bits because that's
>> just the overhead you can't avoid if you what to send anything. Also
>> of course you can generate a larger percentage of overhead if you send
>> smaller packets.
>
> That's why I think it is reasonable to include the IP header (and its
> options) that immediately encapsulates the ConEx dest opt.
Okay I misread your text.

I just though, if you detect a CDO in the header you are currently 
looking at, you will simply look at the playload length and next hop 
fields of this header and than add another 40 bytes for the header 
itself. But in fact that might be wrong if you have another IP header 
encapsulated in the payload... so what is actually the right number of 
bytes here?

I was about to write the following but I'm not sure if that is actually 
write and/or clear:
"... IP packet (including the IP header that carries the CDO and all 
associated options)..."?
That just doesn't say if you should look what the next header is and 
subtract all other IP header bytes you can find. Is this needed?

> I have never thought the X flag was needed. There's probably some email
> on the list somewhere in the past from me that says that.
>
I thought the X bit unset means: I'm a ConEx aware sender and i want to 
follow the rules but I don't have any feedback for this (control) data 
so I'm unable to give you useful ConEx information and if you use this 
packet for your estimation of the current congestion level, you might 
underestimate it.

Doesn't that make sense...?

>> I believe this really needs to be a MUST. I know that might restrict
>> the use of ConEx with potential other options that might have the same
> A SHOULD still means it will be the first option in all current
> implementations. However, I suggest a SHOULD, precisely because
> performance reasons are not absolute, so they don't require a MUST. If
> another dest opt cannot work at all unless it is first, that would be a
> valid reason for CDO coming second, because it still works, it's /just/
> slower.
>
> The IESG will (rightly) be very wary of any draft that says an option
> MUST be the first option.
>
So our fast path implementation must simply assume that there is no CDO 
in case it cannot find it as the first option. Otherwise all non-ConEx 
packets would need to go to the slow path to make sure there is no ConEx 
option. That means to me that this must be a MUST...?

> This one wasn't just clarity. I've actually contradicted what was said,
> so pls make sure there wasn't a good reason for why it was like it was.
>
> I was most concerned about suggesting this change, because it was the
> only one that caused a technical difference.
Ohh, I didn't read your comments carefully and was just looking at the 
text changes... this whole accounting is a mess :-(
Maybe we should only account the IPv6 header itself and the destination 
options...?

Moreover, isn't this here the same case than with tunneling in general. 
Only if the node that does the encapsulation is ConEx-aware it can copy 
the CDO, otherwise it will be not visible anymore.

So this should either be a should, or we have to say something like: if 
the node is ConEx-aware is MUST copy the CDO...?

> See earlier. Tunnels don't normally understand dest opts, which is why I
> said SHOULD NOT. But the MAY is a performance optimisation. Am I helping?
>
>
>
> It's not always obvious which aspects relate to security. Especially
> when the security is structural rather than crypto. So I think these
> For anyone watching on the list, the tentative idea that Mirja has
> reminded me of is documented in 11.3.1 of my PhD thesis entitled "Covert
> Markings as a Policer Signal".
>
> The potential problem: A ConEx policer punishes punishment. If a
> congestion policer starts dropping packets because the user has
> contributed excessively to congestion, in subsequent rounds the user has
> to re-echo 'L' markings for the policer drops as well. This can drive
> the policer further into 'debit'. This might make it difficult for the
> user to get out of trouble once she's started getting into trouble.
>
> The basic idea was that when a congestion policer drops packets (because
> the user is causing more congestion than her allowance), it will also
> remove ConEx markings. Then (if there is some way for the receiver to
> feed this back), the sender knows not to send more ConEx marks because
> these aren't congestion drops, they are policer drops.
>
> We didn't that double punishment made it hard to get out of trouble in
> any policer experiments so far, so let's not allow for a possible
> solution to a problem that we probably don't even have. The current crop
> of ConEx drafts are experimental anyway. If this problem does surface,
> then we can reconsider.
>
>
>
>
>

Room ETZ G93
phone: +41 44 63 26932

---

From: Mirja Kühlewind <[email-masked]>
Date: Thu, 14 Aug 2014 18:51:35 +0200

Hi again,

sorry; pressed sent without being done... see further below....

> the node is ConEx-aware is MUST copy the CDO...?
And then we can the same thing for tunneling in general...?

>> said SHOULD NOT. But the MAY is a performance optimisation. Am I helping?
Okay, understood. But why SHOULD NOT? Isn't it sufficient to say MAY...? 
(or even MUST/SHOULD if ConEx-aware...?)

Mirja

Room ETZ G93
phone: +41 44 63 26932

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 14 Aug 2014 20:15:31 +0100

Mirja,

>>evaluate the proposed encoding of the ConEx signals."

Why resist the good practice of defining full 
criteria for success and duration of experiment? 
This doc (not TCP) is the primary basis of any 
ConEx experiment. I don't mind if it's not in a 
separate section, but these elements should be in an expt RFC.

I think "requirements in abstract-mech" would be more objective.

<RANT>I would like it recorded somewhere that it 
is going to take many years to find a scenario 
where a v6-only experiment in traffic management 
(on a tiny %age of the traffic) makes any sense 
whatsoever. It will be near-impossible to measure whether there is any benefit.
</RANT>

>>+ I added your paragraph on visibility at the end.

How about not including the 2 extra requirements 
and changing your last sentence now I understand what you want to say:

"The choice in this document to implement the 
ConEx information in a destination option aims to 
satisfy those requirements that constrain the placement of ConEx information:"

>>subtract all other IP header bytes you can find. Is this needed?

I'm happy with your something like your last 
sentence. For precision I suggest rewording:

IP packet (including the IP header that directly 
encapsulates the CDO and everything that IP header encapsulates).

It doesn't have to go searching for any more 
deeply encapsulated CDO. If there is one, that 
will be dealt with by whatever higher layer it 
was written in, at the point when the outer 
layers have been peeled off and some function at 
that higher layer is processing these inner dest opts.

>>Doesn't that make sense...?

Not to me. What does "feedback for this (control) 
data" mean? Feedback is about a path used by a 
5-tuple. This control data is about to be sent 
over such a path. If the sender has feedback 
about that path, the feedback applies to 
everything sent over the path, at the IP layer, 
whatever categorisation the next packet has at L4.
(Even if control data is somehow being sent over 
a different path, e.g. using MPTCP or something, 
and there has never been feedback over that path, 
then that would warrant Credit, not absence of ConEx.)

>>option. That means to me that this must be a MUST...?

OK, I see the problem, but how much of a 
performance problem would it really be for the 
fast path of a ConEx function to step along dest 
opts until it gets to CDO then stops (rather than stop if CDO is not first)?

Then "CDO SHOULD be first" would give no 
different performance to "CDO MUST be first", if 
CDO actually was first. If CDO had to be placed 
second on a certain packet, "CDO SHOULD be first" 
would take just one more op than "CDO MUST be first".

Note: I've just re-read the spec of the IPv6 
header. We need to specify that CDO goes in the 
"Destination Options (before routing header)", 
not the "Destination Options (before upper-layer 
header)". Then it won't be encrypted by an ESP header.

>>text changes... this whole accounting is a mess :-(

I don't think it has to be, if we keep to the rule we just agreed above.

>>Maybe we should only account the IPv6 header itself and the destination

Why? I really don't understand why the IPsec 
accounting was written like it was. Pls explain.

>And then we can the same thing for tunneling in general...?

That's surely a circular argument. What would 
make a tunnel endpoint into a ConEx-aware tunnel 
endpoint, so that it would have to copy the CDO? 
It would only become ConEx-aware if it had code 
added to look for the CDO, and why would it have 
that code added unless it was going to do 
something with CDO? That's why I think my 'MAY 
copy as a performance optimisation' formula is the best we can do.

There is no point trying to fix the IPv6 
facilities for tunnelling new extension headers. 
The people whose job it was to design this didn't 
do their job. Their design is now burned into 
IPv6 hardware processors everywhere. Full stop.

All we can hope to do is ensure that CDO is not 
encrypted with ESP. That is feasible.

Whatever we do, in many cases, the IPv6 header 
containing the CDO will be encapsulated in other 
IP headers. So ConEx functions will just have to 
live with that. To find CDO, they will have to 
look for an IP header that encapsulates an upper 
layer protocol header. And even then, they will 
have to look one level deeper in case IP headers 
start again. There's loads of kit these days that 
has to do that anyway (e.g. CGNATs looking for 
the transport header or DPI looking for the 
app-layer). This is all we can hope to do at this experimental stage.

We need to prove ConEx is useful, then it can be 
performance optimised. Header parsing performance 
is generally not a big problem these days.

>Okay, understood. But why SHOULD NOT? Isn't it 
>sufficient to say MAY...? (or even MUST/SHOULD if ConEx-aware...?)

You're right, we could leave out the SHOULD NOT. I suggest:

"As with any destination option, an ingress 
tunnel endpoint will not natively copy the CDO 
when adding an encapsulating outer IP header. However, it MAY..."

Bob

---

From: Mirja Kühlewind <[email-masked]>
Date: Mon, 25 Aug 2014 19:36:26 +0200

On 14.08.2014 21:15, Bob Briscoe wrote:
> Why resist the good practice of defining full criteria for success and
> duration of experiment? This doc (not TCP) is the primary basis of any
> ConEx experiment. I don't mind if it's not in a separate section, but
> these elements should be in an expt RFC.
>
> <RANT>I would like it recorded somewhere that it is going to take many
> years to find a scenario where a v6-only experiment in traffic
> management (on a tiny %age of the traffic) makes any sense whatsoever.
> It will be near-impossible to measure whether there is any benefit.
> </RANT>

Okay added one more paragraph in the intro based on our initial proposal:
"The duration of this experiment is expected to be no less than two 
years from publication of this document as infrastructure is needed to 
be set up to determine the outcome of this experiment. Given ConEx is 
only chartered for IPv6, it might take longer to find a suitable test 
scenario where only IPv6 traffic is managed using ConEx."
Okay?

> How about not including the 2 extra requirements and changing your last
> sentence now I understand what you want to say:
>
> "The choice in this document to implement the ConEx information in a
> destination option aims to satisfy those requirements that constrain the
> placement of ConEx information:"
>
Done.

> I'm happy with your something like your last sentence. For precision I
> suggest rewording:
>
> IP packet (including the IP header that directly encapsulates the CDO
> and everything that IP header encapsulates).
Done

>
> It doesn't have to go searching for any more deeply encapsulated CDO. If
> there is one, that will be dealt with by whatever higher layer it was
> written in, at the point when the outer layers have been peeled off and
> some function at that higher layer is processing these inner dest opts.
My point actually was that you will get a different number of marked 
bytes if you look at a point with or without encapsulation. But as 
usually all packets (at least of the same flow) get encapsulated or not, 
the ratio between marked and not-marked bytes should still be the same, 
so I guess that's fine.

> Not to me. What does "feedback for this (control) data" mean? Feedback
> is about a path used by a 5-tuple. This control data is about to be sent
> over such a path. If the sender has feedback about that path, the
> feedback applies to everything sent over the path, at the IP layer,
If you do not get any feedback on a path, e.g. a receiver only sending 
ACKs, you will never be able to send any ConEx markings. So what's the 
point about marking a packet as ConEx-enabled?

Further note, in the TCP mods we only look at the payload because we 
assume, for simplification, all packets have the same size. Therefore a 
packet that carries no data would not decrease the CEG/LEG. If ACKs 
should get marked, we need to rewrite all this stuff in the tcp mods doc...

> (Even if control data is somehow being sent over a different path, e.g.
> using MPTCP or something, and there has never been feedback over that
> path, then that would warrant Credit, not absence of ConEx.)
I don't think credit does help here. Note credit cannot replace 
ConEx-markings anymore. And if you only send a small amount of control 
data, it is not very likely that your packets gets drop and thus 
probably you do not sent any credit.

> OK, I see the problem, but how much of a performance problem would it
> really be for the fast path of a ConEx function to step along dest opts
> until it gets to CDO then stops (rather than stop if CDO is not first)?
So that's the different between you looking at one bit at a defined 
position or having a chain of conditional look-ups where the length is 
unknown. I believe that is something you would avoid to implement in 
fast path as the processing time is not fixed anymore... that would be 
my guess but I'm not an expert in this area.

>
> Then "CDO SHOULD be first" would give no different performance to "CDO
> MUST be first", if CDO actually was first. If CDO had to be placed
> second on a certain packet, "CDO SHOULD be first" would take just one
> more op than "CDO MUST be first".
>
> Note: I've just re-read the spec of the IPv6 header. We need to specify
> that CDO goes in the "Destination Options (before routing header)", not
> the "Destination Options (before upper-layer header)". Then it won't be
> encrypted by an ESP header.
Thanks. I wasn't fully aware of this. But the difference for my 
understanding is if immediate node listed in the routing header should 
proceed this option or not. In our case it is probably not important 
which one we choose as it should be processed by none of the receivers. 
Where did you read that the later one is not encrypted though?

If so, I can simply add one sentence to the first paragraph of section 4:
"The CDO MUST be placed in the destination option before routing header 
such that it does not get encrypted and can be read by immediate 
ConEx-aware nodes."
And then remove the first paragraph of the IPSec section (and probably 
move the other paragraph somewhere else so that the section is removed 
completely)...?

> Why? I really don't understand why the IPsec accounting was written like
> it was. Pls explain.
The problem about tunneling is that the number of ConEx marked bytes 
might be different depending on where at the path you look at the 
packets. But I guess that's less a problem than I initially though. If 
so I guess I can remove this paragraph about accounting in the IPSec 
section (if still needed at all).

> That's surely a circular argument. What would make a tunnel endpoint
> into a ConEx-aware tunnel endpoint, so that it would have to copy the
> CDO? It would only become ConEx-aware if it had code added to look for
> the CDO, and why would it have that code added unless it was going to do
> something with CDO? That's why I think my 'MAY copy as a performance
> optimisation' formula is the best we can do.
What you say above is the point. If the node does not know anything 
about ConEx, it simple cannot copy the option, which is the case for all 
currently existent nodes. So we cannot say MUST in general. But if the 
node does know that ConEx exists for any reason, it really must copy the 
CDO...? But you right that is a little pathologic. I'm will to change if 
that helps understanding/is less confusing.

>
> There is no point trying to fix the IPv6 facilities for tunnelling new
> extension headers. The people whose job it was to design this didn't do
> their job. Their design is now burned into IPv6 hardware processors
> everywhere. Full stop.
>
> All we can hope to do is ensure that CDO is not encrypted with ESP. That
> is feasible.
>
> Whatever we do, in many cases, the IPv6 header containing the CDO will
> be encapsulated in other IP headers. So ConEx functions will just have
> to live with that. To find CDO, they will have to look for an IP header
> that encapsulates an upper layer protocol header. And even then, they
> will have to look one level deeper in case IP headers start again.
> There's loads of kit these days that has to do that anyway (e.g. CGNATs
> looking for the transport header or DPI looking for the app-layer). This
> is all we can hope to do at this experimental stage.
I guess we should write this point more explicitly:
"A network node that assesses ConEx information SHOULD search for 
encapsulated IP headers until a CDO is found or no further IP headers 
can be found." (should or SHOULD?)

>
> We need to prove ConEx is useful, then it can be performance optimised.
> Header parsing performance is generally not a big problem these days.
>
>
>> Okay, understood. But why SHOULD NOT? Isn't it sufficient to say
>> MAY...? (or even MUST/SHOULD if ConEx-aware...?)
>
> "As with any destination option, an ingress tunnel endpoint will not
> natively copy the CDO when adding an encapsulating outer IP header.
> However, it MAY..."

Done. But one question: Why MAY and not SHOULD? Wouldn't it actually be 
nice if all future tunneling nodes would copy the header.

Room ETZ G93
phone: +41 44 63 26932

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 26 Aug 2014 18:27:45 +0100

Mirja,

Inline.

[I've re-added Suresh who seems to have got lost 
from the distr - he's a good person to double check what we've decided.]

At 18:36 25/08/2014, Mirja Kühlewind wrote:

>"The duration of this experiment is expected to 
>be no less than two years from publication of 
>this document as infrastructure is needed to be 
>set up to determine the outcome of this 
>experiment. Given ConEx is only chartered for 
>IPv6, it might take longer to find a suitable 
>test scenario where only IPv6 traffic is managed using ConEx."
>Okay?

Yup. Thx.

BTW, a para (or section) about experiment goals 
doesn't have to be in the Intro. If you'd prefer 
to keep the intro focused, this could be near the end of the doc.

>My point actually was that you will get a 
>different number of marked bytes if you look at 
>a point with or without encapsulation. But as 
>usually all packets (at least of the same flow) 
>get encapsulated or not, the ratio between 
>marked and not-marked bytes should still be the same, so I guess that's fine.

Yup. When relative proportions are used, there's not a problem.

Even where absolute packet size matters (e.g. a 
congestion policer or more generally a policy 
device configured with a certain rate of 
congestion allowance), the proposed definition can work,... as follows

There are two possibilities where there might be 
a more deeply encapsulated CDO:
i) Two independent systems are initiating 
different congestion feedback loops, and they are 
both reinserting ConEx markings.
ii) The inner CDO has at some point been copied to the outer

I was talking about case (i) before, where the 
sender and all policy devices are working at 
their own layer on their own control loop, 
unaware of another ConEx control loop at another layer (which is fine).

Case (ii) could lead to a bias in the amount of 
congestion counted. This is why I suggested that 
the only time a tunnel endpoint copies CDO to the 
outer is as a performance optimisation. E.g. 
where an operator is tunnelling packets across 
its own network and it knows there are ConEx 
policy devices within its own network, so it 
saves them the hassle of burying into the tunnel.

So case (ii) is also fine, as long as the 
operator doing the optimisation is aware of both 
its tunnels and its policy devices, and 
configures the policy device to make the 
(constant) allowance for the additional tunnel headers.

Perhaps we need to add text where we suggest this 
optimisation to say that the operator should 
ensure all packets are decapsulated back how they 
were before passing on to another operator that 
is not aware of the optimisation. But, this is getting over-detailed...

>If you do not get any feedback on a path, e.g. a 
>receiver only sending ACKs, you will never be 
>able to send any ConEx markings. So what's the 

OK, this is a good example for when a 
ConEx-enabled flag might be useful. However,...

...This doesn't justify marking pure ACKs as 
not-ConEx-enabled. If a sender sends a pure ACK 
now, all it knows is that it might not have 
enough feedback to be able to set ConEx markings 
on a whole sequence of packets later in the 
flow,... but only if it keeps sending solely pure 
ACKs from now on. However, a sender can't be sure 
that it won't have enough feedback in future, 
because usually an app (let alone the transport 
layer) cannot predict whether there will be more 
data to send later, even if it's not sending any now.

Once a sender has had no feedback for at least a 
round trip, it has 2 options for subsequent packets:
a) turn off ConEx-enabled;
b) keep sending packets with ConEx-enabled set, 
but conservatively add some credit.

Even if it subsequently sends some data, it will 
still have to do (a) or (b) on these data 
packets, at least for one further round trip, 
until it gets the feedback. So this is nothing to 
do with whether the packet being sent is a pure 
ACK. It is to do with whether feedback has recently been received.

>Further note, in the TCP mods we only look at 
>the payload because we assume, for 
>simplification, all packets have the same size. 
>Therefore a packet that carries no data would 
>not decrease the CEG/LEG. If ACKs should get 
>marked, we need to rewrite all this stuff in the tcp mods doc...

I don't think we should avoid changing tcp-mods if its 'not right'.

I hope you see the problem from my explanation 
above - whether there is enough feedback /now/ to 
ConEx-mark a packet has nothing to do with 
whether the packet being sent /now/ is capable of 
generating feedback /in the next round/.

If you want to make a simplifying assumption, it 
is on the safe side for a sender to assume that 
all incoming feedback is about packets of the 
same size. It's not safe for a sender to assume 
that all packets it is sending are the same size. 
Anyway, it knows what size it is sending, so it 
doesn't need this simplification.

The simplification I propose (that feedback is 
all about the same size packets, rather than all 
the sent packets are the same size) is likely to 
be pretty good, given the receiver doesn't get 
loss or ECN info about pure ACKs, so they are 
automatically removed from the set of packets 
that the sender assumes to be the same size. And, 
and if some of the feedback is about smaller data 
packets, at least this simplification will always be on the safe side.

If I correctly understand the simplification you 
propose, a ConEx sender will more often 
under-declare congestion than over-declaring, which is not safe.

>I don't think credit does help here. Note credit 
>cannot replace ConEx-markings anymore.

OK, I should have said "then that would warrant 
conservatively sending some credit and corresponding L or E markings."

A sender can always send more ConEx marks than 
actual congestion, and if it doesn't know actual 
congestion, it can at least hold an initial 
estimate of what worst-case congestion might be (e.g. 1%).

However, I admit that, if it only sends pure ACKs 
and this estimate is too low, it will never know 
it is too low, and the audit function might be 
dropping loads of its pure ACKs. Ug. This is a 
new issue I hadn't thought of before... so I 
think we should recommend option a), not b).

>  And if you only send a small amount of control 
> data, it is not very likely that your packets 
> gets drop and thus probably you do not sent any credit.

That's reasonable.

>So that's the different between you looking at 
>one bit at a defined position or having a chain 
>of conditional look-ups where the length is 
>unknown. I believe that is something you would 
>avoid to implement in fast path as the 
>processing time is not fixed anymore... that 
>would be my guess but I'm not an expert in this area.

AFAICT, fast path implementations generally work 
along sequences of extensions. So I don't think 
this is a problem. Bear in mind that we are not 
asking general fast path forwarding 
implementations to do this. Only ConEx functions 
specifically written to find the ConEx header.{Note 1}

{Note 1} OK, we do suggest that general 
forwarding functions could do DoS protection 
using the ConEx header. But that's stated as 
optional and 'aspirational'. If such an 
experiment proves useful, you never know, there 
could be demand for ConEx to migrate into the 
hop-by-hop options (according to the v6 spec, 
hop-by-hop and dest options share the same option 
number space, so this would be a straightforward 
migration, just moving where the CDO is placed, 
but using the same option number and format).

>Thanks. I wasn't fully aware of this. But the 
>difference for my understanding is if immediate 
>node listed in the routing header should proceed 
>this option or not. In our case it is probably 
>not important which one we choose as it should 
>be processed by none of the receivers.

You're correct that CDO isn't processed by any of 
the nodes listed in the routing header as 
destinations. The phrase "before routing header" 
is just how its placement is described. We should 
clarify that this isn't anything to do with the 
processing of the routing header.

>Where did you read that the later one is not encrypted though?

ESP encrypts everything after the ESP header, and 
it comes just before the second dest opts. So it 
would be no good putting CDO after it.

See the ESP spec, on "ESP Header Location":
<http://tools.ietf.org/html/rfc2406#section-3.1>
"  The destination options extension header(s) could appear
    either before or after the ESP header depending on the semantics
    desired.  However, since ESP protects only fields after the ESP
    header, it generally may be desirable to place the destination
    options header(s) after the ESP header.
"

Also see the IPv6 spec on "Extension Header Order":
<http://tools.ietf.org/html/rfc2460#section-4.1>

I believe one reason there are two places for the 
dest opt is because if ESP is encrypting 
everything for the destination, it will normally 
be expected that the dest opts need to be 
encrypted too. But this wouldn't work if you have 
multiple destinations on the path in the routing 
header (that probably don't hold the relevant 
key).  Fortunately, this exception is also needed for ConEx.

>"The CDO MUST be placed in the destination 
>option before routing header such that it does 
>not get encrypted and can be read by immediate ConEx-aware nodes."
>And then remove the first paragraph of the IPSec 
>section (and probably move the other paragraph 
>somewhere else so that the section is removed completely)...?

I've lost track of all the proposed changes to 
the IPsec section. But I think there is value in 
spelling out exactly how ConEx and IPsec 
interact, so I wouldn't remove the section 
completely, even if it repeats info elsewhere.

>The problem about tunneling is that the number 
>of ConEx marked bytes might be different 
>depending on where at the path you look at the 
>packets. But I guess that's less a problem than 
>I initially though. If so I guess I can remove 
>this paragraph about accounting in the IPSec section (if still needed at all).

If you're saying that the new definition of what 
to count removes the problem, because counting is 
no longer dependent on whether there are encapsulating headers, I agree.

>What you say above is the point. If the node 
>does not know anything about ConEx, it simple 
>cannot copy the option, which is the case for 
>all currently existent nodes. So we cannot say 
>MUST in general. But if the node does know that 
>ConEx exists for any reason, it really must copy 
>the CDO...? But you right that is a little 
>pathologic. I'm will to change if that helps understanding/is less confusing.

I think we're talking past each other. Given we 
cannot copy CDO to the outer everywhere, for 
consistency I don't think that copying CDO to the 
outer at all is a good idea, UNLESS it's done 
deliberately as part of an operator's whole 
approach to handling ConEx. Ie. tunnel endpoints 
SHOULD NOT copy CDO to the outer by default, but 
they MAY copy CDO to the outer for a specific 
purpose (e.g. optimisation for ConEx functions 
elsewhere in the same operator's network).

>"A network node that assesses ConEx information 
>SHOULD search for encapsulated IP headers until 
>a CDO is found or no further IP headers can be found." (should or SHOULD?)

SHOULD. But I wouldn't word the last clause like 
that, because searching until no further IP 
headers can be found could go on and on and on and on.

How about:
"A network node that assesses ConEx information 
SHOULD search for encapsulated IP headers until a 
CDO is found. At any specific network location, 
the maximum necessary depth of search is likely 
to be the same for all packets." ?

>Done. But one question: Why MAY and not SHOULD? 
>Wouldn't it actually be nice if all future 
>tunneling nodes would copy the header.

See above about consistency, and ensuring it only 
happens if the operator is fully aware of the 
consequences. If it can't be a large majority, it's best not to be any (IMO).

HTH
(Delayed 'cos it was a public holday in the UK yesterday.)

Bob

---

From: Mirja Kühlewind <[email-masked]>
Date: Thu, 28 Aug 2014 17:43:59 +0200

Hi Bob,

again inline...

On 26.08.2014 19:27, Bob Briscoe wrote:
>> usually all packets (at least of the same flow) get encapsulated or
>> not, the ratio between marked and not-marked bytes should still be the
>> same, so I guess that's fine.
>
> Even where absolute packet size matters (e.g. a congestion policer or
> more generally a policy device configured with a certain rate of
> There are two possibilities where there might be a more deeply
> encapsulated CDO:
> i) Two independent systems are initiating different congestion feedback
> loops, and they are both reinserting ConEx markings.
> I was talking about case (i) before, where the sender and all policy
> devices are working at their own layer on their own control loop,
> Case (ii) could lead to a bias in the amount of congestion counted. This
> is why I suggested that the only time a tunnel endpoint copies CDO to
> the outer is as a performance optimisation. E.g. where an operator is
> tunnelling packets across its own network and it knows there are ConEx
> policy devices within its own network, so it saves them the hassle of
> burying into the tunnel.
>
> So case (ii) is also fine, as long as the operator doing the
> optimisation is aware of both its tunnels and its policy devices, and
> configures the policy device to make the (constant) allowance for the
> additional tunnel headers.
>
> Perhaps we need to add text where we suggest this optimisation to say
> that the operator should ensure all packets are decapsulated back how
> they were before passing on to another operator that is not aware of the
> optimisation. But, this is getting over-detailed...
>
>
>
> OK, this is a good example for when a ConEx-enabled flag might be
> useful. However,...
>
> ...This doesn't justify marking pure ACKs as not-ConEx-enabled. If a
> sender sends a pure ACK now, all it knows is that it might not have
> enough feedback to be able to set ConEx markings on a whole sequence of
> packets later in the flow,... but only if it keeps sending solely pure
> ACKs from now on. However, a sender can't be sure that it won't have
> enough feedback in future, because usually an app (let alone the
> transport layer) cannot predict whether there will be more data to send
> later, even if it's not sending any now.
>
> Once a sender has had no feedback for at least a round trip, it has 2
> options for subsequent packets:
> b) keep sending packets with ConEx-enabled set, but conservatively add
> some credit.
>
> Even if it subsequently sends some data, it will still have to do (a) or
> (b) on these data packets, at least for one further round trip, until it
> gets the feedback. So this is nothing to do with whether the packet
> being sent is a pure ACK. It is to do with whether feedback has recently
> been received.

Okay, rewrote the paragraph slightly:

"If the X bit is zero all other three bits are undefined and thus should 
be ignored and forwarded unchanged by network nodes. The X bit set to 
zero means that the connection is ConEx-capable but this	packet MUST NOT 
be accounted when determining ConEx information in an audit function. 
This can be the case if no feedback on the congestion status is 
(currently) available for e.g. for control packets (not carrying any 
user data). As an example a TCP receiver that only sends pure ACKs will 
usually send them as ACK are usually not ECN-capable as ACK usually are 
not ECN-capable and TCP does not have a mechanism to announce ACK lost. 
Thus congestion information about ACKs are not available."

Is this okay?

>> assume, for simplification, all packets have the same size. Therefore
>> a packet that carries no data would not decrease the CEG/LEG. If ACKs
> I hope you see the problem from my explanation above - whether there is
> enough feedback /now/ to ConEx-mark a packet has nothing to do with
> whether the packet being sent /now/ is capable of generating feedback
> /in the next round/.
>
> If you want to make a simplifying assumption, it is on the safe side for
> a sender to assume that all incoming feedback is about packets of the
> same size. It's not safe for a sender to assume that all packets it is
> sending are the same size. Anyway, it knows what size it is sending, so
> it doesn't need this simplification.
Okay, the assumption is (only) that feedback is based on packets that 
are the same size. If we send you a packet we of course decrease the 
LEG/CEG by the actually payload bytes. But taking this assumption be 
simply do not account for headers at all (nor incoming neither 
outcoming) because we can anyway just estimated the header bits and 
there simply assume it will equal out. Which mean if we send a pure ACK 
we will not decrease the LEG/CEG because there are no payload bytes. I 
believe that this simplification makes thing much simpler and is 
therefore useful but will not allow for marking pure ACKs...

You didn't convince me (yet) that this should be changed but this would 
need to be changed in the tcp mods doc and not this one anyway.

>
> The simplification I propose (that feedback is all about the same size
> packets, rather than all the sent packets are the same size) is likely
> to be pretty good, given the receiver doesn't get loss or ECN info about
> pure ACKs, so they are automatically removed from the set of packets
> that the sender assumes to be the same size. And, and if some of the
> feedback is about smaller data packets, at least this simplification
> will always be on the safe side.
>
> If I correctly understand the simplification you propose, a ConEx sender
> will more often under-declare congestion than over-declaring, which is
> not safe.
I don't believe so. Was this just of a different understanding of what 
we proposed or can you explain further...?

>> ConEx-markings anymore.
> OK, I should have said "then that would warrant conservatively sending
> some credit and corresponding L or E markings."
>
> A sender can always send more ConEx marks than actual congestion, and if
> it doesn't know actual congestion, it can at least hold an initial
> However, I admit that, if it only sends pure ACKs and this estimate is
> too low, it will never know it is too low, and the audit function might
> be dropping loads of its pure ACKs. Ug. This is a new issue I hadn't
> thought of before... so I think we should recommend option a), not b).
>
>>  And if you only send a small amount of control data, it is not very
>> likely that your packets gets drop and thus probably you do not sent
>> any credit.
>
> That's reasonable.
>
>
>
>
>
>>>>>>>> ==Fast-path==
>>>>>>>>
> AFAICT, fast path implementations generally work along sequences of
> extensions. So I don't think this is a problem. Bear in mind that we are
> not asking general fast path forwarding implementations to do this. Only
> ConEx functions specifically written to find the ConEx header.{Note 1}
>
> {Note 1} OK, we do suggest that general forwarding functions could do
> DoS protection using the ConEx header. But that's stated as optional and
> 'aspirational'. If such an experiment proves useful, you never know,
> there could be demand for ConEx to migrate into the hop-by-hop options
> (according to the v6 spec, hop-by-hop and dest options share the same
> option number space, so this would be a straightforward migration, just
> moving where the CDO is placed, but using the same option number and
> format).

There might be also further use cases for e.g. traffic management or 
multipath routing where general forwarding nodes need to access this 
information.

So what's the solution here?

> You're correct that CDO isn't processed by any of the nodes listed in
> the routing header as destinations. The phrase "before routing header"
> is just how its placement is described. We should clarify that this
> isn't anything to do with the processing of the routing header.
>
> ESP encrypts everything after the ESP header, and it comes just before
> the second dest opts. So it would be no good putting CDO after it.
>
Thanks. Wasn't able to find this sentence!

> I believe one reason there are two places for the dest opt is because if
> ESP is encrypting everything for the destination, it will normally be
> expected that the dest opts need to be encrypted too. But this wouldn't
> work if you have multiple destinations on the path in the routing header
> (that probably don't hold the relevant key).  Fortunately, this
> exception is also needed for ConEx.
>
>> "The CDO MUST be placed in the destination option before routing
>> header such that it does not get encrypted and can be read by
>> immediate ConEx-aware nodes."
> I've lost track of all the proposed changes to the IPsec section. But I
> think there is value in spelling out exactly how ConEx and IPsec
> interact, so I wouldn't remove the section completely, even if it
> repeats info elsewhere.

Okay I just realized that we recommend to to use TPSec for 
authentication but I believe if the ConEx option should not be encrypted 
by using the respective header, it will also not be authenticated...? So 
you can have either one of the two...? I believe we still need the IPSec 
section but right now I'm not sure what to right in there...? Any proposal?

> If you're saying that the new definition of what to count removes the
> problem, because counting is no longer dependent on whether there are
> encapsulating headers, I agree.

Done.

>> about ConEx, it simple cannot copy the option, which is the case for
>> all currently existent nodes. So we cannot say MUST in general. But if
>> the node does know that ConEx exists for any reason, it really must
>> copy the CDO...? But you right that is a little pathologic. I'm will
>> to change if that helps understanding/is less confusing.
>
> I think we're talking past each other. Given we cannot copy CDO to the
> outer everywhere, for consistency I don't think that copying CDO to the
> outer at all is a good idea, UNLESS it's done deliberately as part of an
> operator's whole approach to handling ConEx. Ie. tunnel endpoints SHOULD
> NOT copy CDO to the outer by default, but they MAY copy CDO to the outer
> for a specific purpose (e.g. optimisation for ConEx functions elsewhere
> in the same operator's network).
Now understood.

I've tried to make this point a little more clear, not sure if I succeeded:
"As with any destination option, an ingress tunnel endpoint will not 
natively copy the CDO when adding an encapsulating outer IP header. In 
general an ingress tunnel SHOULD not copy the CDO to the outer header as 
this would changed the number of bytes that would be accounted. However, 
it MAY copy the CDO to the outer in order to facilitate visibility by 
subsequent on-path ConEx functions if the tunnel ingree is aware of 
these nodes and theses nodes are aware of the tunneling. This trades off 
the performance of ConEx functions against that of tunnel processing. "

> SHOULD. But I wouldn't word the last clause like that, because searching
> until no further IP headers can be found could go on and on and on and on.
>
> How about:
> encapsulated IP headers until a CDO is found. At any specific network
> location, the maximum necessary depth of search is likely to be the same
> for all packets." ?

Done.

>> Done. But one question: Why MAY and not SHOULD? Wouldn't it actually
>> be nice if all future tunneling nodes would copy the header.
>
> See above about consistency, and ensuring it only happens if the
> operator is fully aware of the consequences. If it can't be a large
> majority, it's best not to be any (IMO).

See text above. Or do you want to add more text to this?

Mirja

Room ETZ G93
phone: +41 44 63 26932

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 28 Aug 2014 21:05:44 +0100

Mirja,

At 16:43 28/08/2014, Mirja Kühlewind wrote:
>"If the X bit is zero all other three bits are 
>undefined and thus should be ignored and 
>forwarded unchanged by network nodes. The X bit 
>set to zero means that the connection is 
>ConEx-capable but this packet MUST NOT be 
>accounted when determining ConEx information in 
>an audit function. This can be the case if no 
>feedback on the congestion status is (currently) 
>available for e.g. for control packets (not 
>carrying any user data). As an example a TCP 
>receiver that only sends pure ACKs will usually 
>send them as ACK are usually not ECN-capable as 
>ACK usually are not ECN-capable and TCP does not 
>have a mechanism to announce ACK lost. Thus 
>congestion information about ACKs are not available."
>
>Is this okay?

The main problem is saying 'not available *for* 
control packets'. But just changing 'for' to 
'from' would still make this too unclear to be understood.

Also need to:
* Make it clear the example is TCP-specific.
* Focus on loss first, then ECN.
* 'mechanism to announce ACK loss' is not really understandable.
* Avoid 'control packets', which is too general, 
given this is an example, so it can be specific.
* Nit: duplicated word (for e.g. for) and 
duplicated phrase (as ACK are usually not 
ECN-capable as ACK usually are not ECN-capable).

How about:

First 2 sentences unchanged, then...
"This can be the case if no congestion feedback 
is (currently) available e.g. in TCP if one 
endpoint has been receiving data but sending 
nothing but pure ACKs (no user data) for some 
time. This is because pure ACKs do not advance 
the sequence number, so the TCP endpoint 
receiving them cannot reliably tell whether any 
have been lost due to congestion. Pure TCP ACKs 
cannot be ECN-marked either [RFC3168]."

>Okay, the assumption is (only) that feedback is 
>based on packets that are the same size. If we 
>send you a packet we of course decrease the 
>LEG/CEG by the actually payload bytes. But 
>taking this assumption be simply do not account 
>for headers at all (nor incoming neither 
>outcoming) because we can anyway just estimated 
>the header bits and there simply assume it will 
>equal out. Which mean if we send a pure ACK we 
>will not decrease the LEG/CEG because there are 
>no payload bytes. I believe that this 
>simplification makes thing much simpler and is 

I thought the earlier definition said that ConEx 
accounts for the size of the IP header that 
contains the CDO and everything within it. Also, 
there's the TCP header size on a pure ACK.

That's the basis on which I am assuming that pure 
ACKs are worth counting. A pure ACK will count as 
at least 86B (and more if there are additional TCP options or IP extensions).

IPv6 header: 40B
CDO dest opt: 6B
TCP header: 40B
Total: 86B

If there are more IP extensions, I guess it will 
be hard for TCP to know though.

>You didn't convince me (yet) that this should be 
>changed but this would need to be changed in the 
>tcp mods doc and not this one anyway.

Agreed (that this would affect tcp-mods, not destopt).

What is 'this' that you aren't yet convinced by?

>I don't believe so. Was this just of a different 
>understanding of what we proposed or can you explain further...?

I thought you were proposing that a TCP sender 
assumes all the packets it sends are full-sized, 
even if they aren't. But I believe you have said that is not what you proposed.

>There might be also further use cases for e.g. 
>traffic management or multipath routing where 
>general forwarding nodes need to access this information.
>

I think this will get thrown back by the IESG if 
we say 'MUST be first'. And I think 'SHOULD be 
first' is a doable implementation for ConEx-aware 
nodes. That is sufficient for experimental. Any 
experiments where general forwarding nodes access 
ConEx will already be reading a destopt at every 
hop, which is not what was intended, but it would 
be doable just for an experiment that wanted to prove ConEx has wider uses.

Everyone involved in IPv6 knows that the attempt 
to design extensibility into v6 failed. It won't 
be news to the IESG that we can't add an 
extension that can be processed at every hop on the fast path.

If a destopt is sufficient to prove ConEx useful, 
then implementers will want to satisfy this demand. Then
* either there is even more pressure on the IETF 
to address this failing in v6 (and maybe someone will),
* or ConEx has to continue with this destopt 
solution, just like everyone else is finding hacks round this failing in v6.

But don't ask me. Ask Suresh.

>Okay I just realized that we recommend to to use 
>TPSec for authentication but I believe if the 
>ConEx option should not be encrypted by using 
>the respective header, it will also not be 
>authenticated...? So you can have either one of 
>the two...? I believe we still need the IPSec 

* How to do ConEx when IPsec is also required 
(tunnel & transport modes, and what to count). 
This may all be obvious now, but (IMO) it would 
still be worth spelling out obvious things.
* How to use IPsec to protect the integrity of CDO.

>when adding an encapsulating outer IP header. In 
>general an ingress tunnel SHOULD not copy the 
>CDO to the outer header as this would changed 
>the number of bytes that would be accounted. 
>However, it MAY copy the CDO to the outer in 
>order to facilitate visibility by subsequent 
>on-path ConEx functions if the tunnel ingree is 
>aware of these nodes and theses nodes are aware 
>of the tunneling. This trades off the 
>performance of ConEx functions against that of tunnel processing. "

OK. Rather than implying that equipment has 
evolved conscious awareness, a better formulation would be something like:
"..the configuration of the tunnel ingress and 
the ConEx nodes is co-ordinated."

Nits:
s/SHOULD not/SHOULD NOT/
s/accounted/counted/
   (in English, accounted is not a transitive 
verb, it has to have 'for' after it)
s/ingree/ingress/
s/theses/these/

>See text above. Or do you want to add more text to this?

Nope, no need for more.

We're getting there!
But we really do need Suresh's expert eye on this.

Cheers

Bob