Subject: [conex] draft-ietf-conex-destopt-01.txt

From: John Leslie <[email-masked]>
Date: Mon, 7 Nov 2011 11:01:35 -0500

[email-masked] <[email-masked]> wrote:
> 
> http://www.ietf.org/internet-drafts/draft-ietf-conex-destopt-01.txt

   The changes are all additions at the end of Section 4.

   In this post, I ask for clarification of the first added sentence:
] 
] All packets of a ConEx-capable connection MUST carry the CDO.

   I'm guessing the intention is that "all packets" must include this
ConEx Destination Option. But I don't want to venture too far before
getting a yes or no...

   Also, we should clarify whether we mean "in both directions".

   Perhaps we should even clarify when we start counting "all".
(Presumably not the first ACK of SYN-ACK; what about the second?)

   Regardless, I'll venture so far as to ask "Why should _all_ packets
carry this option?" I'd like to see a _lot_ more justification for
adding this much overhead.

--
John Leslie <[email-masked]>

---

From: John Leslie <[email-masked]>
Date: Mon, 7 Nov 2011 12:05:15 -0500

> http://www.ietf.org/internet-drafts/draft-ietf-conex-destopt-01.txt

   In this post I treat the byte-count language added.
] 
] If the X bit is set, all three other bit (L, E, C) MAY be set. When
] ever one if this bits is set,

   I'm guessing the intent is "Whenever one of these bits is set".

] the number of bytes carried by this IP packet (incl. IP header)
] SHOULD be accounted when determining congestion or credit information.

   Implicit byte-count, IMHO, is a bad idea. Inevitably it leads to
different parties counting differently. And, we have a synchronization
problem too, where sometimes we're counting bytes in this packet and
sometimes we're counting bytes in a previous packet (at least one RTT
ago).

   There's plenty of room for an _explicit_ byte count; and IMHO if
we want to go down this path of auditing congestion volume in bytes,
we owe it to implementors to be explicit.

] In IPv6 the length can easily be calculated by the value given in
] the Payload Length header field (payload length + option space)
] plus a fixed value of 40 Bytes for the IP header itself.

   Yes, this is explicit about counting bytes-on-the-wire at IP layer;
but even so, I see room for confusion: I'd reword slightly (again,
if we want to go down this path at all).

] In principle all of these three bits (L, E, C) MAY be set in the
] same packet. In this case the packet size MUST be accounted more
] than once for each respective ConEx information counter.

   This is confusing. There's no justification given for keeping
three separate counters, nor an algorithm for how to relate them.

] In practice loss and ECN marks can not occur at the same time,

   Actually, this isn't quite true. Loss must be detected by
timeout or funky ACKs, while ECN marks are known at packet receipt.
Thus, both loss and ECN could be detected at the same time.

] so there should usually be a way to signal the respective ConEx
] information in different packets.

   "Where there's a will, there's a way..."

] In many cases if congestion occurs the sender will not sent
] additional credit bit,

   I think there's a typo there, but I'm not sure what it is...

] but if e.g. a sender assumes losses because of an audit function

   I'm not clear what is intended by "losses because of an audit
function": some proposed audit functions may actually drop packets
due to the audit information...

] or needs to maintain a certain sending rate to make an application
] layer service work, the occurrence of credit bits (c) in parallel
] to congestion exposure bit (L, E) is reasonable.

   I'm not sure I'd go as far as "reasonable", but it's certainly
"plausible".

   However, there's really no reason to believe that a sender will
desire _exactly_ the same byte-count of "credit" as "loss".

   Again, there's plenty of room for an explicit byte-count _if_
we want to go down that path.

] If a network node extracts the ConEx information from a connection,
] this node is usually supposed to hold this information byte-wise,
] e.g. comparing the total number of bytes sent with the number of
] bytes sent with ConEx congestion mark (L, E) to determine the current
] whole path congestion level.

   There's that "congestion level" term again. I'm hoping we can
avoid it...

   I remain unconvinced that _I_ want to go down the path of counting
bytes. I'd argue that in the case of packet loss, we're tending to
a active-queue-managment paradigm where drop is correlated to packet
count. (Agreed, for ECN there's a "cost" correlated to bytes allowed
through with ECN marks.) It's _hard_ to cure ISPs of thinking in
terms of percent of packets lost when they consider "congestion".

   The argument in favor of counting bytes seems to rest on "academic
purity", which IMHO will always lose to "pragmatic" in the RealWorld.

] When equally sized packets can be assumed accounting the number of
] packets (and comparing the total number to marked once) should
] deliver the same result.

   One can _always_ "assume"... :^(

] But a network node MUST be aware that this estimation can be quite
] wrong

   (That's not a correct use of 2119 MUSTard.)

   "wrong" is in the eye of the beholder. We can at most talk of
"compliant" with a standard. (And I don't believe this is the right
document for that.)

] and thus is not reliable if e.g. different sized packed are send.

   (Presumably "sent" is intended.)

   "reliable" doesn't seem the right word here.

====

   Let me argue a bit about the difficulties of synchronizing byte
counts.

   If we insist on implicit byte counts, at some point we will force
the sending of small packets to get byte counts right. (IMHO this
will lead to greater packet drops.) And, we can't drive the byte
count near zero because of IPv6 overhead. Indeed, the sender will
have to guess at IPv6 overhead, because the OS composes the actual
packet. Worse yet, no amount of jawboning in RFCs will prevent
middleboxes from fidddling with packets. :^( :^( :^(

   The smaller the packet, the greater the problem of miscounting
bytes-on-the-wire. And we can't count bytes-on-the-wire accurately
except at the node in question. Even if we could count accurately,
we don't know an appropriate estimate of byte-overhead per packet
_beyond_ bytes-on-the-wire.

   Necessarily, any useful audit function must try to match actual
congestion to congestion-expected marks. These _cannot_ happen at
the identical time: typically they're one RTT apart. But in the
'Net we work in, RTT is unknowabble except to end-points. We must,
therefore, work in approximations.

   IMHO, approximating as packet-count is much easier and not
significantly less useful. I'd really appreciate if we could stop
arguing "academic purity" and instead list problems of utility.

---

From: Mirja Kuehlewind <[email-masked]>
Date: Wed, 9 Nov 2011 17:19:54 +0100

Hi John,

first of all: I added some text to this draft about how exactly the ConEx bits 
need to be interpreted and which kind of packets should/can have which 
markings, because there is some more specific text need in this draft. What 
every is written there is of cause open for discussion. So thanks for your 
feedback so far.

Now to your point: The idea behind the MUST below was that a network node can 
figure out based on just one packet if the connection is ConEx-capable or not 
even though the packet itself might not have any ConEx bit set. I'm not sure 
if this is requirement. If not it might be unnecessary overhead and we could 
remove this. If we remove this we don't need the X bit neither anymore. But 
in fact I believe most packets will still need to have a CDO (except pure 
ACKs...?).

Another good point is, if it is possible to have one half-connection 
ConEx-enable and the other not? Maybe it might make sense if e.g. use data 
only flow in one direction and the other direction which sends only pure ACKs 
is not ConEx-capable...? Don't no... further options here?

Mirja 

On Monday 07 November 2011 17:01:35 John Leslie wrote:
> _______________________________________________
> conex mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/conex

-- 
-------------------------------------------------------------------
Dipl.-Ing. Mirja Kühlewind
Institute of Communication Networks and Computer Engineering (IKR)
University of Stuttgart, Germany
Pfaffenwaldring 47, D-70569 Stuttgart

tel: +49(0)711/685-67973
email: [email-masked]
web: www.ikr.uni-stuttgart.de
-------------------------------------------------------------------

---

From: Mirja Kühlewind <[email-masked]>
Date: Wed, 9 Nov 2011 17:59:31 +0100

Hi John,

I guess there is some more discussion needed on this issue (in the next 
meeting). For now, my understanding was that the abstract mechanism draft 
says that ConEx bits should be accounted byte-wise and I just try to reflect 
this here.

Some more points see below.

Mirja

> we owe it to implementors to be explicit.
What do you mean by 'explicit'? You want to put the number of bytes in the 
CDO? 

> if we want to go down this path at all).
Feel free to make a text proposal here! Its all open for discussion and 
re-wording!

> three separate counters, nor an algorithm for how to relate them.
I though the justification was given in the abstract mechanism draft. 
Basically these signals are not related...

> Thus, both loss and ECN could be detected at the same time.
That's really true! I will change this! Was just a quick first effort to get 
some text here.

> due to the audit information...
Yes and the receiver/sender will not be able to distinguish if there was a 
loss because of congestion or if the audit was dropping something.

> desire _exactly_ the same byte-count of "credit" as "loss".
No, a sender should try to have at least as much credit as loss is expected. I 
guess usually there will be (slightly) more credits.

>    There's that "congestion level" term again. I'm hoping we can
I would actually prefer 'congestion level' against only 'congestion' to make 
sure that this time not 'congestion volume' was meant. But of course I will 
go in line with the use case draft...

> terms of percent of packets lost when they consider "congestion".
If you drop every _packet_ with the same probability, that's perfectly right. 

Assume two senders that send the same amount of data (e.g. 150kbyte), but 
sender A sends only 10 packets and sender B sends 100. Now there is a 
congestion (level) of 10%, so 10% of all packets get a marking.

sender A -> 1 packet marked
sender B -> 10 packets marked

But both have the same amount of data/bytes marked!

packet size of A: 1500 byte
packet size of B: 150 byte

sender A -> 1 * 1500byte = 1500byte
sender B -> 10 * 150byte = 1500byte

If all packets are equal sized (as implicitly assumed above) and you count the 
number of marked packets, you will see on both connections a congestion level 
of 1/10 = 10/100 = 10%.

Now let's calculate the congestion (level) on a per byte base:

(1500 * 1)/(1500*10) = 10%
(150 *10)/(150*100) = 10%

Works as well! Great!

The only problem occurs now, if you do not have equal sized packets. In this 
case byte-wise accounting gives still the right congestion level but 
packet-wise accounting not.

> purity", which IMHO will always lose to "pragmatic" in the RealWorld.
I understand this. I still would like to put in the draft that conex is 
defined byte-wise (because its the only way to do this correctly and 
otherwise there are ways to cheat by playing with the packet sizes) but to 
also make clear in the draft that in the usually case of equal sized packets 
it doesn't make a different. Thus it might be sufficient in many cases to 
count packets (depending on what you gone use the ConEx infos for).

> the sending of small packets to get byte counts right. 
No, ConEx is not indented to change the packet size. In the TCP mod draft we 
currently basically say that you can send more bytes as ConEx marked than 
needed and store this information (for a certain time) such that you 
basically have sent those marked bytes in advance. In fact there is no need 
to get the conex marked bytes absolutely accurate (only make sure to send 
enough bytes such that the audit will be happy).

> (IMHO this 
We still work on approximations. The RTT is a totally different issue with the 
audit device...

---

From: Mirja Kuehlewind <[email-masked]>
Date: Wed, 9 Nov 2011 20:58:27 +0100

Hi again,

I just want to quickly answer/rephrase my own question:

> only flow in one direction and the other direction which sends only pure
> ACKs is not ConEx-capable...? Don't no... further options here?

As ConEx requires only sender-side modifications, it is possible to one 
half-connection ConEx-enabled and the other direction not. But the question 
would be: Should a sender that is generally ConEx-capable not enable ConEx on 
a (half-)connection that does not send any use data (but e.g. just pure 
ACKs), as no congestion feedback is available for those packets...?

Mirja

---

From: Ingemar Johansson S <[email-masked]>
Date: Mon, 14 Nov 2011 11:16:04 +0100

Hi

Have not been able to follow the ConEx list in detail but reading  
http://tools.ietf.org/id/draft-briscoe-conex-initial-deploy-00.txt
I can see that received side modifications are optional.
This is of course interesting at least if consinder the normal server client architecture as it is easier to modify a million servers than a zillion clients.
Assuming that I read right...
How does it work with TCP, I know TCP modifications have been considered to make TCP echo back the exact correct number of ECN-CE to the server. 
Does this then mean that a TCP flow (unmodified TCP) will state a higher congestion level in the dest-opts than the actual ?

/Ingemar

=================================
INGEMAR JOHANSSON  M.Sc. 
Senior Researcher

Ericsson AB
Wireless Access Networks
Labratoriegränd 11
971 28, Luleå, Sweden
Phone +46-1071 43042
SMS/MMS +46-73 078 3289
[email-masked]
www.ericsson.com
=================================

---

From: Mirja Kuehlewind <[email-masked]>
Date: Mon, 14 Nov 2011 17:42:38 +0100

Hi Ingemar,

yes there are only sender-side modification needed. If you only have 
the 'classic' ECN, you will only be able to get (at max) one congestion 
notification per RTT. If there is more than one CE marking per RTT you will 
underestimate the congestion.

Mirja

On Monday 14 November 2011 11:16:04 Ingemar Johansson S wrote:
> This is of course interesting at least if consinder the normal server
> client architecture as it is easier to modify a million servers than a
> zillion clients. Assuming that I read right...
> How does it work with TCP, I know TCP modifications have been considered to
> make TCP echo back the exact correct number of ECN-CE to the server. Does
> this then mean that a TCP flow (unmodified TCP) will state a higher
> congestion level in the dest-opts than the actual ?
>
> /Ingemar
>

tel: +49(0)711/685-67973

---

From: Ingemar Johansson S <[email-masked]>
Date: Tue, 15 Nov 2011 09:25:57 +0100

Thanks

This then means that a policer need to give some extra slack for normal TCP. It is perhaps doable with a policer that somehow has access to the TCP acks in the reverse direction and then can determine both RTT with a reasonable accuracy and also if TCP ECE is modified according to (http://tools.ietf.org/id/draft-kuehlewind-conex-tcp-modifications-01.txt) 
Or is this perhaps simpler ?

/Ingemar

PS
Realized that I got things a bit backwards in my question below, thought that ECE is clamped to 1 for an RTT...

> -----Original Message-----
> From: Mirja Kuehlewind [mailto:[email-masked]] 
> Sent: den 14 november 2011 17:43
> To: [email-masked]
> Cc: Ingemar Johansson S
> Subject: Re: [conex] ConEx as sender side only modification
> 
> Hi Ingemar,
> 
> yes there are only sender-side modification needed. If you 
> only have the 'classic' ECN, you will only be able to get (at 
> max) one congestion notification per RTT. If there is more 
> than one CE marking per RTT you will underestimate the congestion.
> 
> Mirja
> 
> 
> > How does it work with TCP, I know TCP modifications have been 
> > considered to make TCP echo back the exact correct number 
> of ECN-CE to 
> > the server. Does this then mean that a TCP flow (unmodified 
> TCP) will 
> > state a higher congestion level in the dest-opts than the actual ?
> >
> > /Ingemar
> >
> Institute of Communication Networks and Computer Engineering 
> (IKR) University of Stuttgart, Germany Pfaffenwaldring 47, 
> D-70569 Stuttgart
> 
> tel: +49(0)711/685-67973

---

From: Mirja Kuehlewind <[email-masked]>
Date: Mon, 21 Nov 2011 16:18:40 +0100

Hi Ingemar,

that sounds really complicated. From point of view, a policer should not do 
anything like this. The ConEx sender should be aware of the risk of 
underestimating congestion. The current document on TCP modification proposes 
two 'hacks' how to get better feedback from a classic ECN receiver. Moreover, 
the draft says in the section about credits that when a sender which 
experiences on-going losses/ECN marks even though the sending rate was 
reduced due to congestion control, it should think about sending credits as 
those losses might be introduced by an audit device. 

In the meeting there was already a comment/question why more than one 
possibility is needed to cope with classic ECN. The answer is all variants 
have draw-backs (e.g. risk of information loss when ACK loss occurs). But I 
guess some further advisement is needed here what the sender should do to 
avoid underestimating the congestion and/or in which situation which action 
is the right one. But actually I'm not really sure what the right advise 
would be....

Mirja

On Tuesday 15 November 2011 09:25:57 Ingemar Johansson S wrote:
> Thanks
>
> This then means that a policer need to give some extra slack for normal
> TCP. It is perhaps doable with a policer that somehow has access to the TCP
> acks in the reverse direction and then can determine both RTT with a
> reasonable accuracy and also if TCP ECE is modified according to
> (http://tools.ietf.org/id/draft-kuehlewind-conex-tcp-modifications-01.txt)
> > > This is of course interesting at least if consinder the
> >
> > normal server
> >
> > > client architecture as it is easier to modify a million
> >
> > servers than a
> >

tel: +49(0)711/685-67973