Subject: Re: [Roll] Flow Label: 12 bits mutable and 8 bits immutable

From: Michael Richardson <[email-masked]>
Date: Tue, 03 Aug 2010 22:05:15 -0400

{there is a thread which started on [email-masked], and [email-masked],
and then seemed to have dropped [email-masked]. I'm not on [email-masked],
so there likely are message there I've missed}

okay, so I've read Carsen's opinion.
I read RFC 3697 (which I didn't know about).

draft-hu-flow-label-cases-00 is a very good read.  I pull out something
from section 4 (Discussion);

   The other choice, for designers who wish to use the flow label to
   control switching or QoS directly, is to bypass the rules within a
   given domain (a set of cooperating nodes) in a way that nodes outside
   the domain cannot detect.  In this case, any deviation from [RFC3697]
   has no possible effect outside the domain in question.

I don't know where this subject line is from: 12 bits/8bits.
Is there a draft that explains that idea that I've missed?

My claim is that ROLL's RPL is a set of cooperating nodes.

But, it's better than that --- it's a set of routers which are tuned to
support specific applications, and the applications want in this case to
be given information like, "what flow label" to use.  RPL's
RPLinstanceID has all the properties required of a flow label (or,
rather, it has no requirements presently, and therefore can have the
flow label requirements imposed upon it, specifically:
   2.  "Nodes MUST NOT assume any mathematical or other properties of
       the Flow Label"
)

The non-mutability of the label isn't a problem either --- the
applications *AT EACH END*, even if one end of the application is
several AS's away (a very unusual case for 95% of RPL's target use),
that application still needs to know something about what label to use.

There are three cases to consider:
      a) traffic between two RPL nodes
      b) traffic exiting an RPL
      c) traffic entering an RPL

case (a) -- is the "set of cooperating nodes", and therefore is no problem.

case (b) -- the flow label is set to get through the RPL/LLN, and out to
            the network, and the flow label has done it's job, and
            the RPL/LLN network could care less what happens to the flow
            label at that point.   The rest of the network might have a
            problem (i.e. a bug) when RPL networks start sending
            non-zero  flow labels, but that's the rest of the network's
            problem.

case (c) - flow labels of zero are not a problem.  There is either a
         default RPLinstanceID to use (and traffic flows, perhaps not
         optimally), or there isn't (and ICMP Host unreachable occurs).

         - non-zero flow labels which do not map to an RPLinstanceID,
           are simply considered zero, see above.
         - non-zero flow labels which map to RPLinstanceIDs are used.
           *If* it is a problem for outsiders to invoke that LLN's
           DODAG, then there are bigger issues, which the flow label can neither
           help nor hinder --- the flowlabel is not a magic security cookie.
           A firewall may still be required.

The only real problem I can see is when a packet needs to do (b) and
(c).   e.g. use label A to exit LLN alpha, and label B to enter LLN beta.

I don't have a solution to this.  Some have suggested IPIP tunnels,
which sound nice in theory, but in practice do not work in the
wilderness found behind the walls of walled gardens.

-- 
]       He who is tired of Weird Al is tired of life!           |  firewalls  [
]   Michael Richardson, Sandelman Software Works, Ottawa, ON    |net architect[
] [email-masked] http://www.sandelman.ottawa.on.ca/ |device driver[
   Kyoto Plus: watch the video <http://www.youtube.com/watch?v=kzx1ycLXQSE>
	               then sign the petition.

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Mon, 9 Aug 2010 12:17:55 +0200

Hi Michael:

With http://tools.ietf.org/html/draft-ietf-roll-rpl-07#section-7.2 I
tried to stay within the lines of RFC 3697 as you also defend in your
mail. 

I think the question we have now is not whether that proposal is lawful
but whether the new law being defined at 6MAN would prevent it in the
future.
If the updated rules allow, then I'll be glad to work on an FL-based
alternate to the IPinIP/HbH. 

It appeared at the 6MAN WG meeting that 12 bits mutable was exactly what
the core network was asking for to do its load balancing.
A first question to the group could be whether 12 mutable bits are
enough for the sensible usages envisioned so far?

Pascal

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf
Of
> Michael Richardson
> Sent: Wednesday, August 04, 2010 4:05 AM
> To: [email-masked]; Carsten Bormann
> Cc: Pascal Thubert (pthubert); [email-masked]
> Subject: Re: Flow Label: 12 bits mutable and 8 bits immutable
> 
> 
and then
> seemed to have dropped [email-masked]. I'm not on [email-masked], so
there likely
> are message there I've missed}
> 
> draft-hu-flow-label-cases-00 is a very good read.  I pull out
something from
> section 4 (Discussion);
> But, it's better than that --- it's a set of routers which are tuned
to support
> specific applications, and the applications want in this case to be
given
> information like, "what flow label" to use.  RPL's RPLinstanceID has
all the
> properties required of a flow label (or, rather, it has no
requirements presently,
> and therefore can have the flow label requirements imposed upon it,
> specifically:
applications *AT
> EACH END*, even if one end of the application is several AS's away (a
very
> unusual case for 95% of RPL's target use), that application still
needs to know
> something about what label to use.
> 
which sound
> nice in theory, but in practice do not work in the wilderness found
behind the
> walls of walled gardens.
> ] [email-masked] http://www.sandelman.ottawa.on.ca/
|device
> driver[
>    Kyoto Plus: watch the video
> <http://www.youtube.com/watch?v=kzx1ycLXQSE>
> 	               then sign the petition.
> 
> 
> 
> 
> --------------------------------------------------------------------
> IETF IPv6 working group mailing list
> [email-masked]
> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
> --------------------------------------------------------------------

---

From: Philip Levis <[email-masked]>
Date: Mon, 9 Aug 2010 08:59:22 -0700

> enough for the sensible usages envisioned so far?

My 2c:

In the case of the hop-by-hop header, 12 bits is clearly enough for an 8-bit RPLInstanceId, but clearly not enough for a 16-bit Rank. I think that trying to compress them (especially Rank) would be a mistake, whose complexity would far outweigh the benefits. 

Phil

---

From: Michael Richardson <[email-masked]>
Date: Tue, 10 Aug 2010 08:57:00 -0400

>>>>> "Brian" == Brian E Carpenter <Brian> writes:
    >> With
    >> http://tools.ietf.org/html/draft-ietf-roll-rpl-07#section-7.2 I
    >> tried to stay within the lines of RFC 3697 as you also defend in
    >> your mail.
    >> 
    >> I think the question we have now is not whether that proposal is
    >> lawful but whether the new law being defined at 6MAN would
    >> prevent it in the future.  If the updated rules allow, then I'll
    >> be glad to work on an FL-based alternate to the IPinIP/HbH.

    Brian> So I understand that ROLL would definitely need the ability
    Brian> to define explicit semantics on the label. Would it also need
    Brian> the label to be mutable?

The only case where there is a problem is when there is a packet that
arrives from the outside, to a ROLL "border router" (we don't have such
a term in ROLL. The ROLL term would be grounded DODAG root).

The problem would be if:
    a) the flow label was non-zero.
    b) the flow label named a RPLinstanceID which did exist.
    c) the originating IP was not permitted to use that instanceID
    d) yet we wanted traffic to continue.

In that case, we have a problem and we might want to set the flow-label
to zero.  I think this situation is simply illegal, and I think the
culprit is part (d).

As I indicated in a previous message, the above could happen when there
were two ROLL networks connected together by the "internet" (lower-case
I intentional, this doesn't have to be the public Internet, but it does
have to cross administrative boundaries of some kind), and one needed
flow-label A to exit network 1, and flow-label B to enter network 2.

("needed" --- only that flow label would name the right RPLinstance,
which had the correct set of routing constraints)

    >> It appeared at the 6MAN WG meeting that 12 bits mutable was
    >> exactly what the core network was asking for to do its load
    >> balancing.  A first question to the group could be whether 12
    >> mutable bits are enough for the sensible usages envisioned so
    >> far?

    Brian> To me it seems that 12 bits could contain enough randomness
    Brian> to drive a load balancing mechanism, but there's no need for
    Brian> those bits to be mutable that I can see. On the other hand,
    Brian> 12 bits seems very small for any kind of nonce or for a flow
    Brian> identification scheme, so it seems that draft-blake and
    Brian> draft-donley would be knocked out.

ROLL is sees the label/RPLinstanceID as being an opaque number with no
specific properties.  We actually punt to the administrator to even
define/configure it.

I doubt that many ROLL networks will have as many as 2^8 instances, let
alone 2^12 instances.  The rational for lots of bits would be to
encourage random allocation such that in the event that two
uncoordinated ROLL networks happened to merge (even for a few
minutes!!!) that the likelyhood of cross talk would be reduced.

A merge like I describe here is not directly envisaged by the current
set of requirements in my opinion, but I think that requirement will
emerge in the future.

- --

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Tue, 10 Aug 2010 15:34:07 +0200

Hi Brian:
 
> On 2010-08-09 22:17, Pascal Thubert (pthubert) wrote:
> > Hi Michael:
> >
> > lawful but whether the new law being defined at 6MAN would prevent it
> > in the future.
> So I understand that ROLL would definitely need the ability to define explicit
> semantics on the label. Would it also need the label to be mutable?
> 

[Pascal] The FL based proposal for RPL uses 12 mutable bits. 

They are used as an in-band control plane that checks the consistency of routing states along a path. Those states can easily get out of sync due to the nature of the links, but the maintenance can be costly. We want to accelerate the repairs on those links that are actually used when an inconsistency is detected that will impact the delivery.

RPL also builds multiple topologies (virtual networks identified by an instanceID) to address various constraints, in terms that can be such as metrics but also administrative. E.g. a power utility Command & Control network at home will have a specific ID that's different from the default general purpose one, so the power related traffic ends up at the power utility GW along a specific graph. 

An opaque instanceID for the topology can be known by the end points. The instanceID is not necessarily related to QoS and does not map to a per hop behavior, but in our case, to acts more like L3 .1Q tag. The Flow Label is the only field available for an application to pass such information to the network. Thus the non-mutable 8 bits in the proposal. 

> >
> > It appeared at the 6MAN WG meeting that 12 bits mutable was exactly
> > what the core network was asking for to do its load balancing.
> To me it seems that 12 bits could contain enough randomness to drive a load
> balancing mechanism, but there's no need for those bits to be mutable that I
> can see. On the other hand, 12 bits seems very small for any kind of nonce or
> for a flow identification scheme, so it seems that draft-blake and draft-donley
> would be knocked out.

I'm unsure why draft_donley does not copy the IPv4 TOS Byte in the IPv6 Traffic Class Octet, if differentiated service is what the draft is after. 
In any case, it appears that the need is not really for a flow but for a flow type, in which case 8 non mutable bits are enough.

I'm also unsure why draft blake can be preferred over syn cookies, but if it is, then, obviously the more bits the better. Still, this nonce is not used by the network, so the network header should not be encumbered by it. If you look at it, a similar problem appears when a node receives a packet back in an ICMP error, in which case it would be safe to make sure that the node really issued that packet in the first place, otherwise actions taken upon the ICMP could be misused by an attacker from anywhere in the Internet.. If we take the path of a nonce, I'd favor a new "source option" with semantics by which the network is not concerned with the header and  that would enable the source to place its own information such as this nonce.

Thanks a bunch for your help,

Pascal

>    Brian

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Tue, 10 Aug 2010 16:24:42 +0200

Salut Rémi,

Please see below:

Pascal

> From: Rémi Després [mailto:[email-masked]]
> Sent: Monday, August 09, 2010 1:50 PM
> To: Pascal Thubert (pthubert)
> Cc: 6man 6man; Michael Richardson; [email-masked]; Carsten Bormann
> Le 9 août 2010 à 12:17, Pascal Thubert (pthubert) a écrit :
> 
> > Hi Michael:
> >
> Would you have more details on WHY it would be "exactly" what was needed
> (???) ?

[Pascal] That does not come from me. I asked at the mike during the WG meeting, and the answer that was given was that for the purpose of a load balancing hash in the SP network, 11 or 12 bits would be enough.

> I'd rather see the first question as being "Are there some improvements of RFC
> 3697 that would make it practicable?".
> 
> If yes, improving this RFC rather than deprecating it after 6+ years of standard-
> track existence would IMHO be better for IPv6 credibility (and IMHO for that of
> IETF in general).
> 
> The two improvements that seem to make sense (see various exchanged mails)
> are:
> - permit stateless support (with flow-ID hashes in place of flow-based stateful
> processing)
> - permit intermediate nodes to set FLs when source hosts haven't done it (still
> as shorthands for flow ISs made available in IPv6 first headers).
> 
[Pascal] We agree here. The point if that there might be enough room for both, depending on what we are doing.
Note that for the latter, the discussion was also that it is not obvious at the egress of the network to reset the FL to zero in order to enable the next AS to reuse it. So the conclusion was rather to use the bits is a DSCP fashion.

Pascal

---

From: Michael Richardson <[email-masked]>
Date: Tue, 10 Aug 2010 10:27:39 -0400

>>>>> "Pascal" == Pascal Thubert <(pthubert)" <[email-masked]>> writes:
 
    >> On 2010-08-09 22:17, Pascal Thubert (pthubert) wrote: > Hi
    >> Michael:
    >> >
    >> > With
    >> http://tools.ietf.org/html/draft-ietf-roll-rpl-07#section-7.2 I >
    >> your > mail.
    >> > I think the question we have now is not whether that proposal
    >> is > lawful but whether the new law being defined at 6MAN would
    >> prevent it > in the future.  > If the updated rules allow, then
    >> I'll be glad to work on an FL-based > alternate to the
    >> IPinIP/HbH.
    >> 
    >> So I understand that ROLL would definitely need the ability to
    >> define explicit semantics on the label. Would it also need the
    >> label to be mutable?
    >> 

    Pascal> [Pascal] The FL based proposal for RPL uses 12 mutable bits.

    Pascal> They are used as an in-band control plane that checks the
    Pascal> consistency of routing states along a path. Those states can
    Pascal> easily get out of sync due to the nature of the links, but
    Pascal> the maintenance can be costly. We want to accelerate the
    Pascal> repairs on those links that are actually used when an
    Pascal> inconsistency is detected that will impact the delivery.

I don't understand how the FL helps us on this!  Probably I missed
something important.  What you write below was my entire understanding
of our need for an in-band tag.

    Pascal> RPL also builds multiple topologies (virtual networks
    Pascal> identified by an instanceID) to address various constraints,
    Pascal> in terms that can be such as metrics but also
    Pascal> administrative. E.g. a power utility Command & Control
    Pascal> network at home will have a specific ID that's different
    Pascal> from the default general purpose one, so the power related
    Pascal> traffic ends up at the power utility GW along a specific
    Pascal> graph.

    Pascal> An opaque instanceID for the topology can be known by the
    Pascal> end points. The instanceID is not necessarily related to QoS
    Pascal> and does not map to a per hop behavior, but in our case, to
    Pascal> acts more like L3 .1Q tag. The Flow Label is the only field
    Pascal> available for an application to pass such information to the
    Pascal> network. Thus the non-mutable 8 bits in the proposal.

+1

---

From: Michael Richardson <[email-masked]>
Date: Tue, 10 Aug 2010 11:21:13 -0400

>>>>> "Joel" == Joel M Halpern <[email-masked]> writes:
    Joel> Off-list, although you may decide that my confusion is wider, and take it
    Joel> back to the list:

okay, I think it's worthwhile.

    Joel> When a ROLL network is talking to the rest of the world, using IPv6, it would
    Joel> seem that packets can arrive with
    Joel> a) a non-zero flow label
    Joel> b) which happened to be an extant RPLinstanceID
    Joel> c) and apparently the sender, since it is outside of ROLL, is not permitted
    Joel> to use that instanceID
    Joel> d) but we clearly want the traffic to continue

    Joel> WHhy is this at all unlikely?  Are you counting on not having
    Joel> collisions between values in a 20 bit field?  It simply isn't
    Joel> big enough for that to be a safe assumption.

I think that (d) is unlikely.
That is, random nodes on the network, not only will not have a reason to
talk to the light bulb in my living room, but also won't be permitted to
do so.
(If it's actually me at the random IP, then likely I'll come through an
IPsec tunnel...) 

I'm not saying that this will apply to *ALL* ROLL deployments, but to
the majority of them that are envisaged today.  (In fact, the things I
want to deploy likely will suffer from this more than the average will)

In particular, the commodity situation where some node on my network
decides to do an HTTP request to commodity site FOO, and the packets in
question here are replies to my request.  

I do not know how the reply flow labels will be set, but if it will be
simply echoed by TCP passive opens, then in fact, (c) might not apply.
That is, the node will have permission to use that instanceID.  

It's not because the node is outside of the ROLL  that it can not use
the instanceID. It's just that nodes outside the ROLL won't know much
about the instanceIDs, if the applications running on them have not been
configured properly.

That is, my network device used instanceID XYZ to send the request out,
and it is appropriate for replies to also use XYZ. There might even be a
stateful firewall opening a pinhole on the ACL for instanceID XYZ.

If it should come that it is accepable for an intermediate node to reset
the FL to zero, that would solve the problem for incoming packets.

I guess, I'd rather we allocate well known FL=1 (or 0b1111...) to
indicate, "has been reset" rather than "was never set", I'd be happier.

If it is permissible for a "firewall" to drop packets with non-zero FLs,
then that might also solve the problem.  I'd hate to see that in any
kind of commodity system: I've had it with "secure" banks that can not
speak ECN, or that have broken PMTU.

- --

---

From: Michael Richardson <[email-masked]>
Date: Tue, 10 Aug 2010 12:09:26 -0400

>>>>> "Rémi" == Rémi Després <[email-masked]> writes:
    Rémi> RFC 3697 isn't concerned with ASes, and doesn't need to be.
    Rémi> The proposal is only that, where load balancing is performed, 
    Rémi> 0 FLs MAY be replaced by meaningful values for this purpose.   
    Rémi> A FL, once set to a non 0 value, never needs to be reset.

okay, so what you are saying is that load balancing uses of the FL are
only upset when they see zero.  So for instance, if layer-4s (i.e. end
points) were mandated that they must now always set a FL consistently on
a flow, and set it to a non-zero value, that this would satisfy the
requirements of load balancers.

In this context, permission would be given to set zero FL to a non-zero
value.

---

From: Michael Richardson <[email-masked]>
Date: Tue, 10 Aug 2010 12:28:07 -0400

>>>>> "Carsten" == Carsten Bormann <[email-masked]> writes:

    >> a term in ROLL. The ROLL term would be grounded DODAG root).

    Carsten> I haven't yet quite found out whether RPL tries to be
    Carsten> useful for networks with hosts, but if it does, the same
    Carsten> kind if problem would occur when a host sets the flow
    Carsten> label. 

RPL networks consists of leafs and routers.  Both typically act as hosts.
Routers are just hosts that happen to be between other nodes.
(Although, some hosts are too weak to be routers)

Little to no traffic on the RPL is unaware of which RPLinstanceID to
use, because the applications there needs to send using a specific set
of constraints, which is encapsulated into the RPLinstanceID.

For ignorant applications, the OS may well be able to set the
instanceID.  In other cases, an instanceID of zero may be most correct.

There may be some cases where there are ordinary hosts which are on a
network which is not RPL, but is connected to an RPL network. 
(The group does not have any consensus if these will be layer-2 bridged,
or layer-3 routed yet).  In these cases the application either will be
aware, or won't care.

- --

---

From: Philip Levis <[email-masked]>
Date: Tue, 10 Aug 2010 10:29:10 -0700

On Aug 10, 2010, at 9:28 AM, Michael Richardson wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> 

...

> or layer-3 routed yet).  In these cases the application either will be

I feel like we're running in circles, in part due to different expectations of how RPL will be used.

It's clear that in proprietary or vertically integrated networks running RPL, it's possible to state how controllers/other nodes set the flow label in order to play nice with RPL. E.g., if my home automation network receives commands from a special device (which itself might have a web/SOAP/etc. interface), then that special device can set the flow label. This isn't the problem case.

The problem case for the use of the flow label is when arbitrary Internet hosts (e.g., my laptop) want to communicate with devices running RPL. E.g., I have a home RPL network and want to directly talk to a RPL device from my laptop. Requiring the packet source to be aware of flow labels and how to set them breaks the whole idea of Internet connectivity: suddenly an end host needs to be aware of what routing protocol is used to reach the other host, as well as configuration parameters of that host. I need to be able to SSH to the RPL-connected device without requiring a new ssh program that uses ioctls to set flow labels. We can't say that only OSes which set the flow label to 0 (or some other value) can be used. Nor can we expect to change how OSes set the flow label.

So the problem is that a packet originating from outside the RPL network, destined to a RPL node, may have an arbitrary flow label. The RPL border router needs to handle the packet in an effective way.  Whatever solution we come up with needs to handle this effectively if the Internet is going to support connectivity to LLN devices. It's critical that the solution not make assumptions about how the flow labels are set or what the behavior of the non-RPL end host is. We want Windows, Linux, OS X, Windows Mobile, Android, Symbian, and other current Internet devices to be able to talk with LLNs.

Phil

---

From: Michael Richardson <[email-masked]>
Date: Tue, 10 Aug 2010 16:09:07 -0400

>>>>> "Philip" == Philip Levis <[email-masked]> writes:
    Philip> I feel like we're running in circles, in part due to
    Philip> different expectations of how RPL will be used. 

    Philip> It's clear that in proprietary or vertically integrated
    Philip> networks running RPL, it's possible to state how
    Philip> controllers/other nodes set the flow label in order to play
    Philip> nice with RPL. E.g., if my home automation network receives
    Philip> commands from a special device (which itself might have a
    Philip> web/SOAP/etc. interface), then that special device can set
    Philip> the flow label. This isn't the problem case. 

    Philip> The problem case for the use of the flow label is when
    Philip> arbitrary Internet hosts (e.g., my laptop) want to
    Philip> communicate with devices running RPL. E.g., I have a home
    Philip> RPL network and want to directly talk to a RPL device from
    Philip> my laptop. Requiring the packet source to be aware of flow
    Philip> labels and how to set them breaks the whole idea of Internet
    Philip> connectivity: suddenly an end host needs to be aware of what
    Philip> routing protocol is used to reach the other host, as well as
    Philip> configuration parameters of that host. I need to be able to
    Philip> SSH to the RPL-connected device without requiring a new ssh
    Philip> program that uses ioctls to set flow labels. We can't say
    Philip> that only OSes which set the flow label to 0 (or some other
    Philip> value) can be used. Nor can we expect to change how OSes set
    Philip> the flow label. 

I guess I don't see the problem to be as big a deal as you suggest.
I'm happy if flow label 0 gets some default RPLinstanceID.  

It would be convenient if the rules were relaxed such that on ingress,
the RPL edge router could *set* the FL if it were zero, and it appears
that this matches up well with what the load balancing people want to
do.

Today, if you ssh or HTTP or what-not, without doing anything special,
you will get a flow label of zero, which with the above rule, means you
get connectivity.  Only if some intermediate node sets the FL=0 would it
be a problem today, and that's illegal so far.

BTW: the flowlabel is not in an ioctl, it's right in the sockaddr_in6.
Too bad that inet_ntop(3)/RFC2373/4291 was not specified to include the flow
label, as then it would be just a matter of user education...  Maybe
there is some way to use the %ifname notation used for ifindex mapping.
(okay, it falls apart once you want to use DNS)

It seems to me that within RPL, if we use RPLinstanceID, we need to
decide what to do when we get a flow label from an untrusted source that
is non-zero.  Inserting an end-host option, with the unstrusted FL
seems like the right thing to do to me.  This facility might be
generally useful to entire Internet.

- --

---

From: Michael Richardson <[email-masked]>
Date: Tue, 10 Aug 2010 16:16:03 -0400

>> (Although, some hosts are too weak to be routers)

    Carsten> OK, I'm not talking of "host" as in originates or
    Carsten> terminates traffic, but "host" in the sense of "does not
    Carsten> participate in routing". 

    Carsten> It appears there is no such thing inside a RPL world then.

Yes, by that definition that's perhaps a fair statement!
I prefer to think of each device as being a router, which an attached host.
(the ip_output()/ip_input() parts... are the host, the ip_forward() is
the router...) 

    >> of constraints, which is encapsulated into the RPLinstanceID.

    Carsten> Hosts (as in "don't participate in routing") would have
    Carsten> little idea of such concepts. 

Yes, but the *applications* do know about such things.

    Carsten> In IP, hosts send packets to an IP address.

You can do that.  Your packet might not get there in time.  It might
take a much longer route than necessary.  It might not get there at all,
if whatever default RPLinstance that exists does not include your destination.

    Carsten> Sometimes there are additional considerations, which we so
    Carsten> far have been addressing [sic] with the TOS byte, i.e., the
    Carsten> DSCP today. 

If you prefer, you may think of the RPLinstance as naming something akin
to an MPLS or ATM "circuit", except that it's NBMA-ish tree rather than P2P.

- --

---

From: Philip Levis <[email-masked]>
Date: Tue, 10 Aug 2010 17:18:09 -0700

> be a problem today, and that's illegal so far.

Can you promise me that every device in the Internet does this, i.e. that there are no IP stacks which set the flow label?

Phil

---

From: Michael Richardson <[email-masked]>
Date: Tue, 10 Aug 2010 21:05:56 -0400

>>>>> "Fred" == Fred Baker <[email-masked]> writes:
    Fred> I would find that surprising. There are ample cases where the
    Fred> originator of a high data rate flow (sensor data from a radio
    Fred> telescope to a number cruncher, to pick one example) might
    Fred> want to use the flow label to send data from one session down
    Fred> multiple paths. Multi-path TCP would be another example. I
    Fred> suppose you could argue that consistently alternating among N
    Fred> flow labels can be thought of as N separate flows, but if it's
    Fred> TCP there are other ramifications. 

    >> requirements of load balancers.

I don't mean to say that the FL has to be a single value: I guess it
could be from a set of values if one wants to do multi-path.

    Fred> In any event, the requirement of the load balancer is not that
    Fred> the originator gives things a tag; we can develop such tags
    Fred> from a hash of the source and destination address quite well,
    Fred> thanks. The important thing is that things are tagged in a

Well... if FL is unique per sender's concept of flow, that's 128+20 bits
for the tcam vs 256+8+32 bits if you use the 5-tuple.  (why use a tcam
for this... I don't know... there are no wildcard bits.  A cam or
patricia-tree would be as good...)

    Fred> manner that helps the load balancer. There are two major uses
    Fred> of load balancers. In data centers, we balance sessions
    Fred> to/from sets of servers, and the question when a new session
    Fred> comes up is which server is most likely to be able to
    Fred> effectively serve it (has the necessary resources including
    Fred> access to the right disks, CPU cycles, etc). In bandwidth
    Fred> allocation uses (spreading load across multiple
    Fred> otherwise-equivalent paths), the question is how to use each
    Fred> of the paths most effectively - we want some traffic on each
    Fred> of them and we want none of the paths to be overloaded or
    Fred> introducing more delay or loss than others. Everyone's
    Fred> instinctive knee-jerk response to the question seems to be
    Fred> "hash something and depend on the law of large numbers to
    Fred> serve the need", and the observation of numerous operators is
    Fred> that the law of large numbers is a good start but is not an
    Fred> adequate solution. You really want the load balancer to be
    Fred> able to be smarter than that. 

    Fred> Which is why people that build load balancers frequently do so
    Fred> using NAT technology or some other way of tagging individual
    Fred> sessions. And why they are looking for mutable flow labels in
    Fred> this thread. 

So, they want to mutate all flow labels, or just non-zero ones?

    > Rémi> A FL, once set to a non 0 value, never needs to be reset.

    mcr> okay, so what you are saying is that load balancing uses of the FL are
    mcr> only upset when they see zero.  So for instance, if layer-4s (i.e. end
    mcr> points) were mandated that they must now always set a FL consistently on
    mcr> a flow, and set it to a non-zero value, that this would satisfy the
    mcr> requirements of load balancers.
    mcr> 
    mcr> In this context, permission would be given to set zero FL to a non-zero
    mcr> value.

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Wed, 11 Aug 2010 12:00:38 +0200

> I don't understand how the FL helps us on this!  Probably I missed
something
> important.  What you write below was my entire understanding of our
need for
> an in-band tag.

[Pascal] This point is not about the flow label specifically but about
the RPL strategy, and it works whether we use the flow label or the hop
by hop techniques to transport the info. In short, the mere fact that
there is a packet on the broken path causes us to 1) detect and 2) fix
the issue. If there's no packet on a broken path then it can stay broken
for a while. There will probably be many broken paths at any point of
time in the network, due to transient link interference, movement,  and
all sorts of routing desyncs. Considering the cost of proactive rapid
resolution, we have chosen not to fix the routing issues as soon as they
appear but 1) lazily and 2) on-demand.
The lazy piece is a global reconstruction of the DAG that happens
periodically. The on-demand piece is a data path validation that allows
us to fix a path as it is being used, either using a mutable flow label
field or the Hop by Hop header.

Pascal

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Wed, 11 Aug 2010 12:00:36 +0200

Hi Brian:

The Hop by Hop is certainly the clean solution. 

The trouble is that it requires additional bytes in every packet for the
header and for the IP-in-IP encapsulation that goes with it; yet RPL
operates in a domain where devices can be strictly constrained in energy
and frames can be very small, so every bit counts quite dramatically.
Thus we're looking for more affordable alternates. We have started a
6MAN document on the IP in IP avoidance for instance.

Here, the point is not to get the flow label definition impacted by RPL
but to see if the flow label can be used by RPL within the definition
that 6MAN will refine. If we have enough mutable bits, then we can allow
an alternate mode in RPL whereby those they are used as opposed to the
HbH. 

For the instance ID, it's certainly a lot more complex since we are
defining an interaction between the app layer and the network. I do not
know if the result of a 6MAN work on such interaction can be used by RPL
in the end, but I think it is a good subject to address before we make
all bits mutable.

My initial take would be that there should at least be one bit somewhere
that indicates that the source does not want (a piece of) the FL to be
modified. 

Pascal

Of
> Brian E Carpenter
> Sent: Tuesday, August 10, 2010 11:53 PM
> To: Fred Baker (fred)
> Cc: Carsten Bormann; [email-masked]; [email-masked]; Michael Richardson
> Subject: ROLL choice to not use the Flow Label
> 
> Thanks for all the enlightment about ROLL.
> 
> My personal conclusion is that the ROLL considerations are too complex
and
> too subtle to be compatible with using a general-purpose IPv6 header
field (i.e.
> the flow label) for ROLL purposes. They seem to be an extreme case of
the
> challenges of defining a local-use regime for the flow label, which
have already
> been a stumbling block for the various versions of
draft-carpenter-6man-flow-
> update.
> 
> So IMHO, ROLL/RPL was quite wise to drop the IPv6 header flow label
proposal.
> 
>      Brian
> 
> On 2010-08-11 08:53, Fred Baker wrote:
> > On Aug 10, 2010, at 11:21 AM, Carsten Bormann wrote:
> >
> >> OK, I'm not talking of "host" as in originates or terminates
traffic, but "host"
> in the sense of "does not participate in routing".
> >> It appears there is no such thing inside a RPL world then.
> >
> > A RPL or Manet world doesn't have the 1970's-mentality limitation
that hosts
> have to be stupid. Of all networking architectures, the Internet is
the only one
> with that delusion.
> >
> > http://www.ipinc.net/IPv4.GIF
> >
> >

---

From: Michael Richardson <[email-masked]>
Date: Wed, 11 Aug 2010 10:48:43 -0400

>>>>> "Pascal" == Pascal Thubert <(pthubert)" <[email-masked]>> writes:
    Pascal> Hi Brian:

    Pascal> The Hop by Hop is certainly the clean solution.

    Pascal> The trouble is that it requires additional bytes in every
    Pascal> packet for the header and for the IP-in-IP encapsulation
    Pascal> that goes with it; yet RPL operates in a domain where
    Pascal> devices can be strictly constrained in energy and frames can
    Pascal> be very small, so every bit counts quite dramatically.  Thus
    Pascal> we're looking for more affordable alternates. We have
    Pascal> started a 6MAN document on the IP in IP avoidance for
    Pascal> instance.

    Pascal> Here, the point is not to get the flow label definition
    Pascal> impacted by RPL but to see if the flow label can be used by
    Pascal> RPL within the definition that 6MAN will refine. If we have
    Pascal> enough mutable bits, then we can allow an alternate mode in
    Pascal> RPL whereby those they are used as opposed to the HbH.

So one solution is for RPL to just do what it wants to inside the LLN,
and use IP-in-IP for packets that arrive at the LLN border with a
non-zero FL.  (ditto for packets that have to leave)

This makes the extra bytes the exception rather than the rule.

It also can be turned off in closed networks where the FL has been
coordinated across the AS.

This would be facilitated by the rule that we can set a zero FL to a
non-zero value.

---

From: Philip Levis <[email-masked]>
Date: Wed, 11 Aug 2010 16:34:32 -0700

On Aug 10, 2010, at 11:12 PM, Rémi Després wrote:

> 
> Le 10 août 2010 à 18:09, Michael Richardson a écrit :
> 
>> 
> Right.
> To be even more precise: 
> - Flow endpoints (sometimes layer 4 and sometimes layer 3) should from now on be mandated to set FLs with non-0 values that statistically differ from a flow to another.

The intention is to have a BCP for network stack implementers to follow?

> - However, we have to face that, so far, they are generally mandated to set FLs to 0.

I apologize for the lack of context (I'm coming from ROLL): your sentence seems to suggest that flow labels today are mandated to be 0. This doesn't seem to be right: among other things, ping6 supports setting the flow label, and by default allocates a random flow label.[1] Basically, I'm confused if you're talking in the present tense of what's done with flow labels today, or the future tense of how flow labels should be used in the future.

Phil

[1] http://linux.die.net/man/8/ping6

---

From: Philip Levis <[email-masked]>
Date: Wed, 11 Aug 2010 16:41:32 -0700

> field or the Hop by Hop header.

Maybe a rewording might make this clearer.

Basically, RPL puts up to two pieces information in packets that it routes. The first is which routing topology this packet should be routed on: RPL supports multiple parallel topologies, e.g., one optimized for low latency and a second optimized for low energy. This information is called the RPLInstanceId. 

The second is information about the "cost" of the forwarder's route to the destination. This hop-by-hop "cost" information is called Rank. Rank is needed to detect routing loops: if a router receives a packet whose prior router's Rank is less than or equal to its own, Rank is not strictly decreasing and there may be a routing loop.

Being able to embed this information in the flow label, rather than requiring additional headers and IP-in-IP, would greatly reduce header overhead. In RPL, we're often concerned about very short, such as 10 byte, packets, so this could be a big savings.

The challenge is that the RPLInstanceId is 8 bits and Rank is 16 bits.

Does that make sense?

Phil

---

From: Philip Levis <[email-masked]>
Date: Wed, 11 Aug 2010 16:45:56 -0700

On Aug 10, 2010, at 2:53 PM, Brian E Carpenter wrote:

> My personal conclusion is that the ROLL considerations
> are too complex and too subtle to be compatible with using
> a general-purpose IPv6 header field (i.e. the flow label)
> for ROLL purposes. They seem to be an extreme case of the
> challenges of defining a local-use regime for the flow label,
> which have already been a stumbling block for the various
> versions of draft-carpenter-6man-flow-update.
> 

Brian,

I agree that the technical challenges seems significant. It was clear that, as we were writing RPL, we didn't yet know how to do it, and so we dropped the idea from the base draft. 

That being said, the possible benefits are pretty enormous to RPL networks. I'm therefore of the opinion that we're doing the right thing, trying to think this through and reach consensus on if there's a way to do use the flow label cleanly and effectively. The answer may be no, but the answer may also be yes: only discussion and good engineering design will let us figure it out.

Phil

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Thu, 12 Aug 2010 10:47:34 +0200

> > the RPL strategy, and it works whether we use the flow label or the
> > hop by hop techniques to transport the info. In short, the mere fact
> > that there is a packet on the broken path causes us to 1) detect and
> > 2) fix the issue. If there's no packet on a broken path then it can
> > stay broken for a while. There will probably be many broken paths at
> > any point of time in the network, due to transient link
interference,
> > movement,  and all sorts of routing desyncs. Considering the cost of
> > proactive rapid resolution, we have chosen not to fix the routing
> > issues as soon as they appear but 1) lazily and 2) on-demand.
> > periodically. The on-demand piece is a data path validation that
> > allows us to fix a path as it is being used, either using a mutable
> > flow label field or the Hop by Hop header.
> 
> Basically, RPL puts up to two pieces information in packets that it
routes. The
> first is which routing topology this packet should be routed on: RPL
supports
> multiple parallel topologies, e.g., one optimized for low latency and
a second
> optimized for low energy. This information is called the
RPLInstanceId.
> 
> The second is information about the "cost" of the forwarder's route to
the
> destination. This hop-by-hop "cost" information is called Rank. Rank
is needed
> to detect routing loops: if a router receives a packet whose prior
router's Rank
> is less than or equal to its own, Rank is not strictly decreasing and
there may be
> a routing loop.
> 
> Being able to embed this information in the flow label, rather than
requiring
> additional headers and IP-in-IP, would greatly reduce header overhead.
In RPL,
> we're often concerned about very short, such as 10 byte, packets, so
this could
> be a big savings.

[Pascal] Agreed. We'll note that the Hop by Hop + IP in IP is costly but
solves the generic problem *within* the RPL network. The use of the Flow
label *within* the RPL network would be an alternate so it could have a
more limited applicability, like a more limited number of instances or a
Rank floor that can be expressed in 1 octet or less, which is probably
more than enough considering that infinity is 16 in RIP. This could
cover a large number of foreseeable deployments.

There is certainly a strong benefit in Low Power Lossy Networks to make
the flow label mutable. Still, making it all mutable will definitely
block a communication path between the app and the network, which I feel
is a mistake.

Pascal

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Thu, 12 Aug 2010 16:00:06 +0200

Yes, Rémi, you are.

We have explained that inserting data requires a new header and thus IP in IP encapsulation. This happens for instance when a packet comes from the Internet into a RPL network. Which we can hardly accept in constrained networks an devices. In most cases, we could squeeze the information we need in the Flow Label should it be mutable.
 
Pascal

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Rémi Després
> Sent: Thursday, August 12, 2010 2:55 PM
> Cc: [email-masked]; Carsten Bormann; [email-masked]; Brian E Carpenter;
> [email-masked]
> Subject: Re: [Roll] Flow Label: 12 bits mutable and 8 bits immutable
> 
> 
> Le 12 août 2010 à 10:47, Pascal Thubert (pthubert) a écrit :
> > We'll note that the Hop by Hop + IP in IP is costly but solves the
> > generic problem *within* the RPL network. The use of the Flow label
> > *within* the RPL network would be an alternate so it could have a more
> > limited applicability, like a more limited number of instances or a
> > There is certainly a strong benefit in Low Power Lossy Networks to
> > make the flow label mutable.
> 
> How strong the benefit isn't clear to me.
> Since RPL networks are recognized as such by hosts, it seems easy to add a
> short field, just before the IP header, to contain whatever is useful for this kind
> of network.
> Mixing functions with those of FLs would thus be avoided.
> 
> Did I miss something?
> RD
> 
> 
>

---

From: Rémi Després <[email-masked]>
Date: Mon, 9 Aug 2010 13:49:58 +0200

> the core network was asking for to do its load balancing.

Would you have more details on WHY it would be "exactly" what was needed (???) ? 

> enough for the sensible usages envisioned so far?

I'd rather see the first question as being "Are there some improvements of RFC 3697 that would make it practicable?".

If yes, improving this RFC rather than deprecating it after 6+ years of standard-track existence would IMHO be better for IPv6 credibility (and IMHO for that of IETF in general).

The two improvements that seem to make sense (see various exchanged mails) are:
- permit stateless support (with flow-ID hashes in place of flow-based stateful processing)
- permit intermediate nodes to set FLs when source hosts haven't done it (still as shorthands for flow ISs made available in IPv6 first headers).

Regards,
RD

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 10 Aug 2010 11:41:39 +1200

Pascal,

Thanks for this and your previous message. I would need to think
very carefully whether this proposal would really have been strictly
compatible with RFC 3697 - it actually depends on how the agreement
to use these semantics imposed on the flow label is made (what is
called 'flow state establishment' in the RFC). I see more weaknesses
in 3697 each time I look at it :-(

More below...

> alternate to the IPinIP/HbH. 

So I understand that ROLL would definitely need the ability to define
explicit semantics on the label. Would it also need the label to be mutable?

> enough for the sensible usages envisioned so far?

To me it seems that 12 bits could contain enough randomness to drive
a load balancing mechanism, but there's no need for those bits to be
mutable that I can see. On the other hand, 12 bits seems very small for
any kind of nonce or for a flow identification scheme, so it seems
that draft-blake and draft-donley would be knocked out.

   Brian

---

From: Rémi Després <[email-masked]>
Date: Tue, 10 Aug 2010 17:11:16 +0200

Hi Pascal,
More comments below.

> [Pascal] That does not come from me. I asked at the mike during the WG meeting, and the answer that was given was that for the purpose of a load balancing hash in the SP network, 11 or 12 bits woul…

What is clear, though, is that a hash of 11-12 bits has a higher probability of collisions than a hash on 20 bits.
If hashes become permitted (which simplifies support of FLs for load balancing), it is then clear that 20 is better than 12.

> Note that for the latter, the discussion was also that it is not obvious at the egress of the network to reset the FL to zero in order to enable the next AS to reuse it.

RFC 3697 isn't concerned with ASes, and doesn't need to be.
The proposal is only that, where load balancing is performed, 0 FLs MAY be replaced by meaningful values for this purpose.
A FL, once set to a non 0 value, never needs to be reset.

> So the conclusion was rather to use the bits is a DSCP fashion.

I am not sure what is meant by used in a DSCP fashion (some DSCP actual uses seem to me rather complex).

In any case, using FLs ONLY as FLs (essentially RFC 3697, but completed to be more practicable), is IMHO the most reasonable approach to improve the situation in a reasonable timeframe.

I understand that 20 unused bits in IP headers is bound to arouse envy, but adding a new role to FLs, with the original role becoming less efficient rather than more efficient, is IMHO a wrong approach. 

Cheers,
RD

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 10 Aug 2010 18:04:33 +0200

> a term in ROLL. The ROLL term would be grounded DODAG root).

I haven't yet quite found out whether RPL tries to be useful for networks with hosts, but if it does, the same kind if problem would occur when a host sets the flow label.

Gruesse, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 10 Aug 2010 18:16:13 +0200

On Aug 10, 2010, at 14:57, Michael Richardson wrote:

> The rational for lots of bits would be to

Hmm.  If this is intended as a routine way to handle the situation, the likelihood is still way to high.
(If this is just intended as a mitigation to increase robustness in a case that already is an error situation that is being actively avoided by other means, your argument might be more interesting.)

> set of requirements in my opinion, but I think that requirement will

DANGER.  Adding complexity to a protocol to cater for requirements that don't exist yet leads to convoluted solutions.  Never do that.
(Note that this situation is different from the selection of one out of a set of same-complexity alternatives, where it is quite useful to conjecture about potential future requirements to generate a tie breaker.  Note also that extensibility by itself is a known current requirement for any protocol, which is quite different from designing in all conceivable future extensions.)

Gruesse, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 10 Aug 2010 20:21:56 +0200

> (Although, some hosts are too weak to be routers)

OK, I'm not talking of "host" as in originates or terminates traffic, but "host" in the sense of "does not participate in routing".
It appears there is no such thing inside a RPL world then.
(Note that the traditional usage I'm using here is slightly different from the definition used in RFC2460, which says "does not forward" -- the old 1980's "hosts with RIP routing" would be hosts in RFC2460, but we have learned not to like these monsters.  That's why IPv6 has a separate ES-IS protocol called ND.)

> of constraints, which is encapsulated into the RPLinstanceID.

Hosts (as in "don't participate in routing") would have little idea of such concepts.

What you are describing is somehow surprisingly remote from IP.
In IP, hosts send packets to an IP address.
Sometimes there are additional considerations, which we so far have been addressing [sic] with the TOS byte, i.e., the DSCP today.
I don't understand yet why RPL needs a different concept.

(Note that Phil's point is also valid, but mostly orthogonal to mine.)

Gruesse, Carsten

---

From: Fred Baker <[email-masked]>
Date: Tue, 10 Aug 2010 12:06:54 -0700

I would find that surprising. There are ample cases where the originator of a high data rate flow (sensor data from a radio telescope to a number cruncher, to pick one example) might want to use the flow label to send data from one session down multiple paths. Multi-path TCP would be another example. I suppose you could argue that consistently alternating among N flow labels can be thought of as N separate flows, but if it's TCP there are other ramifications.

In any event, the requirement of the load balancer is not that the originator gives things a tag; we can develop such tags from a hash of the source and destination address quite well, thanks. The important thing is that things are tagged in a manner that helps the load balancer. There are two major uses of load balancers. In data centers, we balance sessions to/from sets of servers, and the question when a new session comes up is which server is most likely to be able to effectively serve it (has the necessary resources including access to the right disks, CPU cycles, etc). In bandwidth allocation uses (spreading load across multiple otherwise-equivalent paths), the question is how to use each of the paths most effectively - we want some traffic on each of them and we want none of the paths to be overloaded or introducing more delay or loss than others. Everyone's instinctive knee-jerk response to the question seems to be "hash something and depend on the law of large numbers to serve the need", and the observation of numerous operators is that the law of large numbers is a good start but is not an adequate solution. You really want the load balancer to be able to be smarter than that.

Which is why people that build load balancers frequently do so using NAT technology or some other way of tagging individual sessions. And why they are looking for mutable flow labels in this thread.

---

From: Fred Baker <[email-masked]>
Date: Tue, 10 Aug 2010 13:53:29 -0700

> It appears there is no such thing inside a RPL world then.

A RPL or Manet world doesn't have the 1970's-mentality limitation that hosts have to be stupid. Of all networking architectures, the Internet is the only one with that delusion.

---

From: Mark Smith <[email-masked]>
Date: Wed, 11 Aug 2010 07:09:34 +0930

Hi Pascal,

On Tue, 10 Aug 2010 16:24:42 +0200

> > Le 9 août 2010 à 12:17, Pascal Thubert (pthubert) a écrit :
<snip>
> Note that for the latter, the discussion was also that it is not obvious at the egress of the network to reset the FL to zero in order to enable the next AS to reuse it. So the conclusion was rathe…

The DSCP domain model use was my inspiration when I originally suggested
the idea of having flow domains. If the flow label stays mutable in
the network, the issue with the mandatory resetting to zero
idea is that downstream domains probably aren't going to trust upstream
domains to always do it - so they'll probably choose to do it
themselves anyway. Which is similar to how upstream DSCP values are or
rather aren't trusted.

One use of IPv4 DSCP I've seen in the past was to label or classify
packets for billing purposes. I disagreed with using DSCP for that,
because it is specifically for QoS use. It seems to me that the IPv6
flow label field is or could be a more general purpose field, and if
that DSCP labelling idea had been carried forward to IPv6, would be the
field to use for it.

Regards,
Mark.

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 11 Aug 2010 09:53:01 +1200

Brian

---

From: Rémi Després <[email-masked]>
Date: Wed, 11 Aug 2010 08:12:36 +0200

Right.

To be even more precise: 
- Flow endpoints (sometimes layer 4 and sometimes layer 3) should from now on be mandated to set FLs with non-0 values that statistically differ from a flow to another.
- However, we have to face that, so far, they are generally mandated to set FLs to 0.
- To mitigate the negative effect of this fact on uses of FLs as heuristic flow discriminants, intermediate nodes may replace 0 values by values that statistically differ from a flow to another.

Note also that intermediate nodes typically don't reassemble multi-packet datagrams. 
In this common case, they can only identify the IPv6 flow of a packet based on its two addresses and its next header. 
Setting the FL with a hash of this ID is still useful for downstream nodes to be able to use it (rather than treating the packet as if in the carryall set of 0-FL packets, or having to recompute its flow ID). 

Regards,
RD

---

From: Rémi Després <[email-masked]>
Date: Wed, 11 Aug 2010 09:06:54 +0200

Le 10 août 2010 à 21:06, Fred Baker a écrit :

> I would find that surprising. There are ample cases where the originator of a high data rate flow (sensor data from a radio telescope to a number cruncher, to pick one example) might want to use the flow label to send data from one session down multiple paths. Multi-path TCP would be another example.

> I suppose you could argue that consistently alternating among N flow labels can be thought of as N separate flows, but if it's TCP there are other ramifications.

Packets that are expected to go down multiple paths should be identified as belonging to different flows as far as FLs are concerned.
In the case of multi-path TCP, where "one or both peers are expected to have multiple addresses", this is already the case: different paths are taken by packets of different MPTCP "subflows", i.e. packets having different 5-tuples.

> 
> In any event, the requirement of the load balancer is not that the originator gives things a tag; we can develop such tags from a hash of the source and destination address quite well, thanks.

Originators of TCP and UDP connections should clearly hash 5-tuples when they exist (not 2-tuples in this case as you seem to assume).

> The important thing is that things are tagged in a manner that helps the load balancer.

Agreed.

> There are two major uses of load balancers.

> In data centers, we balance sessions to/from sets of servers, and the question when a new session comes up is which server is most likely to be able to effectively serve it (has the necessary resources including access to the right disks, CPU cycles, etc). In bandwidth allocation uses (spreading load across multiple otherwise-equivalent paths), the question is how to use each of the paths most effectively - we want some traffic on each of them and we want none of the paths to be overloaded or introducing more delay or loss than others. Everyone's instinctive knee-jerk response to the question seems to be "hash something and depend on the law of large numbers to serve the need", and the observation of numerous operators is that the law of large numbers is a good start but is not an adequate solution.

"Hash something" (and possibly a 2-tuple even when a 5-tuple is available) is clearly not as good as "hash 5 tuples if available", which I propose to become a SHOULD.

Taking a "good start" as starting point, and trying to improve from there, is what I try to propose.  

> You really want the load balancer to be able to be smarter than that.
> 
> Which is why people that build load balancers frequently do so using NAT technology or some other way of tagging individual sessions.

Are you arguing in favor of NATs in IPv6, adding load balancing as a reason to use them :-(?

As a convinced promoter of IPv6, and of its restored e2e transparency, I find that IPv6 load balancing solutions that don't need NATs are worth specifying.

> And why they are looking for mutable flow labels in this thread.

I don't see why 0-FL mutability wouldn't be sufficient for them (provided new values must statistically different for different flows, including subflows, micro-flows, whatever can be identified).

But more data on what would be their problem are most welcome. 

Regards,
RD

---

From: Brian E Carpenter <[email-masked]>
Date: Thu, 12 Aug 2010 14:02:31 +1200

> The intention is to have a BCP for network stack implementers to follow?

I don't there is a clear intention just yet, but my personal view is coming
round to a 3697bis document, which would presumably be Proposed Standard,
not BCP. But certainly we need more precise normative guidance.

> I apologize for the lack of context (I'm coming from ROLL): your sentence seems to suggest that flow labels today are mandated to be 0. This doesn't seem to be right: among other things, ping6 supp…

What 3697 says is
"A Flow Label of zero is used to indicate packets not part
of any flow.
...
A source node which does not assign traffic to flows MUST set
the Flow Label to zero."

It's a little self-referential, since flow *by definition* is
a set of packets with the same flow label (and address pair).
The rule is there to prevent accidental flow labels because
the programmer simply forgot to zero the field.

That being so, it isn't clear to me that we need a mandatory
flow label, although I can make some arguments for a SHOULD.

    Brian

---

From: Rémi Després <[email-masked]>
Date: Thu, 12 Aug 2010 14:54:46 +0200

Le 12 août 2010 à 10:47, Pascal Thubert (pthubert) a écrit :
> We'll note that the Hop by Hop + IP in IP is costly but
> the flow label mutable.

How strong the benefit isn't clear to me.
Since RPL networks are recognized as such by hosts, it seems easy to add a short field, just before the IP header, to contain whatever is useful for this kind of network.
Mixing functions with those of FLs would thus be avoided.

Did I miss something?
RD

---

From: Rémi Després <[email-masked]>
Date: Thu, 12 Aug 2010 15:01:12 +0200

Le 12 août 2010 à 04:02, Brian E Carpenter a écrit :

> not BCP.

+1

> But certainly we need more precise normative guidance.

Yes.
RD