Subject: Re: [mpls] Comments on MPLS-in-UDP

From: Eric Rosen <[email-masked]>
Date: Mon, 06 Aug 2012 15:51:04 -0400

> As for MPLS-in-GRE or MPLS-in-L2TPv3, it requires P routers to be capable
> of performing hash calculations on the specific "load-balancing
> field" contained in the encapsulating header (e.g., L2TPv3 session ID
> field or GRE key field).  Such requirement can not be met by some P routers
> which could only perform hash calulations on the five tuple of TCP/UDP
> packets. MPLS-in-UDP is only proposed to be used in such specific cases.

I see the logic of this argument. On the other hand, we already have three
standardized MPLS-in-IP encapsulations, each of which has been argued to
have an important advantage over the other two.  I don't think any of them
has really caught on though.  There's also been work on MPLS-in-IPsec, which
similarly has failed to generate much interest in practice.  After a dozen
years, do we really think there's going to be a lot of interest in yet
another MPLS-in-IP encapsulation?

Note that each new encapsulation impacts the data plane of the PE routers,
and the draft requires that each encapsulation header have a "source port"
field whose value is a computed hash of the TCP/UDP header of the payload.
This is not something that is completely trivial to implement.

A few miscellaneous technical comments.

Re the IANA Considerations:

>   Two distinct UDP destination port numbers indicating MPLS unicast 
>   and MPLS multicast respectively need to be assigned by IANA.

Since RFC 5332 we do not have codepoints for "MPLS unicast" and "MPLS
multicast".  Sections 4-8 of RFC 5332 allocate codepoints for a variety of
encapsulating protocols, you should take a look at that.

Do you expect to support MVPN over the UDP encapsulation?  

To get an idea of what the Security Area ADs are likely to require with
regard to the Security Considerations, please look at the Security
Considerations section of RFC 4023.

---

From: Thomas Nadeau <[email-masked]>
Date: Mon, 6 Aug 2012 20:42:36 -0400

>This is not something that is completely trivial to implement.

TOM: This was basically the argument made against the draft last week at
the meeting. I did not hear any convincing technical or otherwise
convincing reasoning as to why we should get behind yet another encap
scheme. Perhaps the draft's authors can ellaborate here on the list?

	--Tom

>_______________________________________________
>mpls mailing list
>[email-masked]
>https://www.ietf.org/mailman/listinfo/mpls

---

From: Xuxiaohu <[email-masked]>
Date: Tue, 7 Aug 2012 09:39:24 +0000

Hi Eric,

Thanks a lot for your insightful comments and suggestions. Please see my response inline.

> -----邮件原件-----
> 发件人: Eric Rosen [mailto:[email-masked]]
> 发送时间: 2012年8月7日 3:51
> 收件人: Xuxiaohu
> 抄送: [email-masked]; [email-masked]
> 主题: Re: Comments on MPLS-in-UDP
> 
> 

As for service provider backbones, most of them are capable of MPLS forwarding and therefore it's understandable that MPLS-in-IP encapsulations are not popular. However, in cloud data center network environments, some data center operators choose to use IP, rather than MPLS in the underlying network due to certain reasons. In addition, MPLS-based L2VPN and L3VPN technologies are considered to be used as scalable data center network solutions to support multi-tenancy. In this case, MPLS-in-IP encapsulations would be useful. To perform fine-grained load-balancing over LAG and ECMP paths in the underlying network, the MPLS-in-IP encapsulation header should contain an entry field so that the underlying network could achieve a fine-grained load-balancing of the encapsulated traffic. With the MPLS-in-UDP encapsulation, the source port field could be directly used as an entry field, and most importantly, most routers and layer3 switches have already been able to perform hash calculations on the five tuple of UDP packets.

> This is not something that is completely trivial to implement.

As for the implementation difficulty of PE routers, it seems there is no big difference between filling a hash value in the GRE key field (or L2TPv3 session ID field) and filling in a hash value in the UDP source port field, IMHO.

> encapsulating protocols, you should take a look at that.

Thanks a lot for reminding me of RFC5332. The corresponding text would be changed as follow " Two distinct UDP destination port numbers indicating MPLS and MPLS with upstream-assigned label respectively need to be assigned by IANA... "  Is that OK?

> Do you expect to support MVPN over the UDP encapsulation?

If data center operators want to use MVPN in their data centers, they could use MPLS-in-UDP encapsulation for VPN unicast traffic and VPN multicast traffic.

> Considerations section of RFC 4023.

It’s a very useful reference. We co-authors would add more detailed security considerations in the next revision.

Thanks again for your insight comments and suggestions. 

Best regards,
Xiaohu

---

From: Thomas Nadeau <[email-masked]>
Date: Tue, 7 Aug 2012 09:03:11 -0400

On 8/7/12 5:39 AM, "Xuxiaohu" <[email-masked]> wrote:

>As for service provider backbones, most of them are capable of MPLS
>forwarding and therefore it's understandable that MPLS-in-IP
>encapsulations are not popular. However, in cloud data center network
>environments, some data center operators choose to use IP, rather than
>MPLS in the underlying network due to certain reasons. In addition,
>MPLS-based L2VPN and L3VPN technologies are considered to be used as
>scalable data center network solutions to support multi-tenancy. In this
>case, MPLS-in-IP encapsulations would be useful.

TOM: I'd like to hear from any data center service providers as to this
point.  As Eric mentioned earlier, there are a number of encapsulations
available already which have mild (at best) adoption rates.  It would seem
to me that we would need some pretty good operational requirement demand
to make up yet another one other than just making for some neat work for
some people to do in standards land.

	--Tom

>To perform fine-grained load-balancing over LAG and ECMP paths in the
>underlying network, the MPLS-in-IP encapsulation header should contain an
>entry field so that the underlying network could achieve a fine-grained
>load-balancing of the encapsulated traffic. With the MPLS-in-UDP
>encapsulation, the source port field could be directly used as an entry
>field, and most importantly, most routers and layer3 switches have
>already been able to perform hash calculations on the five tuple of UDP
>packets.
>
>As for the implementation difficulty of PE routers, it seems there is no
>big difference between filling a hash value in the GRE key field (or
>L2TPv3 session ID field) and filling in a hash value in the UDP source
>port field, IMHO.
>
>Thanks a lot for reminding me of RFC5332. The corresponding text would be
>changed as follow " Two distinct UDP destination port numbers indicating
>MPLS and MPLS with upstream-assigned label respectively need to be
>assigned by IANA... "  Is that OK?
>
>If data center operators want to use MVPN in their data centers, they
>could use MPLS-in-UDP encapsulation for VPN unicast traffic and VPN
>multicast traffic.
>
>It’s a very useful reference. We co-authors would add more detailed
>security considerations in the next revision.
>

---

From: "S. Davari" <[email-masked]>
Date: Tue, 7 Aug 2012 07:30:50 -0700

Hi Xiaohu,

The complexity of 

Regards,
Shahram

---

From: "S. Davari" <[email-masked]>
Date: Tue, 7 Aug 2012 07:35:52 -0700

Hi Xiaohu

The complexity of computing the GRE key is same as UDP source. However routers are available that do GRE Key calculations while You need new hardware for your proposal.

Also data center architectures are now leaning toward VXLAN and NVGRE. 

Regards,
Shahram

---

From: "Eric Osborne (eosborne)" <[email-masked]>
Date: Tue, 7 Aug 2012 15:08:04 +0000

[please note...different Eric]

...

> As for the implementation difficulty of PE routers, it seems there is no big
> difference between filling a hash value in the GRE key field (or L2TPv3
> session ID field) and filling in a hash value in the UDP source port field, IMHO.

What happens if the UDP source port you generate is an ephemeral number already in use?  I realize that the draft doesn't specify a particular hashing algorithm but collisions are unavoidable.  Does the hardware that's imposing the hashed source port now need to track allocated source ports by all applications on the PE?  I could see some on-router firewalls choking on a packet that shared a source port with some other application.

eric

---

From: Xuxiaohu <[email-masked]>
Date: Wed, 8 Aug 2012 01:15:45 +0000

> -----邮件原件-----
> 发件人: Eric Osborne (eosborne) [mailto:[email-masked]]
> 发送时间: 2012年8月7日 23:08
> 收件人: Xuxiaohu; Eric Rosen (erosen)
> 抄送: [email-masked]
> 主题: RE: Comments on MPLS-in-UDP
> 
> What happens if the UDP source port you generate is an ephemeral number
> already in use?  I realize that the draft doesn't specify a particular hashing
> algorithm but collisions are unavoidable.  Does the hardware that's imposing
> the hashed source port now need to track allocated source ports by all
> applications on the PE?  I could see some on-router firewalls choking on a
> packet that shared a source port with some other application.

Hi Eric,

It's an interesting question. As said in section 3: " To ensure that the source port number is always in the range 49152 to 65535 which is required in some cases, instead of calculating a 16-bit hash, the ingress PE router would calculate a 14-bit hash and use those 14 bits as the least significant bits of the source port field while the most significant two bits would be set to binary 11."  Can this trick deal with the above concern you mentioned?

Best regards,
Xiaohu

---

From: Xuxiaohu <[email-masked]>
Date: Wed, 8 Aug 2012 01:28:28 +0000

> -----邮件原件-----
> 发件人: S. Davari [mailto:[email-masked]]
> 发送时间: 2012年8月7日 22:36
> 收件人: Xuxiaohu
> 主题: Re: [mpls] Comments on MPLS-in-UDP
> 
> Hi Xiaohu
> 
> The complexity of computing the GRE key is same as UDP source. However
> routers are available that do GRE Key calculations while You need new
> hardware for your proposal.

Hi Shahram,

The approach of using the GRE key field as an entry field relies on the assumption that core routers are capable of performing hash calculations on the GRE key field. However, such capability is not widely supported as far as I know. In contrast, performing hash calculations on the five-tuple of TCP/UDP packets is a default feature for most routers/layer3 switches.

> Also data center architectures are now leaning toward VXLAN and NVGRE.

It's just one possible option but not the only option, isn’t it?

Best regards,
Xiaohu

> forwarding and therefore it's understandable that MPLS-in-IP encapsulations
> are not popular. However, in cloud data center network environments, some
> data center operators choose to use IP, rather than MPLS in the underlying
> network due to certain reasons. In addition, MPLS-based L2VPN and L3VPN
> technologies are considered to be used as scalable data center network
> solutions to support multi-tenancy. In this case, MPLS-in-IP encapsulations
> would be useful. To perform fine-grained load-balancing over LAG and ECMP
> paths in the underlying network, the MPLS-in-IP encapsulation header should
> contain an entry field so that the underlying network could achieve a
> fine-grained load-balancing of the encapsulated traffic. With the MPLS-in-UDP
> encapsulation, the source port field could be directly used as an entry field, and
> most importantly, most routers and layer3 switches have already been able to
> perform hash calculations on the five tuple of UDP packets.
> >
> difference between filling a hash value in the GRE key field (or L2TPv3 session
> ID field) and filling in a hash value in the UDP source port field, IMHO.
> >
> changed as follow " Two distinct UDP destination port numbers indicating MPLS
> and MPLS with upstream-assigned label respectively need to be assigned by
> IANA... "  Is that OK?
> >
> > If data center operators want to use MVPN in their data centers, they could
> use MPLS-in-UDP encapsulation for VPN unicast traffic and VPN multicast
> traffic.
> >

---

From: "Eric Osborne (eosborne)" <[email-masked]>
Date: Wed, 8 Aug 2012 11:33:02 +0000

> It's an interesting question. As said in section 3: " To ensure that the source
> port number is always in the range 49152 to 65535 which is required in some
> cases, instead of calculating a 16-bit hash, the ingress PE router would
> calculate a 14-bit hash and use those 14 bits as the least significant bits of the
> source port field while the most significant two bits would be set to binary
> 11."  Can this trick deal with the above concern you mentioned?
> 

No.

See, for example, rfc6335 section 6.  49152-65535 is the ephemeral port range.  So effectively the entire 16-bit UDP port space is 100% allocated. 

eric

---

From: Xuxiaohu <[email-masked]>
Date: Thu, 9 Aug 2012 01:49:42 +0000

> 发送时间: 2012年8月8日 19:33
> See, for example, rfc6335 section 6.  49152-65535 is the ephemeral port
> range.  So effectively the entire 16-bit UDP port space is 100% allocated.

Hi Eric,

As stated in section 6 of RFC6335, "...the Dynamic Ports, also known as the Private or Ephemeral Ports, from 49152-65535 (never assigned)..."  In addition, the specific destination port number to be assigned by IANA can indicate the payload of the UDP is a MPLS packet. Hence I don't understand why it would cause any problem. Could you please explain your concerns with a concrete example?

Best regards,
Xiaohu

---

From: "Eric Osborne (eosborne)" <[email-masked]>
Date: Thu, 9 Aug 2012 12:22:22 +0000

...

> Hi Eric,
> 
> As stated in section 6 of RFC6335, "...the Dynamic Ports, also known as the
> Private or Ephemeral Ports, from 49152-65535 (never assigned)..."  In
> addition, the specific destination port number to be assigned by IANA can
> indicate the payload of the UDP is a MPLS packet. Hence I don't understand
> why it would cause any problem. Could you please explain your concerns
> with a concrete example?

The ephemeral range is used for source port numbers.   
http://en.wikipedia.org/wiki/Ephemeral_port

If a host stack on a router allocates source port 50000 to be used to talk to a particular server and your scheme also uses source port 50000 at the same time due to hashing, you have two distinct applications using the same source port.  This is Bad, as I can see it breaking things like on-router firewalls and anything in the middle.

In order to avoid this collision you need to ensure that your hashing doesn't collide with your ephemeral port allocation, which could mean a ton of real-time coordination between forwarding hardware and the host stack.

eric

---

From: Eric Rosen <[email-masked]>
Date: Thu, 09 Aug 2012 10:24:29 -0400

A few more miscellaneous comments.

How many possible hash values are actually needed?  Probably not 2**14.  If
ECMP in the transit nodes uses only four equal cost paths, say, can't one
achieve the necessary entropy with just four hash values?

If one uses a small number of hash values, I wonder if it would be better to
use the "destination port" field rather than the "source port" field to
convey the entropy.  The possible destination port values could even be
dynamically allocated by the receiving endpoint of the UDP "tunnel" and
signaled to the transmitting endpoint.  (We know there's already some
signaling between the tunnel endpoints, since the MPLS labels have to be
signaled.)

It's convenient to use the port number to indicate whether the top label of
the payload packet is downstream-assigned or upstream-assigned, but it's not
strictly necessary to do so, as one could devise a way to indicate this in
the first word of the payload.

---

From: Alia Atlas <[email-masked]>
Date: Thu, 9 Aug 2012 10:36:40 -0400

The number of hash values needed depends on how many transit nodes
there are, the number of paths at each of those, and how well hash
diversity is maintained on each path.   Say there are 3 transit nodes
with 4 paths each- with perfect splitting, then one needs to specify
64 different hash values.  Perfect  splitting isn't likely either.

This isn't to say that a new encapsulation method is needed - but the
problem of hashing is real.

Alia

---

From: "Eric Osborne (eosborne)" <[email-masked]>
Date: Thu, 9 Aug 2012 14:43:22 +0000

If you have four paths and each hop has an arbitrary (proprietary, in most cases) hash algorithm, don't you need far more than four hash values to ensure decent distribution across the buckets?

eric

> -----Original Message-----
> From: Eric Rosen (erosen)
> Sent: Thursday, August 09, 2012 10:24 AM
> To: Xuxiaohu
> Cc: Eric Osborne (eosborne); Eric Rosen (erosen); [email-masked]
> Subject: Re: Comments on MPLS-in-UDP
>

---

From: Eric Rosen <[email-masked]>
Date: Thu, 09 Aug 2012 11:16:36 -0400

Alia> Say there are 3 transit nodes with 4 paths each- with perfect
Alia> splitting, then one needs to specify 64 different hash values.

Eric O> If you have four paths and each hop has an arbitrary (proprietary, in
Eric O> most cases) hash algorithm, don't you need far more than four hash
Eric O> values to ensure decent distribution across the buckets?

Yes, sorry, don't know what I was thinking.  If the DC environment requires
anything like a 2**n-way (or more) split, and/or if the hashing algorithms
are proprietary, you want as many hash buckets as possible.  

But the more hash values you need, the less attractive it becomes to store
the hash value in either of the UDP port fields.

---

From: Xuxiaohu <[email-masked]>
Date: Fri, 10 Aug 2012 01:01:05 +0000

> 发送时间: 2012年8月9日 20:22
> If a host stack on a router allocates source port 50000 to be used to talk to a
> particular server and your scheme also uses source port 50000 at the same
> time due to hashing, you have two distinct applications using the same source
> port.  This is Bad, as I can see it breaking things like on-router firewalls and
> anything in the middle.

Do you believe that on-router firewalls or NAT boxes would be deployed on the path between PE routers in practice?

Best regards,
Xiaohu

> In order to avoid this collision you need to ensure that your hashing doesn't
> collide with your ephemeral port allocation, which could mean a ton of
> real-time coordination between forwarding hardware and the host stack.
> 
> 
> eric
> 
> 
> >
> > Best regards,
> > Xiaohu
> >
> >
> > >
> > >
> > >
> > > eric
> > >

---

From: Xuxiaohu <[email-masked]>
Date: Fri, 10 Aug 2012 01:18:39 +0000

> 发送时间: 2012年8月9日 22:24
> 抄送: Eric Osborne (eosborne); Eric Rosen (erosen); [email-masked]

Hi Eric,

It's the routine method, IMHO.

> the first word of the payload.

Did you mean the MPLS header by saying the first word of the UDP payload? If so, which field can be used to fill this role?

Best regards,
Xiaohu

---

From: Xuxiaohu <[email-masked]>
Date: Fri, 10 Aug 2012 02:01:15 +0000

> 发送时间: 2012年8月9日 23:17
> 收件人: Alia Atlas
> 抄送: [email-masked]; Xuxiaohu; [email-masked]

Yes, the more, the better. However, isn't this 14-bit entry label enough in the DC environment?

Best regards,
Xiaohu

---

From: "Eric Osborne (eosborne)" <[email-masked]>
Date: Fri, 10 Aug 2012 13:15:00 +0000

...

> the path between PE routers in practice?

1)  I'm aware of at least one implementation where the PE itself creates a dynamic stateful firewall to help protect against DDOS.  Carving a 14-bit hole for ephemeral ports seems like a bad idea, and real-time coordination between hashing hardware and socket software is hard.

2) I have, over the years, discovered that I'm extremely bad at predicting what people will never, ever do with their networks.  This applies doubly to things that we think are 'sensible' - one operator's best practice is another operator's anathema.  

3) It is bad practice to assume that nobody will mind if you overload a field and then overload the field based on that assumption.  

While it may be possible to overload a long used and well understood field in the right circumstances, you'd have to have pretty strong justification.  I don't believe that "some hardware may not be able to handle other solutions to this problem" is strong enough.  

eric

---

From: Xuxiaohu <[email-masked]>
Date: Tue, 14 Aug 2012 09:20:31 +0000

Hi Eric,

Thanks a lot for your comments.

In fact, using UDP encapsulation and filling the source port field with a hash value isn't a totally novel idea since it had been proposed in many other encapsulation related protocols. One example is LISP, please see section 6.5 (http://tools.ietf.org/html/draft-ietf-lisp-23#section-6.5) for more details.

Any further comments are more than welcome.

Best regards,
Xiaohu

> 发送时间: 2012年8月10日 21:15
> 1)  I'm aware of at least one implementation where the PE itself creates a
> dynamic stateful firewall to help protect against DDOS.  Carving a 14-bit hole
> for ephemeral ports seems like a bad idea, and real-time coordination between
> hashing hardware and socket software is hard.
> 
> 2) I have, over the years, discovered that I'm extremely bad at predicting what
> people will never, ever do with their networks.  This applies doubly to things
> that we think are 'sensible' - one operator's best practice is another operator's
> anathema.
> 
> 3) It is bad practice to assume that nobody will mind if you overload a field and
> then overload the field based on that assumption.
> 
> 
> While it may be possible to overload a long used and well understood field in
> the right circumstances, you'd have to have pretty strong justification.  I don't
> believe that "some hardware may not be able to handle other solutions to this
> problem" is strong enough.