Subject: [armd] address resolution requirement from hosts to overlay edge nodes. Any opinion?

From: Linda Dunbar <[email-masked]>
Date: Fri, 10 Feb 2012 21:28:20 +0000

We all believe that the current trend for Data Center network is going towards overlay network, be it TRILL, MAC-in-MAC (SPB), NV03, etc. In Overlay network, each Overlay Border Node (OBN) has to map the destination address of a data packet to an egress Overlay Border Node. Very much like ARP/ND which is to resolve physical addresses (i.e. MAC Addresses) from target IP addresses.

Here are some distinct characteristics of hosts in Data Center:

1.  Even though the communication between DC hosts to external peers may be small in volume comparing with the east-west (intra-data center) traffic,  majority of hosts in data center still do communicate with external peers.

2.  Traffic entering/exiting the Data Center Overlay could be via multiple Overlay Border nodes.

3.  The placement of VMs/hosts to server racks is orchestrated by Management Entities. So Overlay Border Nodes can, at least in theory, get the hosts <-> OverlayBorderNode mapping information from directory server(s). There could be multiple directory servers serving a very large data center.

>From the very high level view, the directory providing mapping from hosts to Overlay Border Node seems very straight forward. However, if you look deeper, following issues come out:

-   When hosts can enter or exit the overlay network via multiple OBNs, is it necessary for each OBN to be aware of which egress OBN can reach the target? Or, is it necessary for each OBN to advertise its connectivity status of all the attached hosts to all other OBNs?

-   When migration is orchestrated by management entity (or entities), there could be multiple instances of directory servers

o   is it necessary for each directory server to have full information? What if some directory servers only have partial information (for very large DC environment)?

o   Is it necessary for each OBN to report connectivity status of all the attached hosts to directory servers?

o   Should resolution requests be load balanced among them?

-   A data frame arriving at OBN normally might have an VID in the data frame, the egress OBN might have a different VID value for the  same virtual network instance. Original VID loses its meaning when traversing to Egress overlay border. Proper Ingress VID -> overlay network instance ID - >to Egress VID  resolution (or mapping) is needed.

-   Etc.

It was suggested that ARMD WG should have a requirement draft on Address Resolution from data center host addresses to Overlay Border Node addresses so that the industry can have a common requirement for various types of overlay for data center environment.

A draft on this topic could potentially trigger more input on various constraints or requirement for hosts <-> OBN mapping in data center environment.

Any opinion on this?

Linda Dunbar

---

From: Anoop Ghanwani <[email-masked]>
Date: Fri, 10 Feb 2012 18:12:25 -0800

I think this is a very interesting problem.  If BOTH the MAC-to-IP binding,
and the MAC-to-tunnel-end-point binding can be resolved using a directory
service, it would eliminate the need for IP multicast support from the network,
provided there are no multicast applications.

If both cannot be done and ARP/ND must still be used to find the MAC-to-IP
binding, then I don't see much value in a directory service to get only the
MAC-to-tunnel-end-point mapping since that can be inferred from the ARP
traffic itself (as long as a MAC address sits behind only a single
tunnel end point).

Anoop

On Fri, Feb 10, 2012 at 1:28 PM, Linda Dunbar <[email-masked]> wrote:
> We all believe that the current trend for Data Center network is going
> towards overlay network, be it TRILL, MAC-in-MAC (SPB), NV03, etc. In
> Overlay network, each Overlay Border Node (OBN) has to map the destination
> address of a data packet to an egress Overlay Border Node. Very much like
> ARP/ND which is to resolve physical addresses (i.e. MAC Addresses) from
> target IP addresses.
>
>
>
> 1.  Even though the communication between DC hosts to external peers may be
> small in volume comparing with the east-west (intra-data center) traffic,
> majority of hosts in data center still do communicate with external peers.
>
>
>
> 3.  The placement of VMs/hosts to server racks is orchestrated by Management
> Entities. So Overlay Border Nodes can, at least in theory, get the hosts <->
> OverlayBorderNode mapping information from directory server(s). There could
> be multiple directory servers serving a very large data center.
>
>
>
> From the very high level view, the directory providing mapping from hosts to
> Overlay Border Node seems very straight forward. However, if you look
> deeper, following issues come out:
>
>
>
> -   When hosts can enter or exit the overlay network via multiple OBNs, is
> it necessary for each OBN to be aware of which egress OBN can reach the
> target? Or, is it necessary for each OBN to advertise its connectivity
> status of all the attached hosts to all other OBNs?
>
>
>
> -   When migration is orchestrated by management entity (or entities), there
> could be multiple instances of directory servers
>
> o   is it necessary for each directory server to have full information? What
> if some directory servers only have partial information (for very large DC
> environment)?
>
> -   A data frame arriving at OBN normally might have an VID in the data
> frame, the egress OBN might have a different VID value for the  same virtual
> network instance. Original VID loses its meaning when traversing to Egress
> overlay border. Proper Ingress VID -> overlay network instance ID - >to
> Egress VID  resolution (or mapping) is needed.
>
>
>
> -   Etc.
>
>
>
> It was suggested that ARMD WG should have a requirement draft on Address
> Resolution from data center host addresses to Overlay Border Node addresses
> so that the industry can have a common requirement for various types of
> overlay for data center environment.
>
>
>
> A draft on this topic could potentially trigger more input on various
> constraints or requirement for hosts <-> OBN mapping in data center
> environment.
>
>
>
> Any opinion on this?
>
>
>
> Linda Dunbar
>
>
>
>
> _______________________________________________
> armd mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/armd
>

---

From: Mike McBride <[email-masked]>
Date: Mon, 13 Feb 2012 16:01:44 -0800

Anoop,

> provided there are no multicast applications.

A directory could eliminate any reliance upon IP Multicast for MAC to
tunnel end point binding but would not eliminate the need for IP
Multicast in the network. You would still want the dynamic nature of
joining and pruning along with the efficient data delivery across the
overlay network. Good solutions will take mcast applications into
proper consideration.

> traffic itself (as long as a MAC address sits behind only a single

You are probably right but, either way, a directory approach is
probably worth further exploration.

mike

---

From: Anoop Ghanwani <[email-masked]>
Date: Tue, 14 Feb 2012 08:23:47 -0800

> overlay network. Good solutions will take mcast applications into

I agree that we would still need IP multicast in the network to
support multicast applications.  My point was that a directory-based
mechanism would eliminate the need for IP multicast if the only reason
it was need was for the purpose of getting ARP traffic through the network.

That said, there are unicast applications (such as Microsoft's Network
Load Balancing) that require multicast at L2, and because they
require multicast at L2, such a network would still need to support
IP multicast.

So really, the directory-based approach would be helpful
only in a narrow set of circumstances -- no multicast applications
including things like NLB.

Anoop

---

From: AshwoodsmithPeter <[email-masked]>
Date: Tue, 14 Feb 2012 18:29:13 +0000

>including things like NLB.

Anoop, even if you have a good multicast in the underlay network and its being used by things like MS NLB there is still enormous value in eliminating the multicast methods that are used to establish the <c-mac, ip-addr, b-mac tunnel> tripple relationships via some form of database etc.

Peter

---

From: Anoop Ghanwani <[email-masked]>
Date: Tue, 14 Feb 2012 10:55:26 -0800

> Anoop, even if you have a good multicast in the underlay network and its being used by things like MS NLB there is still enormous value in eliminating the multicast methods that are used to establi…

That's correct, Peter.  As I said in my first response, we have
to solve both problems -- ipaddr->cmac and cmac->bmac/tunnel.
If we solve only one, then the value proposition is significantly
diluted.  As an added benefit, if we solve this problem and there
is no other need for L2 multicast in the network (NLB, directed
broadcast), then the overlay does not need multicast support from
the underlying network.  Requiring IP multicast (especially PIM bidir)
from the network is really a big deal, especially at the kind of
scale that the VXLAN/NVGRE proposals need it.

Anoop

---

From: David Allan I <[email-masked]>
Date: Tue, 14 Feb 2012 14:05:30 -0500

I have to admit I am a bit confused by the direction this is going. 

There seems to be a view that multicast is bad, yet it is a useful tool to permit a significant amount of functionality to be delegated to end systems, and is a part of useful address aggregation in the form of the subnet. IMO virtualized broadcast domains are good things, not something to be avoided, and they permit a trusted resource community to do a lot of self-organizing. 

Perhaps you should be thinking about improving multicast, not evicerating everything else. And as a general design principle, I believe that applies to most areas where the tail is wagging the dog in these deliberations.

My 2 cents
Dave

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Anoop Ghanwani
Sent: Tuesday, February 14, 2012 10:55 AM
To: AshwoodsmithPeter
Cc: Thomas Narten; [email-masked]; Igor Gashinsky
Subject: Re: [armd] address resolution requirement from hosts to overlay edge nodes. Any opinion?

On Tue, Feb 14, 2012 at 10:29 AM, AshwoodsmithPeter <[email-masked]> wrote:

>>Load Balancing) that require multicast at L2, and because they require 
>>multicast at L2, such a network would still need to support IP 
>>multicast.
>
>>So really, the directory-based approach would be helpful only in a 
>>narrow set of circumstances -- no multicast applications including 
>>things like NLB.
>

That's correct, Peter.  As I said in my first response, we have to solve both problems -- ipaddr->cmac and cmac->bmac/tunnel.
If we solve only one, then the value proposition is significantly diluted.  As an added benefit, if we solve this problem and there is no other need for L2 multicast in the network (NLB, directed broadcast), then the overlay does not need multicast support from the underlying network.  Requiring IP multicast (especially PIM bidir) from the network is really a big deal, especially at the kind of scale that the VXLAN/NVGRE proposals need it.

Anoop

---

From: Anoop Ghanwani <[email-masked]>
Date: Tue, 14 Feb 2012 11:36:00 -0800

> Perhaps you should be thinking about improving multicast, not evicerating everything else. And as a general design principle, I believe that applies to most areas where the tail is wagging the dog…

Theoretically speaking, it is easy to brush this off and say "let's just improve
multicast", and if we do nothing then that is pretty much what will happen.
However, as a practical matter, there is a lot of hardware out there that
has significant problems with multicast.  This means we delay the deployment
of such solutions until the hardware catches up (if it ever does), or let smart
people who don't care about standards figure out ways around it that are
non-interoperable, and worry about about the standardization aspect later.

Anoop

---

From: David Allan I <[email-masked]>
Date: Tue, 14 Feb 2012 14:58:54 -0500

Hi Anoop:

Eeesh, no offence but you are advocating overlaying complexity on complexity vs. cutting to the heart of the problem. 

"No-hardware left behind" IMO is social engineering, not technology.

Cheers
Dave 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Anoop Ghanwani
Sent: Tuesday, February 14, 2012 11:36 AM
To: David Allan I
Cc: AshwoodsmithPeter; Thomas Narten; [email-masked]; Igor Gashinsky
Subject: Re: [armd] address resolution requirement from hosts to overlay edge nodes. Any opinion?

> Perhaps you should be thinking about improving multicast, not evicerating everything else. And as a general design principle, I believe that applies to most areas where the tail is wagging the dog…

Theoretically speaking, it is easy to brush this off and say "let's just improve multicast", and if we do nothing then that is pretty much what will happen.
However, as a practical matter, there is a lot of hardware out there that has significant problems with multicast.  This means we delay the deployment of such solutions until the hardware catches up (if it ever does), or let smart people who don't care about standards figure out ways around it that are non-interoperable, and worry about about the standardization aspect later.

Anoop

---

From: Mike McBride <[email-masked]>
Date: Tue, 14 Feb 2012 12:14:44 -0800

And what protocol improvements are needed for multicast with overlays?
The multicast community is ready to help. Multicast makes us happy. Or
are you solely talking about vendor implementation, s/w release, hw
support problems? AFAIK multicast works fairly well with VXLAN, TRILL,
L2VPN... What's missing?

mike

---

From: Anoop Ghanwani <[email-masked]>
Date: Tue, 14 Feb 2012 12:32:58 -0800

> support problems? AFAIK multicast works fairly well with VXLAN, TRILL,

The whole problem of sending L2 multicast over a campus or data center
backbone, in any sort of significant way, is a new one enabled for the first
time by overlays.  There are interesting challenges when pushing large
amounts of multicast traffic through a network, and  have thus far been dealt
with using purpose-built networks.  While the overlay proposals have
been careful not to impose new protocol requirements, they have not
addressed the issues of performance and scalability, nor the large-scale
availability of these protocols.

Anoop

---

From: Mike McBride <[email-masked]>
Date: Tue, 14 Feb 2012 13:16:28 -0800

Anoop,

> availability of these protocols.

There are interesting challenges with unicast and multicast in big
flat DC networks. That's why we are here. One of the constant themes
we hear with multicast is the FUD that it doesn't scale. Especially
from vendors who are pushing another solution or who have a crappy
multicast implementation. Meanwhile multicast is being deployed in
large scale networks without scaling issue. Large L2 overlay networks?
I don't know. Would be good to find out from the community about
performance and scalability of multicast in the DC.

mike

---

From: Anoop Ghanwani <[email-masked]>
Date: Tue, 14 Feb 2012 13:53:20 -0800

On Tue, Feb 14, 2012 at 1:16 PM, Mike McBride <[email-masked]> wrote:

> large scale networks without scaling issue.

Like I said earlier...we have large-scale multicast today, but
those are purpose-built networks, and at least today, they are no
where near the kinds of scale where people want to go with overlay
technology (in terms of size of network and traffic patterns).

> Large L2 overlay networks?

Perhaps it's best we wait until the problems are little more obvious.
Or may be they're just non-existent and I get proven wrong.  Either
way, it looks like the correct thing for now is to do nothing.

Anoop

---

From: AshwoodsmithPeter <[email-masked]>
Date: Tue, 14 Feb 2012 22:15:29 +0000

Irrespective of what is right or wrong etc. technically, I think it would be very useful to have a taxonomy of the different applications that are using multicast / broadcast for things other than ARP.

The complexity of the underlay depends a lot on what happens with the use of multicast by the overlay.

For example. Assume Microsoft load balancing (for better or worse). If I remember correctly, and sombody correct me if I'm wrong, it generates multicast frames from otherwise unicast packets at the router so that multiple hosts all get copies and then only one responds for a given flow... or something like that.

So if we are using something like a VXLAN overlay we need to map the L2 multicast in the overlay to L3 multicast in the underlay or do head end replication in the overlay and get a wack load of duplicate frames on the first link from the router to the core "switch" (really it's a router). Not good. The solution presumably is to run potentially thousands of PIM-SM or something messages to generate/maintain the required multicast state in the IP underlay.

Anyway the behavior of the upper layer with respect to broadcast / multicast affects the choice of head end, (*,G) or (S,G) replication in the underlay, which affects the opex and capex of the entire solution, so I think its an important question to get a handle on earlier rather than later.

Peter

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Anoop Ghanwani
Sent: Tuesday, February 14, 2012 4:53 PM
To: Mike McBride
Cc: Thomas Narten; [email-masked]; Igor Gashinsky
Subject: Re: [armd] address resolution requirement from hosts to overlay edge nodes. Any opinion?

Anoop

---

From: "Michael K. Smith - Adhost" <[email-masked]>
Date: Tue, 14 Feb 2012 22:22:49 +0000

> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> AshwoodsmithPeter
> Sent: Tuesday, February 14, 2012 2:15 PM
> To: Anoop Ghanwani; Mike McBride
> Cc: Thomas Narten; Igor Gashinsky; [email-masked]
> Irrespective of what is right or wrong etc. technically, I think it would be very
> useful to have a taxonomy of the different applications that are using
> multicast / broadcast for things other than ARP.
> 
> For example. Assume Microsoft load balancing (for better or worse). If I
> remember correctly, and sombody correct me if I'm wrong, it generates
> multicast frames from otherwise unicast packets at the router so that
> multiple hosts all get copies and then only one responds for a given flow... or
> something like that.
> 
> So if we are using something like a VXLAN overlay we need to map the L2
> multicast in the overlay to L3 multicast in the underlay or do head end
> replication in the overlay and get a wack load of duplicate frames on the first
> link from the router to the core "switch" (really it's a router). Not good. The
> solution presumably is to run potentially thousands of PIM-SM or something
> messages to generate/maintain the required multicast state in the IP
> underlay.
> 
> Anyway the behavior of the upper layer with respect to broadcast / multicast
> affects the choice of head end, (*,G) or (S,G) replication in the underlay,
> which affects the opex and capex of the entire solution, so I think its an
> important question to get a handle on earlier rather than later.
> 

Microsoft is configurable for IGMP Multicast and Unicast.  There are also the other LB protocols such as CARP, VRRP, GLBP, HSRP, etc.  

Mike

---

From: Joel jaeggli <[email-masked]>
Date: Tue, 14 Feb 2012 21:36:25 -0800

On 2/14/12 14:15 , AshwoodsmithPeter wrote:
> 
> Irrespective of what is right or wrong etc. technically, I think it
> would be very useful to have a taxonomy of the different applications
> that are using multicast / broadcast for things other than ARP.

ganglia is a common datacenter multicast user

vertica clusters can use broadcast for communication and span hundres of
nodes, however unwise that approach may be.

> For example. Assume Microsoft load balancing (for better or worse).
> If I remember correctly, and sombody correct me if I'm wrong, it
> generates multicast frames from otherwise unicast packets at the
> router so that multiple hosts all get copies and then only one
> responds for a given flow... or something like that.
> 
> So if we are using something like a VXLAN overlay we need to map the
> L2 multicast in the overlay to L3 multicast in the underlay or do
> head end replication in the overlay and get a wack load of duplicate
> frames on the first link from the router to the core "switch" (really
> it's a router). Not good. The solution presumably is to run
> potentially thousands of PIM-SM or something messages to
> generate/maintain the required multicast state in the IP underlay.
> 
> Anyway the behavior of the upper layer with respect to broadcast /
> multicast affects the choice of head end, (*,G) or (S,G) replication
> in the underlay, which affects the opex and capex of the entire
> solution, so I think its an important question to get a handle on
> earlier rather than later.
> 
> Peter
> 
> -----Original Message----- From: [email-masked]
> [mailto:[email-masked]] On Behalf Of Anoop Ghanwani Sent:
> Tuesday, February 14, 2012 4:53 PM To: Mike McBride Cc: Thomas
> Narten; [email-masked]; Igor Gashinsky Subject: Re: [armd] address
> resolution requirement from hosts to overlay edge nodes. Any
> opinion?
> 
>> flat DC networks. That's why we are here. One of the constant
>> themes we hear with multicast is the FUD that it doesn't scale.
>> Especially from vendors who are pushing another solution or who
>> have a crappy multicast implementation. Meanwhile multicast is
>> being deployed in large scale networks without scaling issue.
> 
> Like I said earlier...we have large-scale multicast today, but those
> are purpose-built networks, and at least today, they are no where
> near the kinds of scale where people want to go with overlay 
>> Large L2 overlay networks? I don't know. Would be good to find out
>> from the community about performance and scalability of multicast
>> in the DC.
> 
> Perhaps it's best we wait until the problems are little more
> obvious. Or may be they're just non-existent and I get proven wrong.
> Either way, it looks like the correct thing for now is to do
> nothing.
> 
> Anoop _______________________________________________ armd mailing
> list [email-masked] https://www.ietf.org/mailman/listinfo/armd 
> _______________________________________________ armd mailing list 
> [email-masked] https://www.ietf.org/mailman/listinfo/armd
>

---

From: Thomas Narten <[email-masked]>
Date: Wed, 15 Feb 2012 09:07:38 -0500

Mike McBride <[email-masked]> writes:

> from the community about performance and scalability of multicast in
> the DC.

My impression is that many data centers do not enable IP multicast on
their routers. That means you can use link-local multicast (which
works fine within one IP subnet and doesn't really have scaling
issues). But if you want multicast that goes beyond one link (and IP
subnet), which is presumably necessary for an overlay like
VXLAN/NVGRE, that is where you have problems.

The question is not even whether L3 multicast scales. It's whether the
DC operater is willing to enable such multicast.

Thomas

---

From: Aldrin Isaac <[email-masked]>
Date: Wed, 15 Feb 2012 09:39:40 -0500

Any DC with applications performing heavy pub-sub are more than likely using multicast.  I am aware of many companies for whom multicast is critical and enabled on their DC routers.

---

From: AshwoodsmithPeter <[email-masked]>
Date: Wed, 15 Feb 2012 14:42:27 +0000

Thanks Joel,

Something else to consider here is just what the multicast is being used for.

Obviously one use is true multicast data, and if that's the case the underlay needs to be efficient (S,G) (*,G) or such when any serious volume of traffic is involved.

However another use is multicast as a way to find/signal other participants in your application. I do not know how the two applications 'ganglia' and 'vertica' behave below, but if they and others fit in this category perhaps a better solution is to give them an API a bit closer to what they are trying to do eg: register to be member in a group, find members in some group, signal members in some group, rather than a lower level 'multicast this message to this group'... because there are lighter ways to do these membership/signalling functions than brute force multicast.

Peter

---

From: Linda Dunbar <[email-masked]>
Date: Wed, 15 Feb 2012 14:53:11 +0000

Aldrin, 

Is "pub-sub" same as "Publish-subscribe" related applications? 
Are all the "subscribers" in the same subnet as the "publisher"? 
If yes, it is almost like many multicast groups being created (in real time?), with subscribers subscribing to a subset of the multicast groups, isn't it? 
If they are in different subnets, is it like L3 multicast?

Linda

> Aldrin Isaac
> Sent: Wednesday, February 15, 2012 8:40 AM
> To: Thomas Narten
> Cc: [email-masked]
> Subject: Re: [armd] Multicast in the data center [was Re: address
> resolution requirement from hosts to overlay edge nodes. Any opinion?]
> 
> Any DC with applications performing heavy pub-sub are more than likely
> using multicast.  I am aware of many companies for whom multicast is
> critical and enabled on their DC routers.
>

---

From: AshwoodsmithPeter <[email-masked]>
Date: Wed, 15 Feb 2012 15:35:10 +0000

> VXLAN/NVGRE, that is where you have problems.

Agreed. If you extrapolate a bit and imagine VXLAN with 1000's of logical groups.. then that maps either to head end replication in the hypervisor, or to IGMP from the hypervisor and PIM between the TOR and CORE 'switches' and the gateway router.

> DC operater is willing to enable such multicast.

No choice if you want to run a Layer 2 Emulation over IP and you care about multicast banwdidh effeciency.

This highlights one of the major differences between the L2 over IP and L2 over L2 solutions, i.e. signalling v.s. computation for the multicast state.

This is why I think an understanding of the true uses of that multicast is important because if 90% of the applications are only finding each other and signalling each other using multicast there are better ways but if they are truly multicasting data in large volumes .. well we need extremely good underlay multicast scale.

Peter

---

From: David Allan I <[email-masked]>
Date: Wed, 15 Feb 2012 10:45:04 -0500

We do need to be clear between protocols that leverage layer 2 broadcast and multicast vs. those that genuinely need layer 3 and the multicast group spans multiple subnets.

I suspect we are really discussing the former, the latter looks to me like complexity well beyond what a pool of VMs supporting an application need to autodiscover and self organize.

That an overlay may use IP multicast at the server layer to virtualize layer 2 broadcast/multicast is a separate issue.

Cheers
Dave

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of AshwoodsmithPeter
Sent: Wednesday, February 15, 2012 7:35 AM
To: Aldrin Isaac; Thomas Narten
Cc: [email-masked]
Subject: Re: [armd] Multicast in the data center [was Re: address resolution requirement from hosts to overlay edge nodes. Any opinion?]

Peter

---

From: Aldrin Isaac <[email-masked]>
Date: Wed, 15 Feb 2012 16:03:09 -0500

Hi Linda,

Yes, pub-sub is the same as publish-subscribe.  The publishers and subscribers are both in the same subnet and across subnets, even on opposite sides of of a WAN.  In fact many of the applications both publish and subscribe simultaneously.  Some applications use it for heart-beat and as bi-directional application control channels, while others use it to send fat (50Mbps to 1Gbps+) one-way data flows to many downstream recipients, and everything in between.  Multicast is generally organized into small, medium and large flow groups to manage state.  Small flows are generally ad-hoc (come and go at will) and forced to share a limited set of small flow groups.  Medium flow groups are shared by communities of interest.  Large flows get their own multicast groups to avoid excessive bandwidth use.  A PGM layer is used for reliable multicast.  Others will have different use-cases and operating models.  Bottom line -- multicast IS being used extensively in the DC, but not by run-of-the-mill applications.

-- aldrin

---

From: Igor Gashinsky <[email-masked]>
Date: Wed, 15 Feb 2012 14:08:30 -0800 (PST)

On Tue, 14 Feb 2012, Mike McBride wrote:

:: Anoop,
:: 
:: On Tue, Feb 14, 2012 at 12:32 PM, Anoop Ghanwani <[email-masked]> wrote:
:: > The whole problem of sending L2 multicast over a campus or data center
:: > backbone, in any sort of significant way, is a new one enabled for the first
:: > time by overlays.  There are interesting challenges when pushing large
:: > amounts of multicast traffic through a network, and  have thus far been dealt
:: > with using purpose-built networks.  While the overlay proposals have
:: > been careful not to impose new protocol requirements, they have not
:: > addressed the issues of performance and scalability, nor the large-scale
:: > availability of these protocols.
:: 
:: There are interesting challenges with unicast and multicast in big
:: flat DC networks. That's why we are here. One of the constant themes
:: we hear with multicast is the FUD that it doesn't scale. Especially
:: from vendors who are pushing another solution or who have a crappy
:: multicast implementation. Meanwhile multicast is being deployed in
:: large scale networks without scaling issue. Large L2 overlay networks?
:: I don't know. Would be good to find out from the community about
:: performance and scalability of multicast in the DC.

I've so far stayed pretty quiet on this, but, on this, I have to strongly 
disagree. It is not FUD that multicast doesn't scale well inside large 
datacenters -- it is a simple fact, and I speak as an operator of what 
several of my vendors called the 2nd largest multicast deployment they 
have ever seen, with many 10's of thousands (S,G) entries. 

Sorry, but many of us who operate really large scale datacenters with 
a lot of variability in the communication flows have removed all 
*application* use of multicast from the network due to scalability 
and stability reasons (ie by the time an (S,G) entry is programmed into 
hardware, we might be 3k+ packets deep into the flow, or the flow is over 
already, or what happens when you need to scale to 20k+ (S,G)'s?). 
Obviously, network protocols are still able to use multicast, but due 
to the very low volumes of flows and (S,G)s that is still acceptable.

Yes, we could have pushed the multicast scaling capabilities a bit higher 
by use of BIDIR, and yes, if the network hardware/software was built 
better it could have scaled a bit further, but, that's not the world we 
live in, and that would have cost significant troubleshooting capability, and a 
lot of extra complexity which we chose to instead solve via unicast, more 
bandwidth, and better application software.

So, whatever overlay protocols are proposed, I would suggest that counting 
on there being multicast underneath to transport you is not a very good 
idea...

Thanks,
-igor

---

From: Dino Farinacci <[email-masked]>
Date: Wed, 15 Feb 2012 14:20:20 -0800

> disagree. It is not FUD that multicast doesn't scale well inside large 

There are others who probably wish to not divulge their proprietary scalign numbers who would disagree with you.

> have ever seen, with many 10's of thousands (S,G) entries. 

It depends what you are comparing 10,000 to. Comparing to unicast numbers would not being comparing apples with apples. Remember the granularity of a multicast route is much finer than a unciast route because it wants to conserve bandwidith and build good distribution trees.

It is a simple bandwidth versus state tradeoff and 10,000 is pretty large.

Many think a data center with 10,000 hosts are large too. I know you can one-mag-up us Igor, but 10,000 is a large number for enterprise sites.

Dino

---

From: Igor Gashinsky <[email-masked]>
Date: Thu, 16 Feb 2012 23:16:05 -0800 (PST)

On Wed, 15 Feb 2012, Dino Farinacci wrote:

:: > I've so far stayed pretty quiet on this, but, on this, I have to strongly 
:: > disagree. It is not FUD that multicast doesn't scale well inside large 
:: 
:: There are others who probably wish to not divulge their proprietary 
:: scalign numbers who would disagree with you.

Would anybody be willing to speak about orders of magnitude here? We had 
to do a *lot* of nasty, ugly, hacky-ish things to get mcast to scale to 
the 10's of Ks (S,G) scale, but, when we looked at what it would take to 
make the next order-of-magnitude jump to 100k+, that's when the 
cost/complexity tradeoff simply made it not practical -- I'd love to hear 
if anybody is actually doing 100k+ (S,G) of random-ish traffic profile.

:: > datacenters -- it is a simple fact, and I speak as an operator of what 
:: > several of my vendors called the 2nd largest multicast deployment they 
:: > have ever seen, with many 10's of thousands (S,G) entries. 
:: 
:: It depends what you are comparing 10,000 to. Comparing to unicast 
:: numbers would not being comparing apples with apples. Remember the 
:: granularity of a multicast route is much finer than a unciast route 
:: because it wants to conserve bandwidith and build good distribution 
:: trees.
:: 
:: It is a simple bandwidth versus state tradeoff and 10,000 is pretty large.

You are absolutely right.. but the tradeoff is mostly dependant on the 
*degree of replication* for all those mcast routes vs the state that they 
cost. For my network, the *average* degree of replication is actually 
quite small (on the order of 2-3 hosts per group, taking up a tiny 
fraction of the bandwidth), so trading that amount of bandwidth for state 
was a no brainer. However, if you have very few groups with a very high 
replication factor (say, 1000 hosts), then you would arrive at a very 
different conclusion, but, then you likely don't have a multicast scaling 
problem, since then you likely only have a few hundred (S,G) entries.. So, 
do people have some other datapoints on degree of replication, amount of 
bw saved by mcast vs state on large datacenter (ie not-designed-for-video) 
multicast networks?

:: Many think a data center with 10,000 hosts are large too. I know you 
:: can one-mag-up us Igor, but 10,000 is a large number for enterprise 
:: sites.

I guess that brings us to the crux of the question -- ARMD = Address 
Resolution for *Massive* numbers of hosts in the Data center (from the 
charter), so, what is the order of magnitude for massive? :)

To me, (and perhaps i'm in a very small minority here?), massive implies 
cloud scale datacenters, so, 10-20k physical hosts *per cluster*, and 
somewhere around 400-500k VM's is the absolute *minimum* for what I would 
concider to be massive (and, really, I think 100k physical, 2M+ logical is 
what i believe a realistic aiming point these days should be). 

If that's the case, then we are looking at about 2-order-of-magnitude 
higher then what most enterprises do.. so, what problem space do we want 
to solve? 

Thanks,
-igor

PS as you can probably tell, Dino and I have had this discussion before, 
quite a few times :)

--------------------+----------------------+------------------
   Igor Gashinsky   | Network Architecture | Yahoo! Inc.
 [email-masked] |  cell 917.807.2213   | Do You... Yahoo?
--------------------+----------------------+------------------

---

From: Linda Dunbar <[email-masked]>
Date: Wed, 22 Feb 2012 22:49:28 +0000

Michael, 

> Microsoft is configurable for IGMP Multicast and Unicast.  There are
> also the other LB protocols such as CARP, VRRP, GLBP, HSRP, etc.
> 

Why do you consider VRRP as LB protocols? Hosts are not even aware of if VRRP is used or not, correct?

Linda

---

From: Igor Gashinsky <[email-masked]>
Date: Wed, 22 Feb 2012 15:08:48 -0800 (PST)

On Wed, 22 Feb 2012, Linda Dunbar wrote:

:: Michael, 
:: 
:: > -----Original Message-----
:: > Microsoft is configurable for IGMP Multicast and Unicast.  There are
:: > also the other LB protocols such as CARP, VRRP, GLBP, HSRP, etc.
:: > 
:: 
:: Why do you consider VRRP as LB protocols? Hosts are not even aware of if VRRP is used or not, correct?

I actually can easily see VRRP being termed an LB protocol  -- while it's 
true that VRRP and HSRP are first-hop redundancy protocols, by striping 
VRRP/HSRP groups along multiple switches, you can manually achieve 
load-ballancing between multiple gateways, in fact, quite a few people do 
that very thing (and have done it for over a decade).

-igor

---

From: Linda Dunbar <[email-masked]>
Date: Wed, 22 Feb 2012 23:17:27 +0000

In VRRP environment unicast address is used by hosts, not the multicast address, correct? 
So there is no multicast address in the Layer 2 domain as some other LB described in the email list (e.g. Microsoft LB)? 

Linda 

> From: Igor Gashinsky [mailto:[email-masked]]
> Sent: Wednesday, February 22, 2012 5:09 PM
> To: Linda Dunbar
> Cc: Michael K. Smith - Adhost; AshwoodsmithPeter; Anoop Ghanwani; Mike
> McBride; Thomas Narten; [email-masked]
> Subject: RE: [armd] address resolution requirement from hosts to
> overlay edge nodes. Any opinion?
>

---

From: "Michael K. Smith - Adhost" <[email-masked]>
Date: Wed, 22 Feb 2012 23:34:51 +0000

> From: Linda Dunbar [mailto:[email-masked]]
> Sent: Wednesday, February 22, 2012 3:17 PM
> To: Igor Gashinsky
> Subject: RE: [armd] address resolution requirement from hosts to overlay
> edge nodes. Any opinion?
> 
> So there is no multicast address in the Layer 2 domain as some other LB
> described in the email list (e.g. Microsoft LB)?
> 
> Linda
> 

VRRP and similar use multicast to communicate to one another (heartbeat).  The communication between the host and the default gateway is unicast.  With that said, the multicast heartbeat can be very chatty when you have thousands of VRRP pairs with sub-second heartbeat calls back and forth.

Mike