Subject: [armd] thoughts on the continuation of the ARMD work

From: Jari Arkko <[email-masked]>
Date: Fri, 14 Jan 2011 00:52:46 +0200

It is time to decide what to do with another BOF or some other
continuation of this work.

I'm not the AD responsible for this work (Ralph is) but I thought it
might be useful to provide some possible ideas that might work.

One approach is to run a second BOF, and attempt to come to a conclusion
that we have or do not have a scaling problem in ARP. If we do have a
scaling problem then we can charter a working group that works on a
solution. Linda and Sue tell me that they are working on experiments
that may help us understand the scaling problems better, so there may be
some material ready from this to the BOF.

Another approach is to just charter a working group that documents
scaling characteristics of ARP and ND, and perhaps also provides some
operational advice on how to best run your networks without running into
issues. This working group would run, say, for six months and would come
up with a couple of RFCs. At that point it would we could perhaps
recharter the group for some protocol work, if the RFCs showed that
there are severe issues.

In either case we need to work on what the scaling characteristics of
ARP and ND are. The difference is that in the BOF approach we need to
focus on finding the "smoking gun" and have a yes-no type discussion. In
the WG approach we can take a bit more time, do an in-depth analysis,
can document the results neutrally without pressure to find the gun, and
may even provide some useful advice to network managers in the process.

Jari

---

From: guyingjie 00107907 <[email-masked]>
Date: Fri, 14 Jan 2011 07:38:28 +0800

Well, personally I would support approach 2. We may first output some Informal RFCs, and if necessary, we can recharter the WG.

Best Regards
Y.J. GU

----- Original mail -----

> One approach is to run a second BOF, and attempt to come to a 
> conclusionthat we have or do not have a scaling problem in ARP. If 
> we do have a
> _______________________________________________
> armd mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/armd
>

---

From: robert sultan <[email-masked]>
Date: Thu, 13 Jan 2011 23:45:43 +0000

I agree with Jari's second suggestion, to charter a group to document and analyze ARP and ND performance in networks of various size.  ...Bob
________________________________________
From: [email-masked] [[email-masked]] on behalf of Jari Arkko [[email-masked]]
Sent: Thursday, January 13, 2011 2:52 PM
To: [email-masked]
Subject: [armd] thoughts on the continuation of the ARMD work

Jari

---

From: Murari Sridharan <[email-masked]>
Date: Tue, 18 Jan 2011 23:38:44 +0000

I like the second approach and support the formation of a WG to study the scaling characteristics. Understanding this is essential. 

Thanks

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Jari Arkko
Sent: Thursday, January 13, 2011 2:53 PM
To: [email-masked]
Subject: [armd] thoughts on the continuation of the ARMD work

It is time to decide what to do with another BOF or some other continuation of this work.

I'm not the AD responsible for this work (Ralph is) but I thought it might be useful to provide some possible ideas that might work.

One approach is to run a second BOF, and attempt to come to a conclusion that we have or do not have a scaling problem in ARP. If we do have a scaling problem then we can charter a working group that works on a solution. Linda and Sue tell me that they are working on experiments that may help us understand the scaling problems better, so there may be some material ready from this to the BOF.

Another approach is to just charter a working group that documents scaling characteristics of ARP and ND, and perhaps also provides some operational advice on how to best run your networks without running into issues. This working group would run, say, for six months and would come up with a couple of RFCs. At that point it would we could perhaps recharter the group for some protocol work, if the RFCs showed that there are severe issues.

In either case we need to work on what the scaling characteristics of ARP and ND are. The difference is that in the BOF approach we need to focus on finding the "smoking gun" and have a yes-no type discussion. In the WG approach we can take a bit more time, do an in-depth analysis, can document the results neutrally without pressure to find the gun, and may even provide some useful advice to network managers in the process.

Jari

---

From: Donald Eastlake <[email-masked]>
Date: Tue, 18 Jan 2011 22:20:50 -0500

The second approach, to charter a working group that documents
scaling characteristics of ARP and ND, and perhaps also provides some
operational advice, seems like a reasonable course to me.

Thanks,
Donald
=============================
 Donald E. Eastlake 3rd   +1-508-333-2270 (cell)
 155 Beaver Street
 Milford, MA 01757 USA
 [email-masked]

---

From: Ross Callon <[email-masked]>
Date: Tue, 18 Jan 2011 22:31:26 -0500

How big a task do we think that this will be? 

(I am asking this with the understanding that there is a very wide range of "amount of work" implied by a WG charter). 

Ross

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Donald Eastlake
Sent: Tuesday, January 18, 2011 10:21 PM
To: [email-masked]
Subject: Re: [armd] thoughts on the continuation of the ARMD work

Thanks,
Donald

---

From: "So, Ning" <[email-masked]>
Date: Wed, 19 Jan 2011 14:35:13 +0000

The data centers, especially the new ones that are being built to host
the next generation "Cloud-based" services, are getting to be VERY big
in anticipation of the forecasted service offerings.  Scalability and
the utilization efficiency (mobility) are certainly major concerns for
the service providers.  Proposed ARMD charter seems to cover those areas
well, from my opinion.  I support the formulation of the WG to start
tackling the problems officially. 

Best regrads,
 
Ning So
Network Evolution Planning
Verizon, Inc.
(office) 972-729-7905
(Cell) 972-955-0914

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Jari Arkko
Sent: Thursday, January 13, 2011 2:53 PM
To: [email-masked]
Subject: [armd] thoughts on the continuation of the ARMD work

Jari

---

From: Linda Dunbar <[email-masked]>
Date: Wed, 19 Jan 2011 09:53:05 -0600

Ross, 

I will revise the Charter statement, scope of work and deliverables based on
the emails posted. We can then evaluate the amount of work. I anticipate
that it will be in several phases: Phase 1 being documenting the problems
and issues, Phase 2 being on solutions, ...

Linda 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Ross
Callon
Sent: Tuesday, January 18, 2011 9:31 PM
To: Donald Eastlake; [email-masked]
Subject: Re: [armd] thoughts on the continuation of the ARMD work

(I am asking this with the understanding that there is a very wide range of
"amount of work" implied by a WG charter). 

Ross

-----Original Message-----
Donald Eastlake

Thanks,
Donald

On Tue, Jan 18, 2011 at 6:38 PM, Murari Sridharan <[email-masked]>
wrote:
> I like the second approach and support the formation of a WG to study the
scaling characteristics. Understanding this is essential.
Jari Arkko
>
> I'm not the AD responsible for this work (Ralph is) but I thought it might
be useful to provide some possible ideas that might work.
that we have or do not have a scaling problem in ARP. If we do have a
scaling problem then we can charter a working group that works on a
solution. Linda and Sue tell me that they are working on experiments that
may help us understand the scaling problems better, so there may be some
material ready from this to the BOF.
>
> Another approach is to just charter a working group that documents scaling
characteristics of ARP and ND, and perhaps also provides some operational
advice on how to best run your networks without running into issues. This
working group would run, say, for six months and would come up with a couple
of RFCs. At that point it would we could perhaps recharter the group for
some protocol work, if the RFCs showed that there are severe issues.
>
> In either case we need to work on what the scaling characteristics of ARP
and ND are. The difference is that in the BOF approach we need to focus on
finding the "smoking gun" and have a yes-no type discussion. In the WG
approach we can take a bit more time, do an in-depth analysis, can document
the results neutrally without pressure to find the gun, and may even provide
some useful advice to network managers in the process.

---

From: Ben Mack-Crane <[email-masked]>
Date: Wed, 19 Jan 2011 10:49:10 -0600

I agree with the sentiments expressed so far. It would be best to charter a
WG to further study and document performance concerns and related
operational advice for very large L2 networks and add to the charter if
protocol work is required.  So I support the second approach described
below.

Regards,
Ben Mack-Crane

> Sent: Thursday, January 13, 2011 4:53 PM
> To: [email-masked]
> conclusion

---

From: [email-masked]
Date: Thu, 20 Jan 2011 09:19:44 +0800

I think the ARP/ND issues depend on not only the network scale but also 
the distributed mechanism adopted among the servers.

The distributed algorithm decides the connectivity in the same LAN, and 
connectivity decides the number of ARP/ND broadcast request triggered by 
the ageing timer.

Russell

Ben Mack-Crane <[email-masked]> 
发件人:  [email-masked]
2011-01-20 00:49

收件人
'Jari Arkko' <[email-masked]>, [email-masked]
抄送

主题
Re: [armd] thoughts on the continuation of the ARMD work

I agree with the sentiments expressed so far. It would be best to charter 
a
WG to further study and document performance concerns and related
operational advice for very large L2 networks and add to the charter if
protocol work is required.  So I support the second approach described
below.

Regards,
Ben Mack-Crane

--------------------------------------------------------
ZTE Information Security Notice: The information contained in this mail is solely property of the sender's organization. This mail communication is confidential. Recipients named above are obligated to maintain secrecy and are not permitted to disclose the contents of this communication to others.
This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the originator of the message. Any views expressed in this message are those of the individual sender.
This message has been scanned for viruses and Spam by ZTE Anti-Spam system.

---

From: Thomas Narten <[email-masked]>
Date: Thu, 20 Jan 2011 09:39:50 -0500

The difficulty I have with this group is that it is based on the
premise is that ARP doesn't scale. Of course that is true. But it is
at some level meaningless. ARP works fine, so long as the subnet (or
link, using IPv6 terminology) doesn't get "too large". This has been
known for 20+ years. If you make your subnet too large and flat, you
will experience pain.

We are dealing with the classic "I want to make my network big and
flat because that gives me certain advantages". Yet there are
downsides to that. We will never make ARP "scale" as big as is
imaginable or as some people might want. Even if we *want* that, and
it would be *nice*. It would be nice to get a Free Lunch too, but that
doesn't happen in real life.

The constraints to "solving" the ARP scaling problem are enormous. We
can't change any protocols, or modify hosts, because it is pretty
clear that those sorts of changes will take too long to become widely
implemented and deployed to be useful. That leaves us with techniques
like Proxy ARP. Do we need a WG to tell us how to use Proxy ARP? I
would hope not...

Thomas

---

From: Linda Dunbar <[email-masked]>
Date: Thu, 20 Jan 2011 10:14:35 -0600

Thomas, 

ARP works fine when all the hosts within one subnet are confined to one
Aggregation switch (or Top of Rack switch) because all the broadcast
messages are confined. 

When VM and VM migration is deployed in Data Centers, hosts in one subnet
(VLAN) may start with being close together and gradually being relocated to
various places as the result of continuous VM migration. 

We are not talking about flat big network. Each subnet is not big, but when
there are so many of them, ARP/ND broadcast messages will cause a lot more
traffic traversing across Aggregation switches. One of our customers' data
centers already have more than 4000 VLANs. 

When VMs belonging to one subnet are spread across multiple TOR switches,
all the ARP broadcast messages and ND multicast messages are sent to all TOR
switches in the broadcast domain. This will cause TOR switch to refresh its
FDB even for the MAC not used by its local hosts. Most likely, the frequency
of ARP/ND by hosts is shorter then TOR FDB MAC aging time. That means TOR
FDB will have all the MAC addresses even if most of them are not used by
local hosts. 

Linda Dunbar 

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Thomas Narten
Sent: Thursday, January 20, 2011 8:40 AM
To: Jari Arkko
Cc: [email-masked]
Subject: Re: [armd] thoughts on the continuation of the ARMD work

Thomas

---

From: "Susan Hares" <[email-masked]>
Date: Thu, 20 Jan 2011 18:11:13 -0500

Thomas:

Let me add my 2 cents to Linda's comment. The comment of 

"I want to make my network big and flat because that gives me certain
advantages".

is arising again due to changes in the Data Center environment.  Please note
this is a specific target environment.  

Please understand that general "ARP Scaling" is not our focus - but a
specific area of ARP scaling for this target environment with business
implications. 

As to the constraints, we are examining the constraints on ARP that must
stay for this specific target work and those which can be relaxed (for this
specific target). 

Linda is providing specific examples for this target environment. IP + glue
protocols has always adapted to specific environments with tuning (for
example mobile). 

We welcome comments based on this specific scope on this list. General
discussion about "general ARP" changes is not our focus. 

Sue Hares

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of
Linda Dunbar
Sent: Thursday, January 20, 2011 11:15 AM
To: 'Thomas Narten'; 'Jari Arkko'
Cc: [email-masked]
Subject: Re: [armd] thoughts on the continuation of the ARMD work

Thomas, 

Linda Dunbar 

-----Original Message-----

Thomas

---

From: Anoop Ghanwani <[email-masked]>
Date: Fri, 21 Jan 2011 12:49:57 -0800

Thought I should also voice my support for the
creation of a working group to document the problems
and a mechanism to deal with them.  Folks are already
starting to implement various things and it would
be nice to have interoperable, consistent solutions.

And to address Thomas' concern, this is really not
so much about making networks flatter, but rather 
about stretching an L2 domain across the data center,
as opposed to keeping it confined to a single 
top-of-the-rack or aggregation switch.  Put another
way, the subnet size stays the same, say 256 hosts, 
but instead of having all 256 sitting under the same
aggregation switch, we would now like to see them
scattered across the data center.

Network administrators have complained about this
problem ("I can't decommission a malfunctioning server
in one part of the network and put a new one in
with the same IP address in a new location")
even before the advent of server virtualization,
but server virtualization with its mobility function
has made this an even more compelling problem.

Anoop

> Susan Hares
> Sent: Thursday, January 20, 2011 3:11 PM
> To: 'Linda Dunbar'; 'Thomas Narten'; 'Jari Arkko'
> Cc: [email-masked]

---

From: Shehzad Merchant <[email-masked]>
Date: Fri, 21 Jan 2011 13:33:54 -0800

Hi Anoop,

I am in agreement around creating a working group.

Also agree that the ability to decommission a server in one part of the DC and bring up a new one in another part of the DC with the same IP address is a problem.

However, we are definitely seeing that L2 domains are growing in size as they start stretching across racks within a DC (driven in part by the need to move VMs around within the DC) and in fact there even seems to be a lot of interest in stretching them across DCs in different locations as well. As such there definitely seems to be a trend where the L2 domains have more and more hosts within them which is directly exacerbating issues related to ARPs and more generally broadcast/multicast processing.

-Shehzad

-----Original Message-----
From: [email-masked] [mailto:[email-masked]] On Behalf Of Anoop Ghanwani
Sent: Friday, January 21, 2011 12:50 PM
To: Susan Hares; 'Linda Dunbar'; 'Thomas Narten'; 'Jari Arkko'
Cc: [email-masked]
Subject: Re: [armd] thoughts on the continuation of the ARMD work

Anoop

DISCLAIMER:
This e-mail and any attachments to it may contain confidential and proprietary material and is solely for the use of the intended recipient. Any review, use, disclosure, distribution or copying of this transmittal is prohibited except by or on behalf of the intended recipient.  If you have received this transmittal in error, please notify the sender and destroy this e-mail and any attachments and all copies, whether electronic or printed.

---

From: Thomas Narten <[email-masked]>
Date: Fri, 21 Jan 2011 17:32:11 -0500

One difficulty I have had with this discussion is lack of clarity
about the actual scenario. I agree (and understand) that the problem
being described is more complicated than just saying we are starting
with one big flat IP subnet.

But there appear to be unstated assumptions about VLANS, where related
hosts are clustered, etc. that are not fully explained.

> Aggregation switch (or Top of Rack switch) because all the broadcast

Is the entire IP subnet confined to the same aggregation switch? Are
they all in one VLAN where all members of a particular VLAN are
confinded to one aggregation switch?

Why is ARP traffic confined by default in the above case? There are
some instated assumptions going on here (or at least, I don't know
what they are.)

> various places as the result of continuous VM migration.

So is the VLAN containing the migrating machine "stretching" and no
longer confined to the same Aggregation switch after the move?

> centers already have more than 4000 VLANs.

Then isn't your problem that you simply have too many nodes in a
single broadcast domain. ARP isn't the root problem. The scaling
problem is really related to size of the L2 broadcast domain, and the
number of VLANs and hosts on that one L2 broadcast domain.

Right?

> When VMs belonging to one subnet are spread across multiple TOR
> switches, all the ARP broadcast messages and ND multicast messages
> are sent to all TOR switches in the broadcast domain.

Here again there are some unstated assumptions about TOR racks, and
how IP subnets (and/or VLANs)are spread across them.

I think it would help explain the real problem to also describe (say)
the VLAN structure before a move and after a move to show why ARP
traffic is sent to only one TOR switch in one case, but sent to more
broadly in other cases.

My guess is that ARP is the straw that breaks the proverbial camel's
back in the datacenter configuration where this is a problem. But ARP
per se, is not the problem.

Thomas

> This will cause TOR switch to refresh its FDB even for the MAC not
> used by its local hosts. Most likely, the frequency of ARP/ND by
> hosts is shorter then TOR FDB MAC aging time. That means TOR FDB
> will have all the MAC addresses even if most of them are not used by
> local hosts.

---

From: Anoop Ghanwani <[email-masked]>
Date: Sat, 22 Jan 2011 11:45:36 -0800

First off, for the most part VLAN = IP subnet.  There are 
some corner cases such as multinetting or the use of 
private VLANs where this equality doesn't hold, but we can
leave those out for this discussion.

Today, all members of a VLAN and thus subnet, are typically
confined to a single aggregation switch (typically a pair running
VRRP for redundancy).  An additional reason why people don't
like to string VLANs across multiple aggregation switches all
over the data center is because of problems they have had with
STP such debugging loops due to misconfiguration of STP
and the fact that STP limits the links on which traffic is sent
to a tree (no ECMP).  This is expected to change with the advent
of technologies such as TRILL.  In fact in many designs, the
subnet termination is done even before the aggregation switch
at the top of rack, so that STP can be completely done away with.

ARP only gets within the subnet, so it only flows on links
that need to get to members of the subnet.  By putting members
of the subnet in arbitrary locations, we now have ARP traffic
going across backbone links.

Correct.  We want it to be able to move anywhere within
the data center, or even across data centers.  And we want
to keep as much ARP traffic as we can off the backbone links.

Anoop

________________________________________
From: [email-masked] [[email-masked]] On Behalf Of Thomas Narten [[email-masked]]
Sent: Friday, January 21, 2011 3:32 PM
To: Linda Dunbar
Cc: [email-masked]
Subject: Re: [armd] thoughts on the continuation of the ARMD work

Right?

Thomas