ARMD: Address Resolution for Massive amount of hosts in one broadcast Domain

Description of Working Group:

As server virtualization is introduced to data centers, the number of hosts in a data center can grow dramatically. Each physical server, which used to host one end-station, now can host hundreds of end-stations or virtual machines (VMs). Virtual machines (VMs) can be easily added and/or deleted and moved between servers. The VM’s flexible addition/deletion and moving around provide foundation for virtual hosts and virtual desktops offered by Cloud Computing services and make it possible to combine pool of servers into one single entity for ease of management and better utilization.

This rapid growth of virtual hosts could tremendously impact networks and servers. One major issue is frequent ARP (IPv4) or neighbor discovery (IPv6) requests from hosts. All hosts send out those requests frequently due to host ARP caches being aged out in minutes. With tens of thousands of hosts (each with a distinct MAC address) in one Data Center, the number of ARP (or ND) packets per second from all hosts could potentially be as high as 1,000 to 10,000/second. This rate imposes tremendous computational burden on many hosts.

Traditional subnet (VLAN) partitioning no longer works well when servers are virtualized. First of all, servers need to be on the same subnet for VM mobility. Secondly, when each server only has one host, there is only one VLAN enabled for the port towards the server. The switch can block all broadcast messages from other subnets (VLANs). When one physical server is supporting >100 Virtual Machines, i.e. >100 hosts, most likely the virtual hosts on one server are on different subnets (VLANs). If there are 50 subnets (VLANs) enabled on the switch port to the server, the server has to handle all the ARP broadcast messages on all 50 subnets (VLANs). The amount of ARP to be processed by each server is still too much.

One characteristic of a next-generation Data Center is the possibility of one subnet spanning multiple sites. For example, a server administrator can combine servers in multiple sites into one huge resource pool for better resource management and ease of allocation. Each of the servers in this pool might be configured with one or more VMs. If mobility is required, like VMware’s vMotion, all the physical servers (and associated VMs) need to be on the same IP subnet. Under those circumstances, one Layer 2 network (or subnet) could span across multiple sites. The multiple sites could be connected by any types of network, like L2VPN, L3VPN, Ethernet network, or simple IP networks. Efficient address resolution across multiple sites should be considered by this working group.

The goal of this working group is to develop interoperable solutions to solve the above problems within the Data Center and multi Data Center environments. This could involve multi-domain (multi-company), multi-site and multi-vendor environments.

The design should consider the following properties:

All solutions developed by ARMD WG should not require any behavior changes on hosts, applications, or Virtual Machines being deployed in the market.
All solutions developed should not break DHCP, or any other broadcast/multicast mechanism used by applications.
Evaluating the impact to IPv6 ND, and develop solutions accordingly if needed.
Should consider variety of solutions, including directory based, proxy based, or cache based solutions.
Include analysis of security concerns of IPv4 ARP requests from malicious users. Evaluating potential security solutions and conclude if the security threat can justify solutions.
ARMD assumes the direct links to individual hosts and virtual machines are IEEE802.3 Ethernet links.
Should consider scenarios of one Ethernet network being interconnected by another network, which can be L2VPN, pure IP, Ethernet, or others.
Should consider isolation of address resolution solutions across different client controlled virtual subnets, i.e. hosts from one client should be limited to getting MAC addresses from virtual hosts within its own controlled subnet(s). Should consider a performance analysis of proposed solutions.

Here are the items which should not be in the scope of the working group:

Re-define DHCP behavior
Re-define IPv6 ND security model
Solution developed should not expect any behavior changes for the guest OS/applications running on the VMs
Direct links from hosts and virtual hosts being non-Ethernet links.

Goals and Milestones:

Problem statement
Gap Analysis Statement within Data Center

o Gaps within ARP
o Gaps within IPv6 ND or autoconfiguration

Survey of Existing Solutions

o Survey of NHRP (RFC2332) & SCSP, and their applicability to Ethernet networks;
o Survey of TRILL work as a potential solution;
o Survey of other potential solutions, like MOOSE or SEATTLE; and
o Other proposals

Security Analysis of Existing Solutions

o Study existing solutions, like Dynamic ARP Inspection, Egress ARP Inspection, etc, and evaluate if any of them or others should be included in the solutions specified by this working group.

Architectural Design for Next-Generation Data Center
Protocol Documents

Time line for BOF presentations:


Initial Problem Statement                          August   
GAP analysis                                       October
Survey of Existing Solutions                       October
Security Analysis of Existing Solutions            October
Architectural Design for NG                        October
Final Problem statement for BOF                    October
Protocol Documents                                 October

Last modified: Mon Sep 13 15:55:25 EEST 2010