Network Design - Choosing a technology for multiple datacenter topology

Running VXLAN over MPLS versus Ethernet VPN (EVPN) Multisite with point-to-point (P2P) links, especially when considering multiple data centers involves several considerations in terms of scalability, complexity, and operational needs.

Key Considerations

  1. VXLAN over MPLS:

    • Scalability: MPLS is widely adopted for its scalability in both Layer 2 (L2) and Layer 3 (L3) VPN scenarios, enabling efficient interconnectivity between data centers. VXLAN overlays on top of MPLS can extend L2 domains across data centers while maintaining Layer 3 isolation and segmentation.
    • Centralized Control with BGP Route Reflectors/Servers: In scenarios with multiple data centers, route reflectors or route servers become useful in scaling the BGP control plane without creating full-mesh BGP sessions. Cisco recommends using route reflectors to avoid the complexity of maintaining a full mesh of eBGP peers.
    • Simplification: Running VXLAN over MPLS simplifies the network by maintaining the MPLS backbone while using VXLAN for tenant traffic and communication between different VLANs or VRFs across sites. It leverages MPLS's existing label-switching capabilities to minimize complexity.

  2. EVPN Multisite (MS) with P2P Links:

    • Optimal for Smaller Sites: Cisco's recommendation to use EVPN Multisite for only two data centers often stems from the additional complexity and control mechanisms required when scaling to more than two DCs. Multisite introduces the challenge of stretching broadcast domains (for L2) over multiple sites. EVPN MS generally becomes more cumbersome when dealing with three or more DCs.
    • Single-Homed or Dual-Homed Sites: EVPN Multisite is typically well-suited to two connected DCs with dual-homing for redundancy. When three or more data centers are involved, the complexity of managing the fabric across the different locations increases.
    • Full-Mesh eBGP or Route Servers: For more than two DCs, you could still use full-mesh eBGP or utilize route reflectors/servers to reduce the number of BGP sessions between border gateways. However, operational overhead grows as more DCs are added.

  3. EVPN/VXLAN with Route Servers:

    • The concept of using a route server in VXLAN environments, especially with EVPN, is intended to reduce the BGP peering overhead. The route server acts as a "BGP route reflector" and simplifies the peering topology for multihop eBGP sessions, removing the need for a full mesh.
    • With the full-mesh requirement across three or more DCs, using route servers can mitigate the complexity. This approach aligns with Cisco's suggestions for larger deployments where scaling control-plane sessions efficiently is a priority.

Which to Choose?

  • For More than Two Data Centers:

    If you are connecting more than two DCs (as depicted in your diagram with DC1, DC2, DC3, and DC4), you might find VXLAN over MPLS with route servers or reflectors to be a better fit. It avoids the operational complexity and scalability issues that arise with EVPN Multisite when dealing with multiple DCs.

  • Layer 3 Communication Between VLANs:

    In your use case of Layer 3 communication between VLANs across multiple DCs, VXLAN over MPLS may provide the necessary L3 segmentation and reachability between the DCs without the need for the extra complexity of managing EVPN MS.

https://networklessons.com/mpls https://networklessons.com/vxlan