Infrastructure Solutions

 View Only

Design and implement Azure ExpressRoute

This thread has been viewed 4 times
  • 1.  Design and implement Azure ExpressRoute

    Posted 08-10-2023 08:28
      |   view attached

    Design and implement Azure ExpressRoute



    ExpressRoute lets you extend your on-premises networks into the Microsoft cloud over a private connection with the help of a connectivity provider. With ExpressRoute, you can establish connections to various Microsoft cloud services, such as Microsoft Azure and Microsoft 365. Connectivity can be from an any-to-any (IP VPN) network, a point-to-point Ethernet network, or a virtual cross-connection through a connectivity provider at a colocation facility. Since ExpressRoute connections do not go over the public Internet, this approach allows ExpressRoute connections to offer more reliability, faster speeds, consistent latencies, and higher security.

    ExpressRoute capabilities

    Some key benefits of ExpressRoute are:

    ·       Layer 3 connectivity between your on-premises network and the Microsoft Cloud through a connectivity provider

    ·       Connectivity can be from an any-to-any (IPVPN) network, a point-to-point Ethernet connection, or through a virtual cross-connection via an Ethernet exchange

    ·       Connectivity to Microsoft cloud services across all regions in the geopolitical region

    ·       Global connectivity to Microsoft services across all regions with the ExpressRoute premium add-on

    ·       Built-in redundancy in every peering location for higher reliability

    Azure ExpressRoute is used to create private connections between Azure datacenters and infrastructure on your premises or in a colocation environment. ExpressRoute connections do not go over the public Internet, and they offer more reliability, faster speeds, and lower latencies than typical Internet connections.


    Understand use cases for Azure ExpressRoute

    Faster and Reliable connection to Azure services - Organizations leveraging Azure services look for reliable connections to Azure services and data centers. Public internet is dependent upon many factors and may not be suitable for a business. Azure ExpressRoute is used to create private connections between Azure data centers and infrastructure on your premises or in a colocation environment. Using ExpressRoute connections to transfer data between on-premises systems and Azure can also give significant cost benefits.

     Storage, backup, and Recovery - Backup and Recovery are important for an organization for business continuity and recovering from outages. ExpressRoute gives you a fast and reliable connection to Azure with bandwidths up to 100 Gbps, which makes it excellent for scenarios such as periodic data migration, replication for business continuity, disaster recovery and other high-availability strategies.

     Extends Data center capabilities - ExpressRoute can be used to connect and add compute and storage capacity to your existing data centers. With high throughput and fast latencies, Azure will feel like a natural extension to or between your data centers, so you enjoy the scale and economics of the public cloud without having to compromise on network performance.

     Predictable, reliable, and high-throughput connections - With predictable, reliable, and high-throughput connections offered by ExpressRoute, enterprises can build applications that span on-premises infrastructure and Azure without compromising privacy or performance. For example, run a corporate intranet application in Azure that authenticates your customers with an on-premises Active Directory service, and serve all your corporate customers without traffic ever routing through the public Internet.

    ExpressRoute connectivity models

    You can create a connection between your on-premises network and the Microsoft cloud in four different ways, CloudExchange Co-location, Point-to-point Ethernet Connection, Any-to-any (IPVPN) Connection, and ExpressRoute Direct. Connectivity providers may offer one or more connectivity models.

    Co-located at a cloud exchange

    If you are co-located in a facility with a cloud exchange, you can order virtual cross-connections to the Microsoft cloud through the co-location provider's Ethernet exchange. Co-location providers can offer either Layer 2 cross-connections, or managed Layer 3 cross-connections between your infrastructure in the co-location facility and the Microsoft cloud.


    Point-to-point Ethernet connections

    You can connect your on-premises datacenters/offices to the Microsoft cloud through point-to-point Ethernet links. Point-to-point Ethernet providers can offer Layer 2 connections, or managed Layer 3 connections between your site and the Microsoft cloud.


    Any-to-any (IPVPN) networks

    You can integrate your WAN with the Microsoft cloud. IPVPN providers (typically MPLS VPN) offer any-to-any connectivity between your branch offices and datacenters. The Microsoft cloud can be interconnected to your WAN to make it look just like any other branch office. WAN providers typically offer managed Layer 3 connectivity.


    Direct from ExpressRoute sites

    You can connect directly into the Microsoft's global network at a peering location strategically distributed across the world. ExpressRoute Direct provides dual 100 Gbps or 10-Gbps connectivity, which supports Active/Active connectivity at scale.

    Design considerations for ExpressRoute deployments

    When planning an ExpressRoute deployment, there are many decisions to make. This section discusses a few key areas that you must consider as you design your deployment.

    Choose between provider and direct model (ExpressRoute Direct)

    ExpressRoute Direct

    ExpressRoute Direct gives you the ability to connect directly into Microsoft's global network at peering locations strategically distributed around the world. ExpressRoute Direct provides dual 100 Gbps or 10-Gbps connectivity, which supports Active/Active connectivity at scale. You can work with any service provider for ExpressRoute Direct.

    Key features that ExpressRoute Direct provides includes:

    ·       Massive Data Ingestion into services like Storage and Cosmos DB

    ·       Physical isolation for industries that are regulated and require dedicated and isolated connectivity like: Banking, Government, and Retail

    ·       Granular control of circuit distribution based on business unit

    Using ExpressRoute direct vs using a Service Provider

    ExpressRoute using a Service Provider

    ExpressRoute Direct

    Uses service providers to enable fast onboarding and connectivity into existing infrastructure

    Requires 100 Gbps/10 Gbps infrastructure and full management of all layers

    Integrates with hundreds of providers including Ethernet and MPLS

    Direct/Dedicated capacity for regulated industries and massive data ingestion

    Circuits SKUs from 50 Mbps to 10 Gbps

    Customer may select a combination of the following circuit SKUs on 100-Gbps ExpressRoute Direct: 5 Gbps 10 Gbps 40 Gbps 100 Gbps Customer may select a combination of the following circuit SKUs on 10-Gbps ExpressRoute Direct: 1 Gbps 2 Gbps 5 Gbps 10 Gbps

    Optimized for single tenant

    Optimized for single tenant with multiple business units and multiple work environments

    Route advertisement

    When Microsoft peering gets configured on your ExpressRoute circuit, the Microsoft Edge routers establish a pair of Border Gateway Protocol (BGP) sessions with your edge routers through your connectivity provider. No routes are advertised to your network. To enable route advertisements to your network, you must associate a route filter.

    In order to associate a route filter:

    ·       You must have an active ExpressRoute circuit that has Microsoft peering provisioned.

    ·       Create an ExpressRoute circuit and have the circuit enabled by your connectivity provider before you continue. The ExpressRoute circuit must be in a provisioned and enabled state.

    ·       Create Microsoft peering if you manage the BGP session directly. Or, have your connectivity provider provision Microsoft peering for your circuit.

    Get a list of BGP community values

    BGP community values associated with services accessible through Microsoft peering is available in the ExpressRoute routing requirements page.


    Make a list of the values that you want to use

    Make a list of BGP community values you want to use in the route filter.

    Bidirectional Forwarding Detection

    ExpressRoute supports Bidirectional Forwarding Detection (BFD) both over private and Microsoft peering. When you enable BFD over ExpressRoute, you can speed up the link failure detection between Microsoft Enterprise edge (MSEE) devices and the routers that your ExpressRoute circuit gets configured (CE/PE). You can configure ExpressRoute over your edge routing devices or your Partner Edge routing devices (if you went with managed Layer 3 connection service). This section walks you through the need for BFD, and how to enable BFD over ExpressRoute.

    You can enable ExpressRoute circuit either by Layer 2 connections or managed Layer 3 connections. In both cases, if there are more than one Layer-2 devices in the ExpressRoute connection path, the responsibility of detecting any link failures in the path lies with the overlying BGP session.

    On the MSEE devices, BGP keep-alive and hold-time are typically configured as 60 and 180 seconds, respectively. For that reason, when a link failure happens it can take up to three minutes to detect any link failure and switch traffic to alternate connection.

    You can control the BGP timers by configuring a lower BGP keep-alive and hold-time on your edge peering device. If the BGP timers are not the same between the two peering devices, the BGP session will establish using the lower time value. The BGP keep-alive can be set as low as three seconds, and the hold-time as low as 10 seconds. However, setting a very aggressive BGP timer isn't recommended because the protocol is process intensive.

    In this scenario, BFD can help. BFD provides low-overhead link failure detection in a sub second time interval.


    The following diagram shows the benefit of enabling BFD over an ExpressRoute circuit:


    Enabling BFD

    BFD is configured by default under all the newly created ExpressRoute private peering interfaces on the MSEEs. As such, to enable BFD, you only need to configure BFD on both your primary and secondary devices. Configuring BFD is two-step process. You configure the BFD on the interface and then link it to the BGP session.

    When you disable a peering, the Border Gateway Protocol (BGP) session for both the primary and the secondary connection of your ExpressRoute circuit is shut down. When you enable a peering, the BGP session on both the primary and the secondary connection of your ExpressRoute circuit is restored.


    The first time you configure the peering on your ExpressRoute circuit, the Peerings are enabled by default.


    Resetting your ExpressRoute Peerings might be helpful in the following scenarios:

    You are testing your disaster recovery design and implementation. For example, assume that you have two ExpressRoute circuits. You can disable the Peerings of one circuit and force your network traffic to use the other circuit.

    You want to enable Bidirectional Forwarding Detection (BFD) on Azure private peering or Microsoft peering. If your ExpressRoute circuit was created before August 1, 2018, on Azure private peering or before January 10, 2020, on Microsoft peering, BFD was not enabled by default. Reset the peering to enable BFD.

    Configure encryption over ExpressRoute

    This section shows you how to use Azure Virtual WAN to establish an IPsec/IKE VPN connection from your on-premises network to Azure over the private peering of an Azure ExpressRoute circuit. This technique can provide an encrypted transit between the on-premises networks and Azure virtual networks over ExpressRoute, without going over the public internet or using public IP addresses.


    Topology and routing

    The following diagram shows an example of VPN connectivity over ExpressRoute private peering:

    The diagram shows a network within the on-premises network connected to the Azure hub VPN gateway over ExpressRoute private peering. The connectivity establishment is straightforward:

    ·       Establish ExpressRoute connectivity with an ExpressRoute circuit and private peering.

    ·       Establish the VPN connectivity.

    An important aspect of this configuration is routing between the on-premises networks and Azure over both the ExpressRoute and VPN paths.

     Traffic from on-premises networks to Azure

    For traffic from on-premises networks to Azure, the Azure prefixes (including the virtual hub and all the spoke virtual networks connected to the hub) are advertised via both the ExpressRoute private peering BGP and the VPN BGP. This results in two network routes (paths) toward Azure from the on-premises networks:

    ·       One over the IPsec-protected path

    ·       One directly over ExpressRoute without IPsec protection

    To apply encryption to the communication, you must make sure that for the VPN-connected network in the diagram, the Azure routes via on-premises VPN gateway are preferred over the direct ExpressRoute path.

     Traffic from Azure to on-premises networks

    The same requirement applies to the traffic from Azure to on-premises networks. To ensure that the IPsec path is preferred over the direct ExpressRoute path (without IPsec), you have two options:

    ·       Advertise more specific prefixes on the VPN BGP session for the VPN-connected network. You can advertise a larger range that encompasses the VPN-connected network over ExpressRoute private peering, then more specific ranges in the VPN BGP session. For example, advertise over ExpressRoute, and over VPN.

    ·       Advertise disjoint prefixes for VPN and ExpressRoute. If the VPN-connected network ranges are disjoint from other ExpressRoute connected networks, you can advertise the prefixes in the VPN and ExpressRoute BGP sessions, respectively. For example, advertise over ExpressRoute, and over VPN.

    In both examples, Azure will send traffic to over the VPN connection rather than directly over ExpressRoute without VPN protection.


    If you advertise the same prefixes over both ExpressRoute and VPN connections, Azure will use the ExpressRoute path directly without VPN protection.

    Design redundancy for an ExpressRoute deployment

    There are 2 ways in which redundancy can be planned for an ExpressRoute deployment.

    ·       Configure ExpressRoute and site to site coexisting connections

    ·       Create a zone redundant VNET gateway in Azure Availability zones

    Configure ExpressRoute and site to site coexisting connections

    This section helps you configure ExpressRoute and Site-to-Site VPN connections that coexist. Having the ability to configure Site-to-Site VPN and ExpressRoute has several advantages. You can configure Site-to-Site VPN as a secure failover path for ExpressRoute or use Site-to-Site VPNs to connect to sites that are not connected through ExpressRoute.

    Configuring Site-to-Site VPN and ExpressRoute coexisting connections has several advantages:

    ·       You can configure a Site-to-Site VPN as a secure failover path for ExpressRoute.

    ·       Alternatively, you can use Site-to-Site VPNs to connect to sites that are not connected through ExpressRoute.

    You can configure either gateway first. Typically, you will incur no downtime when adding a new gateway or gateway connection.

    Network Limits and limitations

    ·       Only route-based VPN gateway is supported. You must use a route-based VPN gateway. You also can use a route-based VPN gateway with a VPN connection configured for 'policy-based traffic selectors'.

    ·       The ASN of Azure VPN Gateway must be set to 65515. Azure VPN Gateway supports the BGP routing protocol. For ExpressRoute and Azure VPN to work together, you must keep the Autonomous System Number of your Azure VPN gateway at its default value, 65515. If you previously selected an ASN other than 65515 and you change the setting to 65515, you must reset the VPN gateway for the setting to take effect.

    ·       The gateway subnet must be /27 or a shorter prefix, (such as /26, /25), or you will receive an error message when you add the ExpressRoute virtual network gateway.

    ·       Coexistence in a dual stack VNet is not supported. If you are using ExpressRoute IPv6 support and a dual-stack ExpressRoute gateway, coexistence with VPN Gateway will not be possible.

    Create a zone redundant VNet gateway in Azure availability zones

    You can deploy VPN and ExpressRoute gateways in Azure Availability Zones. This brings resiliency, scalability, and higher availability to virtual network gateways. Deploying gateways in Azure Availability Zones physically and logically separates gateways within a region, while protecting your on-premises network connectivity to Azure from zone-level failures.

     Zone-redundant gateways

    To automatically deploy your virtual network gateways across availability zones, you can use zone-redundant virtual network gateways. With zone-redundant gateways, you can benefit from zone-resiliency to access your mission-critical, scalable services on Azure.


    Zonal gateways

    To deploy gateways in a specific zone, you can use zonal gateways. When you deploy a zonal gateway, all instances of the gateway are deployed in the same Availability Zone.

    Gateway SKUs

    Zone-redundant and zonal gateways are available as gateway SKUs. There is a new virtual network gateway SKUs in Azure AZ regions. These SKUs are like the corresponding existing SKUs for ExpressRoute and VPN Gateway, except that they are specific to zone-redundant and zonal gateways. You can identify these SKUs by the "AZ" in the SKU name.

     Public IP SKUs

    Zone-redundant gateways and zonal gateways both rely on the Azure public IP resource Standard SKU. The configuration of the Azure public IP resource determines whether the gateway that you deploy is zone-redundant, or zonal. If you create a public IP resource with a Basic SKU, the gateway will not have any zone redundancy, and the gateway resources will be regional.

    ·       Zone-redundant gateways

    o   When you create a public IP address using the Standard public IP SKU without specifying a zone, the behavior differs depending on whether the gateway is a VPN gateway, or an ExpressRoute gateway.

    o   For a VPN gateway, the two gateway instances will be deployed in any 2 out of these three zones to provide zone-redundancy.

    o   For an ExpressRoute gateway, since there can be more than two instances, the gateway can span across all the three zones.

    ·       Zonal gateways

    o   When you create a public IP address using the Standard public IP SKU and specify the Zone (1, 2, or 3), all the gateway instances will be deployed in the same zone.

    ·       Regional gateways

    o   When you create a public IP address using the Basic public IP SKU, the gateway is deployed as a regional gateway and does not have any zone-redundancy built into the gateway.

    Configure a Site-to-Site VPN as a failover path for ExpressRoute

    You can configure a Site-to-Site VPN connection as a backup for ExpressRoute. This connection applies only to virtual networks linked to the Azure private peering path. There is no VPN-based failover solution for services accessible through Azure Microsoft peering. The ExpressRoute circuit is always the primary link. Data flows through the Site-to-Site VPN path only if the ExpressRoute circuit fails. To avoid asymmetrical routing, your local network configuration should also prefer the ExpressRoute circuit over the Site-to-Site VPN. You can prefer the ExpressRoute path by setting higher local preference for the routes received the ExpressRoute.


    If you have ExpressRoute Microsoft Peering enabled, you can receive the public IP address of your Azure VPN gateway on the ExpressRoute connection. To set up your site-to-site VPN connection as a backup, you must configure your on-premises network so that the VPN connection is routed to the Internet.


    While ExpressRoute circuit is preferred over Site-to-Site VPN when both routes are the same, Azure will use the longest prefix match to choose the route towards the packet's destination.

    Design an ExpressRoute deployment

    ExpressRoute enables us to connect on Premises to Azure services seamlessly. lets review some design decisions you will make before deploying an ExpressRoute circuit.

    ExpressRoute circuit SKUs

    Azure ExpressRoute has three different circuit SKUs: Local, Standard, and Premium. The way you are charged for your ExpressRoute usage varies between these three SKU types.

    ·       Local SKU - With Local SKU, you are automatically charged with an Unlimited data plan.

    ·       Standard and Premium SKU - You can select between a Metered or an Unlimited data plan. All ingress data are free of charge except when using the Global Reach add-on.



    Based on requirements of workloads and data plan, selection of SKU types can help optimize cost and budget.

    Explore pricing based on ExpressRoute SKU

    SKU models have been discussed previously as Local, Standard and Premium. It is a good practice to estimate costs before using Azure ExpressRoute as the price might affect your design decisions.

    Use the Azure pricing calculator to estimate costs before you create an Azure ExpressRoute circuit.

    1.     On the left, select Networking, then select Azure ExpressRoute to begin.

    2.     Select the appropriate Zone depending on your peering location.

    3.     Then select the SKU, Circuit Speed, and the Data Plan you would like an estimate for.

    4.     In the Additional outbound data transfer, enter an estimate in GB of how much outbound data you might use over the course of a month.

    5.     Lastly, you can add the Global Reach Add-on to the estimate.

    Choose a peering location

    Peering location is of importance when working with ExpressRoute.


    Azure regions and ExpressRoute locations are two distinct and different concepts, understanding the difference between the two is critical to exploring Azure hybrid networking connectivity.


    Azure regions

    Azure regions are global datacenters where Azure compute, networking and storage resources are located. When creating an Azure resource, a customer needs to select a resource location. The resource location determines which Azure datacenter (or availability zone) the resource is created in.


    ExpressRoute locations (Peering locations)

    ExpressRoute locations (sometimes referred to as peering locations or meet-me-locations) are co-location facilities where Microsoft Enterprise Edge (MSEE) devices are located. ExpressRoute locations are the entry point to Microsoft's network – and are globally distributed, providing customers the opportunity to connect to Microsoft's network around the world. These locations are where ExpressRoute partners and ExpressRoute Direct customers issue cross connections to Microsoft's network. You would have access to Azure services across all regions within a geopolitical region if you connected to at least one ExpressRoute location within the geopolitical region.


    Azure regions to ExpressRoute locations within a geopolitical region.

    The following link provides a list of Azure regions to ExpressRoute locations within a geopolitical region. This page is kept up to date with the latest ExpressRoute locations and providers.


    ExpressRoute connectivity providers

    The following link list's locations by service provider. This page is kept up to date with the latest available providers by location, see Service providers by location.


    Connectivity through Exchange providers

    If your connectivity provider is not listed in previous sections, you can still create a connection. Several connectivity providers are already connected to Ethernet exchanges.


    Connectivity through satellite operators

    If you are remote and do not have fiber connectivity or want to explore other connectivity options, you can check the following satellite operators.

    Additional Connectivity options:

    ·       Through additional service providers

    ·       Datacenter providers

    ·       National Research and Education networks (NERN)

    ·       System integrators

    Choose the right ExpressRoute circuit and billing model

    Microsoft offers various Express Route options depending on the desired bandwidth of this private connection between the customer on premises network and the selected Azure region. Typically, enterprises need to evaluate their current usage and determine how much data they use monthly to start with.

    The next step is to figure out which of the available ExpressRoute is the best choice depending upon the requirements of the Enterprise keeping in mind the budget and SLA requirements.

    When you deploy ExpressRoute, you must choose between the Local, Standard and Premium SKUs. The Standard and Premium SKU are available in a metered version, where you pay per used GB and an unlimited option.

    The other option is the ExpressRoute Direct, connecting your network to the closest Microsoft Edge node which then connects to the Microsoft Global Network, to connect to other customers offices or factories and any Azure Region. The usage of the Microsoft Global Network is charged on top of the ExpressRoute Direct.

    Please refer to the Express Route pricing for details on metered and unlimited data plan based on the bandwidth.

    You can purchase ExpressRoute circuits for a wide range of bandwidths. The supported bandwidths are listed as follows. Be sure to check with your connectivity provider to determine the bandwidths they support.

    50 Mbps

    100 Mbps

    200 Mbps

    500 Mbps

    1 Gbps

    2 Gbps

    5 Gbps

    10 Gbps

    Choose a billing model

    You can pick a billing model that works best for you. Choose between the billing models listed as followed.


    ·       Unlimited data. Billing is based on a monthly fee; all inbound and outbound data transfer is included free of charge.

    ·       Metered data. Billing is based on a monthly fee; all inbound data transfer is free of charge. Outbound data transfer is charged per GB of data transfer. Data transfer rates vary by region.

    ·       ExpressRoute premium add-on. ExpressRoute premium is an add-on to the ExpressRoute circuit. The ExpressRoute premium add-on provides the following capabilities:

    o   Increased route limits for Azure public and Azure private peering from 4,000 routes to 10,000 routes.

    o   Global connectivity for services. An ExpressRoute circuit created in any region (excluding national clouds) will have access to resources across every other region in the world. For example, a virtual network created in West Europe can be accessed through an ExpressRoute circuit provisioned in Silicon Valley.

    o   Increased number of VNet links per ExpressRoute circuit from 10 to a larger limit, depending on the bandwidth of the circuit.


    Configure peering for an ExpressRoute deployment

    An ExpressRoute circuit has two peering options associated with it: Azure private, and Microsoft. Each peering is configured identically on a pair of routers (in active-active or load sharing configuration) for high availability. Azure services are categorized as Azure public and Azure private to represent the IP addressing schemes.


    Create Peering configuration

    ·       You can configure private peering and Microsoft peering for an ExpressRoute circuit. Peering's can be configured in any order you choose. However, you must make sure that you complete the configuration of each peering one at a time.

    ·       You must have an active ExpressRoute circuit. Have the circuit enabled by your connectivity provider before you continue. To configure peering(s), the ExpressRoute circuit must be in a provisioned and enabled state.

    ·       If you plan to use a shared key/MD5 hash, be sure to use the key on both sides of the tunnel. The limit is a maximum of 25 alphanumeric characters. Special characters are not supported.

    ·       This only applies to circuits created with service providers offering Layer 2 connectivity services. If you are using a service provider that offers managed Layer 3 services (typically an IPVPN, like MPLS), your connectivity provider configures and manages the routing for you.

    Choose between private peering only, Microsoft peering only, or both

    The following table compares the two peering. Public peering is deprecated for new peering.


    Private Peering

    Microsoft Peering

    Max. # prefixes supported per peering

    4000 by default, 10,000 with ExpressRoute Premium


    IP address ranges supported

    Any valid IP address within your WAN.

    Public IP addresses owned by you or your connectivity provider.

    AS Number requirements

    Private and public AS numbers. You must own the public AS number if you choose to use one.

    Private and public AS numbers. However, you must prove ownership of public IP addresses.

    IP protocols supported

    IPv4, IPv6 (preview)

    IPv4, IPv6

    Routing Interface IP addresses

    RFC1918 and public IP addresses

    Public IP addresses registered to you in routing registries.

    MD5 Hash support