WAN edge deployment

2.2.b i Onboarding new edge routers

General information on “SDW Onboarding new edge routers”:

• Pre-onboarding steps:
  • WAN Edge device list (*.viptela) file must be imported into vManage
• vEdge non-ZTP onboarding:
  1. Configure System (host-name, system-ip, site-id, organization, vBond address, NTP server)
  2. Configure VPN 0 (set default route, set DNS, assign interface, assign IP address, “no shut” interface, enable/configure tunnel-interface)
  3. Verify connection to the controllers via ping
  4. Install Root CA manually via CLI with (when using a local CA):
     • request root-cert-chain install <CERT-PATH>
  5. Grab unused UUID/OTP from the vManage GUI and activate the vEdge manually via CLI with:
     • request vedge-cloud activate chassis <UUID> token <OTP>
• cEdge (IOS-XE) non-ZTP onboarding:
  1. Configure System (host-name, system-ip, site-id, organization, vBond address, NTP server)
  2. Configure physical Interface(s) later used for the underlay network (set default route, assign IP address, “no shut” interface)
  3. Verify connection to the controllers via ping
  4. Install Root CA manually via CLI with (when using a local CA):
     • request platform software sdwan root-cert-chain install <CERT-PATH>
  5. Configure virtual tunnel interface(s) (IP unnumbered, tunnel source, tunnel mode sdwan)
  • Import: The virtual tunnel interface number, IP unnumbered, tunnel source correlates to the physical interface, eg. Tunnel1 for Gi0/0/1, Tunnel10 for Gi0/1/0, Tunnel 100 for Gi1/0/0, …
  6. Configure SDWAN function (assign physical interface, enable/configure tunnel-interface, set color/encapsulation)
  7. Grab unused UUID/OTP from the vManage GUI and activate the vEdge manually via CLI with:
     • request platform software sdwan vedge_cloud activate chassis-number <UUID> token <OTP>

2.2.b ii Orchestration with zero-touch provisioning/Plug-And-Play

// Graphic missing - Coming soon //

General information on “SDW ZTP/PNP”:

• Process of ZTP/PNP:
  1. WAN Edge device boots up and tries to obtain an IP address on the WAN side via DHCP
  2. • vEdge: tries to contact the ZTP Service (call home to Cisco on ztp.viptela.com)
     • cEdge: tries to contact the Device Helper Service (call home to devicehelper.cisco.com)
  3. ZTP service redirects the WAN Edge device to the customers orchestrator (vBond)
  4. WAN Edge device establishes a transient connection to the vBond and sends its chassis ID and serial number vBond verifies the information and sends the IP address of the vManage to the WAN Edge device
  5. WAN Edge device establishes a connection to the vManage and gets verified by it vManage sends WAN Edge device its to-be-used system IP address
  6. WAN Edge device re-establishes the connection to the vBond using its new system IP address
  7. WAN Edge device re-establishes the connection to the vManage using its new system IP address
  8. If necessary, the vManage pushes out a software update to the WAN Edge device
  9. After reboot, WAN Edge device re-establishes the connection to the vBond which verifies the WAN Edge device again
  10. WAN Edge device re-establishes the connection to the vManage which then pushes the full configuration
  11. The router joins the SD-WAN overlay network
• When there’s no DHCP service available on the WAN side, the “Auto IP discovery” feature will be used
• “Auto IP discovery” works by examining the ARP packets on the subnetwork and tries to infer the IP address of the ZTP/PNP interface
• The whole device bring-up procedure can be verified within the vManage to see if all steps were successful
• A device template must be attached to the WAN Edge device within vManage before initiating the ZTP/PNP proccess or else it will fail

Cisco PnP Portal/vManage:

• WAN Edge devices can be imported into vManage either via a CSV file created on the Cisco PnP portal (automatic or manual device import) or a Cisco Smart account can be used for automatic synchronization

2.2.b iii OMP

// Graphics missing - Coming soon //

Overlay Management Protocol

General information on “SDW OMP”:

• TCP-based extensible control plane protocol (functionality similar to BGP)
• Runs inside the TLS/DTLS tunnel between the vEdge and vSmart as well as between the vSmarts themselves
• Therefor all data is authenticated as well as encrypted (= not visible to the outside)
• Dramatically simplifies the control plane complexity (star topology instead of full-mesh) which is comparable to BGP route-reflector vs. BGP full-mesh
• OMP peerings are automatically established using the system-ip and don’t need be enabled explicitly
• OMP updates include reachability (routes, TLOCs), security (Encryption Keys), policy (Data/Application-Aware Policies)
• Immediately after the vEdges receive OMP updates, they try to create IPSec tunnels to whatever TLOC they can reach
• By default, connections between different TLOCs are also tried to be established (eg. Internet <-> MPLS) (can be disabled with the restrict keyword if needed)
• vSmarts learn through OMP routes the topology of the overlay network and the services available within
• vEdges use OMP to advertise local prefixes/routes (Static/EIGRP (since 19.1)/OSPF/BGP) along with the corresponding TLOCs to the vSmarts
• vSmarts also push out all neccessary IPSEC information for each TLOC so that WAN Edge devices know how to establish a secure IPSEC tunnel to each others TLOCs
• vSmarts use OMP to push/advertise routes to the vEdges
• vSmarts push/advertise all routes unmanipulated (as they come in) by default but this behavior can be modified by using centralized Control Policies created on vManage
• OMP also advertises policies that are executed on the vEdges (including application-routing policies, data policies, …)
• OMP supports graceful restart which allows the data plane to continue functioning even if the control plane stops working

OMP advertises the following route types:

• OMP routes (vRoutes): Routes that the vEdges have learned from its site-local network. These can be Connected/Static/BGP/OSPF (EIGRP since 19.1 and only with IOS-XE SD-WAN Images). In addition to the prefixes, a number of attributes are advertised like the TLOC (comparable to BGP NEXT_HOP), Site ID, Label (VPN), Tag, Preference, Originator System IP, Origin Protocol, Origin Metric, AS_PATH.
• Transport Locations (TLOCs): TLOC routes identify transport locations. TLOCs can be seen as physical WAN interfaces connecting to different transports (eg. MPLS). A TLOC is denoted by a 3-tuple consisting of the system IP address, color (eg. MPLS, Internet, …) and encapsulation type. Each TLOC is advertised separately. Each TLOC contains the attributes TLOC private address, TLOC public address, Carrier, Color, Encapsulation Type, Preference, Site ID, Tag, Weight.
• Service routes: Represent services that are connected to a vEdge or the local-site network like Firewalls, …

OMP route redistribution:

• Automatic redistribution from routing protocols running on the vEdge (into OMP): Connected, Static
• Requires explicit configuration (into OMP): OSPF, OSPF, BGP, EIGRP
• vEdge doesn’t automatically redistribute routes received via OMP into any routing protocols running on the vEdge
• Redistribution must be explicitly enabled locally on each vEdge router (if needed)
• OMP sets the origin and sub-origin type in each OMP route to indicate the route’s origin:

• Metric of the original route is carried along (metric of 0 indicates a connected route)
• Important regarding BGP redistribution: When doing BGP to OMP redistribution on a WAN Edge router, the ASN of the WAN Edge router won’t get added to the BGP AS_PATH attribute. This is because the BGP NLRI is redistributed into OMP and not advertised to another BGP peer.

OMP redistribution loop prevention:

• OMP to OSPF: The DN-Bit will be set.
• OMP to BGP: The BGP extended community SoO will be set to 0:<Site-ID>, whereas the <Site-ID> is the value of the receiving site.

Administrative Distance within OMP:

• Default AD values used by SD-WAN:

OMP Best-Path Selection Process:

1. Route Resolvability (is next-hop TLOC is reachable?)
2. Route Source Preference (Prefer vEdge over vSmart routes)
3. Administrative Distance (prefer OMP route with lowest AD)
4. Route Preference (prefer route with highest OMP route preference)
5. Only on vEdge: TLOC preference (prefer route with highest TLOC preference)
6. • Origin (prefer route with best origin => Connected, Static, eBGP, EIGRP, OSPF Intra/Inter/External, iBGP, …)
   • If two equal routes have the same AD, the lower origin cost/metric will decide
7. Only on vEdge: Prefer route from highest origin system-ip (aka router-id within OSPF/EIGRP)
8. • Only on vEdge: Prefer route from highest private TLOC IP address
   • Only on vSmart: Prefer vEdge-received routes over vSmart-received routes

OMP Best-Path Selection information:

• By default, 4 equal-cost paths (ECMP) to a destination are advertised by vSmart (16 maximum)
• Backup routes can be advertised to vEdges to provide faster convergence
• vEdge installs an OMP route in the FIB only if the TLOC to which it points is active
• This means, an active BFD session must be associated with that TLOC
• If a vEdge is marked as invalid (offline), all routes etc. will be immediately removed from the SD-WAN fabric (= no stale routes)
• Outbound policies are applied AFTER the best path(s) are chosen

OMP WAN Edge Best-Path MNEMONIC:

• Resolvability
• Source Preference
• Administrative Distance
• OMP Preference
• TLOC Preference
• Origin
• System-IP
• Private TLOC

MNEMONIC Sentence: Routers Should Always Offer Tracking Of System Paths.

OMP support for traditional unicast routing protocols:

• BGP, EIGRP (since 19.1, IOS-XE only) and OSPF are supported in any VPN except VPN 0 and VPN 512
• Routes can be redistributed into OMP so that OMP can better chose paths within the overlay network
• When connecting to a MPLS network, the vEdge acts as a MPLS CE device (L3 VPN only!)
• When the vEdge is several hops away from the WAN and connects indirectly through a non-Viptela hub router, standard routing must be enabled on the vEdge DTLS connections so that they can reach the WAN cloud (BGP, EIGRP or OSPF)

OMP support for multicast routing:

• Only PIM-SM is supported for multicast routing within the overlay network
• PIM-SM RP function must be provided by a non-Viptela router since this function isn’t implemented (yet?)
• The RP can only be defined by using the Cisco AutoRP feature
• PIM must be enabled everywhere, whereas IGMP and Multicast Replication is only required at the receiver sites
• PIM/IGMP must be enabled on a per-interface basis

2.2.b iv TLOC

// Graphic missing - Coming soon //

Transport Location

General information on “SDW TLOC”:

• A TLOC is an attachment point into the fabric which equals to the next-hop in traditional routing
• The advantage to traditional routing next-hop is, that the TLOC IP is not the “real” WAN IP address, but rather the virtual system address of a vEdge
• This means, that even if the WAN IP address of a vEdge changes, the TLOC IP address will always stay the same
• By default, all vEdges advertise all of their TLOCs to the vSmart who then advertises them to all other vEdges
• The TLOC is a 3-tuple denotation (system-ip value, color, encapsulation)
• A TLOC is identified by several properties, such as the “IP Address <-> Color”-pair (primary identifier)
• The TLOC IP address is inherited from the system-ip value
• The color is a static identifier used to define the connected transport of a TLOC
• The TLOC color is either private (mpls, private1-6, metro-ethernet) or public (default, red, blue, public-internet, …)
• If the color is private, WAN Edges will establish a tunnel with the native underlay IP address
• If the color is public, WAN Edges will establish a tunnel with the post-NAT IP address (if NAT is involved)
• By default, every color tries to connect with every other colors
• This can be changed by configuring the color command with the restrict keyword under the tunnel-interface configuration
• A single color can be only assigned to one physical interface at any time
• Only a single control connection to the vManage controller is established, even if there are multiple TLOCs

TLOC Extension:

// Graphic missing - Coming soon //

• Used to extend the TLOC (= WAN Transport) of a vEdge to another vEdge who has (either/or):
  • no direct WAN/MPLS/… connection at all, or…
  • a TLOC of another color
• Used mainly for transport redundancy reasons
• Could also be done via NAT or static/dynamic routing but you would lose all SD-WAN features like AAR, …
• Example: vEdge_A has a TLOC with color mpls. vEdge_B has a TLOC with color public-internet. Both have a direct connection to each to each other. By default, each router only establishes tunnels with his own TLOCs. TLOC Extension allows each router to use each others Transpot VPN TLOC color and establish another tunnel over it.
• Configuration considerations:
  • LTE interfaces don’t support TLOC extensions
  • For the vEdge who wants to “share” his TLOC:
    • The transit interface needs to be configured under VPN0
    • tloc-extension <if> command needs to be configured, where the <if> is the TLOC to be shared
  • For the vEdge who wants to “use” another vEdge' “shared” TLOC:
    • The transit interface needs to be configured under VPN0
    • The tunnel-interface command including the color needs to be configured
  • For public network TLOCs:
    1. NAT needs to be configured on the “sharing” vEdge, if the outgoing interface is connected to a public network
    2. A static route needs be configured of the “using” vEdge, pointing to the transit IP of the “sharing” vEdge
  • For private network TLOCs:
    1. The transit network between the vEdges has to be advertised into the private network
• A working TLOC extension can be validated using the show control connections command

2.2b Everything else (not on blueprint)

General information on “SDW Everything else”:

• When creating sub-interfaces, there are two possible configuration options:
  1. The respective physical interface must be put in VPN0 without an IP address and the MTU must be changed to 1504
  2. The respective physical interface must be put in VPN0 without an IP address and the MTU of the sub-interfaces must be adjusted to 1496