Troubleshoot DMVPN Phase 3 with dual-hub

Dynamic Multipoint Virtual Private Network

3.2.a Configuration (Not on blueprint)

General information on “DMVPN Configuration”:

DMVPN consists of several mandatory and optional elements
DMVPN key elements:
- mGRE (mandatory)
- NHRP (mandatory)
- IPsec (optional)
- Routing Protocol (optional)

Configuration considerations:

Configuration parameters:
- The network-id <id> parameter is only locally significant and is used to create multiple NHRP databases in case there are multiple DMVPN domains configured/terminated on the router
- The nhrp authentication <key> parameter is domain-wide significant is a pseudo-authentication (=not encrypted) and used to keep NHRP domains separate from each other
- The tunnel key <key> parameter is domain-wide significant and used as a distinguisher in case there are multiple DMVPN domains configured/terminated on the router so that GRE knows which packet belongs to which tunnel interface
Protocol overhead:
- To avoid fragmentation, it’s recommended to adjust the MTU and MSS of the tunnel interface
- IPv4: MTU 1400, MSS 1360
- IPv6: MTU 1400, MSS 1340
Dual/Multi-Hub setup:
- Important: The second hub must be registered to the first hub as spoke!
- For each hub, there must be a separate ip nhrp nhs command configured on the spoke(s)
EIGRP over DMVPN:
- Split-Horizon must be disabled on the hub(s)
OSPF over DMVPN:
- Network type point-to-multipoint is recommended because of no DR/BDR election
- Network type broadcast can be used, but then DMVPN is limited to two hubs and the priority must be explicitly configured (255 on the hubs, 0 on the spokes) so that no spoke router becomes the DR/BDR
DMVPN with IPv6:
- IPv4 only: Only use ip nhrp commands
- IPv4 over IPv6: The ip nhrp nhs must be mapped to the IPv6 NBMA address
- IPv6 only: Only use ipv6 nhrp commands
- IPv6 over IPv4: The ipv6 nhrp nhs must be mapped to the IPv4 NBMA address
- When doing IPv4 over IPv6, the command tunnel mode gre multipoint ipv6 must be configured under the tunnel interface

“DMVPN Configuration” CLI configuration commands:

## =========
## DMVPN hub
## =========

## Basic DMVPN hub router configuration
Router(config)# interface tunnel <id>
Router(config-if)# ip address <ip> <mask>
Router(config-if)# ip nhrp authentication <key>
Router(config-if)# ip nhrp map multicast dynamic
Router(config-if)# ip nhrp network-id <id>
Router(config-if)# tunnel source <interface>
Router(config-if)# tunnel mode gre multipoint
Router(config-if)# tunnel key <key>
Router(config-if)# ip nhrp redirect


## ===========
## DMVPN spoke
## ===========

## Basic DMVPN spoke router configuration
Router(config)# interface tunnel <id>
Router(config-if)# ip address <ip> <mask>
Router(config-if)# ip nhrp authentication <key>
Router(config-if)# ip nhrp nhs <hub-tunnel-ip> nbma <hub-nbma-ip> multicast
Router(config-if)# ip nhrp network-id <id>
Router(config-if)# tunnel source <interface>
Router(config-if)# tunnel mode gre multipoint
Router(config-if)# tunnel key <key>
Router(config-if)# ip nhrp shortcut


## ==================
## MTU/MSS adjustment
## ==================

## Adjusting MTU and MSS (IPv4)
Router(config)# interface tunnel <id>
Router(config-if)# ip mtu 1400
Router(config-if)# ip tcp adjust-mss 1360

## Adjusting MTU and MSS (IPv6)
Router(config)# interface tunnel <id>
Router(config-if)# ipv6 mtu 1400
Router(config-if)# ipv6 tcp adjust-mss 1340

“DMVPN Configuration” CLI show commands:

## Showing the DMVPN status and tunnels in detail
Router# show dmvpn

## Showing the NHRP cache
Router# show ip nhrp

## Showing the NHRP redirect table
Router# show ip nhrp redirect

## Showing the NHRP shortcut table
Router# show ip nhrp shortcut

## Showing all routes which' next-hop address has been overwritten by NHRP
Router# show ip route next-hop-override

3.2.a i NHRP

Next Hop Resolution Protocol

General information on “NHRP”:

Used for spokes to find out the NBMA address (public IP address) of other spokes
NHRP server-client principle:
- NHRP server: Keeps track of all NBMA addresses (public IP addresses) in its NHRP cache
- NHRP client: Register themselves to the server and report their own NBMA address (public IP address)
When a Spoke wants to connect to another spoke, it queries the NHRP server for the NBMA address (public IP address) of the other Spoke
NHRP message types:
- Registration Request: Spokes register their NBMA address and Tunnel IP address to the NHRP server
- Resolution Request: Spokes query the NHRP server for NBMA addresses of other spokes
- Redirect: NHRP server answer to Spoke-to-Spoke data-plane packet through it
DMVPN underlay and overlay:
- Underlay: The network used for connectivity between the different routers
- Overlay: The established GRE tunnels
NHRP Flags:
- Authoritative: NHRP information was obtained from a NHS.
- Implicit: NHRP information was obtained/learned from a NHRP resolution request or packet.
- Local: NHRP mapping entry for a network that is local to this router.
- NAT: Indicates that the remote device supports NHRP NAT extensions which allows dynamic spoke-to-spoke tunnels to/from spokes behind a NAT router.
- Negative: Indicates that the requested NBMA address for a NHRP mapping could not be obtained.
- (no socket): NHRP mapping for which the router won’t set up IPsec encryption because there’s no data traffic that uses this tunnel.
- Registered: NHRP mapping entry was created from a received an NHRP registration request.
- Router: NHRP mapping entries for a remote router itself to access a network/host behind it.
- Unique: NHRP mapping entry can’t be overwritten by a different NHRP mapping entry with the same tunnel address but different NBMA address.
- Used: Data packets are being process-switched for the NHRP mapping.

NHRP Next-Hop Override (DMVPN phase 3):

When the tunnels (Hub to Spokes) are established for the first time and a dynamic routing protocol used, the Hubs Tunnel IP address is set as the next-hop IP address for each route on the Spokes
When traffic starts to flow, dynamic tunnels between the Spokes are created
Since NHRP redirect and NHRP shortcut is used, the next-hop IP address gets changed to the router who is actually advertising the route
In the routing table a percent-sign (%) is displayed next to the route which indicates NHO
This is because a the Spoke learns a better way (metric-wise) to reach the destination
The CEF entry gets also overwritten

DMVPN route types:

T1: NHRP-learned routes
T2: NHO routes

3.2.a ii IPsec/IKEv2 and IPsec/ISAKMP (IKEv1) using pre-shared key

General information on “IPsec/IKEv2 and IPsec/ISAKMP (IKEv1) using pre-shared key”:

Without IPsec, no data is encrypted when being transferred over DMVPN
This means the packet payload is in clear text and can be sniffed
To overcome this, IPsec with IKEv2 is recommended but IKEv1 is also possible

DMVPN Order of operations:

First: Crypto
Second: NHRP
Third: Routing

Configuration considerations:

5 steps are required to configure IPsec/ISAKMP (IKEv1) using PSK:

Steps	Mandatory/Optional	Comments
ISAKMP Policy (Phase 1)	Optional	Defaults are defined
ISAKMP Keyring (Phase 1)	Mandatory
ISAKMP Profile (Phase 1)	Mandatory
IPsec Transform-Set (Phase 2)	Mandatory
IPsec Profile (Phase 2)	Mandatory
Applying IPsec Profile to Tunnel Interface	Mandatory

7 steps are required to configure IPsec/IKEv2 using PSK:

Steps	Mandatory/Optional	Comments
IKEv2 Proposal (Phase 1)	Optional	Defaults are defined
IKEv2 Policy (Phase 1)	Optional	Mandatory for F-VRF
IKEv2 Keyring (Phase 1)	Optional
IKEv2 Profile (Phase 1)	Mandatory
IPsec Transform-Set (Phase 2)	Mandatory
IPsec Profile (Phase 2)	Mandatory
Applying IPsec Profile to Tunnel Interface	Mandatory

Showing commands output:
- ISAKMP SA status has to be QM_IDLE in order for Phase 2 to work
- IKEv2 SA status has to be READY in order for Phase 2 to work
- IPsec SA status has to be…
  - Equal for pkts encaps/encrypt/digest
  - Equal for decaps/decrypt/verify

Important: It’s always recommended to get a working DMVPN tunnel first before applying IPsec to it!

“DMVPN IPsec/IKEv2 and IPsec/ISAKMP (IKEv1) using pre-shared key” CLI configuration commands:

## ============================
## IPsec/ISAKMP (IKEv1) Phase 1
## ============================

## IPsec/ISAKMP (IKEv1) Phase 1 configuration
Router(config)# crypto isakmp policy <priority>
Router(config-isakmp)# encryption [options]
Router(config-isakmp)# hash [options]
Router(config-isakmp)# group [options]
Router(config-isakmp)# authentication [options]

Router(config)# crypto keyring [NAME]
Router(conf-keyring)# pre-shared-key address <ip> <mask> key <KEY>

Router(config)# crypto isakmp profile [NAME]
Router(conf-isa-profile)# match identity address <peer-nbma-ip> <mask>


## ===================
## IPsec/IKEv2 Phase 1
## ===================

## IPsec/IKEv2 Phase 1 configuration
Router(config)# crypto ikev2 proposal [NAME]
Router(config-ikev2-proposal)# encryption [options]
Router(config-ikev2-proposal)# integrity [options]
Router(config-ikev2-proposal)# group [options]

Router(config)# crypto ikev2 policy [NAME]
Router(config-ikev2-policy)# proposal [NAME]

Router(config)# crypto ikev2 keyring [NAME]
Router(config-ikev2-keyring)# peer [NAME]
Router(config-ikev2-keyring-peer)# address <peer-nbma-ip>
Router(config-ikev2-keyring-peer)# pre-shared-key <key>

Router(config)# crypto ikev2 profile [NAME]
Router(config-ikev2-profile)# authentication local pre-share
Router(config-ikev2-profile)# authentication remote pre-share
Router(config-ikev2-profile)# match identity remote address <ip>
Router(config-ikev2-profile)# keyring local [NAME]
Router(config-ikev2-profile)# dpd <sec> <sec> on-demand


## =============
## IPsec Phase 2
## =============

## IPsec Phase 2 transfrom set configuration (valid for IKEv1/IKEv2)
Router(config)# crypto ipsec transform-set [NAME] [options]
Router(cfg-crypto-trans)# mode transport

## IPsec Phase 2 IPsec profile configuration (valid for IKEv1/IKEv2)
Router(config)# crypto ipsec profile <name>
Router(ipsec-profile)# set transform-set [NAME]

For IPsec/ISAKMP (IKEv1):
Router(ipsec-profile)# set isakmp-profile [NAME]

For IPsec/IKEv2:
Router(ipsec-profile)# set ikev2-profile [NAME]


## ==============
## Enabling IPsec
## ==============

## Enabling IPsec on the tunnel interface
Router(config)# interface <if>
Router(config-if)# tunnel protection ipsec profile <ipsec-profile-name>

“DMVPN IPsec/IKEv2 and IPsec/ISAKMP (IKEv1) using pre-shared key” CLI show commands:

## Showing IPsec/ISAKMP (IKEv1) Phase 1 security associations
Router# show crypto isakmp sa

## Showing IPsec/IKEv2 Phase 1 security associations
Router# show crypto ikev2 sa

## Showing IPsec Phase 2 security associations
Router# show crypto ipsec sa

3.2.a iii Per-Tunnel QoS

General information on “DMVPN Per-Tunnel QoS”:

Used to apply QoS on a per hub-to-spoke-tunnel basis
QoS only applies between hub and spoke
QoS doesn’t apply between dynamic spoke tunnels

Configuration considerations:

Tunnels need to be flapped after applying the configuration in order for QoS to work

“DMVPN Per-Tunnel QoS” CLI configuration commands:

## Per-Tunnel QoS configuration on DMVPN hub routers
Router(config)# interface tunnel <id>
Router(config-if)# nhrp map group <group-name> service-policy output <policy-name>

## Per-Tunnel QoS configuration on DMVPN spoke routers
Router(config)# interface tunnel <id>
Router(config-if)# nhrp group <group-name>

“DMVPN Per-Tunnel QoS” CLI show commands:

## Showing QoS policies applied to multipoint interfaces
Router# show policy-map multipoint

3.2.a x Front-Door VRF (F-VRF) (not on blueprint)

General information on “DMVPN Front-Door VRF (F-VRF)":

The interface facing towards the underlay network (“front-door interface”) is put in a separate VRF
This separates the underlay transport routing table from the overlay transport routing table

“DMVPN Front-Door VRF (F-VRF)” CLI configuration commands:

## ============================
## IPsec/ISAKMP (IKEv1) Phase 1
## ============================

## IPsec/ISAKMP (IKEv1) F-VRF configuration
Router(config)# crypto keyring [NAME] vrf [F-VRF]
Router(conf-keyring)# pre-shared-key address <ip> <mask> key <KEY>

Router(config)# crypto isakmp profile [NAME]
Router(conf-isa-profile)# match identity address <peer-nbma-ip> <mask> [F-VRF]


## ===================
## IPsec/IKEv2 Phase 1
## ===================

## IPsec/IKEv2 F-VRF configuration
Router(config)# crypto ikev2 proposal [NAME]
Router(config-ikev2-proposal)# encryption [options]
Router(config-ikev2-proposal)# integrity [options]
Router(config-ikev2-proposal)# group [options]

Router(config)# crypto ikev2 policy [NAME]
Router(config-ikev2-policy)# proposal [NAME]
Router(config-ikev2-policy)# match fvrf <fvrf-name>

Router(config)# crypto ikev2 profile [NAME]
Router(config-ikev2-profile)# match fvrf <fvrf-name>


## ==================
## Enabling the F-VRF
## ==================

## F-VRF interface configuration (valid for IPsec/ISAKMP (IKEv1) and IPsec/IKEv2)
Router(config)# interface <if>
Router(config-if)# vrf forwarding <fvrf-name>

Router(config)# interface tunnel <id>
Router(config-if)# tunnel vrf <fvrf-name>

“DMVPN Front-Door VRF (F-VRF)” CLI show commands:

## Showing IPsec/ISAKMP (IKEv1) Phase 1 security associations
Router# show crypto isakmp sa

## Showing IPsec/IKEv2 Phase 1 security associations
Router# show crypto ikev2 sa

## Showing IPsec Phase 2 security associations
Router# show crypto ipsec sa

3.2.a x Inside VRF (I-VRF) (not on blueprint)

General information on “DMVPN Inside VRF (I-VRF)":

The virtual tunnel interface (“inside interface”) is put in a separate VRF
This separates the overlay transport routing table from the underlay transport routing table

“DMVPN Inside VRF (I-VRF)” CLI configuration commands:

## I-VRF interface configuration (valid for IPsec ISAKMP (IKEv1) and IPsec/IKEv2):
Router(config)# interface tunnel <id>
Router(config-if)# vrf forwarding <ivrf-name>

“DMVPN Inside VRF (I-VRF)” CLI show commands:

## Showing IPsec/ISAKMP (IKEv1) Phase 1 security associations
Router# show crypto isakmp sa

## Showing IPsec/IKEv2 Phase 1 security associations
Router# show crypto ikev2 sa

## Showing IPsec Phase 2 security associations
Router# show crypto ipsec sa

3.2.b x MPLS over DMVPN (not on blueprint)

General information on “MPLS over DMVPN (not on blueprint)":

Used for separation of different networks sometimes w/ overlapping address space) (just like “normal” MPLS)
Supported MPLS operations: pop (dispose), route, push (impose)
Doesn’t support MPLS swap operation (= no label switching!)
Based on NHRP instead of LDP because LDP uses keep-alives and therefor spoke-to-spoke tunnels would never be terminated

Configuration considerations:

Only option:
- Hub and Spokes: Configure mpls nhrp on all tunnel interfaces which belong to the DMVPN solution
- Hub and Spokes: Configure iBGP VPNv4 between spokes and hub
- Hub only: Summarize the routes for each VRF
Using the hub as RR is not supported/recommended by Cisco!

“MPLS over DMVPN” CLI configuration commands:

## Enabling MPLS NHRP on all DMVPN interfaces
Router(config)# interface <if>
Router(config-if)# mpls nhrp