As part of a temporary solution I had to join a new office to our existing Data Centres and OSPF core using a Gig circuit over the Internet. To flesh out this idea and test its viability I thought I would try and solve an ESX capacity problem I have at home and move vCentre into the cloud.
When ESX 6.5 came out I looked at upgrading but could never get vCentre to run stably on anything less than the 8GB of RAM. I use a NUC for my ESX host so am limited to a max of 32gb RAM so wasting a quarter of that for features I occasionally use didnt appeal me so I stuck with 5.5 as I can get away running vCentre with 3GB. However recently I have been having issues with Win10 on 5.5, and if I upgrade past 1511 IT causes the windows machine to crash when upgrading.
The initial setup will using Server2012 (vcentre) and CSR1000v (GRE & IPsec) in azure, whilst at home a ASA5505 (IPSEC) and CSR1000v (GRE). I could have terminated the IPsec tunnel on ASR at both ends, however at work we have Nexus7K on one end that doesn’t support IPSEC so am using the ASA to replicate our production envirnomet.
I am using Azure over AWS simply because I have a Visual Studio Enterprise subscription which gives I get free credits with, I am sure you could do the exact same thing in AWS. In the future once I am finished with testing I will remove the ASRs and use Azure VPN as the CSRs are quite pricey and trial license is only 3 months at a time.
There are 3 core real elements to this solution:
-IPSEC tunnel that will allow only the GRE traffic.
-GRE tunnel which will allow all traffic between home and Azure.
-OSPF peering over the GRE tunnel.
1. License ASRs
By default when you deploy the ASRs they have an AX license, meaning you can use all the features but are limited to only 100kbps of throughput.
show platform hardware throughput level Check current throughput show version | in license To see if it has a valid license
You can obtain a free 60 day trial license for cisco, however after assigning 6 you can’t get anymore without contacting Cisco support.
The 3 main types of license are:
IPBase: Basic Networking Routing (Routing, HSRP, NAT, ACL, VRF, GRE)
Security: IPBase package + Security features (IP Security VPN, Firewall, MPLS, Multicast, QoS)
AX: IPBase package + Security features + Advanced Networking features (AppNav, AVC, OTV and LISP)
To get license you first need to get the UID from your ASR as that is bound to the license.
show license udi
Then got http:/www.cisco.com/go/license
Get Licenses >> Demo and Evalaution >> routers & switches >>. cisco cloud services router 1000v
Choose 2.5 Gbps Full Featured (AX) 60 day evaluation license and enter the UID.
Upload the license to the ASR and install it, you shouldn’t need a reboot unless your changing the license level.
license install bootflash:lic-name.xml (config)# license boot level ax Change boot level if not ax show license detail show version | in license show platform hardware throughput level
2. HM-ASA to HM-CSR connectivity.
I am running OSPF area0 between these devices and in the core of my network. Loopback0 is advertised into this as will be used for the GRE tunnel source/destination.
HM-ASA1
interface Vlan10 nameif mgmt security-level 100 ip address 10.10.10.1 255.255.255.0 ! interface Vlan101 nameif gre-transit security-level 100 ip address 172.168.255.17 255.255.255.240 ! route-map CONN->OSPF permit 10 match interface mgmt ! router ospf 1 router-id 172.168.255.1 network 172.168.255.16 255.255.255.240 area 0 redistribute connected subnets route-map CONN->OSPF
HM-ASA1# show ospf int brief
Interface PID Area IP Address/Mask Cost State Nbrs F/C
gre-transit 1 0 172.168.255.17/255.255.255.240 10 DR 1/1
HM-ASA1# show ospf neighbor
Neighbor ID Pri State Dead Time Address Interface
192.168.201.1 1 FULL/BDR 0:00:34 172.168.255.18 gre-transit
HM-CSR1
router ospf 1 router-id 192.168.201.1 ! interface GigabitEthernet1 ip address 172.168.255.18 255.255.255.240 ip ospf 1 area 0 ! interface Loopback1 ip address 192.168.201.1 255.255.255.255 ip ospf 1 area 0
HM-CSR1#show ip ospf int bri
Interface PID Area IP Address/Mask Cost State Nbrs F/C
Lo1 1 0 192.168.201.1/32 1 LOOP 0/0
Gi1 1 0 172.168.255.18/28 1 BDR 1/1
HM-CSR1#show ip ospf neighbor
Neighbor ID Pri State Dead Time Address Interface
172.168.255.1 1 FULL/DR 00:00:36 172.168.255.17 GigabitEthernet1
3. ASA IPsec configuration
The IPsec VPN interesting traffic only needs to carry the CSR loopback interfaces that will be used as the source and destination by the GRE tunnel.
Interesting traffic:
object-group network HM-to-AZ-LOCAL network-object host 192.168.201.1 object-group network HM-to-AZ-REMOTE network-object host 172.168.201.1 ! access-list VPN-HM-to-AZ extended permit ip object-group HM-to-AZ-LOCAL object-group HM-to-AZ-REMOTE
If have sysopt connection permit-vpn disabled also need a rule in the outside ACL.
access-list outside extended permit ip object-group HM-to-AZ-REMOTE object-group HM-to-AZ-LOCAL
Will need to noNAT the traffic if it is matched by another NAT such as PAT. Easy way to check is with packet-tracer.
packet-tracer input gre-transit udp 192.168.201.1 500 172.168.201.1 500
Is best to put this rule near the top as you don’t want it to be accidentally be caught by another NAT in the future.
nat (gre-trasnit,outside) source static HM-to-AZ-LOCAL HM-to-AZ-LOCAL destination static HM-to-AZ-REMOTE HM-to-AZ-REMOTE
Phase1 and Phase2:
crypto ikev1 policy 10 authentication pre-share encryption aes-256 hash sha group 5 lifetime 86400 crypto ikev1 enable outside ! crypto ipsec ikev1 transform-set ESP-AES-256-SHA esp-aes-256 esp-sha-hmac crypto map outside_map 10 match address VPN-HM-to-AZ crypto map outside_map 10 set pfs group5 crypto map outside_map 10 set peer azure-public -ip crypto map outside_map 10 set ikev1 transform-set ESP-AES-256-SHA crypto map outside_map interface outside ! tunnel-group azure-public-ip type ipsec-l2l tunnel-group azure-public-ip ipsec-attributes ikev1 pre-shared-key myvpnisnotsecure
4. Azure CSR base and IPsec configuration
The Internet facing interface of the ASR will be in its own VRF so I have isolation between the Internet and my private network. This also allows me to advertise a default route over OSPF making all traffic at the remote site use the GRE tunnel whilst ensuring it doesn’t affect the GRE control traffic. As the loopback is used to source the GRE tunnel that traverses the Internet it will also have to in this VRF.
vrf definition INET rd 1:0 address-family ipv4 ! interface GigabitEthernet1 vrf forwarding INET ip address dhcp <<<< Needs to be DHCP due to Azure ! interface Loopback1 vrf forwarding INET ip address 172.168.201.1 255.255.255.255 ! ip route vrf INET 0.0.0.0 0.0.0.0 172.60.99.1
Interesting traffic:
object-group ip access-list extended VPN-HM-to-AZ permit ip host 172.168.201.1 host 192.168.201.1
Phase1 and Phase2:
crypto isakmp policy 10 encr aes 256 authentication pre-share group 5 ! crypto isakmp keepalive 10 2 crypto ipsec transform-set ESP-AES-256-SHA esp-aes 256 esp-sha-hmac crypto map INET 1 ipsec-isakmp set peer home-public-ip set transform-set ESP-AES-256-SHA set pfs group5 match address VPN-HM-to-AZ ! interface GigabitEthernet1 crypto map INET
As I am using a VRF for the transport I need to use a keyring for the pre-shared key.
crypto keyring FG-to-DC1 vrf INET pre-shared-key address home-public-ip 255.255.255.255 key myvpnisnotsecure
The CSR required the DPD keepalive command where as the ASA had this by default. The least this can be set to is an interval of 10 seconds and retry twice. So in theory if the VPN had issues and was stuck in a state it would take a max of 29 seconds for this to cleared.
Now you should be able to bring the tunnel up from the ASR
AZ-CSR1#ping vrf INET 192.168.201.1 source loop1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.201.1, timeout is 2 seconds:
Packet sent with a source address of 172.168.201.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 17/17/19 ms
You should see the tunnel as up and packets encapsulated and decapsulated.
AZ-CSR1#show crypto session detail
Crypto session current status
Code: C – IKE Configuration mode, D – Dead Peer Detection
K – Keepalives, N – NAT-traversal, T – cTCP encapsulation
X – IKE Extended Authentication, F – IKE Fragmentation
R – IKE Auto Reconnect
Interface: GigabitEthernet1
Uptime: 00:32:07
Session status: UP-ACTIVE
Peer: home-public-ip port 4500 fvrf: INET ivrf: INET
Phase1_id: 172.168.255.1
Desc: (none)
Session ID: 0
IKEv1 SA: local 172.60.99.4/4500 remote home-public-ip/4500 Active
Capabilities:DKN connid:1049 lifetime:23:27:51
IPSEC FLOW: permit ip host 172.168.201.1 host 192.168.201.1
Active SAs: 2, origin: crypto map
Inbound: #pkts dec’ed 573576 drop 0 life (KB/Sec) 4607909/1672
Outbound: #pkts enc’ed 511359 drop 0 life (KB/Sec) 4607946/1672
HM-ASA1# show vpn-sessiondb l2l
Session Type: LAN-to-LAN
Connection : azure-public-ip
Index : 33 IP Addr : azure-public-ip
Protocol : IKEv1 IPsecOverNatT
Encryption : IKEv1: (1)AES256 IPsecOverNatT: (1)AES256
Hashing : IKEv1: (1)SHA1 IPsecOverNatT: (1)SHA1
Bytes Tx : 61476 Bytes Rx : 61600
Login Time : 10:31:53 GMT Sun Nov 26 2017
Duration : 0h:36m:09s
5. GRE Tunnels
Now the VPN tunnel is up just need to build the GRE tunnel on the Home ASR and the Azure ASR using the loopbacks configured earlier and then run OPSF area 10 over this.
To account for the GRE and ESP overhead it is recommended to lower MTU to 1400 bytes and TCP Maximum Segment Size (MSS) to 1360 bytes (IP header 20 Bytes, TCP/UDP header 20 Bytes) for GRE over IPsec. This way it avoids ESP packets being fragmented and they can still be hardware switching. Otherwise ESP packets might need fragmentation and the fragments will need to be software switching reducing the performance and throughput of the tunnel.
As the tunnel is using source and destination that is in a VRF the tunnel interface will need the tunnel vrf command. By default all tunnel interfaces are mode GRE over IP.
GRE keepalives are used to dynamically bring down the tunnel interface if the remote end is down. I set these timers to match the IPsec DPD keepalives as the GRE tunnel relies on the IPsec tunnel as its mode of transport.
The OSPF hello time was shortened to an interval of 5 and dead of 20 so would have a maximum 24 seconds before OSPF neighbour is taken down. I wanted it slightly shorter than the DPD and GRE keepalives, but not too short so that the neighbours are reset every time the VPN flaps.
AZ-CSR1
router ospf 1 router-id 192.168.201.1 passive-interface GigabitEthernet2 ! interface GigabitEthernet2 ip address dhcp <<<< Has to be DHCP as Azure, is 172.60.1.4/24 ip ospf 1 area 10 ! interface Tunnel10 ip address 172.168.220.2 255.255.255.252 ip mtu 1400 ip tcp adjust-mss 1360 ip ospf hello-interval 5 ip ospf 1 area 10 keepalive 10 2 tunnel source Loopback1 tunnel destination 192.168.201.1 tunnel vrf INET
On Home ASR
interface Tunnel10 ip address 172.168.220.1 255.255.255.252 ip mtu 1400 ip tcp adjust-mss 1360 ip ospf hello-interval 5 ip ospf 1 area 10 keepalive 10 2 tunnel source Loopback1 tunnel destination 172.168.201.1
HM-CSR1#show ip ospf neighbor
Neighbor ID Pri State Dead Time Address Interface
172.168.255.1 1 FULL/DR 00:00:39 172.168.255.17 GigabitEthernet1
172.168.201.1 0 FULL/ – 00:00:17 172.168.220.2 Tunnel10
HM-CSR1#show ip ospf interface brief
Interface PID Area IP Address/Mask Cost State Nbrs F/C
Lo1 1 0 192.168.201.1/32 1 LOOP 0/0
Gi1 1 0 172.168.255.18/28 1 BDR 1/1
Tu10 1 10 172.168.220.1/30 1000 P2P 1/1
HM-CSR1#show ip route ospf
Gateway of last resort is 172.168.255.17 to network 0.0.0.0
O*E1 0.0.0.0/0 [110/12] via 172.168.255.17, 04:48:39, GigabitEthernet1
10.0.0.0/24 is subnetted, 1 subnets
O E1 10.10.10.0 [110/12] via 172.168.255.17, 04:48:39, GigabitEthernet1
172.60.0.0/24 is subnetted, 1 subnets
O 172.60.1.0 [110/1001] via 172.168.220.2, 04:08:31, Tunnel10
172.168.0.0/16 is variably subnetted, 5 subnets, 3 masks
O 172.168.255.0/28
[110/11] via 172.168.255.17, 04:48:39, GigabitEthernet1
AZ-CSR1#show ip ospf int brief
Interface PID Area IP Address/Mask Cost State Nbrs F/C
Gi2 1 10 172.60.1.4/24 1 DR 0/0
Tu10 1 10 172.168.220.2/30 1000 P2P 1/1
AZ-CSR1#show ip ospf neighbor
Neighbor ID Pri State Dead Time Address Interface
192.168.201.1 0 FULL/ – 00:00:18 172.168.220.1 Tunnel10
AZ-CSR1#show ip route ospf
Gateway of last resort is 172.168.220.1 to network 0.0.0.0
O*E1 0.0.0.0/0 [110/1012] via 172.168.220.1, 04:11:20, Tunnel10
10.0.0.0/24 is subnetted, 5 subnets
O E1 10.10.10.0 [110/1012] via 172.168.220.1, 04:11:20, Tunnel10
172.168.0.0/16 is variably subnetted, 3 subnets, 3 masks
O IA 172.168.255.16/28 [110/1001] via 172.168.220.1, 04:11:20, Tunnel10
192.168.201.0/32 is subnetted, 1 subnets
O IA 192.168.201.1 [110/1001] via 172.168.220.1, 04:11:20, Tunnel10
QoS can be added to the physical interface and use qos pre-classify under the tunnel interface to make sure the TOS byte is passed onto the ESP packets, however can only uses if IPSEC and GRE tunnels terminate on the same device. An alternative method that also allows you to cause congestion at less that the interfaces physical speed is using child policy like below. This will shape at 100Mbps and then apply what QoS as per the child policy.
policy-map PM-CHILD class CONTROL bandwidth percent 5 class class-default ! policy-map PM-PARENT class class-default shape average percent 100 service-policy PM-CHILD ! int tun 10 bandwidth qos-reference 1000000 service-policy output PM-PARENT
You can also add a zone-based firewall on the ASR just to be certain that the only traffic it will accept is IPSEC and GRE from your peer.
ip access-list extended VRF_OUTSIDE->SELF permit icmp any any echo permit icmp any any echo-reply permit icmp any any port-unreachable permit icmp any any time-exceeded remark IPSEC TRAFFIC permit esp host home-public-ip host azure-public-ip permit udp host home-public-ip host azure-public-ip eq isakmp permit udp host home-public-ip host azure-public-ip eq 4500 remark GRE TRAFFIC permit gre host 192.168.201.1host 172.168.201.1 ! class-map type inspect match-all CM-VRF_OUTSIDE->SELF match access-group name VRF_OUTSIDE->SELF ! policy-map type inspect PM-VRF_OUTSIDE->SELF class type inspect CM-VRF_OUTSIDE->SELF inspect class class-default drop ! zone security VRF_OUTSIDE ! zone-pair security ZP_VRF_OUTSIDE->SELF source VRF_OUTSIDE destination self service-policy type inspect PM-VRF_OUTSIDE->SELF ! ip access-list extended SELF->VRF_OUTSIDE permit icmp any any permit udp any any range 33434 33464 remark IPSEC TRAFFIC permit esp host azure-public-ip host home-public-ip permit udp host azure-public-ip host home-public-ip eq isakmp permit udp host azure-public-ip host home-public-ip eq 4500 remark GRE TRAFFIC permit gre host 172.168.201.1 host 192.168.201.1 ! class-map type inspect match-all CM-SELF->VRF_OUTSIDE match access-group name SELF->VRF_OUTSIDE ! policy-map type inspect PM-SELF->VRF_OUTSIDE class type inspect CM-SELF->VRF_OUTSIDE inspect class class-default drop ! zone-pair security ZP_SELF->VRF_OUTSIDE source self destination VRF_OUTSIDE service-policy type inspect PM-SELF->VRF_OUTSIDE ! int gi 1 zone-member security VRF_OUTSIDE
my internets not working 