Your company has an IPsec tunnel with another company for achieving network connectivity between servers in 10.10.10.0/24 on your side to 10.20.20.0/24 on theirs.
Lately they complained that their equipment has problems dealing with ESP and requested to migrate this existing IPsec tunnel from Encapsulating Security Payloads (ESP) to Authentication Headers (AH), since encryption/confidentiality was never a requirement for this tunnel. Please note that network connectivity is ok in the initial state, using ESP.


Quiz 25 - Troubleshooting IPsec AH


The only change that you did was adding a new transform set T_SET_AH to use ah-md5-hmac and updating the crypto map to use it.
Together with the partner company you establish a maintenance window and perform the migration from ESP to AH. The new tunnel using AH comes up, but unfortunately you do not have network connectivity between the internal networks.

You start troubleshooting and conclude the following:

  • ISAKMP phase gets established
  • IPSEC phase gets established
  • the new parameters (AH & MD5) are agreed correctly
  • the protected internal subnets 10.10.10.0/24 and 10.20.20.0/24 are agreed correctly
  • but there is no network connectivity between them
  • you see packets being sent but nothing received back
! No connectivity
R1#ping 10.20.20.20 source 10.10.10.10

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.20.20.20, timeout is 2 seconds:
Packet sent with a source address of 10.10.10.10
.....
Success rate is 0 percent (0/5)
R1#

! Phase 1 OK
R1#sh crypto isakmp sa
IPv4 Crypto ISAKMP SA
dst             src             state          conn-id slot status
44.44.44.44     12.12.12.1      QM_IDLE           1001    0 ACTIVE

IPv6 Crypto ISAKMP SA

! Phase 2 OK
R1#sh crypto ipsec sa detail

interface: FastEthernet0/0
    Crypto map tag: IPSEC_VPN, local addr 12.12.12.1

   protected vrf: (none)
   local  ident (addr/mask/prot/port): (10.10.10.0/255.255.255.0/0/0)
   remote ident (addr/mask/prot/port): (10.20.20.0/255.255.255.0/0/0)
   current_peer 44.44.44.44 port 4500
     PERMIT, flags={origin_is_acl,}
    #pkts encaps: 5, #pkts encrypt: 5, #pkts digest: 5
    #pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
    #pkts compressed: 0, #pkts decompressed: 0
    #pkts not compressed: 0, #pkts compr. failed: 0
    #pkts not decompressed: 0, #pkts decompress failed: 0
    #pkts no sa (send) 5, #pkts invalid sa (rcv) 0
    #pkts encaps failed (send) 0, #pkts decaps failed (rcv) 0
    #pkts invalid prot (recv) 0, #pkts verify failed: 0
    #pkts invalid identity (recv) 0, #pkts invalid len (rcv) 0
    #pkts replay rollover (send): 0, #pkts replay rollover (rcv) 0
    ##pkts replay failed (rcv): 0
    #pkts internal err (send): 0, #pkts internal err (recv) 0

     local crypto endpt.: 12.12.12.1, remote crypto endpt.: 44.44.44.44
     path mtu 1500, ip mtu 1500, ip mtu idb FastEthernet0/0
     current outbound spi: 0xDEE2A85C(3739396188)

     inbound esp sas:

     inbound ah sas:
      spi: 0x7612ECC1(1980951745)
        transform: ah-md5-hmac ,
        in use settings ={Tunnel UDP-Encaps, }
        conn id: 1, flow_id: SW:1, crypto map: IPSEC_VPN
        sa timing: remaining key lifetime (k/sec): (4530158/3488)
        replay detection support: Y
        Status: ACTIVE

     inbound pcp sas:

     outbound esp sas:

     outbound ah sas:
      spi: 0xDEE2A85C(3739396188)
        transform: ah-md5-hmac ,
        in use settings ={Tunnel UDP-Encaps, }
        conn id: 2, flow_id: SW:2, crypto map: IPSEC_VPN

What is wrong ? Why everything works if the IPsec tunnel uses ESP but it is not working with AH?

Even though you requested your peer network engineer working for the Partner Company to give you the config from his router (R3), due to internal policies, he rejected that !... but he confirms your above conclusion on his side: tunnel goes up but no connectivity !

Eventually, you enable crypto debugging for both phases. You also bring in a senior network consultant. He looks at the show crypto outputs and the debug crypto below and concludes that you cannot use AH between the two companies, so you'll have to keep ESP !

Here is the debug output:

R1#deb crypto isakmp
Crypto ISAKMP debugging is on
R1#
R1#deb crypto ipsec
Crypto IPSEC debugging is on
R1#
R1#sh crypto isakmp sa
IPv4 Crypto ISAKMP SA
dst             src             state          conn-id slot status

IPv6 Crypto ISAKMP SA

R1#
R1#
R1#
R1#ping 10.20.20.20 sou
R1#ping 10.20.20.20 source 10.10.10.10

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.20.20.20, timeout is 2 seconds:
Packet sent with a source address of 10.10.10.10

*Mar  1 00:02:06.175: IPSEC(sa_request): ,
  (key eng. msg.) OUTBOUND local= 12.12.12.1, remote= 44.44.44.44,
    local_proxy= 10.10.10.0/255.255.255.0/0/0 (type=4),
    remote_proxy= 10.20.20.0/255.255.255.0/0/0 (type=4),
    protocol= AH, transform= ah-md5-hmac  (Tunnel),
    lifedur= 3600s and 4608000kb,
    spi= 0x0(0), conn_id= 0, keysize= 0, flags= 0x0
*Mar  1 00:02:06.179: ISAKMP:(0): SA request profile is (NULL)
*Mar  1 00:02:06.183: ISAKMP: Created a peer struct for 44.44.44.44, peer port 500
*Mar  1 00:02:06.183: ISAKMP: New peer created peer = 0x663A6274 peer_handle = 0x80000002
*Mar  1 00:02:06.187: ISAKMP: Locking peer struct 0x663A6274, refcount 1 for isakmp_initiator
*Mar  1 00:02:06.191: ISAKMP: local port 500, remote port 500
*Mar  1 00:02:06.191: ISAKMP: set new node 0 to QM_IDLE
*Mar  1 00:02:06.191: insert sa successfully sa = 66981718
*Mar  1 00:02:06.191: ISAKMP:(0):Can not start Aggressive mode, trying Main mode.
*Mar  1 00:02:06.195: ISAKMP:(0):found peer pre-shared key matching 44.44.44.44
*Mar  1 00:02:06.203: ISAKMP:(0): constructed NAT-T vendor-rfc3947 ID
*Mar  1 00:02:06.203: ISAKMP:(0): constructed NAT-T vendor-07 ID
*Mar  1 00:02:06.207: ISAKMP:(0): constructed NAT-T vendor-03 ID
*Mar  1 00:02:06.207: ISAKMP:(0): constructed NAT-T vendor-02 ID
*Mar  1 00:02:06.207: ISAKMP:(0):Input = IKE_MESG_FROM_IPSEC, IKE_SA_REQ_MM
*Mar  1 00:02:06.207: ISAKMP:(0):Old State = IKE_READY  New State = IKE_I_M.M1

*Mar  1 00:02:06.207: ISAKMP:(0): beginning Main Mode exchange
*Mar  1 00:02:06.207: ISAKMP:(0): sending packet to 44.44.44.44 my_port 500 peer_port 500 (I) MM_NO_STATE
*Mar  1 00:02:06.207: ISAKMP:(0):Sending an IKE IPv4 Packet.....
Success rate is 0 percent (0/5)
R1#
*Mar  1 00:02:16.207: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE...
*Mar  1 00:02:16.207: ISAKMP (0:0): incrementing error counter on sa, attempt 1 of 5: retransmit phase 1
*Mar  1 00:02:16.207: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE
*Mar  1 00:02:16.207: ISAKMP:(0): sending packet to 44.44.44.44 my_port 500 peer_port 500 (I) MM_NO_STATE
*Mar  1 00:02:16.207: ISAKMP:(0):Sending an IKE IPv4 Packet.
R1#
*Mar  1 00:02:26.207: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE...
*Mar  1 00:02:26.211: ISAKMP (0:0): incrementing error counter on sa, attempt 2 of 5: retransmit phase 1
*Mar  1 00:02:26.215: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE
*Mar  1 00:02:26.215: ISAKMP:(0): sending packet to 44.44.44.44 my_port 500 peer_port 500 (I) MM_NO_STATE
*Mar  1 00:02:26.219: ISAKMP:(0):Sending an IKE IPv4 Packet.
*Mar  1 00:02:26.331: ISAKMP (0:0): received packet from 44.44.44.44 dport 500 sport 500 Global (I) MM_NO_STATE
*Mar  1 00:02:26.347: ISAKMP:(0):Input = IKE_MESG_FROM_PEER, IKE_MM_EXCH
*Mar  1 00:02:26.351: ISAKMP:(0):Old State = IKE_I_MM1  New State = IKE_I_MM2

*Mar  1 00:02:26.363: ISAKMP:(0): processing SA payload. message ID = 0
*Mar  1 00:02:26.363: ISAKMP:(0): processing vendor id payload
*Mar  1 00:02:26.363: ISAKMP:(0): vendor ID seems Unity/DPD but major 245 mismatch
*Mar  1 00:02:26.363: ISAKMP (0:0): vendor ID is NAT-T v7
*Mar  1 00:02:26.363: ISAKMP:(0):found peer pre-shared key matching 44.44.44.44
*Mar  1 00:02:26.363: ISAKMP:(0): local preshared key found
*Mar  1 00:02:26.363: ISAKMP : Scanning profiles for xauth ...
*Mar  1 00:02:26.363: ISAKMP:(0):Checking ISAKMP transform 1 against priority 10 policy
*Mar  1 00:02:26.363: ISAKMP:      encryption DES-CBC
*Mar  1 00:02:26.363: ISAKMP:      hash SHA
*Mar  1 00:02:26.363: ISAKMP:      default group 1
*Mar  1 00:02:26.363: ISAKMP:      auth pre-share
*Mar  1 00:02:26.363: ISAKMP:      life type in seconds
*Mar  1 00:02:26.363: ISAKMP:      life duration (VPI) of  0x0 0x1 0x51 0x80
*Mar  1 00:02:26.367: ISAKMP:(0):atts are acceptable. Next payload is 0
*Mar  1 00:02:26.367: ISAKMP:(0):Acceptable atts:actual life: 0
*Mar  1 00:02:26.371: ISAKMP:(0):Acceptable atts:life: 0
*Mar  1 00:02:26.371: ISAKMP:(0):Fill atts in sa vpi_length:4
*Mar  1 00:02:26.375: ISAKMP:(0):Fill atts in sa life_in_seconds:86400
*Mar  1 00:02:26.379: ISAKMP:(0):Returning Actual lifetime: 86400
*Mar  1 00:02:26.379: ISAKMP:(0)::Started lifetime timer: 86400.

*Mar  1 00:02:26.379: ISAKMP:(0): processing vendor id payload
*Mar  1 00:02:26.379: ISAKMP:(0): vendor ID seems Unity/DPD but major 245 mismatch
*Mar  1 00:02:26.379: ISAKMP (0:0): vendor ID is NAT-T v7
*Mar  1 00:02:26.379: ISAKMP:(0):Input = IKE_MESG_INTERNAL, IKE_PROCESS_MAIN_MODE
*Mar  1 00:02:26.379: ISAKMP:(0):Old State = IKE_I_MM2  New State = IKE_I_MM2

*Mar  1 00:02:26.383: ISAKMP:(0): sending packet to 44.44.44.44 my_port 500 peer_port 500 (I) MM_SA_SETUP
*Mar  1 00:02:26.387: ISAKMP:(0):Sending an IKE IPv4 Packet.
*Mar  1 00:02:26.391: ISAKMP:(0):Input = IKE_MESG_INTERNAL, IKE_PROCESS_COMPLETE
*Mar  1 00:02:26.391: ISAKMP:(0):Old State = IKE_I_MM2  New State = IKE_I_MM3

*Mar  1 00:02:26.491: ISAKMP (0:0): received packet from 44.44.44.44 dport 500 sport 500 Global (I) MM_SA_SETUP
*Mar  1 00:02:26.495: ISAKMP:(0):Input = IKE_MESG_FROM_PEER, IKE_MM_EXCH
*Mar  1 00:02:26.499: ISAKMP:(0):Old State = IKE_I_MM3  New State = IKE_I_MM4

*Mar  1 00:02:26.511: ISAKMP:(0): processing KE payload. message ID = 0
*Mar  1 00:02:26.531: ISAKMP:(0): processing NONCE payload. message ID = 0
*Mar  1 00:02:26.531: ISAKMP:(0):found peer pre-shared key matching 44.44.44.44
*Mar  1 00:02:26.531: ISAKMP:(1001): processing vendor id payload
*Mar  1 00:02:26.531: ISAKMP:(1001): vendor ID is Unity
*Mar  1 00:02:26.531: ISAKMP:(1001): processing vendor id payload
*Mar  1 00:02:26.531: ISAKMP:(1001): vendor ID is DPD
*Mar  1 00:02:26.531: ISAKMP:(1001): processing vendor id payload
*Mar  1 00:02:26.531: ISAKMP:(1001): speaking to another IOS box!
*Mar  1 00:02:26.531: ISAKMP (0:1001): NAT found, the node outside NAT
*Mar  1 00:02:26.531: ISAKMP:(1001):Input = IKE_MESG_INTERNAL, IKE_PROCESS_MAIN_MODE
*Mar  1 00:02:26.531: ISAKMP:(1001):Old State = IKE_I_MM4  New State = IKE_I_MM4

*Mar  1 00:02:26.531: ISAKMP:(1001):Send initial contact
*Mar  1 00:02:26.531: ISAKMP:(1001):SA is doing pre-shared key authentication using id type ID_IPV4_ADDR
*Mar  1 00:02:26.531: ISAKMP (0:1001): ID payload
        next-payload : 8
        type         : 1
        address      : 12.12.12.1
        protocol     : 17
        port         : 0
        length       : 12
*Mar  1 00:02:26.531: ISAKMP:(1001):Total payload length: 12
*Mar  1 00:02:26.531: ISAKMP:(1001): sending packet to 44.44.44.44 my_port 4500 peer_port 4500 (I) MM_KEY_EXCH
*Mar  1 00:02:26.531: ISAKMP:(1001):Sending an IKE IPv4 Packet.
*Mar  1 00:02:26.531: ISAKMP:(1001):Input = IKE_MESG_INTERNAL, IKE_PROCESS_COMPLETE
*Mar  1 00:02:26.535: ISAKMP:(1001):Old State = IKE_I_MM4  New State = IKE_I_MM5

*Mar  1 00:02:26.639: ISAKMP (0:1001): received packet from 44.44.44.44 dport 4500 sport 4500 Global (I) MM_KEY_EXCH
*Mar  1 00:02:26.647: ISAKMP:(1001): processing ID payload. message ID = 0
*Mar  1 00:02:26.647: ISAKMP (0:1001): ID payload
        next-payload : 8
        type         : 1
        address      : 192.168.1.2
        protocol     : 17
        port         : 0
        length       : 12
*Mar  1 00:02:26.655: ISAKMP:(0):: peer matches *none* of the profiles
*Mar  1 00:02:26.655: ISAKMP:(1001): processing HASH payload. message ID = 0
*Mar  1 00:02:26.663: ISAKMP:(1001):SA authentication status:
        authenticated
*Mar  1 00:02:26.663: ISAKMP:(1001):SA has been authenticated with 44.44.44.44
*Mar  1 00:02:26.663: ISAKMP: Trying to insert a peer 12.12.12.1/44.44.44.44/4500/,  and inserted successfully 663A6274.
*Mar  1 00:02:26.663: ISAKMP:(1001):Input = IKE_MESG_FROM_PEER, IKE_MM_EXCH
*Mar  1 00:02:26.663: ISAKMP:(1001):Old State = IKE_I_MM5  New State = IKE_I_MM6

*Mar  1 00:02:26.663: ISAKMP:(1001):Input = IKE_MESG_INTERNAL, IKE_PROCESS_MAIN_MODE
*Mar  1 00:02:26.663: ISAKMP:(1001):Old State = IKE_I_MM6  New State = IKE_I_MM6

*Mar  1 00:02:26.663: ISAKMP:(1001):Input = IKE_MESG_INTERNAL, IKE_PROCESS_COMPLETE
*Mar  1 00:02:26.663: ISAKMP:(1001):Old State = IKE_I_MM6  New State = IKE_P1_COMPLETE

*Mar  1 00:02:26.663: ISAKMP:(1001):beginning Quick Mode exchange, M-ID of 2112503013
*Mar  1 00:02:26.663: ISAKMP:(1001):QM Initiator gets spi
*Mar  1 00:02:26.671: ISAKMP:(1001): sending packet to 44.44.44.44 my_port 4500 peer_port 4500 (I) QM_IDLE
*Mar  1 00:02:26.675: ISAKMP:(1001):Sending an IKE IPv4 Packet.
*Mar  1 00:02:26.679: ISAKMP:(1001):Node 2112503013, Input = IKE_MESG_INTERNAL, IKE_INIT_QM
*Mar  1 00:02:26.679: ISAKMP:(1001):Old State = IKE_QM_READY  New State = IKE_QM_I_QM1
*Mar  1 00:02:26.679: ISAKMP:(1001):Input = IKE_MESG_INTERNAL, IKE_PHASE1_COMPLETE
*Mar  1 00:02:26.679: ISAKMP:(1001):Old State = IKE_P1_COMPLETE  New State = IKE_P1_COMPLETE

*Mar  1 00:02:26.775: ISAKMP (0:1001): received packet from 44.44.44.44 dport 4500 sport 4500 Global (I) QM_IDLE
*Mar  1 00:02:26.783: ISAKMP:(1001): processing HASH payload. message ID = 2112503013
*Mar  1 00:02:26.787: ISAKMP:(1001): processing SA payload. message ID = 2112503013
*Mar  1 00:02:26.787: ISAKMP:(1001):Checking IPSec proposal 1
*Mar  1 00:02:26.791: ISAKMP: transform 1, AH_MD5
*Mar  1 00:02:26.791: ISAKMP:   attributes in transform:
*Mar  1 00:02:26.791: ISAKMP:      encaps is 3 (Tunnel-UDP)
*Mar  1 00:02:26.795: ISAKMP:      SA life type in seconds
*Mar  1 00:02:26.795: ISAKMP:      SA life duration (basic) of 3600
*Mar  1 00:02:26.795: ISAKMP:      SA life type in kilobytes
*Mar  1 00:02:26.795: ISAKMP:      SA life duration (VPI) of  0x0 0x46 0x50 0x0
*Mar  1 00:02:26.795: ISAKMP:      authenticator is HMAC-MD5
*Mar  1 00:02:26.795: ISAKMP:(1001):atts are acceptable.
*Mar  1 00:02:26.795: IPSEC(validate_proposal_request): proposal part #1
*Mar  1 00:02:26.795: IPSEC(validate_proposal_request): proposal part #1,
  (key eng. msg.) INBOUND local= 12.12.12.1, remote= 44.44.44.44,
    local_proxy= 10.10.10.0/255.255.255.0/0/0 (type=4),
    remote_proxy= 10.20.20.0/255.255.255.0/0/0 (type=4),
    protocol= AH, transform= NONE  (Tunnel-UDP),
    lifedur= 0s and 0kb,
    spi= 0x0(0), conn_id= 0, keysize= 0, flags= 0x0
*Mar  1 00:02:26.795: Crypto mapdb : proxy_match
        src addr     : 10.10.10.0
        dst addr     : 10.20.20.0
        protocol     : 0
        src port     : 0
        dst port     : 0
*Mar  1 00:02:26.795: ISAKMP:(1001): processing NONCE payload. message ID = 2112503013
*Mar  1 00:02:26.799: ISAKMP:(1001): processing ID payload. message ID = 2112503013
*Mar  1 00:02:26.803: ISAKMP:(1001): processing ID payload. message ID = 2112503013
*Mar  1 00:02:26.811: ISAKMP:(1001): Creating IPSec SAs
*Mar  1 00:02:26.811:         inbound SA from 44.44.44.44 to 12.12.12.1 (f/i)  0/ 0
        (proxy 10.20.20.0 to 10.10.10.0)
*Mar  1 00:02:26.811:         has spi 0x7612ECC1 and conn_id 0
*Mar  1 00:02:26.811:         lifetime of 3600 seconds
*Mar  1 00:02:26.811:         lifetime of 4608000 kilobytes
*Mar  1 00:02:26.811:         outbound SA from 12.12.12.1 to 44.44.44.44 (f/i) 0/0
        (proxy 10.10.10.0 to 10.20.20.0)
*Mar  1 00:02:26.811:         has spi  0xDEE2A85C and conn_id 0
*Mar  1 00:02:26.811:         lifetime of 3600 seconds
*Mar  1 00:02:26.811:         lifetime of 4608000 kilobytes
*Mar  1 00:02:26.811: ISAKMP:(1001): sending packet to 44.44.44.44 my_port 4500 peer_port 4500 (I) QM_IDLE
*Mar  1 00:02:26.811: ISAKMP:(1001):Sending an IKE IPv4 Packet.
*Mar  1 00:02:26.811: ISAKMP:(1001):deleting node 2112503013 error FALSE reason "No Error"
*Mar  1 00:02:26.815: ISAKMP:(1001):Node 2112503013, Input = IKE_MESG_FROM_PEER, IKE_QM_EXCH
*Mar  1 00:02:26.819: ISAKMP:(1001):Old State = IKE_QM_I_QM1  New State = IKE_QM_PHASE2_COMPLETE
*Mar  1 00:02:26.827: IPSEC(key_engine): got a queue event with 1 KMI message(s)
*Mar  1 00:02:26.827: Crypto mapdb : proxy_match
        src addr     : 10.10.10.0
        dst addr     : 10.20.20.0
        protocol     : 0
        src port     : 0
        dst port     : 0
*Mar  1 00:02:26.827: IPSEC(crypto_ipsec_sa_find_ident_head): reconnecting with the same proxies and peer 44.44.44.44
*Mar  1 00:02:26.831: IPSEC(policy_db_add_ident): src 10.10.10.0, dest 10.20.20.0, dest_port 0

*Mar  1 00:02:26.831: IPSEC(create_sa): sa created,
  (sa) sa_dest= 12.12.12.1, sa_proto= 51,
    sa_spi= 0x7612ECC1(1980951745),
    sa_trans= ah-md5-hmac , sa_conn_id= 1
*Mar  1 00:02:26.831: IPSEC(create_sa): sa created,
  (sa) sa_dest= 44.44.44.44, sa_proto= 51,
    sa_spi= 0xDEE2A85C(3739396188),
    sa_trans= ah-md5-hmac , sa_conn_id= 2
*Mar  1 00:02:26.831: IPSEC(update_current_outbound_sa): updated peer 44.44.44.44 current outbound sa to SPI DEE2A85C
R1#
R1#

What did the senior engineer found ? What is the problem?


Post your answer in the 'Comments' section below and subscribe to this blog to get the detailed solution and more interesting quizzes.