High Availability NAT for AWS VPC with Multiple Private Subnets.

VPCs have become the de-facto architectural choice when deploying enterprise applications. They are highly secure, keep your infrastructure logically separated which when combined with the power that AWS has to offer become highly available and robust.
The one thing, however, that still feels like a point of failure are the NAT instances that need to be created for all private subnet outbound connectivity to the internet or any service that is external to the VPC. This brings us to create a sort of NAT failover mechanism. High Availability for Amazon VPC NAT Instances by Jinesh Varia offers a solution for a simple VPC that has active NATs per private subnet and each keeps pinging the other till one of them loses connectivity. The active NAT, at that moment, "takes over" the route for the failed NAT. An edge case to this solution is when you have private subnets per AZ and there is a loss of connectivity between AZs.

We were faced with the same issue, which brought the two NATs into a deadlock. Since each NAT detected the other to be down due to loss of connectivity, it took over the route of the other and issued an API signal to stop the other NAT. Which meant each stopped the other NAT, rendering the production application unable to make server side calls since both NATs were unable to start the other NAT. Also we could not use cloudformation to set up the above NAT dependencies since each required the other NAT's instance id and route table id and entered into a circular dependency. We tried to solve the above issue with a slight change in the cloudformation and architecture of the HA NATs.

This approach requires setting up two NATs per AZ. A primary NAT and a failover NAT. The privste subnets would be served by the promary NAT as default. All HA logic goes into the Failover NAT. The failover NATs would run a script at startup that would poll for the status of the primary NATs in their respective AZs.

In case a NAT fails, in this case NAT Device 1, the Failover NAT would take over the route and continue serving the instances in the private subnet within the same AZ. The infrastructure in the other AZ remains unaffected. 

In case the connectivity between the two AZs fails, each AZ independently can make server side calls. Also in case both primary NATs fails, the respective Failover NATs would take over the route seamlessly. 

Once the primary NAT springs back in action, the Failover NAT would surrender the route and continue checking status of the primary NAT.

The above architecture reduces coupling of NATs across AZs and makes each AZ independent in terms of operations.

Below is the script that needs to go on the Failover NATs.

#!/bin/sh
. /etc/profile.d/aws-apitools-common.sh

#Set Global Variables
Num_Pings=10
Ping_Timeout=1
Wait_Between_Pings=5
EC2_URL=https://ec2.ap-southeast-1.amazonaws.com
NAT_ID=""
ROUTE_CHANGED=0
NAT_RT_ID=""
My_ID=`/usr/bin/curl --silent http://169.254.169.254/latest/meta-data/instance-id`
NAT_IP=`/opt/aws/bin/ec2-describe-instances $NAT_ID -U $EC2_URL | grep PRIVATEIPADDRESS -m 1 | awk '{print $2;}\'`

#Starting service
echo `date` "-- Starting NAT monitor"
while [ . ]; do
  pingresult=`ping -c $Num_Pings -W $Ping_Timeout $NAT_IP | grep time= | wc -l`
  while [ "$pingresult" == "0" ]; do
    if [ "$ROUTE_CHANGED" == "0" ]; then
      echo `date` "-- Primary NAT heartbeat failed, taking over route table $NAT_RT_ID"
      /opt/aws/bin/ec2-replace-route $NAT_RT_ID -r 0.0.0.0/0 -i $My_ID -U $EC2_URL
      #Update variable when route has been taken over.
      ROUTE_CHANGED=1
    fi
    sleep $Wait_Between_Pings
    pingresult=`ping -c $Num_Pings -W $Ping_Timeout $NAT_IP | grep time= | wc -l`
  done
   if [ "$ROUTE_CHANGED" == "1" ]; then
     echo `date` "-- Primary NAT heartbeat passed, surrendering route table $NAT_RT_ID"
     /opt/aws/bin/ec2-replace-route $NAT_RT_ID -r 0.0.0.0/0 -i $NAT_ID -U $EC2_URL
      #Update variable when route has been surrendered.
     ROUTE_CHANGED=0
   fi
  sleep $Wait_Between_Pings
done

When used within UserData for the Failover NAT in AWS Cloudformation, the above script would change to

#!/bin/sh
. /etc/profile.d/aws-apitools-common.sh
Num_Pings=10
Ping_Timeout=1
Wait_Between_Pings=5
EC2_URL=https://ec2.",{ "Ref": "AWS::Region" },".amazonaws.com
NAT_ID=", { "Ref": "NATDevice1" }, "
ROUTE_CHANGED=0
NAT_RT_ID=`/opt/aws/bin/ec2-describe-route-tables -U $EC2_URL --filter "tag:Name=Private Subnet - 1 Route Table" | grep ROUTETABLE -m 1 | awk '{print $2;}'`
My_ID=`/usr/bin/curl --silent http://169.254.169.254/latest/meta-data/instance-id`
NAT_IP=`/opt/aws/bin/ec2-describe-instances $NAT_ID -U $EC2_URL | grep PRIVATEIPADDRESS -m 1 | awk '{print $2;}'`
echo `date` "-- Starting NAT monitor"
while [ . ]; do
  pingresult=`ping -c $Num_Pings -W $Ping_Timeout $NAT_IP | grep time= | wc -l`
  while [ "$pingresult" == "0" ]; do
    if [ "$ROUTE_CHANGED" == "0" ]; then
      echo `date` "-- Primary NAT heartbeat failed, taking over route table $NAT_RT_ID"
      /opt/aws/bin/ec2-replace-route $NAT_RT_ID -r 0.0.0.0/0 -i $My_ID -U $EC2_URL
      ROUTE_CHANGED=1
    fi
    sleep $Wait_Between_Pings
    pingresult=`ping -c $Num_Pings -W $Ping_Timeout $NAT_IP | grep time= | wc -l`
  done
   if [ "$ROUTE_CHANGED" == "1" ]; then
     echo `date` "-- Primary NAT heartbeat passed, surrendering route table $NAT_RT_ID"
     /opt/aws/bin/ec2-replace-route $NAT_RT_ID -r 0.0.0.0/0 -i $NAT_ID -U $EC2_URL
     ROUTE_CHANGED=0
   fi
  sleep $Wait_Between_Pings
done