Last night I accidentally locked myself out of a production EC2 instance. Arg! Panic!

How I regained access:

1: Take a snapshot of the instance.  (Note: if you require 100% uptime, this is a good time to restore the snapshot to a new instance and switch the Elastic IP to it while you fix the issue. )
2: Launch new Ubuntu recovery instance *in the same AZ* using a key file you have access to.
3: Start and SSH to your new recovery instance.
4: Create a new volume *in the same AZ* from your snapshot
5: Attach the volume you created as device /dev/sdf to your recovery instance. (You need to attach the volume after the instance is running because Linux may boot to the attached volume instead of the boot volume and you’ll still be locked out.)
6: On your new instance, run lsblk. You should see the default 8GB volume and the backed up volume you just attached.  (More @ AWS Support):


ubuntu@ip-172-31-3-214:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 8G 0 disk
└─xvda1 202:1 0 8G 0 part /
xvdf 202:80 0 100G 0 disk
└─xvdf1 202:81 0 100G 0 part

7: Mount the backed up volume using the name from lsblk:


sudo mkdir recovery
sudo mount /dev/xvddf1 /recovery

8: Now you can cd /recovery/home and fix your login issue.
If you lost your access key, edit /recovery/home/ubuntu/.ssh/authorized_keys
You can copy the private key from the new ubuntu instance that you know you have access to.  Worst case, copy the .ssh or the entire /home/ubuntu folder from the new instance to the locked-out volume.

9: Assuming you fixed your permission issue, stop the instance and detach the repaired volume.
10: Detach the old locked-out volume from your original instance and attach the repaired volume under /dev/sda1
11: Start the instance – you should have access now.  Whew.  Next time, take a snapshot before making configuration changes!

Thanks for StackOverflow for help with some tricky bits.