ansible stuffs - how to securely get SSH fingerprints/public keys from newly created AWS instances

A spin off from Ansible SSH bastion host for dynamic infrastructure in AWS, this post documents how to gather EC2 instance SSH public key from the AWS System Log.

background

For a while I was stumped at how to deal with AWS, AMIs and SSH fingerprints/public keys. Initially, it was through pre-baked AMIs. But what if pre-baking AMIs isn’t realistic? Or the AMIs are baked without SSH host keys generated and aren’t known?

This very issue popped up here while working at PokitDok.

So that’s the rub. The SSH public key must be known and verified in order for there to be no risk of being man-in-the-middle’d (at least as much as can be helped). How to do that…

Cue the Amazon feature Get System Log. When an instance boots up, it actually writes the SSH host key to this output. And it’s available programmatically via aws ec2 get-console-output. All that’s needed is some logic to handle the dynamic nature of the AWS infrastructure. By using this trusted SSL interface, the SSH public key can be collected and local SSH configurations can be updated.

 

bill-image-blog-1

Now to get that data.

the localhost.aws_ssh_keys role

The core of this role starts with scrubbing the ~/.ssh/known_hosts file. All tagged entries (using the commented marker_vars) are removed. Once scrubbed, the entries are re-added. Each playbook. This is done to ensure that OLD entries/instances that are no longer valid get removed. Once removed, they are added using the AWS CLI tool, pulling the Get System Log programatically:

 

 

 

This handy task repeatedly attempts to get the line from the AWS system log that contains ecdsa-sha2-nistp256, which has the public key data. sed carves up the output and grabs only that public key. It does this for ever host retreived from ec2_facts.instances and registers the output to host_key_results.ec2_facts.instances are the results of the aws.ec2_facts role, a role which collects EC2 facts using a specific filter. It has a slew of data relating to all instances within our VPC. This task uses that result as the with_items list, so the task retreives the System Log for every instance found in that fact. It does this, waiting for instances that haven’t started up yet (like newly created ones), using the handy new until feature in Ansible. 75 tries is a bit excessive, but there is a lag between the instance starting up and the AWS system log producing the results. Quite a handy trick. The AWS access and secret keys are passed in as environment variables. Through this task, the SSH public keys have been added, through a safe and secure channel (HTTPS via AWS CLI). These entries are registered in the variablehost_key_results.

Another minor issue… it’s only as reliable as the AWS Get System Log. Sometimes, this data never appears. I’ve seen this once or twice, usually when deploying a lot of instances at once… The way forward was to terminate the instance and redeploy (it was the initial boot, so still an unconfigured instance). :sadpanda:

Charging forward…. The playbook takes the host_key_results public key gathering and add BOTH the private DNS name and public DNS name, if it has one. Both entries are added to ~/.ssh/known_hosts.

 


with_together is very handy for parallel lists. Since the host_key_results output is derived from the list output of ec2_facts.instances, they are 1:1 lists. The host keys are mapped to the instance private and public DNS names. All SSH public keys are accounted for.

Once this completes, the ~/.ssh/known_hosts will have lines similar to this:

The markers provide the way of modifying ONLY the entries that are relevant and avoid stomping on other entries. It also allows cleaning of the file as in a dynamic inventory, some instances/IP addresses may no longer exist.

instance.ssh_aws_public_key

There is a minor catch to grabbing the AWS System Log; this data is not available in the AWS System Log on reboot or after a shutdown. It is only printed once, at the initial boot. That’s not good; if a reboot occurs, if a shutdown and startup occurs, if an AWS infrastructure occurs, the playbook won’t work. How should this be solved?

The AWS System Log prints everything during the initial boot sequence. So the trick is, get the SSH public key to always print during the initial boot. A very quick and seemingly durable way to do this is is to add a command to the script /etc/rc.local:

Another way would be to just cat /etc/ssh/ssh_host_ecdsa_key.pub. This step ensures that each reboot, the key is available and captured by the playbook.

summary

Hopefully others find this little trick useful. It really should be it’s own module, I find it that handy. There’s a level of infosec comfort in knowing that the SSH keys have been verified and are trustworthy. It also feeds into the dynamic nature of AWS and the instances; the SSH keys don’t need to be known, they don’t need to be added ahead of time or blindly trusted on initial connection.

-b

links

 

About Bill Cawthra

Bill Cawthra, a senior DevOps engineer at PokitDok, is working to change the way healthcare systems and applications communicate.  Through automation and infrastructure-as-a-service, he and the DevOps team deploy infrastructure using tools and technologies such as Ansible, Terraform, Docker, AWS, and Azure.
Bill graduated from the Pennsylvania State University electrical engineering program in 2001 and holds CISSP, RHCE, and multiple GIAC security certifications. With a background in system administration, system engineering, security engineering, and DevOps, working in both the U.S. Department of Defense and private sector, he is focused on finding innovative technical solutions to tricky business problems.

View All Posts

1 comment

  1. Reply

    I was surfing the web for AWS and I saw your Blog. I read a few of your posts and think they were awesome. Thank you.

Leave a Reply

Your email address will not be published.