BLESSing away SSH worries | Ultimate Practical Guide to setup Netflix Bless in under 60 minutes from scratch with the Go Client

This article is a concise, hands-on guide to setup Netflix Bless for your infrastructure in less than an hour using the Go Client for Bless.

What is Netflix Bless?

Directly from Netflix Bless readme.

BLESS is an SSH Certificate Authority that runs as an AWS Lambda function and is used to sign SSH public keys.

SSH Certificates are an excellent way to authorize users to access a particular SSH host, as they can be restricted for a single-use case, and can be short-lived. Instead of managing the authorized_keys of a host, or controlling who has access to SSH Private Keys, hosts need to be configured to trust an SSH Certificate Authority.

Essentially, A certificate is a data structure that includes a public key, name, and ancillary data like an expiration date and permissions. The data structure is signed by a certificate authority (CA) and helps us authenticate for the SSH handshake.

Advantages

Easy SSH Public Key Management for granting access to our infrastructure. No need to manage public keys on every host in your infrastructure.

The task of rotating/archiving/deleting keys becomes easy and we can use multi-factor authentication or 2- step authentication for enabling access.

What are we going to build?

Architecture

1. We will create three bastion hosts in our VPC.
2. Each Bastion host will communicate with the Bless CA hosted inside a Lambda Function. The Lambda Function will have the Bless Private Key and will return signed certificates to the bastion host.
3. The user on the Bastion host will use these certificates to SSH into the remote hosts. The remote hosts will be configured to trust the Certificate Authority using the public part of the Bless Key Pair, and hence, the entire authentication process would occur.
4. The users are created on the Bastion user using IAM Sync Service(more ahead). We need to upload the IAM user’s public key in the AWS IAM console. Our IAMSyncService will also add the users’ public keys to the authorized_users file in the user’s home directory.

Source Code

The source code is separated between two repositories:

https://github.com/kshitijcode/platform

• Terraform: It helps us create a new platform tooling VPC with three bastion hosts(clients) in three private subnets.
• Ansible: This helps us set up the bastion hosts, and our servers with the necessary configurations for using the bless certificate authority.

https://github.com/kshitijcode/blessMe

• terraform: Sets up the Netflix Bless Certificate authority as an AWS Lambda function creating the required AWS resources. ( IAM Policies, Roles, KMS, Lambda )

Getting Hands Dirty

Well, just before we get started, kindly ensure you have the repositories cloned, terraform, ansible, and docker installed.

Setting up the AWS Infrastructure

We will be deploying our infrastructure in the AWS ‘ap-south-1’ region. Navigate to environments > prod under the platform repository.

~/me/platform/environments/prod > terraform init
~/me/platform/environments/prod > terraform plan
~/me/platform/environments/prod > terraform apply

This will create the following resources in the ap-south-1 region.

• VPC with 10.90.0.0/18 with three private and public subnets
• Three Bastions Hosts(acting as clients for Bless) with IAM instance profile
• Lambda Function with the appropriate IAM Roles and Policies
• KMS Key: A KMS key(bless-decrypt-ap-south-1) will be created in AWS to encrypt/decrypt the passphrase of the SSH Key Pair ( to be generated in the next steps ). We need to create KMS keys for each AWS region we wish to deploy our Bless Service.
• A remote backend is configured in the main.yml in the environments directory. Kindly change it to set for your backend.

Build the Lambda function dependencies

We need to build the dependencies( read build additional libraries for running the code which we will deploy to our Lambda function ) using docker.

After cloning the Netflix bless repository, we will execute the following command to build the dependencies :

~/me/bless > make lambda-dep

In the Makefile, the ‘lambda-dep’ trigger consists of all the steps to create the Lambda dependencies. This builds all the artifacts in the aws_lambda_libs directory under the bless folder.

Generate an SSH Key Pair for Certificate Authority

Let’s generate an SSH Keypair for our Bless Service. The key pair will be used to sign temporary certificates and validate the authenticity of the SSH connection.

~/me/bless > mkdir lambda_configs
~/me/bless > cd lambda_configs
~/me/bless/lambda_configs > ssh-keygen -t rsa -b 4096 -f bless-ca-

Enter passphrase for the SSH key and keep it saved.

Encrypt the Passphrase using KMS Key (bless-decrypt-ap-south-1)

Copy the key-encrypter.py to the lambda_configs directory under bless and execute the following :

~/me/bless/lambda_configs > python3 key-encrypter.py --region ap-south-1 --keyID  bless-decrypt-ap-south-1 --passphrase yourPassphraseFromPreviousStep

We will have an encrypted passphrase string. Save this text somewhere.

Expected Output: Save the string between the single quotes. Ignore the b.’

Creating the Bless Deploy Configuration file

The bless_deploy.cfg consists of ALL the configuration for our Bless Service(Certificate Authority) to be hosted on AWS Lambda.

Let’s begin by creating a bless_deploy.cfg

~/me/bless/lambda_configs > chmod 444 bless-ca-
~/me/bless/lambda_configs > touch bless_deploy.cfg

We will be configuring the following parameters for our Bless Lambda in our bless_deploy.cfg.

• certificate_validity_after_seconds: 120
• certificate_validity_before_seconds: 120
• ca_private_key_file = bless-ca-
• ap-south-1_password = [ Encrypted passphrase text generated in previous step ]

Download the bless_deploy.cfg to get started easily.

Generating the Artifact(ZIP) to upload to Lambda

~/me/bless/ > make publish

The artifact is created in the publish folder under the bless directory.

Copy this zip to the artifact folder in the bless repository and execute the following commands again.

~/me/platform/environments/prod > terraform init
~/me/platform/environments/prod > terraform plan
~/me/platform/environments/prod > terraform apply

It will upload the zip to Lambda.

Voila, we have done all the steps to have a Certificate Authority up to and running.

Let’s configure our clients and our hosts(where we wish to SSH into )

Create an IAM User with Public Key

Let’s create an IAM user in AWS and add your local (laptop) public key. It is NOT the one which we generated previously but the one which you usually use to SSH into instances.

Configuring the Clients(Bastion Hosts)

We are going to set up the Bless Go client for generating our temporary certificates. Navigate to the ‘skel/files’ directory under the ansible folder in the bless repository.

Modify bless_client_config according to your requirements and infrastructure configuration. Please ensure you specify the remote users that will mint into your certificate for access.

Download bless_client_config for inspiration.

Execute the following command :

~/me/blessMe/ansible > ansible-playbook -i inventories/prod/ec2.py bless.yml -e "target_hosts=tag_role_bastion" -t bastion

It will perform the following tasks on the bastion-hosts(clients):

• Install goes bless client in the user home directory.
• Setup SSH config to use temporary generated certificates.
• iamsync.service: Syncs public SSH keys for IAM users and creates their home directories.

Configuring the Hosts

Navigate to defaults directory in blessMe/ansible/bless-ca/default and add the public key( bless-ca-.pub) generated in the /me/bless/lambda_configs folder to main.yml. This public key will be used to validate certificates signed by the Bless CA whenever a user tries to log in to the host.

Execute the following ansible command :

~/me/blessMe/ansible > ansible-playbook -i inventories/prod/ec2.py bless.yml -t hosts -l 10.90.45.35

It will configure sshd on host 10.90.45.35(where we wish to SSH into). We need to tell sshd to trust the public CA keys generated by us previously. Here we’ve said sshd to look in /etc/ssh/cas.pub for trusted keys.

We have completed the setup success. Let’s test our setup.

~/ ssh iamuser@10.90.1.5
iamuser@ip-10-90-0-177: ssh ubuntu@10.90.45.35

That’s it. We can SSH into the instance using our temporary generated certificates and don’t need to set up user public keys to every instance in our cloud infrastructure. We have a secure SSH key and user management process in place.

Feel blessed!

Thanks to Conor for inspiration :)