Setting Up Hashicorp Vault in Azure Container Instance

Secret management can be extremely difficult. There are many things involved with managing secrets: expiration, revoking a secret, secret access control, rotate keys, etc. These things are difficult and error prone if done by hand. Thankfully, there are some tools out there to assist us in our desire to remain secure from the first line of code! One of those tools, Hashicorp Vault, provides a tremendously flexible, verifiably secure, centrally managed secret management engine, that can be adapted to just about any use case. In this article, I will show you how to setup a simple Hashicorp Vault using Docker and Azure Container Instance (ACI) and access it using a service connection from Azure DevOps.

Pre-Reqs for this walkthrough:

Problem:

We have many different secrets, to many different resources that we need to maintain securely for an organization and its applications. These secrets could be certificates, usernames and passwords to databases, access credentials to a cloud, or even ssh-keys.

Where should we put these secrets? Hashicorp Vault

Hashicorp Vault (Vault from here on out) is an extremely powerful secret management platform. One of Vault’s best features is that we can use it to dynamically create secrets based on policies using secret engines. To Vault, not all secrets are stored as encrypted usernames/passwords…they can be created on demand from a variety of different services, devices, etc that need secrets to access them. For example, Vault can talk directly to PostgreSQL to dynamically create a new username and password _on demand_ that has pre-determined expiration. Another secret engine, KV, can be used to store the more traditional key/value based secrets. This is the secret engine we will use in this article.

Much like the secret engines allow us pluggable secret sources, Vault provides auth methods , which are configurable authentication mechanisms to Vault itself. Examples of authentication methods that we can use are username/password, token, or even Azure Active Directory credentials. We can configure both how we authenticate to Vault, and we can configure the secret engines that Vault can supply. This makes for a highly configurable and flexible secret store.

For this example we will just deploy a basic Vault to Azure Container Instance (ACI) using a container image off of docker hub and a storage account file share for persistance so that we don’t lose our vault configuration and data (this is not a highly available setup, but it will be enough for our development and testing).

Update the RG (resource group), SA_NAME (storage account name), and FILE_SHARE (file share name) in the code below using values from your storage account and file share (you will need to create a file share if one doesn’t exist). Also, make sure do update the DNS_LABEL. This will be in the URL to our new Vault instance once the ACI is created.

#!/usr/env/env bash

RG="<resource group name>"
VAULT_IMAGE="dankydoo/vault-test" # preconfigured vault image for ACI
SA_NAME="<storage account name>"
FILE_SHARE="<file share name>"
DNS_LABEL="vault-azdo"
LOCATION="eastus2"

SA_KEY=$(az storage account keys list -g ${RG} -n ${SA_NAME} --query "[0].value" -o tsv)

az container create \
--name "vault-test" \
--resource-group $RG \
--image $VAULT_IMAGE \
--dns-name-label $DNS_LABEL \
--ports 8200 \
--location $LOCATION \
--azure-file-volume-account-name $SA_NAME \
--azure-file-volume-account-key $SA_KEY \
--azure-file-volume-share-name $FILE_SHARE \
--azure-file-volume-mount-path "/opt/vault/data"

After a few minutes, you will now have a live vault instance accessible via:

<dnslabel>.<location>.azurecontainer.io:8200

In the case of the example, Vault is at available at: http://vault-azdo.eastus2.azurecontainer.io:8200/.

Vault Start Page

The first time that Vault starts, it is sealed. When the Vault is sealed, nothing can be accessed until it is unsealed. The data within Vault is encrypted with a master key, that can be reconstituted via a number of shares using an algorithm called Shamir’s Secret Sharing. The basic gist is that when we initialize Vault (which we do only once at creation time), we specify how many key shares that we want, and how many of those key shares it should take to unlock. The idea is that no one person should ever really hold all of the keys to the kingdom! An example: we initialize with 5 keys, and we specify that it takes 2 keys to unseal the Vault. We can distribute these 5 keys to 5 trusted individuals, and any two of them can coordinate to unseal the Vault.

Vault Initialization

We will go ahead and enter 5 for keyshares, and 2 for the key threshold. Vault will now present us with the root token (admin password) and the 5 key shares as requested. Record these now!! Once we leave this screen, they can never be seen again. Treat them very securely, like you would root certificates! We could have also used the vault cli to do this.

Your root token will look something like: s.FIIteDEOVQDJeTJ8tifmyWER and the key shares will be of the format: zTTsqyXXb3KskiXXSSQATxxHN/aZYkiPEjN87vk+zgtz

Now that we have generated our keys, we need to actually unseal the Vault. We will use the Vault CLI which can be downloaded from here. Select the download that is appropriate for your system then place the vault binary in your path. A nice feature of vault is that the CLI, server, and agent are all contained in the same executable. It makes managing vault a bit easier.

I cannot stress enough to keep the root token (super user/admin) and key shares protected. In fact, it’s best to enable username/password authentication and delete the root token as soon as possible so that it isn’t shared around or lost.

Now that we have vault in the path, we need to point it at our newly created Vault running in Azure Container Instance. Remember from earlier that the example vault deployed to http://vault-azdo.eastus2.azurecontainer.io:8200. To have the CLI use this Vault instance, we set the VAULT_ADDR environment variable equal to this address.

export VAULT_ADDR=http://vault-azdo.eastus2.azurecontainer.io:8200

We can test this by typing: vault status

Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          2
Unseal Progress    0/2
Unseal Nonce       n/a
Version            1.1.1
HA Enabled         false

We notice that our vault is initialized, but it’s still sealed. We need to use the keys we created earlier to unlock it and be on our way! We do this by typing: vault operator unseal:

 $ vault operator unseal
Unseal Key (will be hidden): 
$ vault login
Token (will be hidden): 

We go ahead and enter any two of our key shares, and the vault unseals. Now that we have unsealed our vault, we need to log into the Vault. Here, we enter the root token.

Now that we have an unsealed vault that we are logged into, it’s time to apply another important Vault concept: policies. Policies allow us to declaratively describe who has access to what secrets. We can store our policies in HCL files and version them just like code. Then we have a source of truth to be able to see who has access to what in our vault.

Initially, all we have with a fresh vault is a root token with all permissions and no policies, and no secret engines. Let’s create our first policy for admins:

path "*" {
        policy = "sudo"
}

We save this into a file, we will call admins.hcl and we create the policy:

vault policy write admins admins.hcl

We need to next enable the standard key/value secret engine. We do this by running:

vault secrets enable -path=/secrets kv

Let’s go ahead and put our first secret into the vault. In the previous step we mounted the kv secret engine onto the /secrets path. We will write our secrets to this path:

$ vault kv put secrets/myapp apikey=xyzabc anothersecret=blahblah
Success! Data written to: secrets/myapp

Next we need to setup username/password authentication to our Vault and then disable the root token. We do this by adding our first vault method: userpass . Run the following commands to enable the userpass auth method and add an admin user:

$ vault auth enable userpass 
Success! Enabled userpass auth method at: userpass/
$ vault write auth/userpass/users/<username> \   
    password=<password> \
    policies=admins
Success! Data written to: auth/userpass/users/<username>

In Vault, everything is a path. Whether it’s an auth method, or a secret engine, it is accessed by path.

We can see in the above example, we gave our new user the “admins” policy. This links our new user to the admin policy we created in the earlier steps. We can go ahead and create a few admin users for our main keyholders. For this example, we will stick with the single user. We test our new user to make sure it’s setup correctly:

vault login -method=userpass username=<username> password=<password>

Now that we have a username and password, let’s revoke that root token so that it’s taken care of!

$ vault token revoke <root token>
Success! Revoked token (if it existed)

If everything has gone as planned, we will be logged in as our user, and we should be able to read our secret that we had added earlier:

$ vault kv list /secrets
Keys
----
myapp
$ vault kv get /secrets/myapp
======== Data ========
Key              Value
---              -----
anothersecret    blahblah
apikey           xyzabc

We now have a fully configured, hands off vault instance that we can learn and utilize in our applications!

Summary:

In this article, we:

  • created a publicly accessible Vault instance using the Azure Container Instance PaaS.
  • Initialized the Vault
  • Unsealed the Vault
  • Mounted the userpass secret engine
  • Added a secret
  • Created a vault policy
  • Created a vault user associated with that policy
  • Logged in as this new user and retrieved our secrets!

With this setup, you’ll be able to get a great feel for Vault and its capabilities. In a future Article, I will show you how to authenticate to this Vault instance using an Azure Managed Service Identity (MSI) from your build pipeline.