Using Vault with Addigy

TABLE OF CONTENTS

• Overview
• Installation
• Set up Vault
  • Start a Dev Vault Server without TLS
  • Start a Vault Server in Ubuntu 20.04 with TLS
    • Create a TLS Certificate with Certbot
    • Set up Vault Configuration
    • Create a Vault Service
    • Viewing Vault Logs
  • Initialize Vault
  • Unseal the Vault and Login as Root
  • Create a Secret to be Accessed
  • Create a Policy that Grants Read Access to our Secret
  • Create a Token Associated With a Policy
  • Use in an Addigy Custom Script
  • Enable Audit Logs
• References

Overview

Vault is an open-source secrets manager maintained by HashiCorp. Secrets stored in Vault are kept encrypted and are accessed as a virtual file system like “secrets/login-creds”. Vault tightly controls access to secrets such as tokens, passwords, certificates, and encryption keys that can be accessed using a GUI, CLI, or HTTP API. 

Permissions to access secrets are “deny by default”, meaning that if a machine or user doesn't have explicit permission to create, read, update, or delete a secret, then they are not able to do so. Permissions are defined in policies in HCL format. Vault also offers audit logging: secrets that are accessed are logged to track who accessed what. 

Authentication in Vault can either be done via passwords or by generating temporary tokens. There are additional authentication methods that integrate with other platforms such as AWS, Azure, Google Cloud, and GitHub. 

Installation

Mac OS:

Download the Vault binary from the official site: https://www.vaultproject.io/downloads 

Unzip the binary into the /usr/local/bin folder.

Verify that Vault works by running:

vault -v

Ubuntu 20.04:

Download the Vault binary for Linux (this command downloads version 1.6.1, for the latest version, check the HashiCorp site https://www.vaultproject.io/downloads):

wget https://releases.hashicorp.com/vault/1.6.1/vault_1.6.1_linux_amd64.zip

Unzip the file:

unzip vault_1.6.1_linux_amd64.zip

Move the vault binary to /usr/local/bin:

sudo mv vault /usr/local/bin/vault

Verify that Vault works:

vault -v

Deploying with Addigy to Macs:

To deploy the binary with Addigy to a client Mac, add the binary as Custom Software and enter the following for the installation script.

DEST=/usr/local/bin/vault
SOURCE='/Library/Addigy/ansible/packages/Vault (1.6.1)/vault'
if [ -f "$DEST" ]; then

    rm -f "$DEST"

fi

echo 'Installing Vault to bin'
cp "$SOURCE" "$DEST"
chmod +x "$DEST"
echo 'Cleaning up'
rm -rf '/Library/Addigy/ansible/packages/Vault (1.6.1)/'

Set up Vault

To start a production-ready Vault server with TLS, go to the next section.

Start a Dev Vault Server without TLS

Create the directory where we’ll keep Vault’s data:

mkdir -p vault/data

Configurations and policies are defined in HashiCorp Configuration Language (HCL) which was created by HashiCorp and is also used in Terraform. Policies can also be defined in JSON format.

Create a file named config.hcl with the following server configuration:

ui = true
disable_mlock = true
storage "raft" {
  path = "./vault/data"
  node_id = "node1"
}

listener "tcp" {
  address = "0.0.0.0:8200"
  tls_disable = 1
}
api_addr = "http://127.0.0.1:8200"
cluster_addr = "https://127.0.0.1:8201"

This will create a server with the raft storage backend and the listener address at the localhost.
NOTE: A production-ready environment would have TLS enabled. 
Now run the server with the config.hcl file:

vault server -config=config.hcl

For more information on Vault configuration: https://www.vaultproject.io/docs/configuration 
For more information on the server command: https://www.vaultproject.io/docs/commands/server 

Start a Vault Server in Ubuntu 20.04 with TLS

Create a TLS Certificate with Certbot

Be sure your Ubuntu server allows TCP traffic through port 8200 and HTTP traffic through port 80.

You’ll also need a DNS A record with a domain pointing to the IP of the machine. For example, vault.addigy.io pointing to the machine at 184.72.89.72.

Now let’s install Certbot which will give us our TLS certificate.

Ensure snapd is up to date:

sudo snap install core; sudo snap refresh core

Install Certbot:

sudo snap install --classic certbot

Run certbot:

sudo certbot certonly --standalone -d your.url.here

NOTE: Your domain name may be different. In this case, we’re using vault.adiggy.io.

If all goes well, it will successfully create a certificate that will expire in 3 months.

This certificate is automatically renewed by default, however, Vault would need to be restarted to use a renewed certificate. A script to do this will be shown at the end of the Create a Vault Service section.

Set up Vault Configuration

Let’s configure Vault.

Make a directory where we’ll store our Vault data.

sudo mkdir -p /var/vault/data

Create the config.hcl file for Vault:

sudo nano /var/vault/config.hcl

Enter the following in config.hcl:

ui = true
disable_mlock = true
storage "raft" {
  path = "/var/vault/data"
  node_id = "node1"
}

listener "tcp" {
  address = "0.0.0.0:8200"
  tls_disable = 0
  tls_cert_file = "/etc/letsencrypt/live/your.url.here/fullchain.pem"
  tls_key_file = "/etc/letsencrypt/live/your.url.here/privkey.pem"
}

api_addr = "https://127.0.0.1:8200"
cluster_addr = "https://127.0.0.1:8201"

NOTE: The tls_cert_file and tls_key_file paths may be different depending on your domain name, in this case, we’re using vault.addigy.io.

Create a Vault Service

Create a file named vault.service as /etc/systemd/system/vault.service:

sudo nano /etc/systemd/system/vault.service

Enter the following in the file:

[Unit]

Description=vault service
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/var/vault/config.hcl

[Service]
EnvironmentFile=-/etc/sysconfig/vault
Environment=GOMAXPROCS=2
Restart=on-failure
ExecStart=/usr/local/bin/vault server -config=/var/vault/config.hcl
LimitMEMLOCK=infinity
ExecReload=/bin/kill -HUP $MAINPID
KillSignal=SIGTERM

[Install]
WantedBy=multi-user.target

Start the service and enable it to start on boot:

sudo systemctl start vault.service
sudo systemctl enable vault.service

Check the status:

sudo systemctl status vault.service

Create a script to reload the Vault service every time Certbot renews the certificate:

sudo sh -c 'printf "#!/bin/sh\nsudo systemctl reload vault.service\n" > /etc/letsencrypt/renewal-hooks/post/vault.sh'

Viewing Vault Logs

To view logs output by the Vault service, the journalctl command can be used.

journalctl -b --no-pager -u vault -f

The -b flag tells journalctl to output logs from the most recent boot. The --no-pager flag prevents launchctl from displaying logs in pager format, this makes the output easier to use with text manipulation tools. The -u flag allows filtering by unit. Finally, the -f flag allows you to follow output as it comes, very similar to tail -f.

Initialize Vault

Open a new terminal and set the VAULT_ADDR environment variable. This is required for the vault command to know where the Vault server is:

export VAULT_ADDR='http://127.0.0.1:8200'

NOTE: The IP should point to the machine where the Vault server is running. In the case that you’re operating on the localhost, you will set the address to 127.0.0.1. In case if you’re operating on a server Vault deployment from somewhere else, you would set the address to the domain of that machine.

Now initialize the Vault server to get the unseal keys and root token:

vault operator init

NOTE: Be sure to securely store the unseal keys and initial root token as they will NOT be shown again.

Conventionally, you would distribute the unseal keys among 5 people. Only 3 of which are required to unseal the Vault.

For more information on the operator init command: https://www.vaultproject.io/docs/commands/operator/init.html

Unseal the Vault and Login as Root

With the unseal keys, unseal the Vault one by one.

vault operator unseal

NOTE: By default, 3 out of 5 of the unseal keys are required to unseal the Vault. 

After unsealing the Vault, login with the root token:

vault login

Enter the root token when prompted, the token should look similar to the following: 

s.1X9qYtFLmUZ5a1234567834324

For more information on the operator unseal command: https://www.vaultproject.io/docs/commands/operator/unseal 

For more information on the login command: https://www.vaultproject.io/docs/commands/login 

Create a Secret to be Accessed

Here we will create a secret that we want other machines to be able to access.

First, we need to enable the key-value secrets engine:

vault secrets enable kv

NOTE: The default path for the kv secrets engine is kv/.

Create a secret in the path kv/creds with the key/value pair password=123456:

vault kv put kv/creds password=123456

You can delete secrets in the key-value secrets engine with: 

vault kv delete [PATH]

vault kv get [PATH]

For more information on the secrets command: https://www.vaultproject.io/docs/commands/secrets 

For more information on the kv command: https://www.vaultproject.io/docs/commands/kv 

Create a Policy that Grants Read Access to our Secret

Here we will create a policy that will allow a machine with our defined permissions to read our secret.

Create a file named read_secret.hcl with the following: 

# Permits reading of secret in kv/creds

path "kv/creds" {

  capabilities = ["read"]

}

Now create the policy in Vault using the read_secret.hcl file:

vault policy write read_secret read_secret.hcl

For more information on Policies: https://www.vaultproject.io/docs/concepts/policies

For more information on the policy command: https://www.vaultproject.io/docs/commands/policy

Create a Token Associated With a Policy

Tokens are used to authenticate with Vault and its permissions are determined by the policy associated with the token. 

For the purposes of security, tokens should be ephemeral meaning that they should have a specified max TTL and/or use limit to prevent a leaked token from being useful as it would be revoked by then. 

Let’s create a token associated with the policy we created earlier:

vault token create -policy=read_secret -use-limit=1 -explicit-max-ttl=1m

This will create a token that is associated with the read_secret policy where it has a use limit of 1 and has a max TTL of 1 minute. The TTL can be specified in seconds, minutes, or hours (ex. 30s, 1m, 1h). 

Copy the created token to use in your Addigy custom script.

NOTE: It is highly recommended that the use limit be set to exactly the amount of times that the token should be used and that the max TTL be set to the shortest time necessary for your custom script(s) to execute.

To manually revoke a token, use the vault token revoke command:

vault token revoke s.Ie7jV4oxoftehFeylBieeoU4

For more information on the token create command: https://www.vaultproject.io/docs/commands/token/create.html 

Use in an Addigy Custom Script

Using Vault in a custom script is not as simple as just running vault kv get kv/creds to get the secret. 

The Vault server’s address and the Vault client token need to be defined in environment variables VAULT_ADDR and VAULT_TOKEN, respectively.

So, for VAULT_ADDR, it should look something like this:

export VAULT_ADDR="http://127.0.0.1:8200"

But replace 127.0.0.1 with the server’s IP or domain name (be sure to change http to https if using TLS ).

For VAULT_TOKEN, enter the token you had created earlier with the associated policy:

export VAULT_TOKEN=s.20Je8g0MJTJDKI80HGEEF4Qn

Your token will be different from the one used in this example.

Finally, the PATH needs to be set. When running scripts in Addigy, the PATH env variable doesn’t include /usr/local/bin and so, by default, it would not recognize the vault command previously installed in that location. 

Modifying the PATH variable should look like this:

export PATH=/usr/local/bin:$PATH

Now you can make API calls with Vault however you choose. 

Here’s an example script that prints out a secret:

export VAULT_ADDR="https://vault.url.here:8200"
export VAULT_TOKEN="Vault_Token_Here"
export PATH=/usr/local/bin:$PATH
PASSWORD=$(vault kv get -field=password kv/creds)
echo "$PASSWORD"

If you had set your token’s usage limit to the exact amount of times it’s used in your script then there’s virtually no risk of a token leak allowing access to your secrets since it would have been revoked by then. It would be good practice to set a max TTL for your token in addition to the max usage if the script fails before the token reaches max usage, otherwise, you run the risk of leaking a token that didn’t get used enough to be revoked.

Enable Audit Logs

One of the key features of Vault is its ability to audit any response and request in the Vault server. This feature needs to be enabled to make use of it.

Be sure to have set the VAULT_ADDR environment variable to the address of the Vault server. Also, be sure to log in with the root token or with a token that has permission to enable audit methods. 

Run the following to enable audit logs:

vault audit enable file file_path=/var/log/vault/vault_audit.log

This will write audit logs to the /var/log/vault_audit.logs path where the Vault server is deployed.

The log file may end up getting too large for the allotted space and will need to be rotated. This can be done using logrotate.

SSH into your Vault server deployment to make the following changes.

Create a file named vault in the /etc/logrotate.d path:

sudo nano /etc/logrotate.d/vault

Enter the following in the file:

/var/log/vault/vault_audit.log {

        rotate 30
        daily
        notifempty
        missingok
        compress
        delaycompress
        postrotate
                systemctl reload vault 2>/dev/null || true
        endscript
        extension log
        dateext
        dateformat %Y-%m-%d.
}

Run logrotate with this file:

sudo logrotate /etc/logrotate.d/vault

Now log rotation should be set up with the audit logs.

For information on querying audit logs, see Querying Audit Device Logs.

References

Vault Docs: https://www.vaultproject.io/docs