I just started a new Club, it's called the Spicy Coderz Club and it is the newest most exclusive club for spicy coderz ONLY. Naturally everyone wants to join, and many people try and sneak in to our clubhouse everyday. But we take security VERY seriously. Everyday we generate a new password, and only by saying that day's password can you enter the clubhouse. As head of security at the SCC, I had to figure out a way to distribute the daily password to all members in the most secure manner. Vault was the obvious choice. This post will give you an basic overview of how we use the Cubbyhole Secret Backend and Response Wrapping, so you can secure your own clubhouse!

Setting up a Vault Server and Logging

The actual Spicy Coderz Club Vault server lives in the "Cloud", but for this post, we are going to use Vault's Dev mode. This is NOT how you would use Vault in a production environment, but will suffice for demonstration purposes.

This post assumes you already have Vault installed on your local computer, if you need more help installing Vault please consult the Documentation.

Set Vault Address to HTTP instead of HTTPS:

export VAULT_ADDR='http://127.0.0.1:8200'

Note: We have to update the Vault Address because we are using Dev mode. This is not something you would run in production.

Start the Vault server:

vault server -dev

This will result in some output like:

==> WARNING: Dev mode is enabled!

In this mode, Vault is completely in-memory and unsealed.
...
Unseal Key: l+L/zOZVw+aNEaMGSyQLAV4iAAGPoFPX2SrkYsazWmk=
Root Token: 00dfe942-df0f-4cbd-66a2-db99c1212654

==> Vault server started! Log data will stream in below: 

The important information right now is the root token. We will use this to finish setting up Vault and storing secrets.

Auditing


Since we take security so seriously at the Spicy Coderz Club, we want to use one of Vault's Audit Backends to track all activity with our Vault server. For the purpose of this post, we will use the File Audit Backend, but you also have the option of Syslog and Socket for your own implementation.

Enable Audit Logging:

export VAULT_TOKEN=00dfe942-df0f-4cbd-66a2-db99c1212654

vault audit-enable file file_path=vault_audit.log log_raw=true hmac_accessor=false
Successfully enabled audit backend 'file' with path 'file'!

Note: For the purpose of this post we are not encrypting the data, so we can more easily see who is interacting, with what on our Vault server. This is NOT what you would do in a production environment. The two settings we modified are:

log_raw optional A string containing a boolean value ('true'/'false'), if set, logs the security sensitive information without hashing, in the raw format. Defaults to false.

hmac_accessor optional A string containing a boolean value ('true'/'false'), if set, enables the hashing of token accessor. Defaults to true. This option is useful only when log_raw is false.

Also Note: If you enable vault auditing without a fully qualified file path (filepath=vault_audit.log versus filepath=/var/logs/vault_audit.log), it will place that file in folder where the Vault Server was started, not the folder where the command was run from.

After we enable Audit logging, we see a file appear called vault_audit.log with some information looking like the following:

{
    "time": "2017-10-22T16:25:00Z",
    "type": "response",
    "auth": {
        "client_token": "00dfe942-df0f-4cbd-66a2-db99c1212654",
        "accessor": "7c2a6947-5dd5-f6c6-5594-971d5b883907",
        "display_name": "root",
        "policies": [
            "root"
        ],
        "metadata": null
    },
    "request": {
        "id": "51455a80-91eb-0571-6348-93a726708727",
        "operation": "update",
        "client_token": "00dfe942-df0f-4cbd-66a2-db99c1212654",
        "client_token_accessor": "7c2a6947-5dd5-f6c6-5594-971d5b883907",
        "path": "sys/audit/file",
        "data": {
            "description": "",
            "local": false,
            "options": {
                "file_path": "vault_audit.log",
                "hmac_accessor": "false",
                "log_raw": "true"
            },
            "type": "file"
        },
        "remote_address": "127.0.0.1",
        "wrap_ttl": 0,
        "headers": {}
    },
    "response": {},
    "error": ""
}

This is the basic format of Vault Audit logs: A request and response, with all relevant information like what token was used for the request, and what the response contained, as well as any errors.

Tail the Audit Logs:

tail -F vault_audit.log | while read line; do echo "$line" | jq; done 

Note: You will need jq installed for this command. We use jq to help format and colorize the Audit logs. If you don't want formatting/coloring, you can just tail the logs with tail -F vault_audit.log.

Disabling and Updating Auditing

If you want to update where you mounted the Vault Audit backend or change the file path you will need to disable the Vault backend first.

Disable Audit Logging:

vault audit-disable file

We now have a Vault server setup with some audit logging, we are ready to start storing and sharing secrets! But first we need to meet our Club members.


Introducing the Spicy Coderz Club

Aside from myself, we currently have 3 members in the Spicy Coders Club:

Pappas
pappas-3

Johnny Utah
johnny-1

Bodhi
bodhi

When they joined we generated a token for each. We added a display name to better track their activity in the audit logs:

vault token-create --display-name=pappas
vault token-create --display-name=johnny
vault token-create --display-name=bodhi

Each of these commands will output something similar to:

Key             Value
---             -----
token           91b1ac6e-ab79-bf44-14db-11438f19af8f
token_accessor  0af7601f-b56c-4bac-d9a6-2264156977c7
token_duration  0s
token_renewable false
token_policies  [root]

We distribute each members token to ONLY them. This will allow us to track down potential leaks to the source in the future.


Generating and Storing the Clubhouse Password

We generate the clubhouse password everyday with some bash magic.

Generate Daily Password

sed "$(jot -r 1 0 $(wc -l < /usr/share/dict/words))q;d" /usr/share/dict/words
# => sesquitertian

sesquitertian

Store Daily Password

We then need to store this password in Vault. Vault provides many options for how to store secrets and authenticate clients, for our case we are going to use the Cubbyhole Secret Backend. This will allow us to generate a new token everyday, and store that daily password in the "Cubbyhole" of that token. This means that secret can only be read out using that particular token. So even if someone was able to steal the token from the day before, they wouldn't be able to use it to steal today's password.

Generate Daily Token:

vault token-create --display-name="$(date +"%Y-%m-%d")"

Note: We generate a new token with a display name of today's date so it is easier to see what day's token was leaked if someone tries to use it later.

We will see some output like:

Key             Value
---             -----
token           8fdaf3e6-0c56-6a49-820b-d268e96ef79e
token_accessor  aec06e13-71f6-2fbc-1084-8ed470ae2d38
token_duration  0s
token_renewable false
token_policies  [root]

And in our Audit Logs we see something like:

{
  "time": "2017-10-22T16:51:44Z",
  "type": "response",
  "auth": {
    "client_token": "00dfe942-df0f-4cbd-66a2-db99c1212654",
    "accessor": "1accfe54-9b16-ea90-14cf-f7e5809c8236",
    "display_name": "root",
    "policies": [
      "root"
    ],
    "metadata": null
  },
  "request": {
    "id": "8fd0deaa-14c8-f856-c3bc-ea3a4b698ddf",
    "operation": "update",
    "client_token": "00dfe942-df0f-4cbd-66a2-db99c1212654",
    "client_token_accessor": "1accfe54-9b16-ea90-14cf-f7e5809c8236",
    "path": "auth/token/create",
    "data": {
      "display_name": "2017-10-22",      
      "num_uses": 0,                     
      "renewable": true                  
    },                                   
    "remote_address": "127.0.0.1",       
    "wrap_ttl": 0,                       
    "headers": {}                        
  },                                     
  "response": {                          
    "auth": {                            
      "client_token": "8fdaf3e6-0c56-6a49-820b-d268e96ef79e",
      "accessor": "aec06e13-71f6-2fbc-1084-8ed470ae2d38",
      "display_name": "token-2017-10-22",
      "policies": [                      
        "root"                           
      ],                                 
      "metadata": null                   
    }                                    
  },                                     
  "error": ""                            
}

Now we can write a secret in the Cubbyhole of that generated token.

Write Password in Cubbyhole of Daily Generated Token:

export VAULT_TOKEN=8fdaf3e6-0c56-6a49-820b-d268e96ef79e

vault write cubbyhole/spicy_coderz_clubhouse password=sesquitertian

Note: 8fdaf3e6-0c56-6a49-820b-d268e96ef79e is the daily generated token in this case.

We will see a message like:

Success! Data written to: cubbyhole/spicy_coderz_clubhouse

Now all we have to do is send each member that day's token and so they can retrieve the password!

Read from the Cubbyhole:

export VAULT_TOKEN=8fdaf3e6-0c56-6a49-820b-d268e96ef79e

vault read cubbyhole/spicy_coderz_clubhouse
Key             Value
---             -----
password        sesquitertian

And in our Audit Logs we see something like:

  {
    "accessor": "324fa11d-49af-587a-b465-d5d3c5d9fd30",                       
    "display_name": "token-2017-10-22",  
    "policies": [                        
      "root"                             
    ],                                   
    "metadata": null                     
  },                                     
  "request": {                           
    "id": "a9d29e6a-a5ed-1f0e-cead-f5d455067ea5",
    "operation": "read",                 
    "client_token": "8fdaf3e6-0c56-6a49-820b-d268e96ef79e",     
    "client_token_accessor": "324fa11d-49af-587a-b465-d5d3c5d9fd30",      
    "path": "cubbyhole/spicy_coderz_clubhouse",
    "data": null,                        
    "remote_address": "127.0.0.1",       
    "wrap_ttl": 0,                       
    "headers": {}                        
  },                                     
  "error": ""                            
}                                        
{                                        
  "time": "2017-10-22T16:56:46Z",        
  "type": "response",                    
  "auth": {                              
    "client_token": "7d99cb23-8b1b-c610-8737-97defc36037b",
    "accessor": "324fa11d-49af-587a-b465-d5d3c5d9fd30",
    "display_name": "token-2017-10-22",  
    "policies": [                        
      "root"                             
    ],                                   
    "metadata": null                     
  },                                     
  "request": {                           
    "id": "a9d29e6a-a5ed-1f0e-cead-f5d455067ea5",
    "operation": "read",                 
    "client_token": "8fdaf3e6-0c56-6a49-820b-d268e96ef79e",                   
    "client_token_accessor": "324fa11d-49af-587a-b465-d5d3c5d9fd30",         
    "path": "cubbyhole/spicy_coderz_clubhouse",                               
    "data": null,                        
    "remote_address": "127.0.0.1",       
    "wrap_ttl": 0,                       
    "headers": {}                        
  },                                     
  "response": {                          
    "data": {                            
      "password": "sesquitertian"        
    }                                    
  },                                     
  "error": ""                            
}

Huzzah! We can see that someone successfully read out the password with the token 8fdaf3e6-0c56-6a49-820b-d268e96ef79e at 2017-10-22T16:56:46Z. Not the most informative, but a start.


Increasing Security

Inevitably as our club gets increasingly popular, more and more people are trying to break in, and we need to beef up our security. Right now, if someone intercepts the token when sending it to one our members, they will be able to read out the password and get into our oh-so-exclusive club. We need to figure out to reduce this attack vector.

Luckily Vault provides some mechanisms to increase our token security. First is the --use-limit flag. Since we know the exact number of members in the Spicy Coderz Club, we can set the use-limit so once every member has used the token once, the token will non-longer be valid, and further the Cubbyhole backend will destroy itself automatically.

Since we have 3 members who need the password, we set the use-limit to 4: once to write the secret in the Cubbyhole of that token, and then once for each of the members of the SCC. If the token is used after already being used 4 times, the user will get a permission denied error.

Example of Use Limit 4 Token


Generate a Token with Use Limit:

vault token-create --use-limit=4
Key             Value
---             -----
token           36e54fae-2e2c-f6c2-5103-75f9e260e49c
token_accessor  6e9eb60a-3415-2e62-db1a-d4d75582adb7
token_duration  0s
token_renewable false
token_policies  [root]

Write Password in Cubbyhole of Daily Generated Token:

export VAULT_TOKEN=36e54fae-2e2c-f6c2-5103-75f9e260e49c

vault write cubbyhole/spicy_coderz_clubhouse password=neologianism

Then we email the Vault Token to each of the members, and they can use it to read the secret from the Cubbyhole, but after 3 uses they will be denied.

johnny-small

export VAULT_TOKEN=36e54fae-2e2c-f6c2-5103-75f9e260e49c

vault read cubbyhole/spicy_coderz_clubhouse
Key             Value
---             -----
password        neologianism

bodhi-small

export VAULT_TOKEN=36e54fae-2e2c-f6c2-5103-75f9e260e49c

vault read cubbyhole/spicy_coderz_clubhouse
Key             Value
---             -----
password        neologianism

pappas-small

export VAULT_TOKEN=36e54fae-2e2c-f6c2-5103-75f9e260e49c

vault read cubbyhole/spicy_coderz_clubhouse
Key             Value
---             -----
password        neologianism

But when Tone tries to get in with the stolen token:
tone-small-2

export VAULT_TOKEN=36e54fae-2e2c-f6c2-5103-75f9e260e49c

vault read cubbyhole/spicy_coderz_clubhouse
Error reading cubbyhole/spicy_coderz_clubhouse: Error making API request.

URL: GET http://127.0.0.1:8200/v1/cubbyhole/spicy_coderz_clubhouse
Code: 403. Errors:

* permission denied
Sorry Tone!


This makes us feel a little safer, but it still leaves us open to someone stealing our token before all the members have used it. Vault can help us with this too by allowing us to set a TTL, which means we can say that after an hour (any any amount of time we determine), this token is no longer valid (and the Cubbyhole will be automatically destroyed!).

Generate a Daily Token with Use Limit and TTL:

vault token-create --use-limit=4 -ttl="1h"
Key             Value
---             -----
token           929ddb51-54a4-58ba-8486-026a656680f2
token_accessor  277cb2f2-c9a1-673f-b8cc-00aad5562346
token_duration  1h0m0s
token_renewable true
token_policies  [root]

You can see the token_duration now says 1h0m0s. So now after an 1 hour, the Cubbyhole will be destroyed, and we don't have to worry about some hooligans who only live to get radical and don't even understand the sea, stealing our secrets and getting into our clubhouse.


Even more Security

We still have a problem in that we are still sending out tokens through email. This makes the security team of the Spicy Coderz Club very nervous. We would like to not send the actual token over the web, as well as get some more detailed information about who is trying to access secrets. This is where Response Wrapping comes in handy.

Response Wrapping allows us to "Wrap" our daily Cubbyhole token in a temporary token, which can only be used Once to "Unwrap" and get the Cubbyhole token value. So we wrap up the daily Cubbyhole token once for each member, and send all the members an individual token. This means we never send the actual Cubbyhole token to anyone and we can get a little more information on who has unwrapped their token. This can help to prevent or potentially track down leaks.

Generate a Daily Token with Use Limit and TTL:

vault token-create --use-limit=4 -ttl="1h"
Key             Value
---             -----
token           be3c7edf-7078-7f6d-bb12-ab53606246bf
token_accessor  acd6f1c6-b5da-a1d9-b5ef-6e6e7df2a0ad
token_duration  1h0m0s
token_renewable true
token_policies  [root]

We can then use the Vault wrap API to Wrap the token.

Wrap the Cubbyhole Token:

vault write sys/wrapping/wrap token=be3c7edf-7078-7f6d-bb12-ab53606246bf
Key                             Value
---                             -----
wrapping_token:                 dfa718bc-0f95-60b0-43c2-e23ed224e042
wrapping_token_ttl:             5m0s
wrapping_token_creation_time:   2017-10-21 18:15:18.256091964 -0700 PDT
wrapping_token_creation_path:   sys/wrapping/wrap

This by default is only valid for 5 minutes, if our SCC member doesn't unwrap in that time, they would have to reach out to get a new "wrapped token". Annoying, but secure!

Unwrap the Token:

vault unwrap dfa718bc-0f95-60b0-43c2-e23ed224e042
Key     Value
---     -----
token   be3c7edf-7078-7f6d-bb12-ab53606246bf

Once a token has been unwrapped, you will get an error if you try and unwrap again.

Unwrapping an expired token:

Error making API request.

URL: PUT http://127.0.0.1:8200/v1/sys/wrapping/unwrap
Code: 400. Errors:

* wrapping token is not valid or does not exist

Also every time a token is unwrapped we can see who unwrapped it in the Audit Logs. For instance if Bodhi unwrapped the token we see a log message like:

{                                        
  "time": "2017-10-22T17:44:52Z",        
  "type": "response",                    
  "auth": {                              
    "client_token": "794a7899-8f7a-766e-0fb7-bda775d2e86c",     
    "accessor": "64fa537a-d643-58da-ebb4-79826f485afe",     
    "display_name": "token-bodhi",       
    "policies": [                        
      "root"                             
    ],                                   
    "metadata": null                     
  },                                     
  "request": {                           
    "id": "6ebcedc3-0385-8333-77eb-7f1e2c2f8629",     
    "operation": "update",               
    "client_token": "794a7899-8f7a-766e-0fb7-bda775d2e86c",     
    "client_token_accessor": "64fa537a-d643-58da-ebb4-79826f485afe",      
    "path": "sys/wrapping/unwrap",       
    "data": {                            
      "token": "adb51644-c387-bfe8-3647-02844abf88f7"     
    },                                   
    "remote_address": "127.0.0.1",       
    "wrap_ttl": 0,                       
    "headers": {}                        
  },                                     
  "response": {                          
    "data": {                            
      "token": "d5b3f384-7800-021d-fb46-e31b581821b6"
    }                                    
  },                                     
  "error": ""                            
}

Now we can see exactly who has and hasn't unwrapped their token yet, and we could even revoke the token if we wanted!

And that's the basics of using the Cubbyhole Secret Backend and Response Wrapping!


A Note on Auto-wrapping Tokens

Vault can also save you some time by automatically wrapping created tokens with the -wrap-ttl flag.

Create an Auto-Wrapped Token:

vault token-create --use-limit=4 -ttl="1h" -wrap-ttl="5m"  
Key                             Value
---                             -----
wrapping_token:                 d8c99b77-bd33-2184-3f3d-7093a934f9bc
wrapping_token_ttl:             5m0s
wrapping_token_creation_time:   2017-10-22 10:48:28.405354574 -0700 PDT
wrapping_token_creation_path:   auth/token/create
wrapped_accessor:               832b1306-babb-d4d7-dd20-b71bae357298

However this auto-wrapping won't work for our particular use case, as we don't ever know what the underlying "Wrapped" token is. So we can't store any secrets in it's Cubbyhole, without unwrapping it first.....which defeats the purpose! This feature is better for creating tokens attached policies, where we don't need to ever know the token.


HTTP versions of all commands

Enable Audit

 curl \
    --header "X-Vault-Token: e7db13dc-beb1-7174-49fe-92f14bb8064f" \ 
    --request PUT \
    --data @enable_audit.json \
    http://127.0.0.1:8200/v1/sys/audit/spicy-coderz-audit

Here is the contents of audit_payload.json:

{
"type": "file",
"options": {
  "path": "vault_audit.log",
  "log_raw": "true",
   "hmac_accessor": "false"
  }
}

Note: If your path is to a folder that doesn't exist, vault will create it, but then fail to to create the actual audit file.
You will see an error like:

{"errors":["sanity check failed; unable to open logs/vault_audit.log for writing: open logs/vault_audit.log: permission denied"]}

If you rerun then it will then be successful. Also this behavior occurs

If you accidentally mounted your logs in the incorrect location, you can not just update it with a PUT request, you must first delete the original path:

curl                                     \
  --header "X-Vault-Token: $VAULT_TOKEN" \
  --request DELETE                       \
  http://127.0.0.1:8200/v1/sys/audit$(pwd)

Create a Token

curl \
    --header "X-Vault-Token: e7db13dc-beb1-7174-49fe-92f14bb8064f" \
    --request POST \
    --data @create_token.json \
    http://127.0.0.1:8200/v1/auth/token/create

The contents of create_token.json:

{
  "display-name":"pappas"
}

Or you can pass in the JSON directly:

curl \
    --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST                         \
    --data '{"display-name":"pappas"}'      \
    http://127.0.0.1:8200/v1/auth/token/create

Create a Daily Token

display_name="$(date +"%Y-%m-%d")"

curl                                                \
    --header "X-Vault-Token: $VAULT_TOKEN"          \
    --request POST                                  \
    --data "{\"display-name\":\"${display_name}\"}" \
    http://127.0.0.1:8200/v1/auth/token/create

Write in Cubbyhole

curl                                           \
    --header "X-Vault-Token: $CUBBYHOLE_TOKEN" \
    --request POST                             \
    --data "{\"password\":\"$DAILY_PASSWORD\"}"      \
    http://127.0.0.1:8200/v1/cubbyhole/spicy_coderz_clubhouse

Read from Cubbyhole

curl                                           \
    --header "X-Vault-Token: $CUBBYHOLE_TOKEN" \
    http://127.0.0.1:8200/v1/cubbyhole/spicy_coderz_clubhouse

Wrap Token

curl                                            \
    --header "X-Vault-Token: $VAULT_TOKEN"      \
    --header "X-Vault-Wrap-TTL: 60"             \
    --request POST                              \
    --data "{\"token\":\"${CUBBYHOLE_TOKEN}\"}" \
    http://127.0.0.1:8200/v1/sys/wrapping/wrap

Unwrap Token

curl                                          \
    --header "X-Vault-Token: $VAULT_TOKEN"    \
    --request POST                            \
    --data "{\"token\":\"${WRAPPED_TOKEN}\"}" \
    http://127.0.0.1:8200/v1/sys/wrapping/unwrap

Generate Token that is Auto-Wrapped

display_name="$(date +"%Y-%m-%d")"

curl                                                \
    --header "X-Vault-Token: $VAULT_TOKEN"          \
    --header "X-Vault-Wrap-TTL: 60"                 \
    --request POST                                  \
    --data "{\"display-name\":\"${display_name}\"}" \
    http://127.0.0.1:8200/v1/auth/token/create

I have also compiled all these commands into a Github Repo Here

Welcome to the Spicy Coderz Club!


Photo by Benjamin Lambert / Unsplash