Vault, From Ground Up
I just ordered a vault and loudspeaker online
They just arrived. Safe and sound
Well, now that we have the unavoidable dad joke out of the way, here we are again to learn something, from scratch.
This story aims to be a main story linking various parts together. Some resources created during external parts are reused elsewhere but they are not necessary for progress. Document is divided in to several parts to make it possible to concentrate on just the topics needed.
Goal: Gaining practical knowledge on Vault in it’s entirety (except prod deployments which is another thing on it’s own and no enterprise stuff unfortunately).
What we need
If you’ve used Kubernetes (or anything else really) to manage your containers before, you have a secret resource where your “secrets” like app creds, tokens, keys are kept as base64 encoded (or if you want, plain text). In case of apps, your secrets are somewhere in a config, known to a few sys admins and manually entered to said config or at best encrypted on a case by case basis. Well, why care about all of this when Vault does it for you?
What is it?
Vault is one of the most used “safe” solutions (created at 2015) for keeping and managing secrets for your apps, clusters or whatever may need it.
Why?
With app and user level access control to secrets and their management, system admins breathe easy. Also, Dynamic Secrets.
Dynamic Secrets?
We will get into this later but in short, Vault is the captain now. Vault manages the credentials, not you. You don’t need to keep any secrets in any config, only a Vault path to where the secret is and your Vault endpoint.
Additionally some secrets can be rotated by Vault so only it knows of your DB creds for example and instead generates short lived credentials for your app to use.
I‘m one of the “let’s try it out until we’re stuck” kind of people so let’s get our local env running.
Installing Vault
First, we need a running Vault, preferably in dev mode. Below story has got you covered if you don’t have one already running on your local.
Vault Part 1- Starting Up Vault
Secrets
The point of Vault is secret management. Vault can create and share secrets securely. See the below story for a brief intro.
Secrets Engines
Secrets engines are components that generate, and encrypt secret data for an app or a user to use.
Using Vault In Apps
Authentication
LDAP, GitHub, Token and AWS IAM authentication steps are below. App role authentication is used by applications, hence the separate blog.
Vault Part 4 - Transferring Authentication Burden to Vault
Vault Part 5 - AppRole Authentication with Vault
Authorization (ACL(Access Control List) Policies)
Vault determines the level of access a user will have with policies.
Vault Part 6 - Vault Authorization: ACL(Access Control List) Policies
Aliases, Entities and Groups are Vault’s way of grouping access control for finer authorization control.
Vault Part 7 - Vault Authorization: Aliases, Entities and Groups
Policy path templating is Vault’s way of creating policies capable of influencing dynamic paths.
Note about HTTP API of Vault
In some actions, UI of Vault could take us only so far and we had to make use of CLI. In a similar manner, some operations cannot be done with the cli as well and we have to make use of the extensive capabilities of Vault Http API. This has been mentioned in other Vault stories in this group but even the CLI communicates with the API so we need to be clear on how it can be used.
Fire up your favorite tool to test out the Http API. Below command simply gets a secret from our t1/secret kv secret engine created from a previous test and a secret in it that we have access to. We have our Vault login token as a header, our http method and the endpoint we use. Policies allow endpoint (path) access right, this is why.
curl -H "X-Vault-Token: s...p" -X GET http://127.0.0.1:8200/v1/t1/secret/data/app_user
Since the JSON return is not prettified (and the secret I entered is a mess), it may not be immediately clear but our secret is within the data bit. Using postman, it’s a bit easier to see.
One more thing to add would be the use of namespaces header which we will come to in a bit but you don’t have to add it. What vault does is add the namespace to the beginning of your requests. So basically like v1/namespace-test1/t1/secret/… for the example above so you can modify your url accordingly instead of a new header. Whichever works for you.
X-Vault-Namespace: namespace-test1The complete documentation is here but I would like to emphasize that the docs have a sample request and , if there is any, a sample payload part on all operations. The ones for KV which we make use of a lot are here.
More on Transit Secrets Engine
We had a brief intro to transit secrets engine in here. However, we did not cover policy requirements for it. Now that we have gone over policies in mid level detail, we can proceed and understand what is happening instead of just memorizing.
With no policy arrangement the below command to encrypt your data will be denied.
vault write transit/encrypt/my-super-secret-encyrption-key plaintext="dGVzdCBlbmNyeXB0IGRhdGE="
You will need to allow below paths in your policies for access to necessary parts by apps or devs. I just listed the basics but there are key rotations, cipher version limiting and other operations available here which should be done by admins.
transit/encrypt/YOURKEY OR * -update
transit/decrypt/YOURKEY OR * -update
transit/keys - list
transit/keys/YOURKEY OR * - readTokens
We have been throwing around tokens left and right but let’s look at tokens in a bit more depth now. This includes batch tokens too.
Response Wrapping
In a previous story about AppRole auth, we gracefully got a roleID and a secretID from an admin(or an adminApp) to be used by our app. It’s essentially our app’s username and password and an admin or an adminApp now technically knows of it.
The admin can get a roleID for our application, attach policies, etc and they can send us (or our app) a wrapped token and our role-id. Our app can ask for the wrapped token instead of asking for a secret-id whenever it needs to use the approle. The adminApp, or an admin if manual, can run the below command to get a wrapped token.
vault write -wrap-ttl=2m -force auth/approle-test/role/testapprole1/secret-id
See how “brave” I am displaying my wrapped token like that. That’s because I (my app) used unwrap to get a secret-id.
VAULT_TOKEN=s.yFdgG7IShU92swgoehtEpiJF vault unwrap
If I were to run the unwrap again using the same wrapped token;
This is how we can remove the burden of knowledge (and the potential of human error) from admins of Vault.
Namespaces
As with k8s and almost everything else, there is a need for multiple projects/teams/versions of an app to coexist in the same cluster. Namespace usage handles this by basically sealing of different bits from each other. This applies to Vault as well.
Each namespace can have it’s own secrets engines, secrets, policies, etc. separated from other namespaces. I would normally say let’s try it out BUT and this is a big but,
there is no namespace selection anywhere and that is because it is a paid feature of Vault Enterprise. This is why namespaces is at the far bottom shunned from the rest since we can’t immediately try it out. There is trial for it as well and it is a very simple feature, I invite everyone to try it out themselves.
Thanks for reading and I hope it was useful to you in some way. See you next time.
