When doing work on many micro services, i have to find a way to manage environment variable for them, previously i used AWS Secret Manager but when working on multiple environment with many account, the effort for manage IAM Account, Role, Permission for cross account is very large, so i switched to Vault, this help me reduce a lot of working on secret management. Now i can centralize all secret in just one Vault server. When i work on many environment i just need to isolate the secret engine path for them. One token with read permission is enough to retrieve the secret on that environment(or account).
In this post i will enable key-vault secret engine for development environment and sync to k8s using https://github.com/external-secrets/external-secrets/
1. Setup Vault server
For demo purpose, i will setup a single node using docker-compose .
❯ tree vault-server
vault-server
├── docker-compose.yml
└── vault.json
0 directories, 2 filesThe content of docker-compose.yml
version: "3.9"
networks:
default:
name: learn-vault
driver: bridge
volumes:
vaultlog:
vaultfile:
services:
vault:
image: vault:1.12.0
ports:
- 8200:8200
restart: always
volumes:
- vaultlog:/vault/logs
- vaultfile:/vault/file
- ./vault.json:/vault/config/vault.json
entrypoint: vault server -config=/vault/config/vault.jsonThe content of vault.json
{
"storage": {
"file": {
"path": "/vault/file"
}
},
"listener": {
"tcp": {
"address": "0.0.0.0:8200",
"tls_disable": "true"
}
},
"default_lease_ttl": "8640h",
"max_lease_ttl": "86400h",
"ui": true,
"log_level": "Info",
"disable_mlock": true
}To start up vault run the up command.
gitops-k8s/vault-server on master on ☁️ (ap-southeast-1)
❯ docker compose up -d
[+] Running 4/4
⠿ Network learn-vault Created 0.0s
⠿ Volume "vault-server_vaultlog" Created 0.0s
⠿ Volume "vault-server_vaultfile" Created 0.0s
⠿ Container vault-server-vault-1 Started 0.4s
gitops-k8s/vault-server on master on ☁️ (ap-southeast-1)
❯Now, you can login to http://localhost:8200 to start using Vault. For me i already deploy it here https://vault.dongnguyen.link
2. Create Vault Secret Key Vault Engine.
2.1 First you need to login to Vault server using Vault token, you can get it from the vault stdout if this is the first time you start the vault server.
❯ export VAULT_ADDR=https://vault.dongnguyen.link
❯ vault login hvs.XXXXXXXXXXXXX
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.
Key Value
--- -----
token hvs.XXXXXXXXXXXXX
token_accessor YYYYYYYYYYYYYYYYY
token_duration ∞
token_renewable false
token_policies ["root"]
identity_policies []
policies ["root"]
2.2 Enable new secret engine for development environment.
❯ vault secrets enable -version=2 --path=development kv
Success! Enabled the kv secrets engine at: development/2.3 Now on Vault UI, you will see this.
2.4 Create a api secret.
On Terminal run vault cli to create a secret with some value.
❯ vault kv put -mount=development api PORT=30006
==== Secret Path ====
development/data/api
======= Metadata =======
Key Value
--- -----
created_time 2023-01-10T04:28:01.729528959Z
custom_metadata <nil>
deletion_time n/a
destroyed false
version 1In the code above, i create a key-value secret that located at development/api on vault server.
Now, open Vault UI you will see this.
Click to view development/api secret.
2.5. Create Policy and Token to use with K8s.
Run Vault CLI to create it. The content of this HCL file is
path "sys/mounts"
{
capabilities = ["read"]
}
path "development/*"
{
capabilities = ["list"]
}
path "development/data/*"
{
capabilities = ["read"]
}With this policy, k8s cluster will has permission to list and read all the secret in development key-value engine.
Now create it
❯ vault policy write development-reader development-reader-policy.hcl
Success! Uploaded policy: development-reader
On Vault UI, go to the policy tab, you will see this.
Click to development-reader policy to see it.
2.6 Create a token with the development-reader policy.
❯ vault token create -policy=development-reader -period=8640h
Key Value
--- -----
token hvs.CAESIMO8teS1Z8Hl-oUjlvumwwqYnVWt2kQebR2g-ZZPWnehGh4KHGh2cy5XaXZocHp4bk9UYzdnVlpzdk8yWW9xdEg
token_accessor kM5etWc7DbFwPtg0Ih42N4uO
token_duration 8640h
token_renewable true
token_policies ["default" "development-reader"]
identity_policies []
policies ["default" "development-reader"]Use Vaut cli to do it, you can customize the period of token, after that period, token is not valid anymore so you need to recreate it.
Open another terminal, login using this token and try to read the development/api secret. Login then get this secret using Cli.
❯ export VAULT_ADDR=https://vault.dongnguyen.link
❯ vault login hvs.CAESIMO8teS1Z8Hl-oUjlvumwwqYnVWt2kQebR2g-ZZPWnehGh4KHGh2cy5XaXZocHp4bk9UYzdnVlpzdk8yWW9xdEg
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.
Key Value
--- -----
token hvs.CAESIMO8teS1Z8Hl-oUjlvumwwqYnVWt2kQebR2g-ZZPWnehGh4KHGh2cy5XaXZocHp4bk9UYzdnVlpzdk8yWW9xdEg
token_accessor kM5etWc7DbFwPtg0Ih42N4uO
token_duration 8639h58m47s
token_renewable true
token_policies ["default" "development-reader"]
identity_policies []
policies ["default" "development-reader"]
❯ vault kv get -format=json development/api | jq ".data.data"
{
"PORT": "30006"
}
gitops-k8s/vault-server on master [?] on ☁️ (ap-southeast-1)
❯
As you can see, we can able to read api secret. Now it time to sync it to k8s using the token above.
3. Install External Secret Operator on K8s.
3.1 Install using Helm
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets \
external-secrets/external-secrets \
-n external-secrets \
--create-namespace \
--set installCRDs=true
4. Sync secret
4.1 Create a secret on k8s to store vault token for development environment. First we need to get the base64 encoded vault for vault token.
Then create a secret to store vault token in external-secrets namespace.
apiVersion: v1
kind: Secret
metadata:
name: development-vault-token
namespace: external-secrets
type: Opaque
data:
token: aHZzLkNBRVNJTU84dGVTMVo4SGwtb1VqbHZ1bXd3cVluVld0MmtRZWJSMmctWlpQV25laEdoNEtIR2gyY3k1WGFYWm9jSHA0Yms5VVl6ZG5WbHB6ZGs4eVdXOXhkRWcK
Now open K8S Dashboard, you will see it.
4.2 Create a ClusterSecretStore.
The content of vault-cluster-secret-store.yml is same as below.
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
name: vault-backend
spec:
provider:
vault:
server: "https://vault.dongnguyen.link"
path: "development"
version: "v2"
auth:
tokenSecretRef:
name: "development-vault-token"
key: "token"
namespace: external-secretsRun the command below
kubectl create -f vault-cluster-secret-store.yml
If you open k8s dashboard, in the CRD session, you will see it.
4.3 Sync development/api secret to development namespace.
Use kubectl to create a secret same as below. Before you you need to create namespace development .
The development-api.yml file content
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: api
namespace: development
spec:
refreshInterval: "15s"
secretStoreRef:
name: vault-backend
kind: ClusterSecretStore
target:
name: api
dataFrom:
- extract:
key: development/apiNow create a external secret , this secret is refer to development/api secret on Vault server and map it to api secret in development namespace on K8s cluster.
Wait some second, now you can check it in development namespace.
As you can see, we have already able to sync secret from vault to k8s, if you want to sync more, you can create secret from vault and creating more external-secret resource on k8s.
That is all, thank for reading !