When doing work on many micro services, i have to find a way to manage environment variable for them, previously i used AWS Secret Manager but when working on multiple environment with many account, the effort for manage IAM Account, Role, Permission for cross account is very large, so i switched to Vault, this help me reduce a lot of working on secret management. Now i can centralize all secret in just one Vault server. When i work on many environment i just need to isolate the secret engine path for them. One token with read permission is enough to retrieve the secret on that environment(or account).

In this post i will enable key-vault secret engine for development environment and sync to k8s using https://github.com/external-secrets/external-secrets/

1. Setup Vault server

For demo purpose, i will setup a single node using docker-compose .

❯ tree vault-server 
vault-server
├── docker-compose.yml
└── vault.json
0 directories, 2 files

The content of docker-compose.yml

version: "3.9"
networks:
  default:
    name: learn-vault
    driver: bridge
volumes:
  vaultlog:
  vaultfile:
services:
  vault:
    image: vault:1.12.0
    ports:
      - 8200:8200
    restart: always
    volumes:
      - vaultlog:/vault/logs
      - vaultfile:/vault/file
      - ./vault.json:/vault/config/vault.json
    entrypoint: vault server -config=/vault/config/vault.json

The content of vault.json

{
  "storage": {
    "file": {
      "path": "/vault/file"
    }
  },
  "listener": {
    "tcp": {
      "address": "0.0.0.0:8200",
      "tls_disable": "true"
    }
  },
  "default_lease_ttl": "8640h",
  "max_lease_ttl": "86400h",
  "ui": true,
  "log_level": "Info",
  "disable_mlock": true
}

To start up vault run the up command.

gitops-k8s/vault-server on  master on ☁️  (ap-southeast-1) 
❯ docker compose up -d  
[+] Running 4/4
 ⠿ Network learn-vault              Created                                                                                                                                             0.0s
 ⠿ Volume "vault-server_vaultlog"   Created                                                                                                                                             0.0s
 ⠿ Volume "vault-server_vaultfile"  Created                                                                                                                                             0.0s
 ⠿ Container vault-server-vault-1   Started                                                                                                                                             0.4s

gitops-k8s/vault-server on  master on ☁️  (ap-southeast-1) 
❯

Now, you can login to http://localhost:8200 to start using Vault. For me i already deploy it here https://vault.dongnguyen.link

2. Create Vault Secret Key Vault Engine.

2.1 First you need to login to Vault server using Vault token, you can get it from the vault stdout if this is the first time you start the vault server.

❯ export VAULT_ADDR=https://vault.dongnguyen.link

❯ vault login hvs.XXXXXXXXXXXXX                            
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                hvs.XXXXXXXXXXXXX
token_accessor       YYYYYYYYYYYYYYYYY
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]

2.2 Enable new secret engine for development environment.

❯ vault secrets enable -version=2 --path=development kv
Success! Enabled the kv secrets engine at: development/

2.3 Now on Vault UI, you will see this.

2.4 Create a api secret.

On Terminal run vault cli to create a secret with some value.

❯ vault kv put -mount=development api PORT=30006              
==== Secret Path ====
development/data/api

======= Metadata =======
Key                Value
---                -----
created_time       2023-01-10T04:28:01.729528959Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

In the code above, i create a key-value secret that located at development/api on vault server.

Now, open Vault UI you will see this.

Click to view development/api secret.

2.5. Create Policy and Token to use with K8s.

Run Vault CLI to create it. The content of this HCL file is

path "sys/mounts"
{
  capabilities = ["read"]
}
path "development/*"
{
  capabilities = ["list"]
}
path "development/data/*"
{
  capabilities = ["read"]
}

With this policy, k8s cluster will has permission to list and read all the secret in development key-value engine.

Now create it

❯ vault policy write  development-reader development-reader-policy.hcl 
Success! Uploaded policy: development-reader

On Vault UI, go to the policy tab, you will see this.

Click to development-reader policy to see it.

2.6 Create a token with the development-reader policy.

❯ vault token create -policy=development-reader -period=8640h                         
Key                  Value
---                  -----
token                hvs.CAESIMO8teS1Z8Hl-oUjlvumwwqYnVWt2kQebR2g-ZZPWnehGh4KHGh2cy5XaXZocHp4bk9UYzdnVlpzdk8yWW9xdEg
token_accessor       kM5etWc7DbFwPtg0Ih42N4uO
token_duration       8640h
token_renewable      true
token_policies       ["default" "development-reader"]
identity_policies    []
policies             ["default" "development-reader"]

Use Vaut cli to do it, you can customize the period of token, after that period, token is not valid anymore so you need to recreate it.

Open another terminal, login using this token and try to read the development/api secret. Login then get this secret using Cli.

❯ export VAULT_ADDR=https://vault.dongnguyen.link

❯ vault login hvs.CAESIMO8teS1Z8Hl-oUjlvumwwqYnVWt2kQebR2g-ZZPWnehGh4KHGh2cy5XaXZocHp4bk9UYzdnVlpzdk8yWW9xdEg                            
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                hvs.CAESIMO8teS1Z8Hl-oUjlvumwwqYnVWt2kQebR2g-ZZPWnehGh4KHGh2cy5XaXZocHp4bk9UYzdnVlpzdk8yWW9xdEg
token_accessor       kM5etWc7DbFwPtg0Ih42N4uO
token_duration       8639h58m47s
token_renewable      true
token_policies       ["default" "development-reader"]
identity_policies    []
policies             ["default" "development-reader"]

❯ vault kv get -format=json development/api  | jq ".data.data"
{
  "PORT": "30006"
}

gitops-k8s/vault-server on  master [?] on ☁️  (ap-southeast-1) 
❯ 

As you can see, we can able to read api secret. Now it time to sync it to k8s using the token above.

3. Install External Secret Operator on K8s.

3.1 Install using Helm

helm repo add external-secrets https://charts.external-secrets.io

helm install external-secrets \
   external-secrets/external-secrets \
    -n external-secrets \
    --create-namespace \
    --set installCRDs=true

4. Sync secret

4.1 Create a secret on k8s to store vault token for development environment. First we need to get the base64 encoded vault for vault token.

Then create a secret to store vault token in external-secrets namespace.

apiVersion: v1
kind: Secret
metadata:
  name: development-vault-token
  namespace: external-secrets
type: Opaque
data:
  token: aHZzLkNBRVNJTU84dGVTMVo4SGwtb1VqbHZ1bXd3cVluVld0MmtRZWJSMmctWlpQV25laEdoNEtIR2gyY3k1WGFYWm9jSHA0Yms5VVl6ZG5WbHB6ZGs4eVdXOXhkRWcK

Now open K8S Dashboard, you will see it.

4.2 Create a ClusterSecretStore.

The content of vault-cluster-secret-store.yml is same as below.

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "https://vault.dongnguyen.link"
      path: "development"
      version: "v2"
      auth:
        tokenSecretRef:
          name: "development-vault-token"
          key: "token"
          namespace: external-secrets

Run the command below

kubectl create -f vault-cluster-secret-store.yml

If you open k8s dashboard, in the CRD session, you will see it.

4.3 Sync development/api secret to development namespace.

Use kubectl to create a secret same as below. Before you you need to create namespace development .

The development-api.yml file content

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: api
  namespace: development
spec:
  refreshInterval: "15s"
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: api
  dataFrom:
    - extract:
        key: development/api

Now create a external secret , this secret is refer to development/api secret on Vault server and map it to api secret in development namespace on K8s cluster.

Wait some second, now you can check it in development namespace.

As you can see, we have already able to sync secret from vault to k8s, if you want to sync more, you can create secret from vault and creating more external-secret resource on k8s.

That is all, thank for reading !