Secrets Management¶

Status: 🟢 Active  |  Owner: Security Engineering  |  Last Reviewed: 2025-Q4

Overview¶

A secret is any value that grants access to a system, proves identity, or enables decryption — API keys, database passwords, TLS private keys, OAuth client secrets, encryption keys, and service account tokens. Secrets are among the most targeted artefacts in a breach. Their mismanagement — hardcoding them in source code, storing them in environment variable files, logging them accidentally, or failing to rotate them — is one of the most common root causes of security incidents.

This page defines how secrets must be managed across the full lifecycle: creation, storage, injection, rotation, and revocation.

The governing rule is simple: secrets never appear in source code, container images, CI logs, or any artefact that is stored alongside the systems they protect.

Approved Secrets Management: HashiCorp Vault¶

HashiCorp Vault (deployed as HCP Vault SaaS) is the organisation's approved secrets store for all secrets used by running services.

All production secrets — database credentials, API keys, encryption keys, service account tokens, TLS certificates — must be stored in Vault and retrieved at runtime. No exceptions without Security Engineering sign-off.

Vault is chosen because it provides: a unified API across all secret types, dynamic secrets with automatic expiry, fine-grained access policies per service, complete audit logging of every secret access, and automatic key rotation.

Vault Architecture¶

                         ┌─────────────────────┐
                         │   HCP Vault (SaaS)  │
                         │                     │
                         │  secret/dev/        │
                         │  secret/staging/    │
                         │  secret/prod/       │
                         │                     │
                         │  pki/               │  ← Dynamic TLS certs
                         │  database/          │  ← Dynamic DB credentials
                         └────────┬────────────┘
                                  │ Vault API (HTTPS)
              ┌───────────────────┼────────────────────┐
              ▼                   ▼                    ▼
    ┌──────────────────┐  ┌──────────────┐   ┌─────────────────┐
    │  Service Pod     │  │  CI Pipeline │   │  Developer      │
    │  (Vault Agent    │  │  (Vault CLI  │   │  Workstation    │
    │   sidecar)       │  │   + OIDC)    │   │  (vault CLI)    │
    └──────────────────┘  └──────────────┘   └─────────────────┘

Secret Path Conventions¶

All secrets are namespaced by environment and service to enforce least-privilege access:

secret/data/<environment>/<service-name>/<secret-name>

# Examples
secret/data/prod/payment-service/db-password
secret/data/prod/payment-service/stripe-api-key
secret/data/staging/payment-service/db-password
secret/data/dev/payment-service/db-password

A service's Vault policy grants read access only to its own namespace. The payment service can read secret/data/prod/payment-service/* and nothing else.

Runtime Secret Injection¶

Kubernetes: Vault Agent Sidecar¶

The recommended pattern for Kubernetes workloads is the Vault Agent sidecar injector. The Vault Agent authenticates to Vault using the pod's Kubernetes ServiceAccount token, retrieves secrets, writes them to a shared in-memory volume, and keeps them refreshed. Application code reads secrets from files — it never calls the Vault API directly.

# Deployment annotation — Vault injects secrets automatically
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "payment-service-prod"
        # Inject the DB password as a file
        vault.hashicorp.com/agent-inject-secret-db-password: >
          secret/data/prod/payment-service/db-password
        vault.hashicorp.com/agent-inject-template-db-password: |
          {{- with secret "secret/data/prod/payment-service/db-password" -}}
          {{ .Data.data.password }}
          {{- end }}
    spec:
      serviceAccountName: payment-service
      containers:
        - name: payment-service
          env:
            # Application reads from file, not from env var
            - name: DB_PASSWORD_FILE
              value: /vault/secrets/db-password

Application code reads the secret from the file:

// Read secret from Vault-injected file (auto-refreshed by Vault Agent)
String dbPassword = Files.readString(Path.of(System.getenv("DB_PASSWORD_FILE"))).strip();

import os
with open(os.environ["DB_PASSWORD_FILE"]) as f:
    db_password = f.read().strip()

Why files and not environment variables? Environment variables are readable by any process in the container, appear in process listings, and are captured in crash dumps. Files on a tmpfs volume are accessible only to processes that mount the volume, not persisted to disk, and more easily revoked.

Dynamic Database Credentials¶

For database access, Vault's dynamic secrets engine generates short-lived, unique credentials per service instance rather than sharing a static password. Each pod gets a unique username and password that expires when the lease expires (default: 1 hour, max: 24 hours). Vault rotates them automatically.

vault.hashicorp.com/agent-inject-secret-db-creds: >
  database/creds/payment-service-prod-role
vault.hashicorp.com/agent-inject-template-db-creds: |
  {{- with secret "database/creds/payment-service-prod-role" -}}
  username={{ .Data.username }}
  password={{ .Data.password }}
  {{- end }}

Benefits: if a credential is leaked, it expires automatically; every access is tied to a specific lease ID in the Vault audit log; revocation is immediate via lease revocation.

CI/CD Pipelines¶

CI pipelines authenticate to Vault using GitHub Actions OIDC — the pipeline proves its identity via a signed JWT from GitHub, not via a stored secret. This avoids the "secret to get secrets" problem.

# GitHub Actions — Vault OIDC authentication
- name: Retrieve secrets from Vault
  uses: hashicorp/vault-action@v3
  id: vault
  with:
    url: https://vault.example.com
    method: jwt
    role: github-ci-payment-service
    jwtGithubAudience: https://github.com/your-org
    secrets: |
      secret/data/ci/payment-service registry_token | REGISTRY_TOKEN ;
      secret/data/ci/payment-service snyk_token     | SNYK_TOKEN

- name: Build image
  run: docker build .
  env:
    REGISTRY_TOKEN: ${{ steps.vault.outputs.REGISTRY_TOKEN }}

Pipeline secrets retrieved from Vault are masked in CI logs automatically by the Vault Action.

Developer Local Development¶

For local development, engineers use one of the following approved patterns. Real production secrets must never be used in local development environments.

Option 1: Vault Dev Server (Recommended)¶

# Start local Vault in dev mode
vault server -dev -dev-root-token-id=dev-root-token &

export VAULT_ADDR=http://127.0.0.1:8200
export VAULT_TOKEN=dev-root-token

# Seed with development secrets (from the team's shared dev seed script)
vault kv put secret/dev/payment-service/db-password password=localdev123

Option 2: direnv + 1Password CLI¶

# .envrc — secrets fetched at shell entry, not stored in files
export DB_PASSWORD=$(op item get "payment-service-dev" --field password)
export STRIPE_API_KEY=$(op item get "stripe-dev-keys" --field api_key)

Option 3: SOPS-Encrypted .env Files¶

Encrypted .env.dev.enc files are committed to the repository. Engineers decrypt locally using their age key. The age private key is distributed out-of-band during onboarding.

# Decrypt for local use (never commit the decrypted file)
sops -d .env.dev.enc > .env.local

What Is Prohibited¶

The following practices are prohibited. Each has been the root cause of real security incidents.

Pre-commit and CI Detection¶

Secret detection is enforced at two stages to catch accidental violations before they become breaches:

Pre-commit (Stage 3 of the SDL): Gitleaks scans every staged change before commit. Detected secrets block the commit.

CI pipeline (Stage 4): Both Gitleaks and TruffleHog scan the full commit history on every pull request — not just the head commit. This catches secrets introduced in earlier commits that were amended but remain in the git object store.

If a secret is detected after it has been pushed, treat it as compromised and rotate immediately — do not wait to confirm whether it was accessed.

Secret Rotation Policy¶

Secrets approaching their rotation deadline generate a Jira ticket assigned to the owning team automatically via Vault's lease notification webhooks. Overdue rotations are escalated to Security Engineering.

Audit and Compliance¶

Every Vault secret access is recorded in the Vault audit log with: timestamp, operation (read/write/delete), path, authentication method, requesting entity (service account, CI job, engineer), and success/failure. Vault audit logs are shipped to the centralised security log stream with 2-year retention to satisfy SOC 2 audit evidence requirements.

Quarterly, Security Engineering reviews: - Secrets not accessed within 90 days (candidates for deletion) - Services with policy grants broader than their declared dependencies - Static credentials that should be migrated to dynamic Vault secrets

References¶

• Secure Development Lifecycle — Stage 2 & 3 — secret detection on developer machines and in pre-commit hooks
• Authentication & Authorization — service-to-service auth using mTLS
• Vulnerability Management — response when a secret is confirmed compromised
• Local Development — Secrets — detailed local setup guides
• HashiCorp Vault documentation

Last reviewed: 2025-Q4  |  Owner: Security Engineering