Designing a Secrets Management System

A secrets management system securely stores, distributes, and rotates credentials, API keys, and certificates.

Problems Without Centralized Secrets

Secrets in code → Git repo leaks database password
Secrets in .env → Deployed to container, easily extracted
No rotation    → Breach undetected for months
No auditing    → Who accessed production DB yesterday?

Architecture

Applications
    ↓ (authenticated request)
Vault / Secrets Service
    ↓
Encrypted Secret Store (DB)
    ↓
HSM (Hardware Security Module) — protects master key

Secret Storage

Master Key → Stored in HSM
Secret: { key: "db_password", value: encrypt("s3cr3t", master_key) }

No plaintext secrets ever stored

Authentication Methods

Application Identity:
  K8s ServiceAccount token
  AWS IAM role
  AppRole (username + secret_id)
  LDAP / OIDC for humans

Dynamic Secrets

Generate credentials on-demand with short TTL:

App requests DB credentials:
  Vault creates PostgreSQL user: vault_user_abc
  Grants SELECT on tables
  Returns: { username: "vault_user_abc", password: "xyz", ttl: "1h" }

After 1 hour:
  Vault revokes user automatically
  App requests fresh credentials

No long-lived credentials!

Secret Rotation

Static secrets rotated automatically:
  1. Generate new password
  2. Update in external service (DB, API)
  3. Update in Vault
  4. Old version retained briefly
  5. Notify applications to refresh

Audit Logging

Every secret access logged:

{
  "time": "2026-03-24T10:00:00Z",
  "auth": "kubernetes/service-account/order-service",
  "operation": "read",
  "path": "secret/database/production"
}

PKI (Certificate Management)

Vault as internal CA:
  Issue TLS certificates for services
  Short TTL (24 hours) → Auto-rotate
  mTLS between microservices

Conclusion

Dynamic secrets with short TTL and automatic rotation dramatically reduce the blast radius of any credential compromise. HashiCorp Vault is the industry standard.