Secure Secret Management and Environment Variables

Hardcoded secrets — API keys, database passwords, TLS private keys, and OAuth tokens embedded directly in source code — are one of the most common and preventable security vulnerabilities. Secrets in source code are exposed in version control history (even after deletion), visible to all developers with repository access, and frequently leaked when repositories are accidentally made public. Tools like GitGuardian and truffleHog continuously scan for leaked secrets on platforms like GitHub.

Environment variables remove secrets from source code by injecting them at runtime through the host OS or container orchestrator. The application reads os.environ['DB_PASSWORD'] rather than a hardcoded value. This is better than hardcoding, but environment variables have weaknesses: they appear in process lists, are inherited by child processes, often end up in crash dumps and debug logs, and require manual rotation. They are appropriate for development but not sufficient alone for production secrets management.

Secrets managers are purpose-built systems for storing, rotating, and auditing access to secrets. Leading solutions include HashiCorp Vault (open source and enterprise), AWS Secrets Manager, Azure Key Vault, and Google Cloud Secret Manager. Applications authenticate to the secrets manager at runtime, retrieve the secret, and use it — no secrets are ever stored on disk or in environment variables. All access is logged, enabling auditing of who accessed which secret and when.

A key advantage of secrets managers over environment variables is automatic rotation. AWS Secrets Manager can automatically rotate RDS database passwords on a schedule (e.g., every 30 days) without requiring application redeployment. The secrets manager updates the password in the database and updates the stored secret simultaneously. Applications that retrieve secrets on each connection automatically receive the new credential. This eliminates the common practice of 'permanent' service account passwords that never rotate.

The first line of defense against committed secrets is a properly maintained .gitignore file that excludes all files that could contain secrets. However, .gitignore only prevents future commits — secrets already committed remain in git history. If secrets are accidentally committed, they must be treated as compromised immediately: rotate the secret, then optionally use tools like git filter-repo to rewrite history (required for compliance but insufficient alone since the secret may already be extracted).

Infrastructure as Code (IaC) files (Terraform, CloudFormation, Kubernetes manifests) frequently contain secrets — database connection strings, API keys in environment variable declarations, and TLS certificates. These files are often committed to version control, creating secret exposure risk. Solutions include Vault dynamic secrets (Vault generates a short-lived credential specifically for each Terraform run), Kubernetes Secrets (stored in etcd, must be encrypted at rest), and external-secrets-operator that syncs from a secrets manager into Kubernetes at runtime.

Each application or service should access only the secrets it specifically requires — the principle of least privilege applied to secrets. A web application needs the database password but not the CA private key. A reporting job needs read-only database credentials, not write access. Secrets managers enforce this through access policies that specify which identities (IAM roles, service accounts, AppRoles) can read which secrets, with all access logged for audit purposes.

Dynamic secrets are generated on-demand for a specific requestor and expire automatically. Vault can generate a temporary database credential that is valid for 1 hour, associated with the specific service that requested it. After expiration, the credential is automatically revoked by the database. This approach means there are no long-lived static credentials to steal — even if an attacker captures a dynamic credential, it expires quickly and is tied to the requesting identity in audit logs.

CI/CD pipelines frequently require secrets — cloud provider credentials for deployment, Docker registry tokens, signing keys. Never store secrets in pipeline scripts or configuration files. Instead, use the pipeline platform's built-in secret store (GitHub Actions Secrets, GitLab CI Variables, Jenkins Credentials Store) or fetch secrets from a central vault at runtime using a machine identity. Mark secret variables as masked in logs to prevent accidental exposure in build output.

Secrets managers provide comprehensive audit logs of every secret access event: which identity accessed which secret, from which IP, at what time, and whether the access was successful or denied. These logs are critical for compliance (SOC 2, PCI-DSS) and incident response. When a credential is suspected of compromise, audit logs reveal which systems accessed it and when — enabling rapid identification of potentially affected systems and containment decisions.

Pre-commit hooks are scripts that run automatically before each git commit is finalized, enabling detection of secrets before they enter version control history. Tools like detect-secrets (Yelp), GitLeaks, and git-secrets (AWS) integrate as pre-commit hooks and scan staged files for patterns matching API keys, connection strings, private keys, and JWT tokens. If a secret is detected, the commit is rejected and the developer is prompted to remove the credential. The pre-commit framework makes it easy to add and share hook configurations across teams.

Test your understanding of CompTIA Security+ (SY0-701) concepts from this lesson.

In this lesson you learned: hardcoded secrets in source code must be eliminated and replaced with secrets managers like Vault or AWS Secrets Manager, automatic rotation removes long-lived credentials that attackers could abuse even after initial compromise, and dynamic secrets and least-privilege access policies minimize the value of any individual secret that is exposed. Next up we explore dependency security and software composition analysis.