Debugging ExternalSecrets

Debugging ExternalSecrets: When ConfigMaps Override Azure Key Vault Secrets in Kubernetes

Recently, I encountered a subtle but important issue in our Kubernetes deployment: secrets from Azure Key Vault weren’t being injected into our application, despite ExternalSecrets being configured correctly. The culprit? ConfigMap values were silently overriding the secrets due to duplicate key names.

This post explores the issue, explains the underlying Kubernetes behavior, and shares best practices to avoid similar problems.

The Problem

In our healthequity-middleware Dev environment:

• ✅ ExternalSecrets Operator was enabled and configured
• ✅ Azure Key Vault integration was working
• ✅ Kubernetes Secrets were being created successfully
• ❌ The application was receiving hardcoded values instead of AKV secrets

The secrets appeared to be configured correctly, but the application never saw them. This was a silent failure—no errors, no warnings, just wrong values.

Understanding Kubernetes ConfigMaps and Secrets

Before diving into the issue, let’s review the basics.

ConfigMaps: Non-Sensitive Configuration

ConfigMaps store configuration data as key-value pairs:

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  DATABASE_HOST: "postgres.example.com"
  LOG_LEVEL: "info"
  API_TIMEOUT: "30s"

Secrets: Sensitive Data

Secrets store sensitive information (base64 encoded):

apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
type: Opaque
data:
  DATABASE_PASSWORD: "cGFzc3dvcmQxMjM="
  API_KEY: "YWJjZGVmZ2hpamts"

Injecting as Environment Variables

Both can be injected into pods using envFrom:

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
      - name: app
        image: myapp:latest
        envFrom:
        - configMapRef:
            name: app-config
        - secretRef:
            name: app-secrets

The Precedence Problem

Here’s where things get tricky. When the same key exists in both a ConfigMap and a Secret, order matters:

envFrom:
- configMapRef:
    name: app-config      # Has API_KEY: "dev-key"
- secretRef:
    name: app-secrets     # Has API_KEY: "prod-secret-key"

Result: The last source wins. In this case, API_KEY gets the value from the Secret.

But if you reverse the order:

envFrom:
- secretRef:
    name: app-secrets     # Has API_KEY: "prod-secret-key"
- configMapRef:
    name: app-config      # Has API_KEY: "dev-key"

Result: API_KEY gets the value from the ConfigMap, completely overriding the secret!

This behavior is by design in Kubernetes, but it’s easy to overlook when managing large configurations.

Enter ExternalSecrets Operator

ExternalSecrets Operator is a powerful tool that syncs secrets from external sources (Azure Key Vault, AWS Secrets Manager, HashiCorp Vault, etc.) into Kubernetes Secrets.

How It Works

flowchart LR
    AKV[Azure Key Vault]
    SS[SecretStore]
    ES[ExternalSecret]
    K8S[Kubernetes Secret]
    POD[Pod]
    
    SS -->|authenticates with| AKV
    ES -->|references| SS
    ES -->|pulls secrets from| AKV
    ES -->|creates/syncs| K8S
    K8S -->|injected as env vars| POD

Step 1: Create a SecretStore

This defines how to authenticate with your secret provider:

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: azure-secretstore
spec:
  provider:
    azurekv:
      authType: WorkloadIdentity
      vaultUrl: https://kv-example.vault.azure.net/

Step 2: Create an ExternalSecret

This defines which secrets to sync:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: app-externalsecret
spec:
  secretStoreRef:
    name: azure-secretstore
  target:
    name: app-secrets
  data:
  - secretKey: API_KEY
    remoteRef:
      key: api-key  # Key name in Azure Key Vault

The ExternalSecrets Operator reads this configuration, fetches secrets from Azure Key Vault, and creates/updates a standard Kubernetes Secret.

Important: The resulting Kubernetes Secret behaves exactly like any manually-created secret, including the precedence rules we discussed earlier.

Our Specific Issue

Here’s what our Dev environment configuration looked like:

ExternalSecret Configuration:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: healthequity-middleware-externalsecret
spec:
  # ... secretStore reference ...
  data:
  - secretKey: MFA_API_CLIENT_ID
    remoteRef:
      key: mfa-api-client-id
  - secretKey: MFA_API_CLIENT_SECRET
    remoteRef:
      key: mfa-api-client-secret
  - secretKey: HEALTH_EQUITY_API_CLIENT_ID
    remoteRef:
      key: he-api-client-id
  # ... more secrets

ConfigMap Configuration (in dev-eus.yaml):

configMap:
  enabled: true
  data:
    # URLs and endpoints - fine
    MFA_API_TOKEN_ENDPOINT: http://stub:3000/oauth2/token
    MFA_API_SCOPE: read:login...
    
    # ⚠️ Problem: Duplicate keys!
    MFA_API_CLIENT_ID: mfa-client
    MFA_API_CLIENT_SECRET: integration-service-secret-key-dev
    HEALTH_EQUITY_API_CLIENT_ID: member-api-client
    HEALTH_EQUITY_API_CLIENT_SECRET: health-equity-secret
    # ...

What Happened

1. ExternalSecrets Operator successfully created a Kubernetes Secret with values from Azure Key Vault
2. Our Helm chart mounted both the Secret and ConfigMap as environment variables
3. The ConfigMap values were processed last (or took precedence), overriding the AKV secrets
4. Application received hardcoded ConfigMap values instead of real secrets

Why It Went Unnoticed

This issue was particularly sneaky because:

• No errors were thrown—both sources provided valid values
• The Dev environment intentionally used stub services with test credentials
• The hardcoded values “worked” for development purposes
• The problem only surfaced when we tried to switch to real AKV secrets

How Our Staging Environment Got It Right

Looking at our Staging configuration revealed the solution:

configMap:
  data:
    # ✅ Only non-sensitive configuration
    MFA_API_TOKEN_ENDPOINT: https://login.microsoftonline.com/...
    MFA_API_SCOPE: api://xxx/.default
    HEALTH_EQUITY_API_HOST: https://api.stg.example.com
    
    # ✅ NO secrets here!
    # CLIENT_ID and CLIENT_SECRET come from ExternalSecret

The secrets came exclusively from ExternalSecrets, with no key overlap in the ConfigMap.

Best Practices

1. Maintain Clear Separation of Concerns

• ConfigMaps: Non-sensitive configuration (URLs, timeouts, feature flags, API scopes)
• Secrets: Sensitive data only (passwords, API keys, tokens, certificates)

2. Never Duplicate Keys

Each environment variable should have exactly one source. If a key exists in a Secret, it should not exist in a ConfigMap.

Use a naming convention to make this obvious:

# Good
configMap:
  API_BASE_URL: "https://api.example.com"
  API_TIMEOUT: "30s"

secret:
  API_KEY: "secret-value"
  API_SECRET: "another-secret"

3. Structure Your Helm Values Clearly

app:
  # Non-sensitive configuration
  config:
    apiUrl: "https://api.example.com"
    timeout: "30s"
  
  # Secrets - reference existing ones from ExternalSecrets
  existingSecrets:
    - app-externalsecret

4. Environment-Specific Strategies

Production/Staging:

• Always use external secret management (ExternalSecrets + AKV/AWS)
• ConfigMaps for non-sensitive config only
• No hardcoded secrets, ever

Development:

• Option A: Use ExternalSecrets with a dev-specific Key Vault (recommended for consistency)
• Option B: Use plain Kubernetes Secrets with test values (simpler for local development)
• Option C: Use ConfigMap with test values only if absolutely necessary (least secure)

5. Review Your envFrom Order

If you must have overlapping keys (though I don’t recommend it), be explicit:

envFrom:
# Secrets should come last to take precedence
- configMapRef:
    name: app-config
- secretRef:
    name: app-secrets  # This wins on conflicts

6. Add Validation

Consider adding an init container to validate critical secrets:

initContainers:
- name: validate-secrets
  image: busybox
  command:
  - sh
  - -c
  - |
    if [ -z "$API_KEY" ]; then
      echo "ERROR: API_KEY not set"
      exit 1
    fi
  envFrom:
  - secretRef:
      name: app-secrets

Debugging Commands

When troubleshooting similar issues:

# Check ExternalSecret sync status
kubectl get externalsecret -n <namespace>
kubectl describe externalsecret <name> -n <namespace>

# Verify the created Secret exists and has data
kubectl get secret <secret-name> -n <namespace> -o yaml

# Check what environment variables the pod actually sees
kubectl exec <pod-name> -n <namespace> -- env | sort

# View ExternalSecrets Operator logs
kubectl logs -n external-secrets-system \
  deployment/external-secrets-operator

The Fix

For our issue, the solution was straightforward:

Remove all sensitive keys from the Dev ConfigMap:

# Before
configMap:
  data:
    MFA_API_CLIENT_ID: mfa-client           # ❌ Remove
    MFA_API_CLIENT_SECRET: xxx              # ❌ Remove
    MFA_API_TOKEN_ENDPOINT: http://...      # ✅ Keep

# After
configMap:
  data:
    MFA_API_TOKEN_ENDPOINT: http://...      # ✅ Keep
    MFA_API_SCOPE: read:login...            # ✅ Keep
    # All secrets now come from ExternalSecret only

Key Takeaways

1. ConfigMaps and Secrets can conflict when they contain the same environment variable names
2. Order matters in envFrom—later sources override earlier ones
3. ExternalSecrets creates standard Kubernetes Secrets, so all normal precedence rules apply
4. Separation of concerns prevents issues—keep secrets in Secrets, config in ConfigMaps
5. Silent failures are dangerous—always validate that your environment variables match expectations
6. Compare your environments—if one works and another doesn’t, diff the configurations

Conclusion

This issue highlights the importance of understanding Kubernetes’ environment variable precedence rules, especially when using tools like ExternalSecrets Operator. While the operator simplifies secret management, it doesn’t change the fundamental behavior of how Kubernetes injects environment variables.

The fix is simple once you understand the problem: maintain strict separation between configuration and secrets, never duplicate keys, and always validate your environment variables in each environment.

Have you encountered similar issues with Kubernetes ConfigMaps and Secrets? I’d love to hear about your experiences in the comments!

Additional Resources:

• ExternalSecrets Operator Documentation
• Kubernetes ConfigMaps Documentation
• Kubernetes Secrets Documentation
• Azure Key Vault Documentation