The Cheapest Resource on Your Bill - Until Idempotency Isn’t

We run our development environment lean. Cloud Run scales to zero, Cloud SQL is on-demand, and when nobody’s working the whole stack costs close to nothing. So when we opened the GCP billing breakdown and found the single largest line item was “Secret version replica storage,” it didn’t add up. Secrets are supposed to be a rounding error.

They’re a rounding error per version. We had hundreds of them.

The billing model nobody reads

Google Secret Manager charges $0.06 per active secret version, per replica location, per month. Two words there do the damage:

• Active - a version bills as long as it’s enabled. Superseding it with a newer version does not stop the meter. Only destroying it does.
• Replica - with automatic replication, Google copies each version across multiple regions, and you pay for each copy.

So your real cost is (enabled versions) × (replica locations) × $0.06. Both multipliers had quietly run away on us.

And critically: this charge accrues continuously, independent of whether your compute is running. Our dev was “switched off” - but the secrets billed every hour anyway, which is exactly why they floated to the top of an otherwise-idle bill.

The root cause: a write with no “if changed”

Our deploy pipeline populated secrets with the obvious command:

printf '%s' "$value" | gcloud secrets versions add "$name" --data-file=-

Here’s the trap: gcloud secrets versions add has no idempotent upsert. Hand it the exact same value it already holds, and it still creates a new, separately-billed version. There’s no “skip if unchanged.”

Multiply that by a deploy step that ran on every push. After ~90 deploys, the numbers were stark:

96  scanner-azure-client-secret
94  app-bootstrap-admin-password
94  scanner-aws-secret-key
94  scanner-aws-access-key-id
94  scanner-azure-sla-client-secret

Five secrets, ~470 versions - and every single one held the same value as the version before it. Around 500 billable version-replicas, of which 484 were pure duplicates. At $0.06 each that’s ~$30/month and climbing, for an environment that’s idle most of the day - and more once automatic replication multiplies it across regions.

The fix, in three parts

1. Stop the bleeding - destroy the duplicates. Only latest is ever read at runtime, so every other enabled version is waste. This keeps the newest and destroys the rest, per secret:

for s in $(gcloud secrets list --format='value(name.basename())'); do
  gcloud secrets versions list "$s" --filter='state=enabled' \
    --sort-by='~createTime' --format='value(name)' \
  | tail -n +2 \
  | xargs -r -I{} gcloud secrets versions destroy {} --secret="$s" --quiet
done

Destroying a non-latest version is safe - it never touches the value your services read. This took us from ~500 enabled versions to 16.

2. Stop it coming back - make the write idempotent. Compare against the current value and only add when it actually differs:

put_secret() {
  local name="$1" value="$2"
  local cur
  cur=$(gcloud secrets versions access latest --secret="$name" 2>/dev/null || true)
  [[ "$cur" == "$value" ]] && { echo "= $name unchanged"; return; }
  printf '%s' "$value" | gcloud secrets versions add "$name" --data-file=-
}

Now a deploy with no secret changes adds zero versions.

3. Cut the replica multiplier. For non-production, single-region replication bills one replica per version instead of the multi-region set automatic picks:

replication {
  user_managed {
    replicas { location = var.region }
  }
}

(Replication policy is immutable, so this recreates the secret container - re-populate right after. Keep automatic in prod where you actually want the DR.)

Two traps we hit so you don’t have to

The version-destroy TTL is not a safe auto-cleaner. It looks perfect - “expire old versions automatically” - but the TTL is keyed off each version’s creation time and applies to every version, including the live one. Set a 2-day TTL and your current credential, if it was created more than 2 days ago, gets scheduled for destruction immediately. It has no concept of “the one in use.” For automated hygiene, destroy the previous version right after a rotation instead - that always spares latest.

This is a GCP-specific failure mode. The same careless re-writes cost nothing on the other major clouds, because their billing models are different:

A pattern that’s invisible on AWS becomes a recurring charge on GCP. Multi-cloud teams especially: a habit that’s free in one account silently bills in another.

We turned the lesson into a guardrail

Finding this once is luck; catching it everywhere is engineering. We’ve added it to Fintropy as an automated scan rule (gcp-gov-003) that flags any Secret Manager secret carrying more enabled versions than it needs, with the exact destroy command and the projected monthly saving attached. The whole point of a FinOps platform is to catch the boring, compounding leaks before they reach an invoice - including the ones in our own backyard.

Secrets really are the cheapest thing on your cloud bill. Just make sure you’re storing one of each - not ninety.

Amit Jethva is the CTO and co-founder of Nuvika Technologies Pvt Ltd, makers of Fintropy, a multi-cloud FinOps platform. Learn more at nuvikatech.com.