# The Cheapest Resource on Your Bill - Until Idempotency Isn't

_How duplicate secret versions quietly became the biggest line item in our GCP dev account._

June 25, 2026 · 5 min · Amit Jethva

We run our development environment lean. Cloud Run scales to zero, Cloud SQL is on-demand, and when nobody's working the whole stack costs close to nothing. So when we opened the GCP billing breakdown and found the single largest line item was "Secret version replica storage," it didn't add up. Secrets are supposed to be a rounding error.

They're a rounding error *per version*. We had hundreds of them.

## The billing model nobody reads

Google Secret Manager charges **$0.06 per active secret version, per replica location, per month**. Two words there do the damage:

- **Active** - a version bills as long as it's enabled. Superseding it with a newer version does *not* stop the meter. Only destroying it does.
- **Replica** - with automatic replication, Google copies each version across multiple regions, and you pay for each copy.

So your real cost is `(enabled versions) × (replica locations) × $0.06`. Both multipliers had quietly run away on us.

And critically: this charge accrues *continuously*, independent of whether your compute is running. Our dev was "switched off" - but the secrets billed every hour anyway, which is exactly why they floated to the top of an otherwise-idle bill.

## The root cause: a write with no "if changed"

Our deploy pipeline populated secrets with the obvious command:

```bash
printf '%s' "$value" | gcloud secrets versions add "$name" --data-file=-
```

Here's the trap: `gcloud secrets versions add` has no idempotent upsert. Hand it the exact same value it already holds, and it still creates a new, separately-billed version. There's no "skip if unchanged."

Multiply that by a deploy step that ran on every push. After ~90 deploys, the numbers were stark:

```text
96  scanner-azure-client-secret
94  app-bootstrap-admin-password
94  scanner-aws-secret-key
94  scanner-aws-access-key-id
94  scanner-azure-sla-client-secret
```

Five secrets, ~470 versions - and every single one held the same value as the version before it. Around 500 billable version-replicas, of which 484 were pure duplicates. At $0.06 each that's ~$30/month and climbing, for an environment that's idle most of the day - and more once automatic replication multiplies it across regions.

## The fix, in three parts

**1. Stop the bleeding - destroy the duplicates.** Only `latest` is ever read at runtime, so every other enabled version is waste. This keeps the newest and destroys the rest, per secret:

```bash
for s in $(gcloud secrets list --format='value(name.basename())'); do
  gcloud secrets versions list "$s" --filter='state=enabled' \
    --sort-by='~createTime' --format='value(name)' \
  | tail -n +2 \
  | xargs -r -I{} gcloud secrets versions destroy {} --secret="$s" --quiet
done
```

Destroying a non-latest version is safe - it never touches the value your services read. This took us from ~500 enabled versions to 16.

**2. Stop it coming back - make the write idempotent.** Compare against the current value and only add when it actually differs:

```bash
put_secret() {
  local name="$1" value="$2"
  local cur
  cur=$(gcloud secrets versions access latest --secret="$name" 2>/dev/null || true)
  [[ "$cur" == "$value" ]] && { echo "= $name unchanged"; return; }
  printf '%s' "$value" | gcloud secrets versions add "$name" --data-file=-
}
```

Now a deploy with no secret changes adds zero versions.

**3. Cut the replica multiplier.** For non-production, single-region replication bills one replica per version instead of the multi-region set `automatic` picks:

```hcl
replication {
  user_managed {
    replicas { location = var.region }
  }
}
```

(Replication policy is immutable, so this recreates the secret container - re-populate right after. Keep `automatic` in prod where you actually want the DR.)

## Two traps we hit so you don't have to

**The version-destroy TTL is not a safe auto-cleaner.** It looks perfect - "expire old versions automatically" - but the TTL is keyed off each version's creation time and applies to *every* version, including the live one. Set a 2-day TTL and your current credential, if it was created more than 2 days ago, gets scheduled for destruction immediately. It has no concept of "the one in use." For automated hygiene, destroy the previous version right after a rotation instead - that always spares `latest`.

**This is a GCP-specific failure mode.** The same careless re-writes cost nothing on the other major clouds, because their billing models are different:

| Provider            | Secret billing                     | Does version pile-up cost money? |
|---------------------|------------------------------------|----------------------------------|
| GCP Secret Manager  | per active version replica / month | Yes                              |
| AWS Secrets Manager | per secret / month (versions free) | No                               |
| Azure Key Vault     | per transaction (versions free)    | No                               |

A pattern that's invisible on AWS becomes a recurring charge on GCP. Multi-cloud teams especially: a habit that's free in one account silently bills in another.

## We turned the lesson into a guardrail

Finding this once is luck; catching it everywhere is engineering. We've added it to [Fintropy](https://www.nuvikatech.com/Fintropy_Overview.html) as an automated scan rule (`gcp-gov-003`) that flags any Secret Manager secret carrying more enabled versions than it needs, with the exact destroy command and the projected monthly saving attached. The whole point of a FinOps platform is to catch the boring, compounding leaks before they reach an invoice - including the ones in our own backyard.

Secrets really are the cheapest thing on your cloud bill. Just make sure you're storing one of each - not ninety.

---

_Amit Jethva is the CTO and co-founder of Nuvika Technologies Pvt Ltd, makers of [Fintropy](https://www.nuvikatech.com/Fintropy_Overview.html), a multi-cloud FinOps platform. Learn more at [nuvikatech.com](https://www.nuvikatech.com)._