Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

```html

Preventing S3 Deploy Regressions: Hard Rules for Multi-Environment CloudFront Invalidation

Over the last 3 hours, a deployment session regressed three critical features on queenofsandiego.com by pushing a stale local index.html to production S3, wiping the hero JADA→BOOK NOW crossfade animation, the Stripe embedded checkout flow, and resurrecting a previously-deleted "For Ranch & Coast readers..." hero line. The root cause: deploying without first diffing local against S3-current, and violating the staging-first rule by deploying staging and prod in a single command.

This post documents the eight hard rules now embedded in our deployment workflow to prevent this class of regression, with exact tooling, file paths, and validation steps.

What Went Wrong

  • Stale Local State: The local checkout at /Users/cb/Documents/repos/sites/queenofsandiego.com/index.html was older than the version already live in S3 bucket queenofsandiego-com-prod.
  • No Pre-Deploy Diff: The session did not run aws s3 cp s3://queenofsandiego-com-prod/index.html ./index.html.prod-current followed by diff -u index.html.prod-current index.html before editing.
  • Dual-Environment Deploy: Both staging and production CloudFront distributions (IDs: E1ABC2DEF3GHI for staging, E4JKL5MNO6PQR for prod) were invalidated in sequence without a human staging review gate between them.
  • Ignored Prior Warning: The previous session summary explicitly flagged "stale local files — pull S3 before editing," but this was overridden.

The Eight Hard Rules (D1–D8)

Rule D1: Pull-S3-and-Diff Before Any Edit

cd /Users/cb/Documents/repos/sites/queenofsandiego.com
aws s3 cp s3://queenofsandiego-com-prod/index.html ./index.html.prod-current
diff -u index.html.prod-current index.html > /tmp/qos-diff.txt
cat /tmp/qos-diff.txt

If the diff shows unexpected deletions (hero section, animation classes, Stripe script tags), abort and ask CB. Do not edit until you understand what's newer in S3.

Rule D2: One Logical Change Per Deploy; No Staging+Prod in One Command

A "logical change" is one feature or fix. Each gets its own aws s3 cp → CloudFront invalidation cycle. Deploy to staging only first. Wait for the CloudFront invalidation to complete (typically 30–90 seconds). Test on staging.queenofsandiego.com. Only then promote to production.

Rule D3: One File Per Logical Change; No Bulk Uploads

Bad: aws s3 cp ./ s3://queenofsandiego-com-prod --recursive

Good: aws s3 cp index.html s3://queenofsandiego-com-prod/index.html

Single-file uploads are atomic, auditable, and reversible. Bulk uploads hide regressions.

Rule D4: Obey Prior Session Summaries and Warnings

Every session ends with a summary that flags known issues, stale state, and blockers. Read it. If it says "local index.html is 2 hours old," treat it as a halt condition until you've pulled S3 and compared.

Rule D5: Snapshot Production Before Overwriting

aws s3 cp s3://queenofsandiego-com-prod/index.html ./snapshots/index.html.prod.$(date +%s)
git add snapshots/
git commit -m "Pre-deploy snapshot: index.html from prod S3"

S3 versioning is not enabled on these buckets. A snapshot in git is your only rollback. Commit it before any cp that overwrites.

Rule D6: Print a Six-Line Proof Block Before Any S3 Copy

echo "=== PRE-DEPLOY PROOF ==="
echo "Source: index.html (local)"
echo "Destination: s3://queenofsandiego-com-prod/index.html"
echo "Staging dist: E1ABC2DEF3GHI (will NOT be invalidated yet)"
echo "Prod dist: E4JKL5MNO6PQR (will NOT be invalidated yet)"
echo "Lines in source: $(wc -l < index.html)"
echo "=== END PROOF ==="

Paste this in chat and ask CB to confirm before running the cp command. This is your human gate.

Rule D7: Maintain a Feature-Token Registry in S3

A feature token is a unique string that marks the presence of a feature. Examples:

  • Hero JADA→BOOK NOW fade: id="hero-jada-crossfade"
  • Stripe embedded checkout: data-stripe-publishable-key=
  • Ranch & Coast hero line: For Ranch & Coast readers

Before and after every prod deploy, grep S3-current for these tokens:

aws s3 cp s3://queenofsandiego-com-prod/index.html /tmp/qos-prod.html
grep -c 'id="hero-jada-crossfade"' /tmp/qos-prod.html
grep -c 'data-stripe-publishable-key=' /tmp/qos-prod.html
grep -c 'For Ranch & Coast readers' /tmp/qos-prod.html

If any count drops to zero after your deploy, that's a regression. Revert immediately.

Rule D8: Escalate to CB if S3 is Ahead of Local

If the diff in Rule D1 shows changes in S3 that you don't have locally, do not merge, edit, or deploy. Message CB with the diff and ask: "S3 is newer; should I git pull, or did you deploy this in another session?" Do not assume.

Infrastructure Context

  • S3 Buckets: queenofsandiego-com-staging (website root), queenofsandiego-com-prod (production website).
  • CloudFront Distributions: Staging alias E1ABC2DEF3GHI, Prod alias E4JKL5MNO6PQR. Both point to their respective S3 bucket origins.
  • Route53 Zone: queenofsandiego.com (hosted zone). CNAME records route staging.qu