```html

Preventing Production Regressions: Hard Rules for S3 Deployments at Scale

Over the last three hours, we discovered and fixed a critical regression on queenofsandiego.com where a stale local index.html overwrote a newer production version in S3, wiping three features: the JADA → BOOK NOW hero crossfade animation, the Stripe embedded checkout booking flow, and accidentally resurrecting a deleted "Ranch & Coast readers..." hero headline. This post documents the root cause, the hard rules we implemented to prevent it, and the infrastructure patterns that made both the regression and the recovery possible.

The Incident: What Happened

The deployment workflow violated two critical assumptions:

• Assumption 1: Before editing /Users/cb/Documents/repos/sites/queenofsandiego.com/index.html, always pull the current version from S3 and diff it against your local copy to detect skew.
• Assumption 2: Deploy to staging first, in isolation, and await review before promoting to prod.

What actually happened: a local index.html—which was 3 commits behind the S3 prod version—was deployed directly to both staging and prod in the same command, overwriting the newer S3 file. The local copy was missing the three features mentioned above because they had been added to S3 in a prior session but never pulled back into the working directory.

Why this matters: S3 is our source-of-truth for production HTML because our CloudFront distribution invalidates on every deploy and serves from S3 directly. Local git repos, while version-controlled, lag behind when changes are made by different agents or sessions. Without a pull-before-edit step, we can lose work silently.

Technical Root Cause

The deployment command was:

aws s3 cp /Users/cb/Documents/repos/sites/queenofsandiego.com/index.html s3://queenofsandiego-web/index.html
aws cloudfront create-invalidation --distribution-id EXAMPLE1D2BUCKET --paths "/*"

This is correct syntax, but it was executed against a stale local file. The missing step before running this was:

aws s3 cp s3://queenofsandiego-web/index.html ./index.html.prod
diff -u index.html index.html.prod | head -50

This would have immediately revealed that the local file was missing the JADA crossfade initialization code (a ~30-line SVG animation handler), the Stripe Session ID injection point (a `