Preventing S3 Deployment Regressions: A Case Study in Stale Local State and Multi-Environment Safeguards

This post documents a deployment incident on queenofsandiego.com and the hard rules we implemented to prevent similar regressions—particularly when local file state drifts ahead of or behind production S3 buckets. The core failure: deploying a stale local index.html that overwrote newer production assets, wiping three working features in a single cp command.

What Happened: The Regression

A development session deployed to the primary S3 bucket (queenofsandiego.com) using a local index.html that was several commits behind the version live in production. The deploy command targeted both staging and production simultaneously, which violated the single-environment-per-deploy rule already in our session memory.

Three features were lost:

• The JADA → BOOK NOW hero section crossfade animation (working in S3 prod)
• The Stripe embedded checkout booking flow (live and verified in prod)
• Deletion of the "For Ranch & Coast readers…" hero line (intentionally removed in a prior commit, resurrected from stale local)

Root causes:

• No aws s3 cp --dry-run or aws s3 ls diff before writing
• Stale local file state (not pulled from S3 since last local edit)
• Simultaneous staging + prod deploy in one command
• Ignored prior session-summary warnings about stale local files
• No snapshot of prod state before overwrite (S3 versioning disabled on the bucket)

Technical Details: The Deploy Chain

The production deployment flows through this chain:

Local /Users/cb/Documents/repos/sites/queenofsandiego.com/
  ↓ (cp command)
S3 bucket: queenofsandiego.com (us-west-1)
  ↓ (CloudFront invalidation)
CloudFront distribution ID: E1ABC2DEF3GHI (queenofsandiego.com CDN)
  ↓
Route53 CNAME: queenofsandiego.com → d123.cloudfront.net

At each stage, there are points of failure if local state is unknown:

• Before cp: No comparison between local and S3 current state
• During cp: No granular file-by-file change control; a single stale file overwrites multiple working features
• After cp: CloudFront cache invalidation clears the edge quickly, but there's no rollback if the wrong version was pushed

The index.html file serves as a single point of failure because it contains:

• All hero section HTML and CSS (including the JADA crossfade and "Ranch & Coast" line)
• Stripe embedded checkout form initialization
• Navigation and footer layouts

A stale index.html is dangerous because it's not a formatting-only change—it reverts feature logic and removes working integrations.

Infrastructure and Deployment Rules (D1–D8)

We implemented eight hard rules, auto-loaded in /Users/cb/Documents/repos/sites/queenofsandiego.com/CLAUDE.md:

D1: Pull and diff S3 before editing or deploying

aws s3 sync s3://queenofsandiego.com /tmp/s3-backup --dryrun
aws s3 ls s3://queenofsandiego.com --recursive --human-readable

This surfaces what's actually live. Compare the S3 version against your local copy before assuming local is correct.

D2: Single-target, staging-first deploys only

Never deploy to staging and prod in the same command. Deploy to staging first:

# Correct:
aws s3 cp index.html s3://staging.queenofsandiego.com/index.html

# Then separately, after review:
aws s3 cp index.html s3://queenofsandiego.com/index.html

This enforces a review gate and allows rollback of staging without touching prod.

D3: One file per logical change

Don't batch multiple unrelated feature edits into index.html`. Split them:

aws s3 cp s3://queenofsandiego.com/index.html ./index.html.prod-backup-$(date +%s)

DEPLOY PROOF (staging → prod for index.html)
Local SHA: [git log -1 --pretty=%H index.html]
S3 prod SHA: [download and sha256sum from S3]
Backup created: ./index.html.prod-backup-[timestamp]
CloudFront dist: E1ABC2DEF3GHI
Target bucket: queenofsandiego.com (staging OR prod, not both)

- [ ] JADA hero BOOK NOW crossfade (lines 234–289, CSS #jada-fade)
- [ ] Stripe embedded checkout (lines 456–512, Stripe Elements init)
- [ ] "Ranch & Coast" hero line (DELETED, do not restore)
- [ ] Contact form validation (lines 620–680)