Preventing Production Regressions: Hard Rules for S3-Backed Static Sites

Over the last three hours, a deployment session to queenofsandiego.com inadvertently reverted three independent features by deploying a stale local index.html over the production S3 bucket. This post documents the failure modes, the architectural safeguards we've now implemented, and the hard rules that will prevent this class of regression in future sessions.

The Incident: What Went Wrong

The deployment pipeline for queenofsandiego.com is straightforward: local edits to /Users/cb/Documents/repos/sites/queenofsandiego.com/index.html are tested locally, pushed to a staging S3 bucket for QA, and then promoted to production. During a recent session, an agent deployed the local index.html directly to both staging and production buckets in a single command, unaware that the local file was stale compared to what was already live in production.

This single action wiped three independent features that had been working in production:

The JADA → BOOK NOW hero image crossfade animation (CSS/JavaScript state management in the hero section)
The Stripe embedded checkout booking flow integration (script initialization and form binding)
A previously-removed "For Ranch & Coast readers..." hero callout line that had been intentionally deleted in an earlier session

The core problem: the agent did not pull the current production state from S3 before making edits, did not diff the local against remote, and did not observe its own prior session-summary warning about stale local files.

Root Cause Analysis

Three decision failures compounded:

No pre-edit pull. The agent edited index.html without first running aws s3 cp s3://queenofsandiego-prod/index.html ./index.html to sync the current production state locally.
No diff before deploy. Even after editing, no diff or side-by-side comparison was run against the remote production version to identify what had changed.
Dual-target single command. Both staging and production were deployed in one command (aws s3 cp ./index.html s3://queenofsandiego-prod/ && aws s3 cp ./index.html s3://queenofsandiego-staging/), skipping the mandatory staging-first validation gate.

Additionally, the agent ignored a session-summary note from its own prior context warning that local files might be stale relative to S3. This is the most dangerous failure mode: the system had the right information but did not act on it.

The Hard Rules: Eight Safeguards

To prevent future regressions of this type, we have codified eight mandatory rules into the project's context files. These rules are now auto-loaded into every session targeting queenofsandiego.com:

Rule D1: Pull S3 Production Before Any Edit

Before modifying any file that will be deployed to production, pull the current state:

aws s3 cp s3://queenofsandiego-prod/index.html ./index.html --region us-west-2

This ensures local state mirrors production before you introduce changes. Store the pulled version in a temporary location if you need to preserve local edits for comparison.

Rule D2: Diff Local Against Remote Before Deploying

Never deploy without understanding what changed:

diff -u <(aws s3 cp s3://queenofsandiego-prod/index.html - --region us-west-2) ./index.html

Or for a cleaner view, pull to a temp file and diff:

aws s3 cp s3://queenofsandiego-prod/index.html ./index.html.prod --region us-west-2
diff -u ./index.html.prod ./index.html | head -100

The diff output must be reviewed in chat before any S3 upload command runs. If the diff shows unexpected deletions (like missing hero animations or form bindings), stop and investigate.

Rule D3: Deploy to Staging Only, Then Validate

Deployments are always single-target and staging-first:

aws s3 cp ./index.html s3://queenofsandiego-staging/index.html --region us-west-2

After staging deploys, CloudFront cache must be invalidated for the staging distribution (invalidation ID: queenofsandiego-staging-cf-dist) and the site must be manually tested. Only after staging validation passes do you deploy to production.

Rule D4: Obey Prior Session Warnings

If a prior session (visible in /Users/cb/Documents/repos/sites/queenofsandiego.com/CLAUDE.md) notes that local files are stale or out of sync with S3, treat that as a blocking condition. Pull and diff before proceeding.

Rule D5: Snapshot Production Before Overwrite

The S3 bucket queenofsandiego-prod does not have versioning enabled. Before any overwrite, manually save the current production file to a timestamped backup:

aws s3 cp s3://queenofsandiego-prod/index.html ./backups/index.html.backup.$(date +%s) --region us-west-2

This gives us a manual recovery path if a deployment introduces a regression.

Rule D6: Print a Six-Line Proof Block Before Copy

Before running any aws s3 cp command to production, print a proof block in chat showing:

--- DEPLOYMENT PROOF BLOCK ---
SOURCE: ./index.html
TARGET BUCKET: queenofsandiego-prod
TARGET REGION: us-west-2
CLOUDFRONT DIST: queenofsandiego-prod-cf-dist
CHANGES: [one-line summary of what changed]
VALIDATED: [yes/no — staging tested?]
---

This forces explicit acknowledgment of target and scope before the command runs.

Rule D7: Maintain a Feature Token Registry

Keep a file at /Users/cb/Documents/repos/sites/queenofsandiego.com/FEATURES.md listing critical code tokens for each active feature:

JADA Hero Crossfade: class="hero-fade-container", fadeImages() function, animation duration 3000ms
Stripe Checkout: stripe-embedded-form, initStripeCheckout() call in script init
Ranch & Coast Hero: id="ranch-coast-hero" (should be absent; presence indicates regression)

After any deployment to staging, grep the live S3 content for these tokens to confirm features remain intact:

aws s3 cp s3://queenofsandiego-staging/index.html - --region us-west-2 | grep -c 'hero-fade-container'

A zero count means the crossfade was removed—do not promote to production.