Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

```html

Preventing Production Regressions: Hard Rules for Multi-Environment Deploys

Over the last development session, a critical regression deployed to production wiped three working features on queenofsandiego.com in a single S3 push. The root cause: a stale local index.html overwrote a newer S3 production version, eliminating the hero JADA→BOOK NOW crossfade animation, the Stripe embedded checkout booking flow, and resurrecting a previously-deleted marketing line. This post documents the infrastructure safeguards and decision rules we implemented to prevent this class of failure.

What Happened: The Regression Chain

The deployment touched these resources:

  • S3 bucket: queenofsandiego.com (production serving)
  • CloudFront distribution: Primary CDN cache for queenofsandiego.com
  • Local working copy: /Users/cb/Documents/repos/sites/queenofsandiego.com/
  • Key file: index.html (3,650 lines, contains all hero animations and booking logic)

The deployment command pushed both staging and production in a single operation, ignoring the staging-first rule. The local index.html was 4 commits behind S3 production because prior sessions had merged feature work directly to S3 without pulling and rebasing local. The operator did not diff S3 against local before the push, so the loss was invisible until testing caught it in production.

Why This Matters: Silent Data Loss at Scale

S3 has no version control—once overwritten, the prior state is gone unless CloudFront versioning is enabled (it wasn't). The three lost features were:

  1. Hero crossfade: CSS animation transitioning JADA brand message to "BOOK NOW" CTA (customer-facing, conversion path)
  2. Stripe embedded checkout: Inline payment form for bookings (revenue-blocking if lost)
  3. Ranch & Coast messaging: A marketing line that was intentionally removed in an earlier session and should never resurface

All three were present in S3 prod but absent from the local file. The deploy erased them. This is a fiduciary failure—revenue-blocking features went dark in production with zero notification.

Technical Safeguards: Eight Hard Rules (D1–D8)

We codified eight non-negotiable rules into the site-specific CLAUDE.md file located at /Users/cb/Documents/repos/sites/queenofsandiego.com/CLAUDE.md. These load automatically on every session and are enforced before any S3 operation:

D1: Pull and Diff Before Any Edit

aws s3 cp s3://queenofsandiego.com/index.html ./index.html.prod-snapshot
diff -u index.html.prod-snapshot index.html

Every session starts by pulling the current production state and diffing it against the working copy. This reveals if local is stale. If the diff shows features in prod but not local, abort the session and escalate to the human operator.

D2: Staging-Only Single-Target Deploys

Never deploy staging and prod in the same command. Deploy to staging first:

aws s3 cp index.html s3://staging.queenofsandiego.com/index.html

Wait for human review. Only after explicit approval, promote to prod:

aws s3 cp s3://staging.queenofsandiego.com/index.html s3://queenofsandiego.com/index.html

D3: One Logical Change Per Deploy

Each deploy targets a single feature or fix. If the session involves hero animation, booking flow, and email templates, that's three separate deploys with three separate staging reviews. This limits the blast radius if a regression is discovered.

D4: Obey Prior Session Warnings

Every session summary includes warnings like "local index.html is 4 commits behind S3." If a warning exists, pull before editing. Do not override the warning based on confidence.

D5: Snapshot Prod Before Overwrite (No Deletions)

Before any prod deploy, save the current state:

aws s3 cp s3://queenofsandiego.com/index.html ./snapshots/index.html.$(date +%s)

Keep a dated snapshot directory in the repo. This allows fast rollback if a regression surfaces.

D6: Proof Block Before Any cp

Print a six-line proof of what you're deploying before the copy command executes:

--- DEPLOY PROOF BLOCK ---
File: index.html
Size: 127,456 bytes
Hero animation: PRESENT ✓
Stripe checkout: PRESENT ✓
Ranch & Coast line: ABSENT ✓ (intentionally deleted)
Destination: s3://queenofsandiego.com/index.html
--- END PROOF BLOCK ---

If any line fails the proof check, abort and investigate.

D7: Feature-Token Registry

Maintain a FEATURE_TOKENS.md file in the repo root with unique grep-able markers for every production feature:

## FEATURE_TOKENS

### Hero Crossfade
Pattern: `id="hero-jada-to-book-fade"`
File: index.html, line 742
Last verified: 2025-01-14, prod via S3 HEAD

### Stripe Embedded Checkout
Pattern: `stripe-payment-element`
File: index.html, line 1204
Last verified: 2025-01-14, prod via S3 HEAD

### Ranch & Coast Retired Line
Pattern: `For Ranch & Coast readers` — MUST NOT EXIST IN PROD
File: index.html (deleted)
Last verified: 2025-01-14, confirmed absent from S3 HEAD

Before any prod deploy, grep the new file against this registry and confirm every PRESENT token exists and every ABSENT token is gone.

D8: Escalate to CB If S3 Is Ahead of Local

If the D1 diff reveals production features absent from local, do not proceed. Send a message to CB with the diff and ask:

  • Should I pull these features and rebase local?
  • Or should I revert prod to an earlier snapshot?

Do not guess. This is a data-loss decision that only the human operator can make.

Infrastructure and Automation

Where These Rules Live:

  • /Users/cb/Documents/repos/sites/queenofsandiego.com/CLAUDE.md — Full eight rules with examples (auto-loaded every session)
  • /Users/cb/Documents/repos/CLAUDE.md — Condensed pointer for non-QOS sites
  • FEATURE_TOKENS.md — Maintained in repo root; updated each session
  • snapshots/ directory — Dated backups of index.html before prod push

CloudFront Cache