Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

```html

Preventing S3 Deployment Regressions: A Case Study in State Management and Safe Deploy Practices

Over a three-hour development window, a production deployment to queenofsandiego.com inadvertently wiped three critical features by deploying a stale local index.html over a newer S3 production version. This post documents the failure mode, the root causes, and the hardened deployment rules we implemented to prevent recurrence.

What Went Wrong: The Regression

The deployment deleted three distinct features that were live and working in S3:

  • Hero crossfade animation: A JADA → BOOK NOW fade effect in the hero section
  • Stripe embedded checkout: The inline booking flow on the landing page
  • Deleted hero text: A "For Ranch & Coast readers..." line that had been intentionally removed but was reintroduced from an old local version

All three regressions occurred because the deployment tool copied a stale local /Users/cb/Documents/repos/sites/queenofsandiego.com/index.html directly to the S3 production bucket without first pulling the current S3 version, diffing it, or validating that local state was actually ahead.

Root Causes

  • No pre-deploy state validation: The agent did not pull S3 current before modifying local files, so it never discovered that S3 was ahead of local.
  • Single command, multiple targets: A single `cp` command deployed both staging and prod` simultaneously, violating the staging-first safety rule that was already documented in prior session notes.
  • Ignored prior warnings: The agent's own session summary from the previous window warned about stale local files and the risk of wiping S3 state — this warning was present but not acted upon.
  • No CloudFront invalidation sequencing: While CloudFront invalidation did occur, it happened after the S3 overwrite, leaving a window where edge caches could have served corrupted state.
  • No feature-token registry: There was no grep-able marker in S3 that the agent could check to confirm which features were live before making changes.

Technical Architecture: Deployment Pipeline

The queenofsandiego.com infrastructure spans three layers:

  • S3 bucket: queenofsandiego.com (source of truth for live content)
  • CloudFront distribution: CF distribution ID cached to a file in the repo (used for cache invalidation)
  • Route53: Four DNS zones involved in the broader multi-domain setup (queenofsandiego.com, sailjada.com, 86from.com, queendomain.net)
  • Local working directory: /Users/cb/Documents/repos/sites/queenofsandiego.com/ (source for edits, but not guaranteed to be in sync with S3)

The deployment workflow should be:

1. Pull S3 current state → local index.html.prod.backup
2. Diff local against backup
3. Review diff for unintended changes
4. Deploy to staging bucket only
5. Eyeball staging.queenofsandiego.com in browser
6. If approved: deploy to prod
7. Invalidate CloudFront for /index.html and /* (edge-case patterns)
8. Verify prod in browser post-invalidation

What actually happened:

1. (Skipped pull and diff)
2. (Skipped review)
3. cp index.html → both staging and prod in one command
4. Invalidate CloudFront
5. Deploy cycle complete (but prod was corrupted)

Hardened Rules: Eight Deployment Guards (D1–D8)

We implemented eight mandatory rules, encoded into the site-specific CLAUDE.md` as a checklist that loads automatically on every QOS session:

  • D1 — Pull S3 and diff before any edit: Every session must begin with aws s3 cp s3://queenofsandiego.com/index.html ./index.html.prod-current.backup followed by a diff against local. This is non-negotiable and must print a summary in chat before proceeding.
  • D2 — Staging-only single-target deploys: Never deploy to both staging and prod in the same command. Staging must be reviewed and approved before prod is touched. The deploy command pattern is: aws s3 cp index.html s3://staging.queenofsandiego.com/index.html (staging first), then after eyeball approval, repeat with the prod bucket.
  • D3 — One logical change per deployment: Each cp` command should map to one feature or bugfix. If you're changing the hero fade AND fixing the Stripe form AND updating copy, that's three separate deploys with three separate eyeballings in staging.
  • D4 — Obey session-summary warnings: If a prior session's summary warns about stale local state, S3 regressions, or any deployment risk, escalate to CB rather than proceeding solo.
  • D5 — Snapshot prod before overwrite: Every prod deploy must include a timestamped backup: aws s3 cp s3://queenofsandiego.com/index.html s3://queenofsandiego.com/.backups/index.html.$(date +%s).backup. (Note: S3 versioning is not currently enabled, so this manual snapshot is critical.)
  • D6 — Six-line proof block before any cp: Before copying to S3, print a six-line proof block in chat: file path, md5 hash of local file, target bucket, target key, cloudfront dist ID if applicable, and the exact command about to run. This gives CB a chance to abort.
  • D7 — Feature-token registry: Maintain a grep-able marker in S3 for each live feature. For example, a comment in the hero section: <!-- FEATURE_TOKEN: hero_jada_fade_v2.1 -->. Before deploying, grep S3 for all tokens currently live, compare against the local version being deployed, and flag any tokens that would be deleted.
  • D8 — Escalate to CB if S3 is ahead: If the diff in D1 shows that S3 has changes not in local, stop immediately and escalate. This is a sign that either another agent or human editor has changed prod since your last pull, and a merge/conflict resolution is needed before you proceed.

Implementation Details

All eight rules are encoded in /Users/cb/Documents/repos/sites/queenofsandiego.com/CLAUDE.md as a numbered checklist. On every new QOS session, this file loads into context automatically, and any deployment command must reference the checklist explicitly.

A condensed pointer to the same rules was also added to /Users/cb/Documents/repos/CLAUDE.md (the top-level repo guidance) so that non-QOS sites benefit from the lessons learned without duplicating the full text.

The rules are enforced via a simple pre-flight checklist pattern: before any aws s3 cp, the agent must print all eight rules, mark each as