Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

```html

Debugging a Cascading Deployment Failure: When AI Agents Break Production and Staging

This post documents a critical incident where an autonomous agent (Claude 4.5) attempted to fix a legitimate race condition in the sailjada.com booking calendar but instead introduced syntax errors that cascaded across 22+ HTML files, corrupted the staging deployment, and required emergency rollback procedures.

The Initial Problem: A Real Race Condition

The sailjada.com booking system had a genuine issue: jadaOpenBook() was opening the calendar modal immediately without waiting for availability data to load from the backend. Users could interact with an empty calendar while the AJAX request was still in flight.

The fix itself was sound—wrap the modal opening in a conditional check for isLoading: false—but the implementation introduced catastrophic syntax errors.

What the Agent Did Wrong

The 4.5 agent modified files across the sailjada deployment:

  • /Users/cb/Documents/repos/sites/sailjada.com/index.html
  • /Users/cb/Documents/repos/sites/sailjada.com/releases/rc1/index.html
  • 22 additional HTML files in the same repository

The critical error: the agent left Python format-string syntax ({{ }} double braces) in the JavaScript code it added. While CSS double-braces like {{ color: red }} are valid (used for CSS custom properties), JavaScript double-braces are not valid syntax in any context.

Example of the corruption introduced:

// BROKEN - Python template syntax in JavaScript
if ({{ isLoading: false }}) {
  jadaOpenBook();
}

// What it should have been
if (!isLoading) {
  jadaOpenBook();
}

The agent then deployed this broken code to s3://queenofsandiego.com/_staging/sailjada/ without testing.

Technical Investigation Process

We conducted a multi-layered diagnosis:

  • Search for double-brace patterns: Used regex to identify all {{ occurrences across HTML files
  • Differentiate CSS from JavaScript: Distinguished legitimate CSS custom property syntax from broken template syntax by examining context
  • Git history analysis: Compared git logs for sailjada.com to identify exactly which commits introduced the syntax errors
  • Production vs. staging comparison: Fetched live files from S3 and compared line-by-line diffs to understand scope of changes
  • Cross-file validation: Checked all 23 affected HTML files for the same pattern of corruption

Key diagnostic commands used:

# Find all Python format-string placeholders
find /path/to/sailjada.com -name "*.html" -type f -exec grep -l "{{" {} \;

# Diff production vs. local staging
diff -u <(aws s3 cp s3://queenofsandiego.com/sailjada/index.html -) ./index.html

# Count scope of changes
git log --oneline sailjada.com | head -20
git diff HEAD~1 -- sailjada.com/**/*.html | wc -l

Remediation Strategy

We executed a four-phase recovery:

Phase 1: Identify All Broken Files

Located all 23 HTML files in the local repository containing the broken jadaBookingState code pattern.

Phase 2: Restore from Production

Mass-restored all broken files directly from the production S3 bucket:

# Restore each corrupted file from production S3
aws s3 cp s3://queenofsandiego.com/sailjada/index.html /Users/cb/Documents/repos/sites/sailjada.com/index.html
aws s3 cp s3://queenofsandiego.com/sailjada/releases/rc1/index.html /Users/cb/Documents/repos/sites/sailjada.com/releases/rc1/index.html
# ... repeat for all 23 files

Phase 3: Validate Restoration

Verified that:

  • jadaBookingState syntax errors were completely removed
  • The full booking system functions were present and intact
  • No Python placeholder syntax remained in JavaScript contexts

Phase 4: Clean Up Staging Deployment

Deleted the corrupted staging deployment at s3://queenofsandiego.com/_staging/sailjada/ to prevent accidental promotion to production.

Infrastructure Context

The sailjada.com site operates on AWS infrastructure:

  • Production bucket: s3://queenofsandiego.com/
  • Staging bucket: s3://queenofsandiego.com/_staging/
  • CloudFront distribution: Configured to serve from production S3 origin with appropriate cache control headers
  • Source repository: /Users/cb/Documents/repos/sites/sailjada.com/

Key Decisions and Lessons

Why restore from production rather than attempt to fix locally?

The scope of corruption (22+ files with inconsistent syntax errors) made surgical fixes risky and time-consuming. Production files were known-good, so restoration provided immediate confidence. The real fix (handling the race condition properly) should be implemented in a controlled code review, not through automated agent scripts.

Why delete staging instead of trying to fix it in place?

The staging deployment represented a "poison" state—files that would break the live site if promoted. Better to maintain a clean staging environment than keep broken files around where they could be accidentally deployed.

Why this reveals an agent capability gap:

Claude 4.5 successfully identified the JavaScript race condition but failed to:

  • Test the syntax of modified JavaScript before deployment
  • Distinguish between valid CSS template syntax and invalid JavaScript template syntax
  • Validate files before pushing to production infrastructure
  • Understand the deployment pipeline and staging/production separation

What's Next

The legitimate race condition fix remains unaddressed. The proper path forward:

  • Code review: Document the race condition in a pull request with test cases
  • Unit testing: Add tests for jadaOpenBook() that verify modal doesn't open until isLoading === false
  • Integration testing: Verify booking calendar loads availability before allowing user interaction
  • Manual QA: Test on staging with network throttling to confirm the race condition is fixed
  • Controlled deployment: Deploy to production only after approval, with rollback plan ready

Autonomous agents are powerful tools for exploration and diagnostics, but production deployments require human oversight, automated tests