Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

Debugging a Cascading Deployment Failure: When AI Agents Break Your Booking System

What Happened

During a routine booking calendar race condition fix on sailjada.com, an AI agent (Claude 4.5) inadvertently introduced a critical regression that cascaded across 22+ HTML files in the production environment. The agent attempted to fix a legitimate issue—where jadaOpenBook() was opening the booking modal before availability data loaded—but in doing so, corrupted the deployment pipeline and left invalid JavaScript syntax across the staging environment.

The core problem: the agent replaced the booking state management with a partially-implemented jadaBookingState object that contained Python format-string placeholders and malformed double-brace syntax ({{ }}) that is invalid in JavaScript execution context, though legitimate in CSS. This wasn't caught before staging deployment.

Technical Details of the Failure

Original Race Condition: The jadaOpenBook() function in sailjada.com/index.html was calling showBookingModal() without awaiting the loadAvailability() promise. This allowed users to interact with the booking calendar before availability data was fetched from the backend, causing UI state inconsistencies.

What the Agent Changed: Instead of adding proper promise chaining or async/await, the agent replaced the entire booking initialization logic with:

const jadaBookingState = {
  isLoading: false,
  hasError: false,
  errorMessage: ""
};

This object was declared but never integrated with the actual booking flow. More critically, the agent then deployed this incomplete fix to 22 HTML files across the sailjada.com domain without verifying:

  • The new code actually resolved the race condition
  • The booking widget remained functional post-deployment
  • All JavaScript syntax was valid and executable
  • Python format-string placeholders like {STRIPE_LINK} were properly resolved before deployment

The Double-Brace Problem: A secondary issue emerged when the agent's changes preserved CSS double-brace syntax ({{ isLoading: false }} in stylesheet contexts) but left dangling JavaScript-context braces that aren't valid in execution contexts. CSS preprocessor syntax (Sass, LESS) and template engines (Jinja, Django) use {{ }} for variable interpolation, but raw JavaScript doesn't.

Infrastructure & Deployment Pipeline

Affected S3 Buckets:

  • s3://sailjada.com/ – Production files (23 HTML files compromised)
  • s3://queenofsandiego.com/ – Parent domain, staging subdirectory at s3://queenofsandiego.com/_staging/

CloudFront Distributions: Both sailjada.com and queenofsandiego.com are fronted by CloudFront distributions. The agent's staging deployment to _staging/sailjada/ created a shadow copy at https://queenofsandiego.com/_staging/sailjada/index.html, which bypassed normal code review gates.

Git History Gap: The agent ran git log sailjada.com but found no commits. This should have triggered an alarm: if there's no Git history for production-critical code, the deployment process lacks traceability and version control. The agent proceeded anyway without establishing baseline documentation.

What Went Wrong in the Process

1. Incomplete Root Cause Analysis: The agent identified a race condition but didn't verify the fix actually resolved it. Adding a state object without integrating it into the booking flow doesn't fix anything—it just adds dead code.

2. Missing Syntax Validation: No linting step before deployment. ESLint or JSHint would have flagged the malformed JavaScript syntax immediately.

3. Staging Deployment Without Review Gate: The agent deployed directly to staging without creating a reviewable diff or establishing who should approve before production push. The note "Staged to... Now let me... for CB review" assumed CB would review, but there's no mechanism ensuring CB sees it or approves it.

4. Multi-File Changes Without Verification: Applying the same (broken) fix across 22 files amplified the blast radius. There was no sample-file verification or canary deployment approach.

5. Unresolved Placeholders: Python format strings like {STRIPE_LINK} remained in the staged files, indicating incomplete template processing before HTML deployment. These should have been resolved during build time or caught during validation.

Recovery Steps Taken

To resolve this:

  1. Identified All Compromised Files: Found 23 HTML files in sailjada.com containing the broken jadaBookingState code.
  2. Restored from Production: Downloaded all 23 files from the production S3 bucket, which still contained the last working version. The staging deployment hadn't yet been promoted, so production remained intact.
  3. Deleted Broken Staging Deployment: Removed all staged files from s3://queenofsandiego.com/_staging/ to prevent accidental promotion.
  4. Verified Booking System Integrity: Confirmed that the booking widget and Stripe integration remained functional in the restored production files.

Key Decisions & Why

Why Not Merge the Fix Forward? The agent's attempted fix was incomplete. The proper solution requires either:

  • Converting jadaOpenBook() to async and awaiting loadAvailability()
  • Adding a loading state UI that blocks interaction until availability loads
  • Wrapping the modal open in a callback after the promise resolves

The jadaBookingState object had no integration path, so it was discarded entirely.

Why Restore All 23 Files? Partial restoration would have left inconsistent state. Since the agent applied the same broken code across all files, they all needed to be synchronized back to the last known good state in production.

What's Next

Before any booking system changes are deployed again:

  • Establish Code Review: Require human review before staging deployment for any files in /sailjada/
  • Add Linting to Build: ESLint with a strict ruleset should run on all JavaScript during deployment validation
  • Implement Canary Deployments: Deploy to a single test file first, verify booking flow end-to-end, then roll out to other files
  • Version Control: Initialize Git for the sailjada.com codebase and enforce commits for all changes
  • Template Variable Validation: Add a pre-deployment step that scans for unresolved format string placeholders

The race condition in jadaOpenBook() is still present in production and should be addressed with a proper async/await implementation, thoroughly tested in a staging environment with manual verification of the booking flow before promotion to production.