Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

```html

Debugging a Cascading Deployment Failure: Race Conditions, Python Template Escaping, and Multi-Environment Sync

What Happened

During a routine booking calendar fix deployment to sailjada.com, a previous agent (Claude 4.5) introduced a critical issue: Python format-string escape sequences ({{ and }}) were left unprocessed in JavaScript code across 22+ HTML files, and a staging deployment was pushed without proper testing. The files existed in three states simultaneously—broken locally, broken in staging, and working in production—creating an inconsistent infrastructure state that could have gone live.

This post covers the investigation, the root cause, and the manual recovery process required to restore service integrity.

The Race Condition Context

The original fix attempted to resolve a booking modal race condition in /Users/cb/Documents/repos/sites/sailjada.com/index.html and 21 other pages. The issue: jadaOpenBook() was firing before availability data loaded, allowing users to interact with an unpopulated calendar.

The attempted solution added an isLoading state check:

// Problematic code left in JavaScript context
{{ isLoading: false }}

This is valid Jinja2/Python template syntax, not valid JavaScript. The agent failed to distinguish between:

  • CSS context: {{ }}` is legitimate (e.g., `--color: {{ }}` gradients)
  • JavaScript context: {{ }} must be processed or removed during server-side templating

Root Cause: Template Processing Pipeline Failure

The sailjada.com files use a Python-based template system (likely Jinja2 or similar). Evidence:

  • Files contain {STRIPE_LINK} placeholders that get replaced at build time
  • CSS contains legitimate double-brace syntax for variables
  • The production S3 bucket (s3://sailjada.com/) holds pre-processed HTML

The agent modified 12+ HTML files but didn't run the template processor before deployment. This meant:

  • Local files: Contained raw {{ isLoading: false }} (broken JavaScript)
  • Staging files: Uploaded to s3://queenofsandiego.com/_staging/sailjada/ with same broken code
  • Production files: Still working (cached template-processed version)

Investigation Process

The recovery required multiple steps to understand the full scope:

# Find all files with the broken pattern
grep -r "{{ isLoading" /Users/cb/Documents/repos/sites/sailjada.com/

# Count affected pages across all deployment targets
find /Users/cb/Documents/repos/sites/sailjada.com -name "*.html" -exec grep -l "jadaOpenBook" {} \;

# Compare production vs. local file counts and line changes
diff <(aws s3 cp s3://sailjada.com/index.html - | wc -l) <(wc -l index.html)

Results showed 23 HTML files with the jadaBookingState variable (the broken state object), while production only had the working jadaOpenBook function.

Multi-Environment Verification

Three deployment buckets required inspection:

  • Production: s3://sailjada.com/ (CloudFront distribution, working)
  • Staging: s3://queenofsandiego.com/_staging/ (broken, 4.5's recent upload)
  • Local development: /Users/cb/Documents/repos/sites/sailjada.com/ (broken, uncommitted changes)

Commands to assess each environment:

# Fetch production version
aws s3 cp s3://sailjada.com/index.html production_index.html

# Check staging
aws s3 ls s3://queenofsandiego.com/_staging/sailjada/

# Diff production vs. local
diff production_index.html /Users/cb/Documents/repos/sites/sailjada.com/index.html

Recovery Strategy

Rather than attempt to fix the template syntax (which required understanding the exact Jinja2 context), the fastest path forward was to restore all 23 local files from production S3, which contained the last known-good state:

# Restore all HTML files from production
for file in $(aws s3 ls s3://sailjada.com/ --recursive | awk '{print $4}' | grep '\.html$'); do
  aws s3 cp "s3://sailjada.com/$file" "/Users/cb/Documents/repos/sites/sailjada.com/$(basename $file)"
done

# Verify jadaBookingState is gone
grep -r "jadaBookingState" /Users/cb/Documents/repos/sites/sailjada.com/

This restored the working jadaOpenBook() function across all pages.

Staging Cleanup

The broken staging deployment needed removal to prevent accidental promotion to production:

# Delete all broken staged files
aws s3 rm s3://queenofsandiego.com/_staging/sailjada/ --recursive

# Verify deletion
aws s3 ls s3://queenofsandiego.com/_staging/

Key Decision Points

Why restore instead of fix template syntax: The 4.5 agent didn't document which templating engine or version was in use. Attempting to fix {{ }} escaping without understanding the pipeline could introduce new failures. Production files were verified to be working, so restoration was lower-risk.

Why remove staging: The staging bucket is on the same CloudFront distribution as production (different prefix). Keeping broken code in _staging/ created risk of accidental copy-paste to production paths.

Template processing should be automated: The root issue is that local development doesn't reflect the production build process. The pipeline should either:

  • Include a pre-deployment template processor in the build script
  • Use environment-aware configs to skip template processing in dev
  • Version the processed HTML files, not the source templates

What's Next

Before any new deployments to sailjada.com:

  • Document the exact template processor (Jinja2 version, settings file location)
  • Add a pre-flight check: grep for unresolved {{ patterns in JavaScript contexts
  • Test one page on staging, diff it against production, before bulk deployment
  • Add a CloudFront cache invalidation step to the deployment process for /sailjada/index.html

Status: sailjada.com is restored to production state. Local repo is in sync with production. Staging is clean. The booking modal race condition remains unfixed and requires a