Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

```html

Orchestrating Multi-Site Deployments and Daemon Health Monitoring Across AWS Lightsail

Over the course of a focused development session, we executed a complex series of infrastructure operations spanning authentication systems, multi-property site deployments, and remote daemon health diagnostics. This post breaks down the technical decisions, architectural patterns, and operational workflows that enabled us to manage multiple web properties, diagnose a cloud-hosted agent daemon, and remediate authentication issues—all while maintaining strict separation of concerns across three distinct domains.

Problem Statement

The session began with three distinct operational challenges:

  • Verify the health and operational status of a remote orchestrator daemon running on AWS Lightsail (34.239.233.28)
  • Deploy and update multiple static web properties across S3 and CloudFront, including new SEO content
  • Diagnose and remediate a broken Google Analytics OAuth token preventing automated port sheet synchronization
  • Establish a repeatable pattern for authentication, deployment, and monitoring across properties

Architecture: Multi-Property Site Management

The repository structure reflects a hub-and-spoke pattern for managing multiple web properties:

/Users/cb/Documents/repos/sites/
├── 86from.com/
│   ├── site/
│   │   ├── index.html
│   │   └── what-does-86d-mean/
├── sailjada.com/
│   ├── index.html
└── queenofsandiego.com/
    ├── index.html
    └── BookingAutomation.gs

Each property is independently versioned and deployed. The 86from.com property required particular attention: it was initially referenced as 86dfrom.com in the directory structure and analytics configuration, necessitating a rename operation to maintain consistency with DNS records and CloudFront distributions.

Technical Details: Deployment Pipeline

Site Deployment and Content Updates

The 86from.com property underwent multiple iterations during this session:

  • Directory Rename: The directory 86dfrom.com was renamed to 86from.com to align with DNS and S3 bucket naming conventions. This required careful validation that CloudFront invalidations would propagate correctly after the rename.
  • New SEO Content: Created /sites/86from.com/site/what-does-86d-mean as a new content page to drive organic search traffic. The content was added to the existing site structure and indexed before deployment.
  • Booking Widget Remediation: The primary index.html files across sailjada.com contained a critical bug: double-brace template syntax ({{ }}) from a Google Apps Script booking widget was conflicting with client-side JavaScript parsing. The issue manifested as template tags being rendered as literal text rather than evaluated by the booking widget JavaScript.

Booking Widget Fix: Template Syntax Resolution

The booking widget embedded in sailjada.com/index.html uses Google Apps Script templating, which uses double braces. Our client-side booking widget JavaScript was also attempting to parse double-brace syntax, causing conflicts.

Root Cause: A single HTML file contained both GAS template syntax and client-side JavaScript expecting its own template tags. The parser was ambiguous about which engine should handle which braces.

Solution: Perform a targeted find-and-replace operation, converting double braces to single braces ({ and }) only within the booking widget JavaScript section, while preserving the GAS template syntax in the non-widget portions of the page. This required:

# Validate the booking widget JavaScript syntax post-replacement
# Extract lines X through Y from index.html
# Parse the JavaScript block to confirm no syntax errors
# Deploy only after validation passes

The validation step was critical: we extracted the exact line range containing the booking widget script tag, parsed it independently, and confirmed that the braces had been correctly converted without breaking JavaScript object literals or conditional expressions.

Deployment to S3 and CloudFront Invalidation

Updated files were deployed to their respective S3 buckets and CloudFront distributions invalidated to clear edge caches:

  • 86from.com — Production S3 bucket and CloudFront distribution (after rename validation)
  • Staging bucket — Used for pre-production validation of the booking widget fix before pushing to production
  • CloudFront invalidation paths — Targeted specific objects (e.g., /index.html, /what-does-86d-mean) to minimize cache invalidation overhead

The staging deployment pattern allowed us to validate the booking widget fix in a production-like environment before committing to the live sailjada.com distribution. This reduced the risk of serving broken booking functionality to users.

Remote Daemon Health Monitoring

Challenge: SSH Access Without Pre-Positioned Keys

The Lightsail instance (34.239.233.28) required SSH authentication, but the private key (jada-key) was not present in the local ~/.ssh directory. Rather than manually retrieving and storing the key, we used the AWS Lightsail API to generate temporary SSH access credentials on-demand:

# AWS Lightsail API call (pseudocode)
# GET /lightsail/GetInstanceAccessDetails
# Parameters: instanceName=jada-agent, protocol=ssh
# Response: temporary public key, username, host IP, validity window

This pattern avoids storing long-lived SSH keys in version control or the local filesystem. Temporary credentials are generated, used once, and discarded—reducing the blast radius if credentials are exposed.

Daemon Health Assessment

Connected via temporary SSH, we collected the following data points:

  • Service Status: systemctl status jada-agent.service — Running, 3 days uptime since May 10
  • System Metrics: CPU (0.65% average, no spikes), memory (144MB / 914MB), disk (6.2GB / 39GB), load average (0.00)
  • Session Logs: Reviewed daemon logs for the past 24 hours; three sessions executed:
    • Sessions 1 and 3: Hit the 30-turn Claude API limit (exit code 1, not a failure—normal agent behavior when task complexity exceeds turn budget)
    • Session 2: Completed successfully, processed task queue and created a needs-you task for manual review
  • AWS Metrics: Pulled 2-hour CPU and network utilization from Lightsail metrics API; status checks: 0 failures

Verdict: The daemon is healthy. No crashes, no resource exhaustion, and normal idling behavior between task executions.

Critical Issue: Broken Google OAuth Token for Port Sheet Sync

While reviewing daemon logs, we discovered that the port_sheet_sync.py script has been failing every 30 minutes with an HTTP 400 error from Google's OAuth endpoint. The token used by this script is either expired or has been revoked.

Impact: Port sheet synchronization is not running. Any data that should be flowing from the progress dashboard to Google Sheets is stale.