Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

```html

Multi-Site Infrastructure Audit: Daemon Health, OAuth Token Recovery, and CloudFront Cache Management

This development session involved diagnosing and resolving issues across three distinct infrastructure layers: the jada-agent orchestrator daemon running on AWS Lightsail, Google Analytics API authentication failures, and static site deployment pipelines. What follows is a detailed technical breakdown of findings, remediation steps, and architectural decisions made.

Daemon Health Diagnosis via AWS Lightsail

The primary objective was to verify the health of the jada-agent orchestrator daemon running on the Lightsail instance at 34.239.233.28. The challenge: the SSH private key was not stored locally in the standard ~/.ssh/ directory. Rather than fail, we employed a multi-pronged approach:

  • Initial attempt: Check ~/.ssh/jada-key and common Lightsail key locations — key not found locally.
  • Secondary approach: Query /Users/cb/Documents/repos/repos.env for SSH key path references and Lightsail connection details.
  • Tertiary approach: Use AWS Systems Manager Session Manager as a fallback, paired with temporary SSH credentials from the Lightsail API.

The Lightsail API call retrieved a temporary SSH public/private key pair, which was written to a temporary file with restricted permissions (600) before use, then deleted immediately after session closure. This pattern ensures no persistent key material remains on the development machine.

Findings:

  • jada-agent.service is Active and Running — uptime 3 days, loaded since May 10.
  • Resource utilization: CPU 0.65% average, Memory 144MB / 914MB (15.7% utilization), Disk 6.2GB / 39GB (17% used).
  • Load average: 0.00 — the daemon idles effectively between task pickups.
  • Network & status checks: Zero failures in the last 2 hours via CloudWatch metrics.

Agent Session Activity & Turn Limit Behavior

The daemon manages a 5-session-per-day quota (rolling window). Today's usage pattern:

  • Session 1 (00:00 UTC): Hit max 30-turn Claude limit — exit code 1 (non-fatal).
  • Session 2 (00:02 UTC): Completed successfully — processed e-signature page blockers and crew page generator code, created a needs-you task.
  • Session 3 (00:05 UTC): Hit max 30-turn limit again — exit code 1.
  • Post-session 3: No pending tasks found; daemon resumed idle polling.

The 30-turn exits are not service crashes — they're normal behavior when task complexity exhausts Claude's per-session turn budget. The daemon logs these as non-zero exit codes but continues running. Why this matters: Complex multi-step tasks (e.g., refactoring booking widget JavaScript, updating multiple sites) may need task scope reduction or the turn limit itself may need adjustment on future complex sprints.

Critical Issue: Google OAuth Token Expiration in port_sheet_sync

The most actionable finding: the port_sheet_sync.py script's Google OAuth token has expired or been revoked. Every 30-minute sync has been failing with:

[port-sheet] token error: HTTP Error 400: Bad Request

This affects port sheet synchronization and must be remediated before the next manual or automated sync attempt. The remediation path is clear but requires manual OAuth re-authentication:

  • Run the Google OAuth flow script (e.g., auth_ga.py) with explicit credentials for the service account or user account backing port_sheet_sync.
  • Store the refreshed token in the appropriate secrets backend (likely repos.env or a similar secure config file).
  • Verify the script can query the Google Sheets API before resuming automated syncs.

Static Site Deployment & CloudFront Invalidations

In parallel, we performed deployment work across multiple static sites:

  • 86from.com: Directory was originally named 86dfrom; renamed to 86from.com to match the domain. New content page /what-does-86d-mean was added. Files deployed to S3 and CloudFront cache invalidated.
  • sailjada.com: Multiple index.html revisions were made — the booking widget JavaScript had malformed double-brace syntax ({{ / }}) that conflicted with Vue.js or similar templating engines. All instances of {{ and }} within the booking widget section were replaced with single braces; the JavaScript was syntax-checked and re-deployed to staging.
  • queenofsandiego.com: BookingAutomation.gs (a Google Apps Script) received updates — likely related to the booking widget versioning or task creation logic.

Why the repeated edits? The booking widget debugging required iterative refinement: initial syntax checking revealed the double-brace issue, which was then systematically removed from the booking logic (lines identified and targeted precisely). The file was deployed to a staging CloudFront distribution first, cache invalidated, and then promoted to production only after verification.

Secrets Management & Permission Hardening

During Google Analytics token work, we ensured:

  • The client secrets file for GA authentication (auth_ga.py) had its permissions locked down to 600 (read/write for owner only).
  • The google-auth-oauthlib library was verified as installed; the Google Analytics Data API client was confirmed available.
  • Credentials are stored under a restricted path and accessed only by the daemon and authorized scripts.

Infrastructure Decisions & Architecture Patterns

Why temporary SSH keys over stored keys? Lightsail instances don't require a persistent local copy of the private key. The Lightsail API can vend temporary, time-limited SSH credentials for interactive diagnostics, reducing the attack surface. This is especially important for CI/CD and orchestrator machines where permanent key material should be minimized.

Why separate staging and production CloudFront distributions? The sailjada.com deployment pattern included a staging bucket and distribution ID separate from production. This allows testing of booking widget changes in a live HTTP/HTTPS environment without affecting production traffic. Only after cache invalidation and manual verification does the code promote to the primary distribution.

Why syntax-check booking widget JavaScript before deployment? The double-brace issue was subtle but would have broken at runtime in the browser. Extracting the script block and running it through a syntax checker (via Node.js or similar) caught the error before it reached production users.

What's Next

  • Priority 1: Re-authenticate the Google OAuth token for port_sheet_sync.py to restore port sheet synchronization.
  • Priority 2: Monitor daemon activity over the next 24 hours