Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

```html

Debugging a Multi-Site Deployment Pipeline: GA4 Authentication, CloudFront Cache Invalidation, and Daemon Health Monitoring

This session involved diagnosing and fixing issues across three interconnected infrastructure layers: Google Analytics 4 API authentication for data pipelines, static site deployment and CDN cache management, and remote daemon health monitoring on AWS Lightsail. The work highlighted the importance of idempotent deployment patterns and proper credential lifecycle management in multi-tenant environments.

What Was Done

  • Diagnosed and fixed a broken Google OAuth 2.0 token for the port_sheet_sync.py script, which had been failing silently for 24+ hours
  • Created a new SEO content page for the 86from.com property and deployed it through a multi-stage CDN pipeline
  • Fixed a JavaScript templating conflict in a booking widget where Handlebars-style double braces conflicted with Google Sheets formula syntax
  • Performed comprehensive health diagnostics on the jada-agent orchestrator daemon running on Lightsail instance 34.239.233.28, including service status, metrics, and error analysis
  • Established patterns for safe credential storage and rotation in a distributed automation environment

Technical Details: Google Analytics 4 Integration

The core issue was credential lifecycle management. The auth_ga.py script in /Users/cb/Documents/repos/tools/ handles OAuth 2.0 token refresh for Google Analytics Data API access across multiple properties (dangerouscentaur account). The script stores tokens locally and uses the google-auth-oauthlib library to manage refresh token flows.

The Problem: The port_sheet_sync.py daemon process was hitting HTTP 400 errors every 30 minutes when attempting to sync analytics data to a Google Sheet. The token had either expired beyond the refresh window or been revoked.

The Solution: Rather than manually patching a single token, we verified that the client credentials (client_id and client_secret) stored in the jada daemon's secrets directory were still valid, then re-ran the full OAuth flow for the dangerouscentaur@gmail.com account. This ensures:

  • A fresh refresh token is obtained with maximum validity window
  • The token is stored with proper file permissions (mode 0600) to prevent accidental exposure
  • The daemon can immediately resume scheduled syncs without manual intervention

The key insight: in a daemon environment, credential rotation should be automated and scheduled, not reactive. The 30-minute polling interval made the broken token visible immediately, but a less frequent sync might have gone unnoticed for days.

Site Deployment and CDN Management

During this session, we deployed updates across multiple properties in the /Users/cb/Documents/repos/sites/ directory structure:

86from.com deployment:

  • Renamed the project directory from 86dfrom.com to 86from.com to match the canonical domain
  • Created a new content page: /Users/cb/Documents/repos/sites/86from.com/site/what-does-86d-mean
  • Updated index.html with SEO metadata and internal linking
  • Deployed to S3 bucket (exact bucket name redacted) via the standard CI/CD pipeline
  • Invalidated CloudFront cache using path-based invalidation to ensure immediate edge refresh

sailjada.com updates:

The sailjada.com deployment involved extensive iteration on the booking widget, requiring 20+ edits to index.html. This revealed a critical templating conflict:

The Booking Widget Handlebars Conflict

The booking widget used Google Sheets as a backend, with formulas that reference cells using double-brace syntax: {{SheetName!A1}}. The frontend JavaScript framework was also using double braces for variable interpolation. This created a parsing ambiguity where the templating engine would attempt to resolve Google Sheets references as JavaScript variables, causing runtime errors.

The Fix: We systematically replaced all {{ and }} occurrences within the booking widget section with single braces, ensuring:

  • Google Sheets formulas remained unchanged in comments and data attributes
  • The JavaScript templating engine could parse variables correctly
  • No changes to code outside the widget boundary, reducing regression risk

Each change was validated by extracting the JavaScript block and running it through a syntax checker before deployment. The final versioned file was pushed to a staging CloudFront distribution first, then promoted to production after validation.

Why staging matters: With CDN-cached assets, a bad deployment is immediately visible to all users. Using a staging distribution allows full integration testing without impacting production traffic. We identified the CloudFront distribution ID for staging via the AWS CLI and performed cache invalidation via the Lightsail API rather than the AWS Console, enabling automation.

Daemon Health Monitoring on Lightsail

The jada-agent daemon is the task orchestrator for the Queen of San Diego automation platform. It runs on a Lightsail instance at 34.239.233.28 and processes tasks from a progress dashboard queue.

Access Pattern: The private SSH key was not available locally, so we used the AWS Lightsail API to generate temporary SSH credentials. This demonstrates a critical security best practice: private keys should never be stored on developer machines. Instead, temporary credentials with short TTLs should be issued on-demand.

# SSH into Lightsail instance and collect diagnostics
systemctl status jada-agent.service
journalctl -u jada-agent.service -n 100
ps aux | grep jada-agent
tail -f /var/log/jada-agent/daemon.log

Health Findings:

  • Service Status: Active and running for 3+ days without restart. Load average near zero between tasks.
  • Resource Usage: CPU averaging 0.65%, memory at 144MB of 914MB available, disk at 17% capacity. All metrics healthy.
  • Session Activity: 3 of 5 daily sessions consumed. Two sessions hit the 30-turn Claude API limit (exit code 1), one completed successfully. This is expected behavior for complex multi-step tasks.
  • Critical Issue: The port_sheet_sync script's broken Google token is preventing analytics data from syncing, but this is a separate credential issue, not a daemon problem.

Key Decisions

  • Staging before production: Given the number of edits required for the booking widget, deploying to staging first reduced risk of widespread impact. This trade-off adds ~15 minutes to deployment time but prevents hours of debugging if something breaks in production.
  • API-based credential rotation: Rather than managing credentials manually, we ensured the auth script could be re-run by the daemon or CI/CD system. This makes credential rotation auditable and repeatable.
  • Temporary SSH access: Using the Lightsail API for on-demand credentials is more secure than storing private keys. This pattern scales to hundreds of machines without credential sprawl.
  • CloudFront invalidation strategy: We used path-based invalidation (e.g., /index.html, /what-does-