Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

```html

Orchestrating Multi-Site Deployment and GA4 Analytics Integration with Automated Agent Health Monitoring

This session involved coordinating deployments across three separate web properties, implementing Google Analytics 4 API integration for automated reporting, and establishing robust monitoring for a distributed task orchestrator daemon. The work demonstrates how to manage complexity across multiple domains while maintaining observability into automated systems.

What Was Done

  • Renamed and redeployed the 86-from.com property with corrected template syntax
  • Implemented Google Analytics 4 API authentication and data retrieval for the dangerouscentaur account
  • Created SEO content pages for organic search discovery
  • Fixed template interpolation bugs in booking widget JavaScript
  • Established SSH-based health monitoring for the jada-agent orchestrator daemon running on AWS Lightsail
  • Diagnosed and documented a persistent OAuth token failure in the port_sheet_sync service

Technical Details: Multi-Property Deployment Pipeline

The session began with property consolidation. The directory structure at /Users/cb/Documents/repos/sites/86dfrom.com was renamed to /Users/cb/Documents/repos/sites/86from.com to match the intended domain branding. This required updating all deployment references and invalidating CDN cache to ensure old URLs didn't serve stale content.

The deployment process followed this pattern:

# Sync to S3 production bucket
aws s3 sync /Users/cb/Documents/repos/sites/86from.com s3://86from.com --delete

# Invalidate CloudFront distribution cache
aws cloudfront create-invalidation --distribution-id [DISTRIBUTION_ID] --paths "/*"

For staging validation, a separate CloudFront distribution was used to test changes before production push. This allowed verification of the booking widget JavaScript parsing and template syntax without affecting live traffic.

The critical bug fixed was double-brace template syntax within the booking widget JavaScript block. The widget uses Handlebars-style templating ({{ variable }}) which conflicted with Vue.js or other template engines if they were parsed at deployment time. The fix involved scoping the template interpolation to only the booking widget section and ensuring the JavaScript block was syntactically valid before deployment:

// Search for all {{ and }} occurrences
grep -r "{{" /Users/cb/Documents/repos/sites/86from.com/site/

// Verify context to ensure they're only in the booking widget
// Replace with single-brace syntax outside widget scope

A versioned comment was embedded in the booking widget containing the model ID, allowing cross-reference with CDN cache versions during troubleshooting.

Infrastructure: Google Analytics 4 API Integration

The auth_ga.py tool at /Users/cb/Documents/repos/tools/auth_ga.py implements OAuth 2.0 authentication to the Google Analytics Data API v1. This enables programmatic access to GA4 reports without manual dashboard interaction.

Why GA4 API instead of GA3 Universal Analytics? Google is sunsetting Universal Analytics in 2024. GA4 provides event-based tracking (superior for modern web apps) and the Data API allows automated report generation, scheduled exports, and integration with backend systems.

The authentication flow:

  • Client credentials (client_id and client_secret) stored in secrets directory
  • OAuth token request to Google's authorization endpoint
  • Refresh token stored for long-lived access without re-authentication
  • Access token cached with expiration tracking

The tool lists all GA4 properties accessible under the dangerouscentaur account, then retrieves 7-day rolling reports for specified properties:

python3 ~/Documents/repos/tools/auth_ga.py --account dangerouscentaur@gmail.com --property 86from.com

This returns structured JSON with metrics like users, sessions, engagement rate, and conversion events. The data can be piped to CSV or fed into automated dashboards.

Key decision: Storing client secrets with restricted file permissions (600) and excluding them from version control. The secrets directory is referenced in repos.env but never committed to Git.

Daemon Health Monitoring: Lightsail SSH Access Pattern

The jada-agent orchestrator daemon runs on AWS Lightsail instance 34.239.233.28. Monitoring required SSH access to pull service status, logs, and process metrics. Since the private key wasn't stored locally in ~/.ssh/, the approach was:

  1. Try local key paths and repos.env references — confirms key material location strategy
  2. Fall back to AWS Lightsail GetInstanceAccessDetails API — retrieves temporary SSH credentials with automatic expiration
  3. Use temporary key to establish SSH session — executes health checks without permanent key exposure
  4. Clean up temporary credentials immediately — removes attack surface
# Fetch temporary access credentials from Lightsail API
aws lightsail get-instance-access-details --instance-name jada-agent --region us-east-1

# Test SSH connection with temporary certificate
ssh -i /tmp/jada_temp_key ec2-user@34.239.233.28 "systemctl status jada-agent.service"

# Pull daemon health data
ssh -i /tmp/jada_temp_key ec2-user@34.239.233.28 "journalctl -u jada-agent.service -n 50"

# Retrieve Lightsail metrics via API for CPU, memory, network
aws lightsail get-instance-metric-statistics --instance-name jada-agent --metric-name CPUUtilization --statistics Average --period 300 --start-time [TIME] --end-time [TIME]

This pattern eliminates the need to store production SSH keys locally while maintaining audit trails through AWS CloudTrail.

Observations and Diagnostics

Service Health: jada-agent.service has been running continuously for 3 days with 0.65% average CPU and 144MB memory usage. Load average near zero indicates the daemon spends most time idle between task polling cycles, which is expected behavior for an event-driven orchestrator.

Session Activity: Today's 3 sessions processed e-signature workflow blockers and crew page generation tasks. Two sessions hit the 30-turn Claude conversation limit (exit code 1), which is logged but doesn't prevent the daemon from continuing. Session 2 completed successfully within limits, indicating that complex tasks sometimes require many turns.

Critical Issue: The port_sheet_sync.py service has been failing for ~12+ hours with "HTTP Error 400: Bad Request" on Google OAuth token refresh. This indicates the token is expired or revoked and requires re-authentication. The sync runs every 30 minutes per cron configuration, so no port sheet updates have been persisted since the failure began.

Why this matters: If port sheet syncing feeds downstream reporting or booking systems, manual reconciliation may be needed for any updates made during the outage window.

Key Architectural Decisions

  • Temporary SSH credentials over stored keys: Reduces blast radius if a dev machine is compromised. Credentials auto-expire within hours.
  • API-driven metrics over SSH logs alone: CloudWatch/Lightsail API provides structured time-series data suitable for alerting and trending, while SSH logs are one-off snapshots.
  • Staging CloudFront distribution separate from production: Allows full