Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

Diagnosing and Fixing a Multi-Site Deployment Pipeline: GA4 Integration, CloudFront Invalidation, and Daemon Health Verification

This session involved debugging a complex multi-site deployment workflow across three separate properties while simultaneously troubleshooting an authentication layer failure in our background orchestrator daemon. The work spans infrastructure, API integration, and CI/CD patterns—here's what we learned.

The Problem Space

We were managing concurrent deployments across /Users/cb/Documents/repos/sites/86from.com, /Users/cb/Documents/repos/sites/sailjada.com, and analytics integration work, while the jada-agent daemon (running on AWS Lightsail instance 34.239.233.28) was showing signs of task processing issues. The session required:

  • Verifying daemon health and task pickup behavior
  • Debugging Google Analytics API authentication and data pipeline
  • Deploying SEO content and fixing JavaScript templating issues
  • Managing CloudFront cache invalidation across multiple distributions

Infrastructure Access: SSH Key Management and AWS Systems Manager

The first challenge was accessing the Lightsail instance. The private key wasn't stored in the standard ~/.ssh/jada-key location, so we took a defense-in-depth approach:

# First, check local SSH directory and environment configs
ls -la ~/.ssh/
grep -i "jada" ~/.ssh/config
cat ~/Documents/repos/repos.env | grep -i "ssh\|lightsail"

# When local key wasn't found, use AWS Lightsail API for temporary credentials
# This avoids storing long-lived SSH keys locally
aws lightsail get-instance-access-details \
  --instance-name jada-agent-orchestrator \
  --region us-east-1

Why this approach: The Lightsail API generates short-lived SSH certificates paired with temporary keys—more secure than a static key file. This pattern aligns with AWS best practices for ephemeral access.

Once connected, we collected comprehensive daemon telemetry via SSH:

# Check service status and uptime
systemctl status jada-agent.service
uptime
free -h
df -h

# Extract recent logs with error context
journalctl -u jada-agent.service -n 100 --no-pager
tail -100 /var/log/jada-agent/daemon.log

# Check running processes and resource usage
ps aux | grep jada
top -b -n 1 | head -20

Daemon Health Findings

The jada-agent.service was healthy overall—active for 3+ days with minimal resource consumption (0.65% CPU, 144MB memory). However, we identified a critical authentication failure in a subprocess:

  • Service Status: Running since May 10, 2026; load average near zero between tasks
  • Session Activity: 3 of 5 daily sessions consumed; 2 hit the 30-turn Claude limit (expected for complex tasks), 1 completed successfully
  • Critical Issue: port_sheet_sync.py Google OAuth token is expired or revoked, failing every 30-minute sync cycle with HTTP 400 errors

The OAuth failure is a permission/credential issue, not a code bug. The script needs re-authentication against Google's OAuth2 endpoint to obtain a fresh access token.

Google Analytics API Integration: Token Management and Data Pipeline

We debugged the GA4 integration by inspecting stored credentials and testing API calls:

# Verify google-auth-oauthlib is installed
pip list | grep google-auth

# Check token structure (without exposing secrets)
cat ~/Documents/repos/tools/auth_ga.py | grep -i "client_id\|client_secret"

# List accessible GA4 accounts and properties
python3 ~/Documents/repos/tools/auth_ga.py \
  --account dangerouscentaur@gmail.com \
  --list-properties

# Pull 7-day GA4 report for a specific property
python3 ~/Documents/repos/tools/analytics_report.py \
  --property 86dfrom.com \
  --days 7

Architecture Pattern: We're using a stored OAuth2 token (client_id + client_secret) to authenticate against the Google Analytics Data API v1. The script reads credentials from a secure config location and exchanges them for an access token, which is cached locally to avoid repeated auth round-trips.

Why stored credentials: Service-to-service authentication requires long-lived credentials when human OAuth isn't feasible. We're protecting these by restricting file permissions (chmod 600) and excluding them from version control.

Site Deployment: Directory Restructuring and Template Fixes

The 86d.com property had a naming inconsistency—the directory was 86dfrom.com but should have been 86from.com. We fixed this and deployed:

# Rename the directory
mv /Users/cb/Documents/repos/sites/86dfrom.com \
   /Users/cb/Documents/repos/sites/86from.com

# Deploy to S3 (no credentials shown)
aws s3 sync /Users/cb/Documents/repos/sites/86from.com \
  s3://86from-production/ \
  --delete

# Invalidate CloudFront to bust cache
aws cloudfront create-invalidation \
  --distribution-id E2ABC1234DEF \
  --paths "/*"

We also created a new SEO landing page at /Users/cb/Documents/repos/sites/86from.com/site/what-does-86d-mean and deployed it alongside the main index update.

JavaScript Template Engine Fix: Double-Brace Conflict

The booking widget embedded in index.html uses double-brace syntax {{ variable }}, which conflicts with server-side template engines (Django, Jinja2, etc.). The fix:

# Identify problematic sections
grep -n "{{" /Users/cb/Documents/repos/sites/86from.com/site/index.html

# Replace {{ and }} with single braces only within the booking widget JavaScript block
# (Verified that double braces don't appear outside the widget section)

# Syntax-check the modified JavaScript
node -c /path/to/extracted/booking-widget.js

# Deploy fixed version to staging first
aws s3 cp index.html s3://86from-staging/index.html

# Invalidate staging CloudFront
aws cloudfront create-invalidation \
  --distribution-id E2ABC1234STG \
  --paths "/index.html"

Why staging first: CloudFront cache invalidation isn't instantaneous across all edge locations. Testing in staging with a separate CloudFront distribution gives us confidence before production deployment.

Key Decisions and Rationale

  • Ephemeral SSH Access: Using Lightsail API credentials instead of storing long-lived keys reduces attack surface.
  • Separate Staging Distribution: Running a separate CloudFront distribution for staging allows full testing without impacting production users.
  • Service Account OAuth: Using a stored client_secret for GA4 API access enables reliable service-to-service communication, though it requires careful credential rotation.
  • Template Syntax Isolation: Keeping JavaScript templates outside server-side template syntax prevents parsing conflicts.