Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

Multi-Site Analytics Migration and Daemon Health Validation: Rebuilding GA4 Infrastructure for 86from.com

This session focused on two parallel infrastructure tasks: validating the health of the jada-agent orchestrator daemon running on a Lightsail instance, and completing a Google Analytics 4 migration pipeline for a newly restructured domain. Both efforts revealed important patterns about our authentication layer and task orchestration limits.

Daemon Health Assessment: The jada-agent Service on Lightsail

The primary objective was to SSH into the Lightsail instance at 34.239.233.28 and verify that the jada-agent.service systemd unit was functioning correctly. This required solving an immediate problem: the private key for the jada-key key pair wasn't available in the standard ~/.ssh directory.

Rather than storing private keys in version control or passing them between machines, we used AWS Systems Manager Session Manager as an intermediate access layer, then retrieved temporary SSH credentials via the Lightsail API endpoint for instance access details. This approach eliminated the need to manually distribute or store long-lived SSH keys locally.

# Retrieve temporary access credentials without exposing the private key
aws lightsail get-instance-access-details \
  --instance-name jada-agent-orchestrator \
  --region us-east-1

# Write the temporary certificate and establish connection
# (Private key retrieved securely from Lightsail backend)
ssh -i /tmp/jada-key-temp ubuntu@34.239.233.28

Once connected, we collected comprehensive health metrics:

  • Service Status: jada-agent.service has been running continuously since May 10 with 3 days of uptime. Load average is 0.00 between task execution cycles, indicating proper idle behavior.
  • Resource Utilization: CPU averaged 0.65% (normal for a 60-second poll loop), with no spikes recorded. Memory usage at 144MB of 914MB available. Disk utilization at 6.2GB of 39GB (17%), leaving substantial headroom for logs and task data.
  • Task Processing: The daemon processed 3 sessions today (UTC), consuming 3 of 5 available daily sessions. Two sessions exited with code 1 after hitting the 30-turn Claude API limit; one session completed successfully and created downstream tasks.
  • Status Checks: Zero failures in the last 2 hours, confirming instance and network health.

Critical Issue: OAuth Token Expiration in port_sheet_sync

The health check revealed a persistent authentication failure in the port_sheet_sync.py script, which runs every 30 minutes to synchronize data with Google Sheets. Every invocation has been failing with:

[port-sheet] token error: HTTP Error 400: Bad Request

This indicates the Google OAuth 2.0 token stored for this service account is either expired or has been revoked. Port sheet syncs have not completed since the token became invalid. The immediate action required is to re-authenticate the service account and refresh the stored token. This pattern is important to document: our authentication layer reuses OAuth tokens across multiple services by storing them in a credentials directory, but those tokens require periodic refresh or explicit re-authentication when they expire.

Analytics Infrastructure: GA4 Property and Data API Integration

In parallel with daemon validation, we migrated Google Analytics 4 access for a domain restructuring. The original domain directory /Users/cb/Documents/repos/sites/86dfrom.com was renamed to 86from.com to reflect updated branding. This required updating GA4 property associations and rebuilding the analytics data pipeline.

The authentication flow for GA4 access was established via a custom Python script (/Users/cb/Documents/repos/tools/auth_ga.py) that leverages Google's oauth2 library to obtain and cache credentials:

# Authenticate against Google Analytics for the dangerouscentaur account
python3 ~/Documents/repos/tools/auth_ga.py --account dangerouscentaur@gmail.com

# Lists all GA4 properties accessible under that account
python3 ~/Documents/repos/tools/list_ga_properties.py

Once authenticated, we queried the GA4 Data API to pull 7-day historical reports for the restructured property. This validated both the OAuth token reuse pattern (credentials were already cached from prior authentication) and confirmed that the property was correctly configured to receive traffic from the new domain.

Content Deployment and SEO Infrastructure

The restructured 86from.com site required updated content, including a new SEO-focused page at the path /what-does-86d-mean. This page was written to disk at /Users/cb/Documents/repos/sites/86from.com/site/what-does-86d-mean and deployed via our S3 + CloudFront architecture.

The main index.html for the site received multiple revisions to repair a critical JavaScript bug: template literal delimiters (double braces {{ and }}) used in a booking widget were conflicting with the page's templating engine. We isolated the booking widget JavaScript block, verified that braces outside this section were legitimate template syntax, and selectively replaced conflicting syntax only within the widget code.

# Deploy updated content to S3
aws s3 sync /Users/cb/Documents/repos/sites/86from.com/site/ \
  s3://86from-production/ \
  --delete

# Invalidate CloudFront cache to force edge nodes to fetch updated assets
aws cloudfront create-invalidation \
  --distribution-id E2ABC123EXAMPLE \
  --paths "/*"

We also deployed a staging version to a separate S3 bucket and CloudFront distribution for QA validation before production rollout, allowing the booking widget JavaScript to be syntax-checked in isolation.

Key Architectural Decisions

Temporary SSH Credentials Over Stored Keys: Rather than rotating and distributing private keys, we used Lightsail's API to generate temporary, short-lived SSH credentials. This reduces the surface area for key compromise and eliminates the operational burden of managing key distribution across development machines.

OAuth Token Caching and Reuse: The GA4 authentication script caches credentials to avoid re-authenticating on every analytics query. However, this pattern requires monitoring for token expiration and a clear refresh strategy (as evidenced by the port_sheet_sync failure).

Staged Deployment with Separate Distributions: Content is deployed to both staging and production S3 buckets with their own CloudFront distributions. This allows for full validation before production traffic is served, reducing the risk of deploying broken code.

What's Next

  • Re-authenticate port_sheet_sync: Run the Google OAuth flow for the port sheet service account to obtain a fresh token and resume 30-minute syncs.
  • Evaluate 30-Turn Limit: Two of three agent sessions today hit the Claude API turn limit. Assess whether task scope should be broken into smaller chunks or if the turn limit needs adjustment for complex orchestration tasks.
  • Monitor GA4 Data Pipeline: Confirm that 86from.com is properly configured in GA4 to receive traffic, and validate that the 7-day historical reports are being generated correctly post-migration.
  • Lock Down Secrets File Permissions: Ensure that credentials directories (containing OAuth tokens and API keys) have restrictive file permissions (mode 0600) to prevent accidental exposure.