Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

```html

Establishing Deterministic Test Coverage for the JADA Booking Pipeline

What Was Done

Defined "done" for critical infrastructure as a deterministic, automated test running on cron, then instrumented the entire JADA booking pipeline with health checks, test suites, and alerting. This covers lead intake through payment processing, crew dispatch, and live site availability.

The Problem: Definition of Done Without Verification

The JADA booking system consists of dozens of interdependent components—Google Sheets automation, Stripe integration, crew dispatch flows, LaunchAgent jobs running on a local macOS system, Lambda functions in AWS, and static sites served via CloudFront. Many were not getting tested before being marked complete, and failures in production weren't detected until clients reported them.

The solution: adopt a hard rule that a feature or fix is only "done" when a deterministic, automated check running on cron verifies it stays fixed.

Technical Details: Issues Found and Fixed

1. Payment Monitor Hung Forever (Socket Timeout Missing)

The /Users/cb/icloud-repos/tools/jada_payment_monitor.py script makes HTTP calls to check Stripe payment status but had no timeout on urlopen(). A single stalled TCP connection would wedge the entire monitor process indefinitely, breaking payment notifications.

Fix: Added a global socket timeout at module import:

import socket
socket.setdefaulttimeout(10)

This wraps both the initial HTTP request and the internal token refresh logic without code duplication. Verified by killing the hung process and restarting via LaunchAgent.

2. Three LaunchAgent Jobs Silently Failing

The com.jada.rebuild-crew-pages, com.jada.magic-link-heartbeat, and com.jada.unsubscribe-monitor jobs were exiting with non-zero codes. Root causes:

  • rebuild-crew-pages: AWS profile configuration mismatch. Script used default AWS creds, but environment was set to queenofsandiego profile. Fixed by standardizing all scripts to use AWS_PROFILE=queenofsandiego.
  • magic-link-heartbeat: Same AWS profile issue. Reloaded the plist at /Users/cb/Library/LaunchAgents/com.jada.magic-link-heartbeat.plist with environment variables.
  • unsubscribe-monitor: New script created (see below); fixed during initial setup.

Verification: Used launchctl list to check exit codes before/after restarts, confirming all four jobs now exit with status 0.

3. Created Three New Test Suites

Built deterministic tests for load-bearing paths:

  • /Users/cb/icloud-repos/sites/queenofsandiego.com/tests/test_launchd_jobs.py — Verifies all JADA LaunchAgent jobs are loaded, running, and exiting cleanly. Checks logs are recent (within 6 hours) and no stale heartbeat files exist.
  • /Users/cb/icloud-repos/sites/queenofsandiego.com/tests/test_crew_pages_gate.py — Validates crew dispatch pages (e.g., g/2026-06-27-cathy-afternoon.html) render without errors and have correct metadata (event codes, noindex tags).
  • /Users/cb/icloud-repos/sites/queenofsandiego.com/tests/test_booking_pipeline.py — End-to-end smoke test: new booking leads can be captured, payment requests initiated, and confirmation emails sent.

4. SMS Alerting for Critical Failures

The nightly test suite now sends SMS alerts on failure. Pattern modeled after existing payment monitor (constants in jada_payment_monitor.py). Added SMS alert block to new /Users/cb/bin/jada-unsub-monitor.py; reused the same approach in test fixtures.

Infrastructure: Cron, LaunchAgents, and Lambda Watchdog

macOS LaunchAgent Jobs

All JADA background work runs via LaunchAgent plists in /Users/cb/Library/LaunchAgents/. Key jobs:

  • com.jada.tests-nightly — Runs /Users/cb/bin/jada-tests-nightly.py every night at 00:43. Executes all three test suites and writes a heartbeat file to ~/icloud-jada-ops/logs/nightly-tests.log.
  • com.jada.payment-monitor — Polls Stripe every 15 minutes, now with socket timeout protection.
  • com.jada.unsubscribe-monitor — New job that scans IMAP inboxes for unsubscribe requests, processes them in bulk, and logs activity. Runs every 6 hours.
  • com.jada.morning-digest — Generates daily summary of crew bookings, payments, and alerts.

Reload command: After updating any plist, run:

launchctl unload ~/Library/LaunchAgents/com.jada.JOBNAME.plist
launchctl load ~/Library/LaunchAgents/com.jada.JOBNAME.plist
launchctl start com.jada.JOBNAME

AWS Lambda Watchdog

Created /Users/cb/icloud-repos/sites/queenofsandiego.com/tools/jada-tests-watchdog/lambda_function.py. This Lambda runs every 6 hours via EventBridge and checks the nightly heartbeat file in S3 (s3://JADA-BUCKET/heartbeats/tests-nightly.txt). If the heartbeat is stale (older than 8 hours) or missing, it sends an SES alert to ops.

Why Lambda instead of cron: The nightly test suite runs on a local macOS machine; a Lambda-based watchdog can detect when the machine has lost network, crashed, or drifted offline without relying on that same machine to report its own failure.

S3 and CloudFront Configuration

Heartbeat and test logs published to S3 bucket under s3://JADA-BUCKET/heartbeats/. CloudFront distribution (ID not disclosed for security) serves both test results and live booking sites. No changes to distribution config; logs already cached appropriately.

Key Decisions

  • Socket timeout as default, not per-call: Setting socket.setdefaulttimeout(10) at module load ensures every network operation—including hidden token refreshes—respects the timeout without scattered try-except blocks.
  • Standardize AWS profile across all scripts: Rather than conditionally check profiles, explicitly set AWS_PROFILE=queenofsandiego in every LaunchAgent plist. One source of truth, no ambiguity.
  • Heartbeat file + Lambda watcher: MacOS LaunchAgent jobs can fail silently if the machine restarts or network drops. A Lambda checking S3 heartbeats is independent infrastructure that detects when the primary machine is down.
  • SMS alerts on test failure: For a business-critical pipeline, SMS is faster and more reliable than email during an incident. SMS block reused from payment monitor to keep maintenance surface small.
  • Separate test suites by layer: LaunchAgent health → Crew page rendering → Booking end-to-end. Each can fail independently; knowing which layer broke is critical for triage.

Verification and Logs

All logs written to ~/icloud-jada-ops/logs/. Key files:

  • nightly-tests.log — Full output of test suite runs; checked for recent timestamp to verify cron execution.
  • payment-monitor.log — Stripe polling logs; now includes timeout markers when connections are interrupted.
  • unsubscribe-monitor.log — IMAP scan results and unsubscribe processing records.

Command to verify all jobs are healthy:

launchctl list | grep com.jada
for job in tests-nightly payment-monitor magic-link-heartbeat unsubscribe-monitor; do
  echo "=== $job ===" 
  launchctl list | grep com.jada.$job
done

What's Next

Future hardening:

  • Extend test coverage to Google Sheets triggers and Apps Script deployments.
  • Add compliance checks (GDPR unsubscribe audit, Stripe refund logs).
  • Monitor CloudFront cache hit ratios and invalidation frequency to catch unexpected traffic patterns.
  • Integrate crew dispatch schedule conflicts into automated gate tests.
  • Build a dashboard in CloudWatch that aggregates all heartbeat and test status signals.

The core principle: if it makes money or keeps clients happy, it gets an automated deterministic check on cron. No exceptions. No manual verification. Done means tested, and tested means cron.

```