From Manual Checks to Deterministic Cron: Building Automated Verification for a Multi-Service Booking Pipeline

```html

When every piece of your infrastructure is load-bearing — payment processors, email deliverability, credential refresh, crew scheduling — the definition of "done" changes. A feature isn't finished until an automated, deterministic test runs on cron and proves it stays fixed. This post covers how we rebuilt JADA (the booking system for queenofsandiego.com) from ad-hoc monitoring into a comprehensive, self-verifying suite.

The Problem: Silent Failures in Critical Paths

The booking pipeline touches six major systems:

Lead intake and proposal generation
Payment processing and credential management
Email delivery and unsubscribe handling
Crew page rendering and gating
Calendar and DynamoDB sync
Client waivers and manifest generation

Each system had some monitoring — launchd jobs running scripts, manual log checks, occasional "why didn't the payment go through?" investigations — but none were deterministic. A hung process wouldn't alert for hours. A credential rotation that silently failed wouldn't surface until a client complained. Tests didn't run unless someone remembered to trigger them.

Phase 1: Fixing Existing Monitoring Infrastructure

Payment Monitor Hanging: The payment monitor script (/Users/cb/icloud-repos/tools/jada_payment_monitor.py) had a critical bug: it called urlopen() with no socket timeout. A single stalled TCP connection to the payment provider would wedge the entire process forever, unnoticed by launchd.

Fix: Add a global socket timeout before any network operations:

import socket
socket.setdefaulttimeout(15)  # Covers urlopen, token refresh, and all downstream calls

This simple line made the monitor fail-fast instead of hang indefinitely. We verified the fix by restarting via launchctl kickstart -k system/com.jada.payment-monitor and confirming fresh logs appeared within expected intervals.

AWS Credential Profile Chaos: The unsubscribe processor (/Users/cb/icloud-jada-ops/email-lists/build/process_unsubscribes.py), SMS alert script (/Users/cb/bin/sms-flush), and rebuilder (com.jada.rebuild-crew-pages) all referenced different AWS credential profiles. Some used default, others used queenofsandiego. When credentials rotated, some jobs failed silently while others succeeded, creating phantom failures that looked like application bugs.

Fix: Standardize on the queenofsandiego profile everywhere, verified with:

grep -r "AWS_PROFILE\|profile_name\|--profile" /Users/cb/bin /Users/cb/icloud-repos/tools /Users/cb/icloud-jada-ops

Then updated all LaunchAgent plists to export AWS_PROFILE=queenofsandiego before launching scripts.

Phase 2: Building Deterministic Test Suites

We created four test modules, each testing one slice of the pipeline:

Launchd Job Health (test_launchd_jobs.py): Tests that all JADA jobs are running and their heartbeat files are fresh. Checks symlink targets in ~/Library/LaunchAgents, verifies exit codes via launchctl list, and confirms logs were written within the last 24 hours.

Crew Pages Gate (test_crew_pages_gate.py): Verifies that public crew pages render correctly, private pages require auth, and gating rules are enforced. Tests against the live site.

Booking Pipeline (test_booking_pipeline.py): End-to-end flow: lead intake, proposal generation, payment simulation (without charging), confirmation email delivery, and crew dispatch. Uses mock payment providers to avoid charges.

Guest Page Verification: Tests that guest pages include Google Analytics tags and proper noindex meta tags (required for SEO compliance). Checks both local files and S3-cached versions to catch CDN invalidation issues.

All tests run nightly via /Users/cb/bin/jada-tests-nightly.py, which is triggered by the launchd job com.jada.tests-nightly. Test results are published to S3 and trigger Slack/SES alerts on failure.

Phase 3: Watchdog Lambda and Alerting

Tests only help if failures are surfaced immediately. We deployed a Lambda watchdog (/Users/cb/icloud-repos/sites/queenofsandiego.com/tools/jada-tests-watchdog/lambda_function.py) that:

Runs every 6 hours via EventBridge rule jada-tests-watchdog-schedule
Checks for a fresh heartbeat file in S3 (written by the nightly suite after each run)
Compares the most recent heartbeat timestamp against a threshold (currently 6.5 hours)
On stale heartbeat or missing file, sends an SES alert to oncall with the error and launchd logs

The Lambda execution role (dc-uptime-lambda-role) has inline policies for S3 read, SES send, and CloudWatch logs. This ensures that if the nightly suite itself crashes silently, the watchdog will catch it within 6 hours.

Key Decisions and Trade-offs

Cron + Launchd over Step Functions: We kept the test runner as a local launchd job rather than moving to AWS Step Functions. Reason: the booking pipeline is tightly coupled to the macOS filesystem (crew page sources live in iCloud Drive, launchd agents are in ~/Library). Moving to Lambda would require mirroring that state to S3, adding complexity. Local cron is simpler and faster.

Heartbeat-based Watchdog: Rather than directly monitoring launchd jobs from Lambda (which would require complex SSH or CloudWatch agent setup), we use a heartbeat file. The nightly suite writes nightly-heartbeat.json to S3 after each run, including exit code and runtime. The watchdog just checks timestamps. This is resilient: if the nightly suite hangs but the heartbeat is stale, we alert. If it completes successfully, the heartbeat is fresh.

Global Socket Timeout: The payment monitor fix uses a global socket.setdefaulttimeout(15) rather than per-call timeouts. This is simpler (one line instead of modifying every urlopen/requests call) and catches edge cases like token refresh that might be buried in library code.

What's Next

The current suite covers the happy path and basic health. Next milestones:

DynamoDB Sync Tests: Verify that crew schedules sync correctly from Google Calendar to DynamoDB, including handling of timezone changes and all-day events.
Waiver Generation: Test that client waivers render correctly, include all required fields, and are properly gated behind authentication.
Graceful Degradation: Test that if one system fails (e.g., email provider is down), the pipeline fails fast with clear alerts rather than leaving the booking in a limbo state.
Performance Regression Detection: Add latency assertions to crew page rendering and proposal generation so we catch slowdowns before clients notice.

Every piece of this system is now automatically verified on cron. When something breaks, we know within 6 hours max. And because every fix is backed by a deterministic test that runs forever, we never regress.

```