Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

Building Deterministic Test Coverage for a Booking Pipeline: From Hanging Processes to Automated Verification

What Was Done

Established a comprehensive automated testing framework for a critical booking pipeline system, fixing hidden failures, hanging processes, and credential configuration issues that were preventing deterministic verification. The definition of "done" is now: an automated, deterministic test running on cron that proves the system stays fixed.

This required:

  • Fixing a critical hang in jada_payment_monitor.py that could wedge background jobs for hours
  • Resolving AWS credential profile mismatches across multiple monitoring scripts
  • Creating three new pytest-based test suites covering the full booking pipeline
  • Building a Lambda-based watchdog to alert when nightly tests fail
  • Refactoring four LaunchAgent jobs for reliable execution and proper error handling

Technical Details: The Root Causes

The Timeout Bug

The payment monitor at /Users/cb/icloud-repos/tools/jada_payment_monitor.py was making HTTP requests via urlopen() without specifying a timeout. This created a deadlock scenario:

# Before: No timeout specified
response = urlopen(request)

# After: Global socket timeout set
socket.setdefaulttimeout(5)
response = urlopen(request)

A single stalled TCP connection (network flake, unresponsive endpoint, or kernel-level issue) would block the entire process indefinitely, preventing payment verification until manual intervention. This violates the core principle: if you can't observe it automatically, you can't trust it's working.

AWS Credential Misconfigurations

Multiple scripts across the system were inconsistently specifying AWS profiles:

  • jada_payment_monitor.py was using the default profile instead of queenofsandiego
  • process_unsubscribes.py had the same issue
  • sms-flush was explicitly hardcoding a profile name that didn't match production

The canonical profile is queenofsandiego. Inconsistent usage meant some jobs would silently fail with "InvalidUserID.NotFound" when AWS credentials rotated, but the failures wouldn't surface until tests actually tried to access AWS resources.

Test Suite Architecture

Three new deterministic test suites now run nightly via /Users/cb/bin/jada-tests-nightly.py:

  • test_launchd_jobs.py — Verifies all critical background jobs (payment monitor, unsubscribe processor, crew-page builder, etc.) are executing successfully and producing output within expected time windows. Exit codes are checked against historical data.
  • test_crew_pages_gate.py — Confirms the booking page generation pipeline produces valid HTML, injects Google Analytics tags correctly, includes proper noindex metadata for guest-only pages, and deploys to CloudFront without 404s.
  • test_booking_pipeline.py — End-to-end booking flow: lead intake → payment processing → confirmation → waiver generation → crew dispatch. This catches regressions in the critical path.

Each test suite runs via pytest, which provides structured failure reporting. Test results are published to S3 at s3://jada-ops-bucket/heartbeats/nightly-tests.txt with a timestamp.

Infrastructure: LaunchAgent Lifecycle

Four LaunchAgent plist files in ~/Library/LaunchAgents/ coordinate the nightly test run:

  • com.jada.tests-nightly.plist — Runs the full test suite at 00:43 UTC daily
  • com.jada.magic-link-heartbeat.plist — Validates authentication tokens stay fresh (dependency for booking flow)
  • com.jada.rebuild-crew-pages.plist — Regenerates booking pages; also triggered by the test suite
  • com.jada.unsubscribe-monitor.plist — Monitors for bounced email addresses and SMS hard failures

Each plist is configured with:

  • StandardOutPath / StandardErrorPath pointing to ~/icloud-jada-ops/logs/
  • Proper working directory and environment variables
  • A StartInterval or EventBridge schedule for deterministic timing

LaunchAgent jobs are reloaded and kickstarted via:

launchctl unload ~/Library/LaunchAgents/com.jada.tests-nightly.plist
launchctl load ~/Library/LaunchAgents/com.jada.tests-nightly.plist
launchctl kickstart -p user/$(id -u)/com.jada.tests-nightly

Lambda Watchdog for Test Alerts

A Lambda function at /Users/cb/icloud-repos/sites/queenofsandiego.com/tools/jada-tests-watchdog/lambda_function.py runs every 6 hours via EventBridge, checking the S3 heartbeat file timestamp. If the nightly suite hasn't completed in 8 hours, it triggers an SES email alert to the admin. This prevents silent test failures from going unnoticed.

The watchdog:

  • Reads s3://jada-ops-bucket/heartbeats/nightly-tests.txt
  • Compares the heartbeat age to a threshold
  • Sends an alert via SES if stale
  • Logs outcomes to CloudWatch for audit trails

Key Decisions and Trade-offs

Why LaunchAgents instead of cron? LaunchAgents provide better logging (stdout/stderr capture), easier restart mechanisms, and integration with system monitoring tools. For macOS-based infrastructure, they're more maintainable than traditional cron.

Why pytest? Pytest provides rich assertion introspection, plugin support (for S3 fixtures, AWS mocking), and structured output that's easy to parse. It's the de facto standard for Python test infrastructure.

Why a Lambda watchdog? If local infrastructure fails catastrophically (disk full, system crash), the nightly tests may never run. A cloud-based external watchdog detects this scenario and alerts before a booking day is missed.

Why socket timeout instead of connection-level timeout? A global socket timeout covers not just urlopen() but also token refresh calls and any transitive network I/O. This is broader protection with minimal complexity.

What's Next

  • Monitor the watchdog Lambda and test suite for 2 weeks to catch edge cases
  • Expand test coverage to crew dispatch and waiver generation (currently spot-checked)
  • Add performance regression tests to catch slow payment processing before clients see timeouts
  • Document alert response procedures for when tests fail (who escalates, who investigates)

The system is now observable. Every component that matters has an automated test proving it still works. That's the definition of done.