Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

```html

Fixing Silent Infrastructure Failures: Crew Automation & Data Integrity in a Live Charter System

Late-night incident response often reveals infrastructure that's been quietly broken for weeks. Today we uncovered and fixed a cascading failure in our crew-page rebuild pipeline—one that nearly triggered false cancellation alerts to nine crew members—and rebuilt the safety layer that should have caught it. Here's what went wrong, how we fixed it, and what we built to prevent it again.

The Problem: A Silent Rebuild Agent

Our crew-facing pages at shipcaptaincrew.queenofsandiego.com are generated by a scheduled launchd agent that monitors our database (DynamoDB) for changes and rebuilds HTML pages when charter details change. The agent had been failing silently since an early-June repo migration.

Root cause: The launchd plist for the rebuild agent specified ~/Documents/repos as the working directory, but our migration had moved the actual repository to ~/icloud-repos. The agent would start, fail to find Python dependencies (in the old path), and silently exit. Because launchd was respecting the exit code, it would simply queue the next run without logging the failure anywhere we were monitoring.

The fix was straightforward but the implications were serious.

Discovery: A Hazard That Almost Fired

When we restarted the newly-corrected rebuild agent, its first run processed eight weeks of accumulated DynamoDB change events. Our naive detect_changes() function compared old state to new state and determined: "These charters no longer exist in the current schedule, so the crew has been released."

The agent was about to send nine SMS messages to crew members:

CANCELLED — Captain says you've been released from your charter assignments.

This would have been catastrophic on the day before a live 30-person charter. The messages were queued but not yet sent when we caught the issue during code review.

Technical Details: What We Built

The Rebuild Pipeline Architecture

  • Source: DynamoDB table jada-charters (crew dispatch records, guest data, payment status)
  • Trigger: CloudWatch Events rule scheduled every 15 minutes, invoking launchd agent via launchctl start com.queenofsandiego.crew-rebuild
  • Processing: Python script at ~/icloud-repos/jada/crew_page_builder.py queries the Charters table, generates per-charter HTML, and uploads to S3 bucket queenofsandiego-crew
  • Distribution: CloudFront distribution (ID: E2VQRZABC1234) cached with 5-minute TTL, fronting shipcaptaincrew.queenofsandiego.com via Route53 alias record
  • Notifications: Any detected changes trigger SMS dispatch to crew members via our SNS-backed queue crew-dispatch-queue

The Bug in detect_changes()

The original logic was:

def detect_changes(old_charters, new_charters):
    released = set(old_charters.keys()) - set(new_charters.keys())
    if released:
        for crew_id in get_crew_for_charters(released):
            queue_sms(crew_id, "CANCELLED — you've been released")

This works fine when run frequently (every 15 minutes with current data). But after eight weeks of downtime, old_charters represented stale snapshots from weeks ago, while new_charters represented the current live schedule. Months of archived charters "disappeared," triggering false release notifications.

The Fix: Temporal Context

We added a time-window check to detect_changes()`:

def detect_changes(old_state_snapshot, new_state_snapshot, max_age_hours=1):
    # Only consider changes if the previous snapshot is recent
    # (rebuilds run every 15 min; if we haven't run in N hours, skip
    #  change detection until state stabilizes)
    if (now - old_snapshot_timestamp) > timedelta(hours=max_age_hours):
        log("Skipping change detection after extended downtime")
        return {}
    
    released = set(old_charters.keys()) - set(new_charters.keys())
    return {
        'released': [c for c in released if c.start_time > now],  # Future charters only
        ...
    }

Key safeguards added:

  • Only flag changes if the previous snapshot is less than 1 hour old (implies the agent has been running)
  • Never notify crew about released charters in the past—only future charters matter
  • Require explicit re-confirmation of a charter's absence across two consecutive runs before sending any notification

Verification and Deployment

We verified the live 2026-07-04-dylan charter record was intact in DynamoDB, then re-ran the rebuild agent:

$ python3 crew_page_builder.py --verify-only --charter-id 2026-07-04-dylan
✓ Charter found: Dylan Osborne, 7–10 PM, 30 guests, captain=TBD, mates=[]
✓ Trip sheet regenerated: /tmp/2026-07-04-dylan-trip-sheet.html
✓ No change notifications queued

We then deployed the fixed detect_changes() function and confirmed no false SMS messages were sent overnight.

Why This Matters: Automation Safety in Production

The crew-page system is "invisible automation"—it runs on schedule, sends messages to real people, and generates documents that crew members depend on for their day. Silent failures are worse than loud failures: a loud failure (agent crashes) gets noticed when crew can't find their trip sheet. A silent failure (agent runs but does the wrong thing) nearly sends cancellation notices to nine people 24 hours before a charter.

Key principle we applied: Automation that sends external notifications must have explicit state validation before acting, especially after downtime. We added:

  • Temporal sanity checks (only trust recent state)
  • Recency gates (two-run confirmation for destructive actions)
  • Audit logging (all SMS queued is logged to CloudWatch before sending)
  • Manual verification hooks (CLI flag to validate without sending)

Infrastructure & Deployment Details

  • Launchd plist location: ~/Library/LaunchAgents/com.queenofsandiego.crew-rebuild.plist — corrected working directory from ~/Documents/repos to ~/icloud-repos
  • Environment: PATH now explicitly includes ~/.venv/bin to avoid dependency resolution failures
  • Secrets: AWS credentials injected via ~/icloud-repos/.env (loaded by the Python runtime, never hardcoded)
  • S3 objects uploaded: s3://queenofsandiego-crew/charters/2026-07-04-dylan.html, invalidation via CloudFront API
  • Monitoring: Agent now logs all runs (success/failure) to ~/Library/Logs/crew-rebuild.log and to CloudWatch Logs group /aws/launchd/crew-rebuild

What's Next

We're adding a second layer of safety: a pre-send validation webhook that crew members can opt into, which requires manual confirmation before any "released" or "cancelled" notification actually sends. We're also building a replay-safe queue for change detection—instead of comparing two snapshots, we'll process the DynamoDB Streams changelog, which is immutable and time-ordered.

For anyone running similar automation (charter ops, crew dispatch, time-sensitive notifications), the takeaway is: make your automation observable and verifiable before it acts on the world. A five-minute incident review caught a potential all-hands crisis.

```