Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

```html

Fixing Four Weeks of Silent Crew-Page Rebuild Failures: LaunchAgent PATH Resolution and DynamoDB Scan Hardening

What Was Done

The crew-page rebuild pipeline, critical for publishing daily crew dispatch, SMS alerts, and manifest pages to S3, had been silently failing for four weeks after a June 3 AWS credential migration. The root cause: bare aws and ~/bin command references in /Users/cb/icloud-jada-ops/crew-pages/rebuild.py that launchd could not resolve without a full shell environment.

The fix involved three layers:

  • Explicit PATH resolution in subprocess calls to aws CLI and local binaries
  • Chunked DynamoDB scans to handle network flakiness and Mac system resource constraints
  • Retry and timeout hardening across the pipeline, plus a fix to SMS change-detection logic that nearly triggered 9 false cancellation alerts to crew

Technical Details

Root Cause: LaunchAgent Environment Isolation

When com.dangerouscentaur.crew-dispatch.plist launches rebuild.py, launchd provides a minimal environment—PATH defaults to /usr/bin:/bin:/usr/sbin:/sbin. Subprocess calls like:

subprocess.run(['aws', 's3', 'cp', ...], ...)

fail silently because aws is installed via Homebrew at /usr/local/bin/aws, and ~/bin tooling isn't on PATH. The script continued executing, but every AWS operation (credential refresh, DDB scans, S3 uploads) was failing with "command not found" buried in stderr that never reached alerting.

Solution: Explicit Path Resolution

All subprocess calls now use full paths resolved at import time:

import os
import subprocess

AWS_PATH = '/usr/local/bin/aws'
LOCAL_TOOLS = os.path.expanduser('~/bin')

def scan_ddb_chunk(last_key=None, limit=50):
    cmd = [
        AWS_PATH, 'dynamodb', 'scan',
        '--table-name', 'jada-crew-dispatch',
        '--region', 'us-west-2',
        '--limit', str(limit)
    ]
    if last_key:
        cmd.extend(['--exclusive-start-key', json.dumps(last_key)])
    
    result = subprocess.run(cmd, capture_output=True, text=True, timeout=30)
    if result.returncode != 0:
        raise RuntimeError(f"DDB scan failed: {result.stderr}")

This approach:

  • Removes dependency on shell PATH resolution within launchd
  • Makes the code testable outside launchd (dry-run verification now works)
  • Fails fast with clear error messages instead of silently returning empty results

DynamoDB Chunked Scans: Handling Network Flakiness

The crew-dispatch table, accessed hundreds of times daily across multiple agents, experiences occasional latency spikes on the Mac's wireless connection. A single full-table scan can hit 800+ items and timeout. Chunking trades increased request count for better fault isolation:

def scan_all_pages(table_name='jada-crew-dispatch', chunk_size=50, max_retries=3):
    all_items = []
    last_key = None
    page_num = 0
    
    while True:
        for attempt in range(max_retries):
            try:
                items, last_key = scan_ddb_chunk(last_key, chunk_size)
                all_items.extend(items)
                page_num += 1
                print(f"[DDB] page {page_num}: {len(items)} items (attempt {attempt+1})")
                break
            except subprocess.TimeoutExpired:
                if attempt == max_retries - 1:
                    raise
                time.sleep(2 ** attempt)  # exponential backoff
        
        if not last_key:
            break
    
    return all_items

Instead of a single 60-second timeout for the entire scan, each 50-item page gets a 30-second timeout. Network hiccups on page 7 don't kill the entire operation; we retry just that page.

SMS Change Detection: The False-Alert Trap

The rebuild runs before sending SMS alerts to crew. After four weeks of downtime, the snapshot in /Users/cb/icloud-jada-ops/crew-pages/snapshots/last-events.json was stale. When rebuild.py revived and compared current DDB state to the stale snapshot, the detect_changes() function incorrectly identified 8 aged-out events and a duplicate Jul-4 record as "cancelled" charters.

This would have triggered 9 SMS messages: "CANCELLED — you've been released" to crew for old events and the Jul 4 Dylan Osborne charter, which DDB verified was ACTIVE.

Fix in detect_changes():

def detect_changes(current, snapshot):
    """Detect real changes; filter false positives from stale snapshots."""
    changes = []
    
    # Only flag as "cancelled" if explicitly marked cancelled in current state
    for event_id, current_event in current.items():
        if event_id not in snapshot:
            if current_event.get('status') == 'CANCELLED':
                changes.append(('cancelled', event_id, current_event))
        elif snapshot[event_id].get('status') != 'CANCELLED' and current_event.get('status') == 'CANCELLED':
            changes.append(('cancelled', event_id, current_event))
    
    # No longer flag missing-from-snapshot as "cancelled"
    # (stale snapshots create false positives on agent restart)

Additionally, the Jul 4 duplicate was removed from the snapshot and verified directly in DDB with: aws dynamodb get-item --table-name jada-crew-dispatch --key '{"id":{"S":"2026-07-04-dylan"}}'. Result: ACTIVE, confirmed.

Infrastructure Changes

  • Modified: /Users/cb/icloud-jada-ops/crew-pages/rebuild.py — explicit AWS path resolution, chunked DDB scans, retry logic, fixed detect_changes()
  • Table: jada-crew-dispatch (us-west-2) — scanned in 50-item chunks instead of one full scan
  • S3 buckets: crew-page snapshots and manifests pushed to their respective CloudFront-fronted buckets; no changes to bucket names or policies
  • LaunchAgent: com.dangerouscentaur.crew-dispatch.plist — no plist changes needed; explicit paths in Python handle the environment isolation
  • Snapshot: /Users/cb/icloud-jada-ops/crew-pages/snapshots/last-events.json — refreshed from live DDB; stale Jul 4 duplicate removed

Key Decisions

Why chunked DDB scans instead of pagination: Pagination tokens require consuming the full result set. Chunking lets us retry individual pages without re-scanning from the beginning, crucial for unreliable network. The tradeoff is more API calls (12 calls × 50 items vs. 1 call × 600 items), but at crew-dispatch's ~hourly update rate and us-west-2 latency, this is negligible.

Why explicit paths instead of shell wrapping: A shell wrapper (e.g., subprocess.run(['bash', '-c', 'source ~/.bashrc && aws ...'])) would work but obscures what launchd is actually executing. Explicit paths make the dependency graph transparent and testable.

Why remove the "stale snapshot = cancelled" heuristic: After four weeks of downtime, that heuristic is a minefield. Better to be conservative: only flag events that are explicitly CANCELLED in DDB. If an event is missing from the old snapshot, it's either new or aged out—not necessarily cancelled.

What's Next

  • Monitor the next five rebuild runs (once per day) to confirm zero alerts and successful S3 uploads
  • Rotate the AWS IAM key exposed in the Lightsail LaunchAgent plist (separate incident, same audit cycle)
  • Consider adding a heartbeat check to the LaunchAgent: if rebuild.py exits with non-zero status, send a digest email; dark operations remain the biggest risk
  • Backport chunked-scan logic to other DDB-dependent agents (unsubscribe watcher, payment dispatcher) after verifying crew-dispatch stability
```