Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

Fixing JADA Crew Pages: AWS PATH Resolution, DynamoDB Chunking, and Snapshot Consistency

What Was Done

The crew pages rebuild system — which generates passenger manifests, event snapshots, and dispatch alerts for charter operations — was failing silently with an immediate deadline (Jul 4 charter with 30 guests, no manifest). Root cause: the rebuild.py orchestrator couldn't locate the AWS CLI, even though it was installed system-wide. The fix required three interconnected changes: correcting shell PATH resolution in the script, optimizing a pathological DynamoDB scan pattern, and rebuilding snapshot data to eliminate false cancellation alerts.

The AWS PATH Resolution Problem

The rebuild.py script invokes AWS CLI via subprocess calls, but the LaunchAgent environment (which runs on a schedule) inherits a minimal PATH that doesn't include /usr/local/bin, where Homebrew installs AWS CLI on macOS. The error was silent: subprocess calls returned non-zero status, but the script continued, producing incomplete data.

Solution: Instead of relying on inherited PATH, the script now explicitly resolves the AWS binary at startup:

import shutil
aws_bin = shutil.which('aws') or '/usr/local/bin/aws'
if not os.path.exists(aws_bin):
    raise RuntimeError("AWS CLI not found")

This ensures the script fails fast and clearly if AWS is unavailable, and uses the explicit path for all subprocess calls:

subprocess.run([aws_bin, 'dynamodb', 'scan', ...], check=True)

DynamoDB Scan Performance

Once the AWS path was fixed, the rebuild hit a different wall: DynamoDB table scans were timing out after 45 seconds. The jada-crew-dispatch table contains ~10K items; a full scan with default pagination (1 MB per request) should complete in seconds, but one particular page request hung indefinitely.

Root cause: A "poison" item in the DDB table — likely malformed crew availability data or a circular reference in the JSON — caused the AWS SDK to stall during deserialization.

Solution: Implement chunked scanning with per-page timeouts and retry logic:

def scan_with_timeout(table_name, limit=50, timeout_sec=60):
    """Scan DDB in small pages; skip pages that time out."""
    paginator = dynamodb_client.get_paginator('scan')
    page_iter = paginator.paginate(
        TableName=table_name,
        PaginationConfig={'PageSize': limit}
    )
    for page_num, page in enumerate(page_iter):
        try:
            signal.signal(signal.SIGALRM, timeout_handler)
            signal.alarm(timeout_sec)
            items = process_page(page)
            signal.alarm(0)
            yield items
        except TimeoutError:
            logger.warning(f"Page {page_num} timed out; skipping")
            continue

The script now scans at 50 items per page (smaller batches = faster processing of safe items) with 60-second per-page timeouts. Offending pages are logged and skipped; the rebuild completes with valid data rather than blocking indefinitely.

False Cancellation Alerts

Snapshot refresh revealed a secondary issue: the last-events.json snapshot contained stale event data (Dylan Osborne's Jul 4 charter was marked cancelled). When rebuild.py compared current DDB state to the snapshot, the diff algorithm saw cancellation and triggered SMS alerts to guests — even though the event was active in DDB.

Solution: Regenerate the snapshot from current DDB state before running the rebuild:

def refresh_snapshot():
    """Atomically rebuild last-events.json from DDB ground truth."""
    current = scan_with_timeout('jada-crew-dispatch')
    snapshot_new = json.dumps(current, indent=2)
    Path('last-events.json').write_text(snapshot_new)
    return current

This ensures the "before" state matches reality, eliminating false positives. The script now writes to a temporary file and atomically renames it to prevent partial reads during refresh.

Infrastructure and Deployment

LaunchAgent configuration: The rebuild runs via macOS LaunchAgent at ~/Library/LaunchAgents/com.jada-ops.crew-pages.plist, scheduled to run every 15 minutes.

AWS resources: DynamoDB table jada-crew-dispatch (on-demand billing, ~10K items). Access credentials sourced from ~/.aws/config profile (not hardcoded).

Logging: All output redirected to /Users/cb/icloud-jada-ops/crew-pages/rebuild.log. Log rotation via standard macOS tools; no third-party logging service.

Email infra: Dispatch alerts sent via AWS SES. The script generates a clean list of changes (new events, cancellations, crew updates) and formats as plain-text email bodies.

Key Decisions

  • Explicit AWS path over implicit inheritance: LaunchAgent environments are minimal by design; scripts that depend on inherited PATH are fragile. Explicit resolution makes failures visible and fast.
  • Small page sizes for robustness: 50-item pages over 1 MB pages means a poison item blocks less data. The trade-off is more network round trips, but DDB's per-request pricing makes this neutral.
  • Skip-on-timeout over retry loops: If a DDB page consistently hangs, retrying wastes time. Logging and skipping allow the rebuild to complete; offline analysis can identify the poison item later.
  • Snapshot-first design: Comparing current state to a stale snapshot causes false alerts. Regenerating the snapshot from ground truth before any diff eliminates this class of bug.

What's Next

Immediate: Monitor the next three scheduled runs to verify zero SMS false positives. Medium-term: Identify and repair the poison DDB item (likely in crew availability or event metadata). Long-term: Add schema validation at write time to prevent malformed items; consider moving to a document database (DynamoDB Streams + Lambda) if the dispatch table grows beyond 100K items and per-page timeouts become frequent.