Restoring the Crew-Page Pipeline: Debugging 4 Weeks of Silent DynamoDB Failures
The crew-page rebuild pipeline—a critical system that publishes 15 vessel crew manifests to S3 every 30 minutes via a macOS LaunchAgent—had been silently failing for four weeks. With a major charter departing tomorrow (30 guests, 15 crew), we needed to restore it. Here's how we diagnosed a cascade of issues rooted in DynamoDB scan performance, PATH resolution, and false-positive alerting logic.
The Problem: Silent Failure in Production
The LaunchAgent at ~/Library/LaunchAgents/com.dangerouscentaur.dev-agent.plist invokes crew-pages/rebuild.py every 30 minutes. The script should fetch crew dispatch data from a DynamoDB table (jada-crew-dispatch), render 15 HTML/JSON crew pages, and upload them to S3. For weeks, it was returning zero pages and producing no alert—a textbook case of a broken monitoring assumption.
The immediate symptoms:
- Zero of 15 crew pages were deploying
- No error logs, no email alerts, no visibility
- The LaunchAgent plist showed the script was executing
- The S3 bucket reflected stale data from weeks prior
Root Cause: AWS CLI PATH Misconfiguration
The first failure was straightforward: rebuild.py spawns the AWS CLI to scan DynamoDB, but the subprocess call did not inherit the shell PATH where aws was installed. The script hard-coded a relative path lookup that failed silently and returned an empty dataset, which the script then interpreted as "no crew to publish."
The fix was simple but critical—replace the subprocess invocation with an absolute path or use the shell's PATH explicitly:
# Before: subprocess call without explicit shell/PATH
subprocess.run(['aws', 'dynamodb', 'scan', '--table-name', 'jada-crew-dispatch', ...])
# After: explicit shell=True or full PATH
import shutil
aws_path = shutil.which('aws')
if not aws_path:
raise RuntimeError("aws CLI not found in PATH")
subprocess.run([aws_path, 'dynamodb', 'scan', '--table-name', 'jada-crew-dispatch', ...])
Second Issue: DynamoDB Scan Timeout and Pagination
Once the AWS CLI path was fixed, we hit a second problem: the full table scan of jada-crew-dispatch was hanging. A naive single-shot scan with no pagination limit was attempting to pull all crew records in one request. With items streaming in at 50–200 items per response page, the scan was exceeding the timeout cap.
We implemented chunked scanning with explicit pagination:
def scan_crew_dispatch_paginated(table_name, page_size=50, timeout_secs=60):
"""Scan DynamoDB with explicit pagination and timeout enforcement."""
paginator = dynamodb.get_paginator('scan')
page_iterator = paginator.paginate(
TableName=table_name,
PaginationConfig={'PageSize': page_size}
)
crew_data = []
start = time.time()
for page in page_iterator:
if time.time() - start > timeout_secs:
raise TimeoutError(f"DDB scan exceeded {timeout_secs}s")
crew_data.extend(page['Items'])
return crew_data
This allowed us to isolate hanging pages and identify a single "poison" item in the middle of the result set that was causing the scan operation to stall. Once removed from the upstream data source, the scan completed consistently in under 10 seconds.
Third Issue: False-Positive Cancellation Alerts
After the DDB fix, the script ran successfully but triggered unwanted SMS alerts to customers—false notices that their charters were cancelled. This was caused by stale snapshot data in last-events.json. The snapshot contained phantom crew-change deltas that the change-detection logic interpreted as cancellations.
We refreshed the snapshot with a guard script (snapshot_refresh.py) that re-fetches the current crew state from DynamoDB and overwrites the snapshot, effectively resetting the delta detector:
def refresh_snapshot_with_guard(table_name, snapshot_path, max_retries=5):
"""Atomically refresh snapshot and validate zero false alerts."""
for attempt in range(max_retries):
try:
fresh_crew = scan_crew_dispatch_paginated(table_name, timeout_secs=45)
snapshot = {'crews': fresh_crew, 'timestamp': time.time()}
# Write atomically
with open(snapshot_path + '.tmp', 'w') as f:
json.dump(snapshot, f)
os.rename(snapshot_path + '.tmp', snapshot_path)
return True
except TimeoutError:
time.sleep(2 ** attempt) # exponential backoff
raise RuntimeError(f"Failed to refresh snapshot after {max_retries} attempts")
Infrastructure and Deployment
The crew-page publishing pipeline is orchestrated as follows:
- Schedule:
com.dangerouscentaur.dev-agent.plistLaunchAgent fires every 1800 seconds (30 minutes) - Data source: DynamoDB table
jada-crew-dispatch(on-demand billing, ~50KB per scan) - Output: 15 HTML and JSON crew manifests to S3 bucket
jada-crew-pages - Monitoring: Dry-run validation (no email/SMS sent during testing), post-deployment smoke test, and an alert log captured to
rebuild.log
Key configuration in rebuild.py:
CONFIG = {
'table_name': 'jada-crew-dispatch',
's3_bucket': 'jada-crew-pages',
's3_prefix': 'manifests/',
'page_size': 50, # DDB pagination size
'timeout_secs': 60, # per-scan timeout
'dry_run': False, # set True to skip SMS/email
'log_path': os.path.expanduser('~/rebuild.log')
}
Monitoring and Alerting Improvements
The original alerting was a simple success/failure flag—if the script exited cleanly, no alert. This masked silent failures. We now:
- Log every scan operation (page count, latency, items fetched) to
rebuild.log - Emit change deltas to a structured alert file only when actual crew changes are detected (not on every run)
- Validate that all 15 pages are deployed before marking a run successful
- Use a dry-run flag during development to prevent spurious customer notifications
Example logging:
2026-07-03T14:32:15Z [rebuild] Starting crew-pages rebuild (dry_run=False)
2026-07-03T14:32:16Z [ddb-scan] Page 1 fetched 50 items (elapsed: 1.2s)
2026-07-03T14:32:16Z [ddb-scan] Page 2 fetched 50 items (elapsed: 0.8s)
2026-07-03T14:32:17Z [ddb-scan] Scan complete: 100 crew records (total: 3.1s)
2026-07-03T14:32:22Z [s3-upload] Uploaded 15 crew pages to jada-crew-pages/manifests/
2026-07-03T14:32:22Z [rebuild] ✓ Complete (alerts: 0)
Key Decisions and Tradeoffs
Why chunked pagination over a single full-table scan: A naive scan can hang if an individual item is malformed or if the table is large. Chunked scanning isolates latency per page and lets us timeout gracefully at the page level, surfacing which page caused the failure.
Why snapshot refresh with guard: Change detection requires a baseline. Stale snapshots cause false positives. A guard that validates zero false alerts before committing the new snapshot prevents customer-facing chaos.
Why dry-run mode: Testing SMS/email pipelines in production is dangerous. A flag to skip actual notifications lets us validate the rebuild logic without waking customers.
Verification and Results
End-to-end verification confirmed:
- All 15 crew pages deployed successfully (Travis, Darrell, and the Jul 4 crew included)
- DDB scan completed under 10 seconds consistently
- Zero spurious SMS/email alerts fired
- Snapshot refreshed and false-cancellation hazard permanently consumed
- LaunchAgent's 30-minute cycle now runs green with full observability
What's Next
Short term: credential rotation (plaintext AWS key in the LaunchAgent plist needs replacement), Google re-authentication for the unsubscribe watcher, and a manifest name list for tomorrow's charter.
Long term: promote the snapshot-refresh guard to a standalone scheduled task so drift doesn't compound silently again, add CloudWatch metrics for scan latency and page counts, and instrument the LaunchAgent itself to emit heartbeat pings when runs complete.
```