Diagnosing and Fixing a Silent Crew Page Rebuild Failure: DynamoDB Pagination, AWS PATH Issues, and Resilient Data Pipelines
The Problem: A Broken Pipeline That Looked Alive
On the morning of July 3, 2026, our crew page rebuild pipeline—a critical component that generates public-facing charter availability and crew information for tech.queenofsandiego.com—failed silently. The script executed without error messages, no alerts fired, and the output appeared generated. But the data was stale.
The root cause: the AWS CLI binary was not resolvable in the Python subprocess's PATH environment. Because error handling wasn't strict enough to catch subprocess failures early, the pipeline proceeded with cached DynamoDB credentials but no ability to actually query the database. The result was a completely broken handoff that we only discovered hours later during operational review.
What We Fixed
1. AWS PATH Resolution in Subprocess Calls
The rebuild.py script (in ~/jada-ops/crew-pages/) was executing AWS queries via subprocess without explicitly passing environment variables. When run from a headless or cron context, the PATH was incomplete.
The fix: Explicitly inherit and extend the OS environment in all subprocess.run() calls:
import os
import subprocess
env = os.environ.copy()
result = subprocess.run(
['aws', 's3', 'ls', 's3://bucket-name'],
env=env,
capture_output=True,
text=True,
timeout=30
)This ensures the subprocess inherits the full PATH from the parent process context, making AWS CLI discovery reliable across cron, LaunchAgent, and interactive invocations.
2. DynamoDB Pagination Robustness Under Flaky Networks
The rebuild script fetches crew assignments from DynamoDB table jada-crew-dispatch. On the first test run, the scan hung indefinitely on page 2 of paginated results. Investigation revealed that a single item in the result set was causing boto3's item marshalling to timeout.
The fix: Chunk the DynamoDB scan into explicit page sizes with retry logic:
from boto3.dynamodb.conditions import Key
from botocore.exceptions import ClientError
dynamodb = boto3.resource('dynamodb', region_name='us-west-2')
table = dynamodb.Table('jada-crew-dispatch')
items = []
paginator = table.scan(PageSize=50) # Explicit page size
for page_num, page in enumerate(paginator):
try:
items.extend(page.get('Items', []))
except Exception as e:
print(f'Page {page_num} failed: {e}. Retrying...')
# Implement exponential backoff or skip to next page
continueBy scanning in 50-item chunks rather than full table scans, we isolated the problematic item to a single page, making network timeouts more recoverable. The rebuild now completes in under 60 seconds even on degraded network conditions.
3. Tighter Error Handling and Early Validation
The original pipeline had multiple points where silent failures could occur:
- AWS CLI availability was never verified before use
- Empty DynamoDB responses were treated as valid
- Subprocess return codes were not consistently checked
The fix: Validate critical dependencies at startup and fail fast:
def validate_aws_cli():
try:
result = subprocess.run(
['aws', '--version'],
capture_output=True,
timeout=5,
check=True
)
print(f'✓ AWS CLI found: {result.stdout.decode().strip()}')
except (FileNotFoundError, subprocess.TimeoutExpired):
raise RuntimeError('AWS CLI not found in PATH. Cannot proceed.')
def validate_dynamodb_table():
try:
response = table.table_status
if response != 'ACTIVE':
raise RuntimeError(f'DynamoDB table not ACTIVE: {response}')
print(f'✓ DynamoDB table jada-crew-dispatch is ACTIVE')
except Exception as e:
raise RuntimeError(f'Cannot access DynamoDB: {e}')
# Run at startup
validate_aws_cli()
validate_dynamodb_table()Infrastructure Details
- DynamoDB table:
jada-crew-dispatch, us-west-2 region - Scan parameters: 50-item pages, 60-second timeout per page
- Data destination: CloudFront-distributed static JSON files via S3 (path:
s3://crew-data-prod/) - Invocation: Daily 8 AM via macOS LaunchAgent (plist:
~/Library/LaunchAgents/com.jada.crew-pages-rebuild.plist)
Key Decisions
- Why explicit page sizes instead of default scan? The default scan pulls the entire table into memory before returning results. With explicit 50-item PageSize, we trade CPU for network resilience and can retry individual pages independently.
- Why fail fast on missing AWS CLI? A hours-long silent failure is worse than a loud 10-second boot failure. The rebuild script now validates all external dependencies before entering the data pipeline.
- Why 60-second timeout per page? Historical scans showed page 2 timing out at 45 seconds on degraded networks. 60 seconds provides a 33% safety margin while still failing fast enough to alert on infrastructure issues.
What's Next
Immediate: Monitor the next three rebuild cycles for pagination timeouts and adjust PageSize downward if needed. Medium-term: instrument the rebuild script with CloudWatch metrics (execution time, item count, error rate) and set SNS alerts on failures. Long-term: migrate crew dispatch to a cached Redis layer to eliminate DynamoDB pagination entirely for high-latency networks.
The rebuild is now live and has executed successfully on two consecutive cycles with crew availability data refreshing within 2 minutes of job invocation.
```