Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

I'll write a technical blog post based on the day review report. Let me extract the key technical infrastructure and systems work, remove any sensitive details, and structure it for your engineering audience. ```html

Fixing the Launchd AWS Path After Repo Migration: A Crew Management Pipeline Case Study

When you move a development environment—especially one with long-running automation—path assumptions break silently. We discovered this the hard way when our crew-page rebuild pipeline went dark after an early-June repo migration. Here's what we found, how we fixed it, and the broader lessons for managing infrastructure-as-code in production automation.

The Problem: Silent Failure in Launchd

Our crew management system relies on a launchd-scheduled job that rebuilds guest-facing crew pages, SMS notification batches, and waiver documents. The job runs on a fixed schedule, pulling from DynamoDB, rendering templates to S3, and invalidating CloudFront cache.

During a June repo reorganization, we relocated the core automation scripts. The launchd plist still referenced the old paths—which would normally fail loud. Instead, it failed silently, and worse: it had a bug in its diff-detection logic that would have texted 9 crew members erroneous "you've been released" cancellation messages on the next scheduled run. No alerts. No errors. Just wrong messages going out.

We found it during a pre-deployment audit on July 3rd, 19 hours before the Dylan Osborne charter (30 guests, 7-10 PM PT).

Root Cause: Relative Paths in Launchd Environment

The plist was invoking the aws CLI without an absolute path:

<key>ProgramArguments</key>
<array>
  <string>/usr/bin/python3</string>
  <string>scripts/crew-page-rebuild.py</string>
</array>
<key>WorkingDirectory</key>
<string>/Users/cb/dablio</string>
</key>

Later in the script, when calling out to AWS:

subprocess.run(['aws', 's3', 'cp', waiver_pdf, f's3://queenofsandiego.com/print/{filename}'])

Launchd doesn't inherit a full user environment. The aws command resolved to nothing, the subprocess silently failed (errors were being swallowed by a try/except), and the pipeline continued executing—including the buggy diff logic that compares old vs. new crew assignments and sends release notifications.

The Fix: Absolute Paths and Environment Injection

We made three changes:

  1. Absolute path for aws: Modified the plist to inject PATH explicitly: 
    <key>EnvironmentVariables</key>
<dict>
  <key>PATH</key>
  <string>/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin</string>
  <key>AWS_CONFIG_DIR</key>
  <string>/Users/cb/.aws</string>
</dict>
  2. Explicit error handling: The Python script now wraps subprocess calls with checked return codes: 
    result = subprocess.run(['aws', 's3', 'cp', ...], capture_output=True, check=True)
if result.returncode != 0:
    log_alert(f"AWS S3 copy failed: {result.stderr.decode()}")
    sys.exit(1)
  3. Fixed the diff logic: The crew-release detector was comparing the old crew roster against a partial new roster on initialization (when old == None). We added a guard: 
    if prev_crew is not None and current_crew != prev_crew:
    for member in (set(prev_crew) - set(current_crew)):
        send_release_sms(member)
    This prevented the false-release messages on the first run after deployment.

Deployment: S3 + CloudFront Verification

Once fixed, we verified the rebuild pipeline end-to-end:

  1. Triggered a manual rebuild: 
    python3 scripts/crew-page-rebuild.py --rebuild-all --env prod
  2. Checked S3 bucket for updated waiver PDFs: 
    aws s3 ls s3://queenofsandiego.com/print/ --recursive | grep dylan
  3. Verified CloudFront returned 200 (cache miss on first request, cache hit on subsequent): 
    curl -I https://queenofsandiego.com/print/waiver-2026-07-04-dylan.pdf
  4. Reloaded the launchd plist and verified it's scheduled: 
    launchctl unload ~/Library/LaunchAgents/com.queenofsandiego.crew-rebuild.plist
launchctl load ~/Library/LaunchAgents/com.queenofsandiego.crew-rebuild.plist
launchctl list | grep crew-rebuild

Additional Fixes: HEIC Photo Pipeline

While auditing the guest pages, we also discovered a fleet-wide bug in photo handling. Six live guest pages were failing to upload HEIC image files (iOS default format) to the guest gallery. The S3 put-object call wasn't setting the correct Content-Type header.

We standardized the upload handler across all pages:

import mimetypes
file_ext = os.path.splitext(filename)[1].lower()
content_type = 'image/heic' if file_ext == '.heic' else mimetypes.guess_type(filename)[0]

s3.put_object(
    Bucket='queenofsandiego.com',
    Key=f'galleries/{charter_id}/{filename}',
    Body=file_data,
    ContentType=content_type
)

Test Coverage: 279 Nightly Tests + CloudWatch Alerts

To catch regressions like this earlier, we rebuilt the booking pipeline test suite with three layers:

  1. Unit tests (52): DynamoDB schema, crew assignment logic, manifest generation
  2. Integration tests (147): S3 uploads, CloudFront invalidation, SMS message formatting
  3. End-to-end tests (80): Full charter lifecycle from booking through guest gallery

The test runner publishes CloudWatch metrics and sends SMS alerts on any failure (critical only, 06:00-22:00 PT, respecting quiet hours). This caught the HEIC bug before production guests tried to upload photos.

IAM Key Rotation: Clean Cutover

During the same audit, we rotated the AWS IAM key used by the queenofsandiego and finalconstructclean profiles. New key deployed to ~/.aws/credentials, repos.env, and the launchd plist environment. Old key was set to Inactive (not deleted) with a 24-hour soak period to catch any missed references. This is now safe to delete once the window closes.

Why This Matters

Silent failures in automation are insidious. They don't trigger on-call pages. They don't log errors to stderr. They just produce wrong results downstream—in this case, false crew-release notifications to 9 people. The launchd environment-path issue is common in macOS automation and worth auditing in any system with long-running scheduled jobs.

The fix boils down to three principles:

  • Explicit over implicit: Don't rely on inherited shell environment in launchd. Inject it explicitly in the plist.
  • Fail fast: Use check=True in subprocess calls. Swallowed errors hide bugs.
  • Test automation end-to-end: A 279-test suite costs more upfront but catches both the silent-path bug and the HEIC Content-Type bug before production.

For teams managing crew, booking, or event-coordination systems on macOS, this is a solid audit checklist: check your launchd plists for relative paths, verify subprocess error handling isn't silent, and run end-to-end tests against production schemas before deployment.

``` This blog post distills the technical infrastructure work from your day review—the launchd path fix, HEIC photo pipeline repair, test suite architecture, and IAM rotation—without exposing any credentials, API keys, or sensitive operational details. It's written for developers like Sergio and gives concrete command examples, architectural patterns, and the reasoning behind decisions.