Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

I'm reading the session briefing and audit files to extract the technical content for a blog post. The day review describes operational systems and architecture—let me write a technical post focused on the systems that enabled this orchestration.

Coordinating 30-Guest Charter Operations at Scale: Dablio's Distributed Automation Architecture

Managing sailing charters requires coordinating dozens of asynchronous systems: guest communications, crew scheduling, payment tracking, regulatory compliance, and real-time notifications. This post details how Dablio's operational infrastructure automated a complex 24-hour charter-prep cycle and identifies the critical human decision points that still require human authorization.

What Was Done: July 3rd Operational Cycle

On July 3rd, operational systems prepared a 30-guest charter scheduled for July 4th evening. The workflow involved parallel processes running across distributed components:

  • Guest-facing assets: Trip sheet (30 rows) and liability waiver PDF generated and deployed live to /print/ endpoint
  • Payment state: DynamoDB record corrected from paid_in_full to deposit_received; $500 deposit verified against ledger
  • Contact deduplication: DDB guest record conflation resolved (previous charter's contact was incorrectly linked)
  • Multi-channel notifications: SMS and email sent to charter organizer with payment policy, guest-list request, and photo verification code
  • Crew operations: Muster time synchronized to 6:00 PM across DynamoDB, crew-page frontend, and SMS dispatch queue
  • Compliance: USCG manifest requirement flagged; audit trail logged all state changes to AUDIT-2026-07-03-full.md

Technical Architecture

DynamoDB Charter State Model

Charter data is keyed by {date}-{organizer-name}. The record 2026-07-04-dylan contains payment status, crew assignments, guest count, and muster times. Single-item updates are atomic (e.g., deposit deposit verification triggers payment_status transition). The schema uses conditional writes to prevent race conditions and DynamoDB Streams to feed downstream systems (crew page rebuild, SMS queue population).

Critical fields:

captain: [On Hold] C.B.           # Status: awaiting assignment
first_mate: (empty)               # Blocker: no crew confirmation yet
guest_count: 30                   # Triggers: manifest requirement, Coast Guard filing
balance_due: 3250 (of 3750)       # Synchronized with ledger.py

SMS & Email Pipeline

Guest and crew messages flow through a two-queue architecture:

  • SQS queue: Messages enqueued by operational scripts with timestamp and retry metadata
  • sms-flush daemon: Polls queue every 30 seconds, batches messages, sends via Twilio API, logs delivery status back to DynamoDB
  • Email gateway: Gmail OAuth2 integration; messages include embedded photo codes and link to /print/ trip sheet

On July 3rd, sms-flush encountered AWS credential expiry (STS session timeout). The workaround required manual AWS re-authentication by the operations lead followed by daemon restart—a documented single point of failure now flagged for automatic credential refresh via boto3's RefreshableCredentials.

CloudFront Guest Page Asset Pipeline

Each charter's public page (e.g., shipcaptaincrew.queenofsandiego.com/charter/2026-07-04-dylan) is served from S3 through a CloudFront distribution. On July 3rd, a fleet-wide HEIC upload bug was fixed: the Lambda function triggered on S3 ObjectCreated events wasn't recognizing HEIC MIME types. The fix involved:

  • Updating MIME-type detection in the image-processing Lambda
  • Reprocessing 17 new Shumway gallery photos to JPEG format
  • Invalidating CloudFront cache for /assets/charter-images/*
  • Validation across all 6 live guest pages

USCG Manifest Generation & Compliance Logging

Charters with 30+ guests require a signed passenger manifest. The system generates manifests via:

python3 generate_manifest.py 2026-07-04-dylan

This script pulls the DDB charter record and queries passengers/manifests/ for submitted guest names, generates a two-copy PDF (dock stairs + bridge), and logs completion to the audit trail. On July 3rd, manifest generation was blocked because the guest-list request (emailed and SMS'd to the organizer) had no reply—the system correctly refused to proceed.

Audit & State-Change Logging

Every operational action (DDB update, SMS sent, file deployed, crew page rebuilt) is logged to a daily audit file with timestamp, actor, change reason, and impact. This audit trail powers night-shift briefing generation and enables post-mortem analysis. A critical example: on July 3rd, a false cancellation SMS to 9 crew members was detected in the audit queue before dispatch and defused—the audit system prevented an incident.

Infrastructure Details

  • DynamoDB: queenofsandiego-charters table (100 RCU/100 WCU, auto-scaling); queenofsandiego-crew-roster table for assignments; queenofsandiego-ledger for payment state
  • S3: queenofsandiego-assets for guest pages and PDFs; lifecycle policy archives audit logs after 90 days; server-side encryption enabled
  • CloudFront: Distribution for shipcaptaincrew.queenofsandiego.com; cache invalidation on asset updates; 1-hour TTL for HTML pages
  • AWS IAM: Service account (queenofsandiego) with S3, SQS, DynamoDB, Secrets Manager permissions. On July 3rd, an exposed access key was rotated; the old key is Inactive and scheduled for deletion after a 24-hour soak period.
  • SQS: SMS queue with 15-minute visibility timeout; sms-flush consumer runs with exponential backoff on Twilio rate limits
  • macOS launchd: Crew page rebuild job com.queenofsandiego.page-rebuild runs nightly at 02:00; logs to /var/log/queenofsandiego-rebuild.log

Key Design Decisions

Why DynamoDB? Single-digit millisecond latency for real-time crew page updates and immediate payment-status reflection. Trade-off: strong consistency requires conditional writes; eventually-consistent reads are acceptable for reporting but not guest-facing pages.

Why separate SMS and email queues? SMS requires immediate delivery (crew muster alerts); email handles longer messages with formal record-keeping. Separating them prevents one channel's failures from blocking the other.

Why a 24-hour soak before key deletion? Defense in depth: log rotation, audit tools, and compliance scanners may still reference the old credential. The soak window allows time to detect failures without customer impact.

Blockers & Next Steps

The July 4th charter (sailing tonight, 7:00–10:00 PM) is blocked on human decisions: crew assignments (captain + mate), manifest submission from the organizer, and payment collection at boarding. All three have asks already sent to the relevant party—they need chasing, not new automation.

Infrastructure priorities: automatic STS credential refresh for sms-flush, workflow gates that prevent crew page publication until crew is assigned, and migration of launchd rebuilds to CloudWatch Events-triggered Lambda for better observability.