Observable Infrastructure by Default: Lessons from a Comprehensive Systems Audit
What Was Found
A complete audit of our automation and operational systems revealed a critical pattern: the factory was built fast and well, but there is almost no instrumentation. Things break silently and stay broken for weeks. Across 26 LaunchAgent jobs, multiple cloud pipelines, and three concurrent businesses, we discovered:
- Email blast system dead for 1+ days — July 2 campaign sent 0 of 3,640 messages due to control characters in mailing list data; SES validation failed silently
- SMS crew dispatch piling up undelivered —
sms-flushfires every 2 minutes but has been failing on an expired AWS session; DynamoDB queue grew to hundreds of undelivered notifications - Google OAuth layer universally expired (~June 9) — app still in Testing mode (7-day revocation); payment monitor, calendar sync, Gmail, and port sheet all broken with no alert
- 7 of 26 LaunchAgents hard-broken — Missing
awsandclaudein PATH, TCC (Full Disk Access) blocking iCloud paths, wrong working directories, zero failure alerting - Plaintext long-term AWS credentials hardcoded in
~/Library/LaunchAgents/com.dangerouscentaur.dev-agent.plist— should be rotated and replaced with IAM profile references - Duplicate-send and suppression-leak risk — Two parallel blast systems (local launchd + cloud Lambda) with no shared suppression state
- No audit trail on critical sequences — Hotel outreach logs go to
/tmp(wiped on reboot), so mid-sequence state is lost
Technical Root Causes
The infrastructure was designed for speed of delivery, not operational visibility. Each system fires on a schedule and reports to stdout, but:
- No centralized logging — LaunchAgent output goes to
~/Library/Logs/with no aggregation, parsing, or alerting - No health checks — A LaunchAgent can fail for weeks; the next success just overwrites stderr from the last run
- No dependency visibility — Email blast depends on Google OAuth (expired), SMS dispatch depends on AWS session (expired), manifest generator depends on a script (aws CLI not in PATH)
- Credentials scattered across config files — LaunchAgent plists, environment files, shell aliases, and Docker secrets; no centralized secret rotation
- No synthetic monitoring — We don't know something is broken until a human notices (or a guest complains)
Infrastructure Changes Required
The fix is a 5-layer observability stack:
- L1 — Job wrapper — Every LaunchAgent invocation wraps with a script that captures exit code, stdout/stderr, duration, and sends to a central log sink (CloudWatch or DynamoDB)
- L2 — Dependency health checks — Before any job runs, verify its dependencies: test AWS credentials (date + list a test S3 object), test Google OAuth (refresh token), test external services (SES SendEmail test)
- L3 — Dead-letter queues — Failures that should have alerted humans go to a DynamoDB "faillog" table with TTL=7d; a nightly scan sends a summary to Slack + email
- L4 — Synthetic tests — Hourly run a minimal version of each critical job (e.g., send a test SMS, post a test crew page, check email blast readiness) and alert on failure within 15 minutes
- L5 — Credential rotation automation — AWS credentials auto-rotate every 30 days via a Lambda; Google OAuth moves to a shared service account with explicit refresh before each job
Key Decisions
Why DynamoDB + launchd, not a full job queue? We have 26 jobs running on a Mac and a Lightsail box. Moving to AWS Batch or Step Functions adds operational overhead and a hard dependency on AWS API availability. Keeping jobs local means they work even if AWS is down (except for jobs that need AWS); centralizing logs to DynamoDB gives us the audit trail without the infrastructure.
Why synthetic tests, not just health checks? A health check tells you "the API is up." A synthetic test tells you "the entire flow works end-to-end." We need both: health checks are fast (run every 1 minute), synthetics are thorough (run every 1 hour).
Why not rebuild LaunchAgents from scratch? We have 26 of them. Instead, we write a launch-wrapper.sh that every plist calls, and every wrapper logs to the same sink. This is backwards-compatible and enforces the pattern immediately.
What's Next
Implementation order:
- Immediate (today) — Rotate plaintext AWS credentials, fix critical LaunchAgent PATH issues, restore email/SMS dispatch, re-auth Google OAuth
- This week — Deploy
launch-wrapper.sh, create DynamoDB faillog table, write dependency health-check scripts - Next week — Centralized nightly faillog digest to Slack, synthetic test suite (crew page, email, SMS, manifest generation)
- Month of July — Credential rotation automation, add Datadog or CloudWatch dashboards, post-mortem on each silent failure found during audit
Goal: By August 1, every job failure is visible within 15 minutes. Silent failures are not tolerated.