Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

Migrating a 63-Job launchd Automation Estate to a New Mac Without a Double-Send

What Was Done

The estate's entire operations layer — charter provisioning, email and SMS pipelines, photo publishing, daily video factories, nightly test suites, payment monitors — runs as 63 launchd agents on a single 2019 Intel MacBook Pro, backed by roughly 50 CLI runners, 6 AWS profiles, 5 separate secrets stores, and about 20 OAuth token files. We designed and tooled a full migration of all of it to a newer Intel i9 machine (8-core, 32GB), treating the new box as a fresh, AI-native host that gets every system permission on day zero. The deliverables are four deterministic scripts (safe export, secrets export, disarm, tiered re-arm) plus a phased roadmap; the scripts are re-runnable and tested.

Technical Details

The export splits into two bundles with different trust levels:

  • Safe bundle (rides cloud file sync): a Brewfile dump, pip and npm freezes, all launchd plists, crontab, the runner-script directory, shell dotfiles, and agent-tooling config including a persistent memory directory. A dry-run mode plus a unit test assert that no secrets-store path can ever appear in this bundle.
  • Transfer bundle (local disk only, chmod 600, moved by AirDrop or direct scp, deleted after restore): the four secrets stores, SSH keys, AWS credentials, tool configs that embed API keys, and local-only workspaces including an isolated browser profile. The script hard-refuses to write anywhere that resolves into cloud-synced storage, and emits a manifest of sha256 checksums so the restore can verify integrity.

The bulk of the estate never needs either bundle: repos and ops trees already live in cloud file sync, bridged into scripts via space-free symlinks in the home directory. On the new machine the restore is: recreate two symlinks, force full local download (no "optimize storage" placeholders — background jobs cannot exec a file that is only a stub), unpack the bundles, brew bundle, reinstall pinned pip/npm globals, and rebuild virtualenvs from scratch. Venvs are never copied between machines; they are build artifacts.

Two path facts made this dramatically simpler. First, both machines are Intel, so Homebrew stays at /usr/local and the three interpreter variants hard-coded across the plists (system python, brew python, versioned brew formula path) resolve identically — zero plist edits. Second, the new machine's account short name is forced to match the old one, so every absolute path in every plist, script, and symlink survives unchanged.

Infrastructure

The dangerous part of migrating an automation fleet is not data loss — it's double-fire: two machines both believing they own the outbound email scheduler, the SMS queue drainers, or the daily content publishers. The cutover is therefore atomic and ordered:

  • disarm_old_mac.sh unloads every estate agent on the old machine and empties the crontab into a dated backup. Plists are left in place, so rollback is a one-line re-arm.
  • arm_new_mac.sh loads the fleet in three tiers: infrastructure daemons (network self-healing, status panes) first, read/produce jobs second, and send-class jobs last — nothing that emails, texts, or publishes loads until everything upstream of it is verified. Known-dead one-shot plists are skipped by pattern.
  • Verification over the first 48 hours checks that every daily artifact arrived exactly once.

A small always-on Linux VPS that runs a headless agent and IP-whitelisted API calls is deliberately excluded from the entire operation. It is a separate authentication domain; the only migration step it gets is a read-only SSH health check. Machine migrations are exactly when "helpful" credential cleanup kills production, so the standing rule is: no credential is rotated, re-minted, or logged out as part of a machine move.

Key Decisions

  • Wipe the used machine again anyway. A seller's "factory reset" is unverifiable. Erase All Content and Settings re-keys the SSD encryption, cryptographically destroying prior contents in minutes; a Recovery-mode reinstall is the paranoia tier. Also: check Startup Security for a leftover firmware password.
  • Skip Migration Assistant. It faithfully copies the one thing we most want to leave behind — years of accreted TCC permission state and configuration drift. A deterministic rebuild from manifests is the point.
  • Grant everything up front. On the old machine, macOS privacy prompts were a chronic production hazard: protected folders silently unreadable by background jobs, screen-capture grants that would not stick, launchd refusing to exec cloud-synced scripts (runners must live in a plain local bin directory). The new machine gets Full Disk Access, Accessibility, Screen Recording, and Developer Tools for the terminal and for the python interpreter binary itself before any workload lands, and the standing policy is: if a permission prompt appears, fixing the grant is the task. Never work around it.
  • Old machine becomes a hot spare for two weeks — disarmed but intact — before it is erased. Rollback is a script, not a project.

What's Next

  • Root-cause the ten agents that are already exiting non-zero on the old machine — migration faithfully preserves your failures too, so each gets an incident-ledger entry and a fix ticket rather than a silent carry-over.
  • Convert two legacy cron entries (pointing at a directory that moved months ago) into proper launchd agents against the current paths.
  • After the soak: erase the old machine and record the final state in the estate's decision log.