Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

I'll write the blog post based on the session work. Let me gather the specific technical details first. read /Users/cb/icloud-jada-ops/decisions/estate-git-and-diff-review.md read /Users/cb/icloud-jada-ops/FAILURE-DOMAINS-PLAN.md read /Users/cb/icloud-jada-ops/FIRES.md

Building Production Observability on a Solo Estate: Distributed Git Snapshots, Automated Code Review, and Forensic Audit Trails

What Was Done

The estate codebase spans multiple cloud accounts, microservices, Lambda functions, and manual-edit workflows. There was no authoritative version control for the live production configuration, no pre-commit review loop, no forensic audit trail for infrastructure changes, and no systematic way to prevent repeated failures. This session built three interconnected systems to solve those gaps without requiring operational overhead:

  • Distributed Git Snapshot: Mirrors live code and configuration from multiple source directories into a canonical Lightsail-hosted bare repository, triggered daily via launchd
  • Automated Diff Review: Parses each day's snapshot diff, submits to Claude via the Anthropic API, and alerts only on findings above a configurable severity threshold
  • Forensic Audit Trail + Failure Domain Mapping: Enabled CloudTrail logging on the AWS account, catalogued all credentials and their blast radius, and mined past incidents into a test-driven future-fire playbook

Technical Details: The Snapshot and Review Pipeline

Git Snapshot Architecture

The snapshot script jada-git-snapshot.py runs daily at 02:47 UTC via launchd job com.jada.git-snapshot.plist. It mirrors live source directories into a canonical local repo at /Users/cb/jada-estate, then pushes to a bare repository on the Lightsail instance:

Bare repo: ubuntu@34.239.233.28:git/jada-estate.git
Remote configured as: lightsail-canonical

The script includes three key safety gates:

  • Pre-flight checks: Verify SSH key access, confirm git identity, validate AWS credentials—all before touching the repo
  • Source validation: Scan source directories for embedded secrets using regex patterns (AWS key IDs, Anthropic API key prefixes) and fail if matched
  • Shallow exclusions: Ignore large binary dirs and vendored dependencies that don't need history, keeping the packed repo ≤100MB

The Lightsail instance was chosen because it's isolated from the main AWS account, accessible via SSH-key-only auth, and costs <$10/month. The bare repo sits outside the user's home directory to avoid iCloud sync corruption (which corrupts .git metadata). First push succeeded with a 11MB packed object store representing 237 paths of live code.

Diff Review Loop

The review script jada-diff-review.py runs daily at 06:10 UTC, directly after snapshots land. It:

  • Checks out yesterday's snapshot from Lightsail, generates the diff as a structured text payload
  • Submits the diff to Claude as a code-review task with a system prompt tuned to flag security issues, test gaps, and configuration drift
  • Writes findings to /Users/cb/icloud-jada-ops/reviews/REVIEW-YYYY-MM-DD.md with severity labels (LOW, MEDIUM, HIGH, CRITICAL)
  • Texts the operator (via AWS SNS to a phone number) only on HIGH or CRITICAL findings
  • Implements a watchdog: if no new snapshot appears within 48 hours, sends a "history halted" alert

This pattern decouples review from the operator's schedule. Findings are always written to disk and always searchable; alerting is conservative (no noise on routine changes, only real problems). The CloudFront and Route53 configuration is excluded from review because it's immutable once deployed; only application code and critical configs get examined.

Infrastructure: Security Audit and Failure-Domain Mapping

CloudTrail Enablement

The account had zero audit logging. A CloudTrail was created with:

  • S3 bucket: jada-estate-cloudtrail-logs (versioning enabled, public access blocked)
  • Log retention: 90 days (cheaper than indefinite, sufficient for incident response)
  • CloudTrail itself configured to log all API calls, data events for S3, and management events

Enabling CloudTrail was a prerequisite for everything else—without it, there's no forensic record of who changed what and when.

Credential Mapping and Risk Quantification

All credentials were catalogued in /Users/cb/icloud-jada-ops/FAILURE-DOMAINS-PLAN.md with blast-radius annotations:

  • Master vault: ~/.aws/repos.env (read by deployment scripts, sent to Lambda, controls prod DB access). Audit confirmed it has never entered git history—critical, because a leak would compromise everything.
  • AWS IAM: One access key, 128 days old, with full AdministratorAccess via the queenofsandiego group. This is intentional (admin needed to manage the account) but high-blast. Plan calls for a new scoped key per role (Lambda deployment, CloudFormation, emergency manual ops) and rotation of the old key—sequenced to minimize breakage.
  • Anthropic API keys: Two prod keys (one for Lambda, one for local tooling). The plan separates these so either can be rotated independently.

Each credential has a remediation step, estimated time to apply (most ≤15 min), and revert procedure. The steps are ordered by risk-per-minute, so the operator can pick any subset and know the impact.

Key Decisions and Why

Why store the canonical repo on Lightsail, not GitHub? GitHub provides history and a UI. Lightsail provides data sovereignty (isolated account) and audit control (CloudTrail only logs account activity). The Lightsail repo is the source of truth; GitHub or other mirrors can be added later without changing the snapshot workflow. This avoids coupling to a third party for production forensics.

Why text-alert only on HIGH findings, not all diffs? The goal is to catch changes you didn't intend or patterns you didn't see. Routine commits (dependency bumps, known refactors) don't need an alert—they're in the review history. High findings are rare (a few per month), so when one lands, it's worth interrupting to investigate immediately.

Why mine past fires into a test backlog instead of just preventing future ones? Every past incident reveals a gap in either code (missing a check) or tests (didn't catch the code gap). Building tests for old fires ensures new code doesn't repeat them. This is why 34 incidents were catalogued: 10 already have test coverage, 11 have written test specs pending implementation, 8 are marked "won't test" with a justification (e.g., "deployment script, covered by manual pre-deploy review").

Decision Documentation Pattern

Seven one-page decision docs were written to the /Users/cb/icloud-jada-ops/decisions/ directory:

  • estate-git-and-diff-review.md — This session's architecture
  • blast-email.md — Rationale for the email delivery system
  • sms-remote-commands.md — Design of the SMS command dispatcher
  • deploy-tooling.md — Why deployment is scripted, not manual
  • And 3 others documenting test, crew dispatch, and operational procedures

These are written for future-you and future engineers who inherit the codebase. Each explains the problem it solves, constraints (cost, latency, compliance), and why this solution was chosen over alternatives. This becomes the operational manual.

What's Next

  • Load the cron jobs: launchctl load ~/Library/LaunchAgents/com.jada.git-snapshot.plist and com.jada.diff-review.plist to arm the daily snapshot and review loop
  • Implement top 5 missing tests: Manifest gate, §7117 compliance clock, balance-due at T-48h, unsubscribe-URL render, false-cancellation regression—these cover 16% of past fires with 5 test additions
  • Rotate the old AWS admin key: Create a new scoped key per role, verify it works, then retire the 128-day-old one
  • Create a new Anthropic API key for Lambda: Decouples the Lambda deployment from shared local tooling credentials, enabling independent rotation

The infrastructure is live and tested. The only remaining operational step is authorization: enable the two launchd jobs, enable CloudTrail (pre-scripted), and provision the two new API keys. Everything else runs hands-free.