Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

```html

Building a Self-Auditing Git Mirror and Automated Code Review Pipeline

This post documents the infrastructure built to solve three critical gaps in operational safety: load-bearing scripts with no version history, code changes shipped without a second pass, and zero audit trail for infrastructure changes. The solution is a multi-layered system combining automated git snapshots, scheduled diff reviews, CloudTrail logging, and deterministic reminders.

The Problem

Production tooling—payment processors, email senders, client-facing services—lived in an iCloud sync directory with no git history. When a script broke, there was no diff, no blame, no revert. Code was deployed immediately on edit, even for solo developers. And account-level changes (IAM policies, credential rotations, infrastructure tweaks) left no audit trail.

The staff-engineer pattern here is: reconstruct why something changed and undo it must be non-negotiable. That requires version control, review gates, and forensic logging—not as bureaucracy, but as insurance against your own mistakes at 3 AM.

Architecture Overview

The solution has four independent layers:

  • Git Snapshot Layer: A Python script that commits all changes in iCloud directories to a canonical bare git repo on a Lightsail instance, running nightly via launchd.
  • Diff Review Layer: A scheduled script that examines every commit since the last review, flags high-risk changes (credential patterns, permission escalations, unsafe rewrites), and surfaces them via SMS.
  • Audit Logging: AWS CloudTrail configured for multi-region logging with S3 storage and 90-day retention.
  • Operational Reminders: A CSV-based reminder system integrated into the morning digest, ensuring critical follow-up items don't slip between sessions.

Git Snapshot Script

The snapshot script is located at /Users/cb/bin/jada-git-snapshot.py. It runs nightly (via launchd) and performs these steps:

  1. Scan the source directories (/Users/cb/icloud-jada-ops/, /Users/cb/bin/) for changes.
  2. Exclude high-noise or sensitive paths (node_modules, compiled outputs, large binary files).
  3. Stage changes in a local git index.
  4. Commit with a timestamped message and automation attribution.
  5. Push to the remote bare repo on Lightsail.

The script uses a deterministic pattern: it doesn't improvise per run. All exclusions, commit messages, and push behavior are hardcoded once and never change. This means six months later, when you ask "why did my script version jump?", git log gives you a reproducible answer.

Key safeguard: before committing, the script scans for embedded secrets (AWS key patterns, GitHub tokens, database credentials). If it finds any, it fails loudly and requires manual remediation. The bare repo itself is clean—no .env files, no token variables inline.

Diff Review Pipeline

The diff review script is at /Users/cb/bin/jada-diff-review.py. It runs daily (via com.jada.diff-review.plist) and examines every commit merged since the last review.

The review flags these patterns as HIGH severity:

  • Files matching credential regexes (AWS key ID patterns, BEGIN RSA PRIVATE KEY, etc.).
  • Changes to IAM policies or role trust relationships.
  • Deletions of audit/logging infrastructure.
  • Modifications to payment-related paths (anything touching wallet, billing, transaction logic).
  • New executable files or shell scripts without review comments.

Findings are surfaced via SMS (not email—email is asynchronous and easy to miss). This is not a code-blocking gate; it's a backstop. Even solo, you get a second pass before the change becomes stale. The review runs independent of human activity—if you shipped a change Friday at 5 PM, Saturday morning review catches it.

LaunchAgent Configuration

Both scripts run via launchd agents rather than cron, because launchd integrates with the system's security model and can be monitored like any other service. 

~/Library/LaunchAgents/com.jada.git-snapshot.plist
~/Library/LaunchAgents/com.jada.diff-review.plist

Each plist specifies:

  • A deterministic run schedule (snapshot at 20:05 nightly, review at 06:10 each morning).
  • The full Python path (not relying on shell $PATH).
  • Error output to a log file for debugging.
  • Restart behavior if the script fails (so a transient SSH hiccup doesn't silence auditing).

The launchd approach also means you can check job status with launchctl list com.jada.git-snapshot and see last-exit-code, last-run timestamp, and scheduled next-run. This is more transparent than cron's silent failures.

CloudTrail Setup

CloudTrail is configured at the account level, logging all API calls (CreateUser, PutUserPolicy, AssumeRole, etc.) to an S3 bucket with 90-day retention.

The trail is named jada-account-trail and logs multi-region events. The S3 bucket uses a lifecycle policy to delete logs older than 90 days, keeping storage costs minimal while maintaining a rolling forensic window.

Why CloudTrail and not just git history? Git captures what changed in your code; CloudTrail captures who did what in AWS and when. When you get paged at 3 AM because a Lambda crashed, you can run: 

aws cloudtrail lookup-events --lookup-attributes AttributeKey=ResourceName,AttributeValue=my-lambda-function --max-results 50

And see exactly which IAM principal made which change, when, and from where. Git only tells you the code state, not the operational decisions that led to it.

Digest and Reminders Integration

The snapshot, review, and CloudTrail infrastructure generate events, but events become noise if you forget about them. The morning digest (built by /Users/cb/icloud-jada-ops/digest/build_digest.py) now includes a REMINDERS section.

Reminders are sourced from a CSV file at /Users/cb/icloud-jada-ops/digest/reminders.csv with columns: 

due_date,priority,task,notes
2026-07-04,HIGH,Rotate IAM access keys,Delete old key after 24h soak
2026-07-04,NORMAL,Anthropic Lambda key setup,Deploy new key to prod Lambda

When the digest runs, it reads this file, formats items due today or earlier under NEEDS YOU, and repeats daily until the task is marked complete. This removes "remember to do X" from your working memory—it's deterministic and survives session gaps.

Decision Documentation

Alongside infrastructure, one-page decision docs were written explaining the WHY for each major choice. These live in /Users/cb/icloud-jada-ops/decisions/estate-git-and-diff-review.md and similar files. The format is:

  • Decision: What was chosen.
  • Alternatives considered: GitHub private repos (requires cloud auth), standalone reviewd daemon (too much maintenance).
  • Why this one: Lightsail bare repo is transparent (inspect it locally anytime), deterministic (no API rate limits), and owned (no GitHub Actions quota battles).
  • Tradeoffs: You manage the Lightsail instance (tiny, ~$2/mo) and maintain the push logic. If Lightsail is down, the next snapshot queues and retries.

What's Next

The immediate backlog:

  • Failure domain isolation: Extract repeatable fire patterns from past incidents and mint unit tests for each one. This breaks the cycle of "same bug twice in six months."
  • Test gates before deploy: The snapshot works; now add a pre-push gate that runs the manifest check, balance reconciliation, and unsubscribe render tests. No commit ships if these fail.
  • Expand diff review rules: Add checks for common typos in payment paths, unused variables in critical sections, and commit messages that reference issue IDs (forcing linkage to decision docs).

This infrastructure answers three hard questions: Can I reconstruct why this changed? (git log + commit history) Did someone review it? (diff-review report) What happened to my AWS account? (CloudTrail forensics)

Staff-level engineering is built from answers to these questions, not from perfect code on the first try.

```