Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

```html

Automating Git Snapshots and Diff Review for Production Tool Versioning

What We Built

Set up an automated git mirroring and diff-review pipeline to version control production tooling and enforce review gates before deployment. The system snapshots local repositories to a remote Lightsail instance on a schedule, captures every diff before it ships, and logs all infrastructure changes to CloudTrail for auditability. This eliminates the gap between "code I wrote locally" and "what's actually running in production."

The Problem We Solved

Production scripts living in ~/bin/ and ~/icloud-jada-ops/tools/ had no version history. When a script changed, there was no diff to review, no blame log, and no way to revert to a known-good state. Staff-level engineering requires "can I reconstruct why this changed and undo it" to be non-negotiable. Local-only files don't meet that bar.

Technical Architecture

Component 1: Automated Git Snapshots

Created /Users/cb/bin/jada-git-snapshot.py — a Python script that runs on a schedule via launchd, captures the current state of production directories into a git repository, and pushes to a remote. Key decisions:

  • Why Python over shell: Structured error handling, easier to audit for command injection, standard logging.
  • Deterministic-first design: Script is idempotent. Running it twice with no changes produces no new commits.
  • Pre-flight checks: Verifies SSH keys exist, git identity is configured, AWS credentials are valid, and the remote is reachable before attempting any write.
  • Commit message structure: Includes ISO timestamp and file count: snapshot 2026-07-03 12:47 (237 paths). Parseable for analytics.

The snapshot repo is a bare git repo on the Lightsail instance at ~/git/estate.git. Local repository at /Users/cb/jada-estate is configured as a worktree with the iCloud/ and .git/ directories exempt from snapshot isolation (so the snapshot process doesn't capture itself).

Component 2: Pre-Deployment Diff Review

Created /Users/cb/bin/jada-diff-review.py — runs on a separate schedule and compares the working directory against the last committed snapshot. Any differences are flagged for human review before production deployment. This answers the question: "What changed since the last safe state?"

  • Runs independently from the snapshot process — no race conditions.
  • Captures diffs in structured format for audit trail.
  • Fails loudly if reviewing changes that touch payment code or client-facing email sends.

Component 3: Scheduling via launchd

Two launchd agents, configured as plist files in ~/Library/LaunchAgents/:

  • com.jada.git-snapshot.plist — runs the snapshot script every 30 minutes.
  • com.jada.diff-review.plist — runs the diff-review script every 6 hours.

Both use absolute paths and redirection to log files so execution is auditable. Agent runs as the user (not root), so SSH keys and AWS credentials in the local keychain are accessible without privilege escalation.

Infrastructure Setup

Lightsail Instance as the Remote

The Lightsail instance serves as the canonical git remote. Created a bare repository and verified SSH access from the local machine: 

ssh -i ~/.ssh/jada-lightsail ec2-user@<lightsail-ip> git init --bare ~/git/estate.git

Local repository configured with the remote: 

git remote add lightsail ssh://ec2-user@<lightsail-ip>/home/ec2-user/git/estate.git

Initial snapshot was verified with: 

ssh -i ~/.ssh/jada-lightsail ec2-user@<lightsail-ip> "cd ~/git/estate.git && git log --oneline | head -5"

CloudTrail and S3 Audit Logging

Created CloudTrail logging to capture all AWS API calls made by this tooling:

  • S3 bucket: jada-estate-cloudtrail-logs with lifecycle policy (90-day expiry).
  • Trail name: jada-estate-audit, logging all regions.
  • CloudTrail confirmed active and writing events.

This ensures that if infrastructure code makes AWS API calls (e.g., to sync state, read secrets, or trigger deployments), the calls are logged and timestamped. Matches the commit timestamp in git for correlation.

Secret Scanning and Security Hardening

Before the first push to Lightsail:

  • Scanned all source directories for embedded credentials (tokens, API keys, passwords). Found references to environment variables (e.g., $AWS_PROFILE) but no literal secrets in files.
  • Audited repos.env (master vault file) to ensure it never entered git history. Confirmed clean.
  • Tightened file permissions on sensitive files to 600 (read/write by owner only).
  • Verified IAM policy scope: the user's AWS credentials grant only the permissions needed for this workflow (git mirror, CloudTrail, S3 lifecycle management).

Key Decisions and Tradeoffs

Why a Remote Repository on Lightsail, Not GitHub Private?

This repository contains production configuration, deployment scripts, and decision history. Keeping it off GitHub adds a layer of isolation: the canonical state lives on infrastructure we control, not a third-party SaaS. GitHub is still valuable for collaboration, but the git-of-record stays internal.

Why Worktree Exemptions for iCloud?

iCloud Drive syncs files to all devices. Without excluding iCloud metadata, the snapshot would capture sync state noise (timestamps changing, atomicity markers). By exempting those directories, snapshots capture only the content that matters.

Why Separate Snapshot and Diff-Review Processes?

Snapshot happens frequently (every 30 minutes) to minimize the window between a change and capture. Diff-review runs less often (every 6 hours) because it's more expensive — it scans the working directory and compares against HEAD. Separating them prevents the diff-review blocking snapshot, and vice versa.

What's Next

  • Test backlog: Mine past incidents (fire logs) and add tests that would have caught each one. The failure-domain plan identifies which tests are highest ROI.
  • Staged rollout: The snapshot and diff-review scripts are now running. Monitor CloudTrail logs for 2 weeks to confirm no regressions, then integrate into deployment gates.
  • Decision doc automation: Every major change to production infrastructure should be accompanied by a one-page decision doc explaining WHY (stored in ~/icloud-jada-ops/decisions/). These live in the repo too, so they're version-controlled with the code they justify.
  • Failure domain shrinking: Each fire becomes a test case. As test coverage grows, the blast radius of bugs shrinks — changes can be validated before they ship, not discovered by oncall at 2 AM.

Verification

First snapshot succeeded:

  • Commit ed6a9d9 created with 237 paths, 11M packed size.
  • Pushed to Lightsail and verified with git log on the remote.
  • Remote HEAD now points at main.
  • CloudTrail confirmed logging S3 and API calls.

The entire git versioning layer is now operational. Every change to production tooling is captured, auditable, and reversible.

```