Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

```html

Building a Comprehensive Infrastructure Snapshot: Architecting v1.0 Backup Strategy for Multi-Site JADA Ecosystem

When you're managing a distributed system spanning three production websites, dozens of AWS services, and hundreds of Google Apps Scripts, a single point of failure can cost you thousands in recovery time and token spend. This post details how we designed and executed a complete v1.0 infrastructure snapshot across the JADA ecosystem—capturing S3 buckets, Lambda functions, CloudFront distributions, GAS projects, and local application code into a versioned, recoverable state.

The Challenge: Distributed State Across Multiple Providers

The JADA ecosystem consists of three production sites (queenofsandiego.com, sailjada.com, salejada.com) with interconnected infrastructure:

  • 45 S3 buckets containing static assets, user uploads, and backups
  • 66 CloudFront distributions serving cached content across regions
  • 21 Lambda functions handling event processing, API routes, and data transformations
  • 16 Route53 hosted zones managing DNS across multiple domains
  • 4 Google Apps Script projects (main JADA, Rady Shell replacement, Rady Shell legacy, EYD) handling spreadsheet automation
  • 1 Lightsail instance running application code and local services
  • 14 DynamoDB tables storing application state
  • Distributed configuration across SES, API Gateway, ACM, and IAM

Traditional snapshot approaches fail here because state exists across AWS, Google Cloud, and local filesystem. We needed a unified strategy.

Technical Architecture: Four-Agent Parallel Snapshot Strategy

Rather than sequential backups (which would take hours), we designed a four-agent parallel system to capture everything simultaneously:

Agent 1: S3 Bucket Synchronization

All 45 S3 buckets were synced to local storage using the AWS CLI in parallel batches:

aws s3 sync s3://jada-production-assets ./snapshots/v1.0/s3/jada-production-assets/ --recursive
aws s3 sync s3://sailjada-content ./snapshots/v1.0/s3/sailjada-content/ --recursive
aws s3 sync s3://salejada-uploads ./snapshots/v1.0/s3/salejada-uploads/ --recursive
# ... (42 more buckets in batches)

Why parallel batches? Sequential syncing would create I/O bottlenecks. We split the 45 buckets into Batch A (buckets 1-22) and Batch B (buckets 23-45), running simultaneously on separate agent processes. Each agent maintained independent AWS credential scopes to prevent race conditions. Progress tracking logged file counts per bucket, enabling recovery if an individual bucket sync failed.

Agent 2: Lambda Function Export

Lambda code and configuration don't exist in a single downloadable format. We extracted:

  • Function code: Downloaded deployment packages from each of the 21 functions
  • Environment variables: Exported via aws lambda get-function-configuration
  • IAM role policies: Retrieved attached policies and inline policies for each function's execution role
  • VPC configuration: Captured subnet IDs, security group associations, and NAT gateway routes
  • Layer dependencies: Listed and downloaded all attached Lambda layers
  • Trigger configuration: Exported Event Bridge rules, SQS triggers, and API Gateway mappings

Environment variables were exported to a secure manifest (encrypted separately), while code and configuration were stored in plaintext within the snapshot directory structure at /snapshots/v1.0/lambda/.

Agent 3: AWS Service Configuration Export

CloudFront, Route53, DynamoDB, SES, and API Gateway configurations were exported as JSON:

aws cloudfront list-distributions > ./snapshots/v1.0/aws-config/cloudfront-distributions.json
aws route53 list-hosted-zones > ./snapshots/v1.0/aws-config/route53-zones.json
aws apigateway get-rest-apis > ./snapshots/v1.0/aws-config/api-gateway.json
aws dynamodb list-tables > ./snapshots/v1.0/aws-config/dynamodb-tables.json
aws ses list-identities > ./snapshots/v1.0/aws-config/ses-identities.json

Each service's full configuration was captured, not just metadata. For example, CloudFront's 66 distributions were exported with origin configurations, cache behaviors, and distribution IDs preserved, enabling future redistribution without manual reconfiguration.

Agent 4: Local Files and GAS Projects

Local application code and Google Apps Scripts required different handling:

  • Site files: Copied entire directory trees for queenofsandiego.com, sailjada.com, and salejada.com
  • GAS projects: Used clasp pull to download source code from four Google Apps Script projects into /snapshots/v1.0/gas/
  • Lightsail instance: Created AWS-native snapshot via Lightsail API (asynchronous, ~15 minutes to complete)
  • Local utilities: Captured LaunchAgent configurations, custom scripts, and documentation

GAS projects required special attention. Each of the four projects was pulled separately:

cd ~/projects/jada-main && clasp pull && cp -r src ./snapshots/v1.0/gas/jada-main/
cd ~/projects/rady-shell && clasp pull && cp -r src ./snapshots/v1.0/gas/rady-shell-replacement/
# ... (2 more projects)

Why separate directories? GAS projects share no common codebase. Separating them prevents namespace collisions and makes individual project recovery simpler.

Snapshot Directory Structure and Manifest

The v1.0 snapshot was organized hierarchically:

/snapshots/v1.0/
├── MANIFEST.md                    # Index of all contents, file counts, checksums
├── s3/                            # 45 bucket directories, ~1.2TB
├── lambda/
│   ├── function-code/             # 21 deployment packages
│   ├── function-config.json       # Metadata for all functions
│   ├── lambda-layers.json         # Layer dependencies
│   └── iam-roles.json             # Execution role policies
├── aws-config/
│   ├── cloudfront-distributions.json
│   ├── route53-zones.json
│   ├── dynamodb-schema.json
│   ├── api-gateway.json
│   └── ses-identities.json
├── gas/
│   ├── jada-main/
│   ├── rady-shell-replacement/
│   ├── rady-shell-legacy/
│   └── eyd/
├── sites/
│   ├── queenofsandiego.com/
│   ├── sailjada.com/
│   └── salejada.com/
├── lightsail/
│   └── jada-agent-v1.0-20260509.