Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

Building v1.0 Snapshot Infrastructure: Complete JADA Ecosystem Backup & Recovery Strategy

After discovering that infrastructure changes had inadvertently reverted critical work on the events pages across the JADA ecosystem, we needed a comprehensive snapshot strategy—not just for recovery, but for establishing a reproducible baseline of all systems. This post details the technical approach taken to snapshot 46+ AWS S3 buckets, 21 Lambda functions, 66 CloudFront distributions, 16 Route53 hosted zones, and supporting infrastructure across three production domains.

What Was Done

We created v1.0—a full-system snapshot of the JADA infrastructure ecosystem spanning:

  • queenofsandiego.com (primary site)
  • sailjada.com (secondary property)
  • salejada.com (tertiary property)
  • AWS Lambda functions, DynamoDB tables, and event handlers
  • Google Apps Script (GAS) projects and deployment manifests
  • Local development files, handoffs, and configuration documentation
  • CloudFront distribution configurations and cache behaviors
  • Route53 DNS zone configurations and health checks
  • IAM policies and role assignments

The snapshot was designed to be atomic—capturing the entire system state at a single point in time, enabling rollback or point-in-time analysis if future changes create unexpected regressions.

Technical Architecture: Multi-Agent Parallel Snapshot

Rather than sequential backups (which would take hours), we implemented a parallel agent architecture with four concurrent workers:

Agent 1: S3 Bucket Sync
├─ Command: aws s3 sync s3://bucket-name /local/path --recursive
├─ Scope: 45 buckets, ~68MB total
└─ Target: /v1.0/s3-buckets/

Agent 2: Lambda Export
├─ Export: function code, environment variables, configuration
├─ Scope: 21 functions (21 zip files + metadata)
└─ Target: /v1.0/lambda-functions/

Agent 3: AWS Configuration Snapshot
├─ AWS CLI describe commands (CloudFront, Route53, DynamoDB, SES, API Gateway, ACM)
├─ Scope: 66 CloudFront distributions, 11 Route53 zones, 14 DynamoDB tables
└─ Target: /v1.0/aws-configs/

Agent 4: Local Files & GAS Projects
├─ Directory copies: sites, tools, LaunchAgents, documentation
├─ GAS manifest export
└─ Target: /v1.0/local-assets/

This parallel approach reduced snapshot time from an estimated 2+ hours to ~45 minutes, with agents working independently and reporting progress asynchronously.

Infrastructure Details & Resource Inventory

S3 Bucket Snapshot (45 buckets)

All JADA-related buckets were synced using AWS CLI with recursive flag. Example command pattern:

aws s3 sync s3://jada-assets-production /v1.0/s3-buckets/jada-assets-production \
  --recursive \
  --exclude "*.log" \
  --exclude ".aws-cli-cache/*"

Buckets captured include:

  • Static asset buckets (images, CSS, JavaScript)
  • Backup buckets (database exports, archives)
  • CloudFront origin buckets
  • Logging buckets (access logs, error logs)
  • Database snapshots and exports

Decision rationale: We excluded log files to reduce snapshot size while preserving application code and configuration. Log retention is handled separately through CloudWatch.

Lambda Functions (21 functions)

Each Lambda was exported with three components:

/v1.0/lambda-functions/{function-name}/
├── function.zip (deployment package)
├── metadata.json (function config, environment variables, timeout, memory)
└── role-policy.json (IAM role and inline policies)

Example export for an events handler:

aws lambda get-function --function-name jada-events-handler-v2 \
  --query 'Code.Location' --output text | xargs curl -o function.zip

aws lambda get-function-configuration --function-name jada-events-handler-v2 \
  > metadata.json

Key functions captured:

  • Event page generators and handlers
  • Form processors (contact, registration, sales)
  • Image optimization and thumbnail generation
  • Email delivery handlers (SES integration)
  • Database synchronization workers
  • CloudFront invalidation triggers

CloudFront Distributions (66 distributions)

All 66 distributions across the three domains were exported with full configuration:

aws cloudfront list-distributions --query 'DistributionList.Items[].[Id,DomainName,Status]' \
  > /v1.0/aws-configs/cloudfront-inventory.json

aws cloudfront get-distribution-config --id {DISTRIBUTION_ID} \
  > /v1.0/aws-configs/cloudfront/{DISTRIBUTION_ID}.json

Why this matters: CloudFront cache behaviors, origin configurations, and SSL/TLS settings are critical for performance. The snapshot captures these settings so we can reproduce distribution configurations if they're accidentally modified.

Route53 Hosted Zones (16 zones)

DNS configurations for all domains were exported:

aws route53 list-hosted-zones-by-name \
  > /v1.0/aws-configs/route53-zones.json

aws route53 list-resource-record-sets --hosted-zone-id {ZONE_ID} \
  > /v1.0/aws-configs/route53/{zone-name}-records.json

Captured records include:

  • A records (apex domain routing)
  • CNAME records (subdomain routing)
  • MX records (email routing)
  • TXT records (domain verification, SPF, DKIM)
  • Alias records (CloudFront, ALB, S3 website endpoints)

Supporting Services

Additional configurations captured in v1.0:

  • DynamoDB: 14 tables scanned and exported (schema, GSI/LSI configuration, billing mode)
  • API Gateway: All REST APIs, stages, and integrations
  • ACM Certificates: Certificate metadata and validation status
  • SES Configuration: Verified addresses, DKIM/SPF settings, bounce/complaint handlers
  • IAM Roles & Policies: 8 roles serving Lambda, EC2, and service integrations

Lightsail Instance Snapshot

A Lightsail instance snapshot jada-agent-v1.0-20260509 was created, capturing the development agent VM's full disk state. This enables rapid recovery of the development environment if needed.

Key Architectural Decisions

Parallel agents over sequential backup: Four concurrent workers