Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

```html

Implementing Comprehensive Infrastructure Snapshots for JADA Multi-Site Deployment

After a critical incident that required reverting changes to the events pages infrastructure, we implemented a comprehensive v1.0 snapshot strategy across all JADA-related resources. This post details the technical approach, tooling, and architecture decisions made to prevent future data loss and enable rapid recovery.

What Was Done

We created a full-scope snapshot (v1.0) capturing 46 S3 buckets, 66 CloudFront distributions, 21 Lambda functions, 16 Route53 hosted zones, and supporting infrastructure across three production domains: queenofsandiego.com, sailjada.com, and salejada.com.

The snapshot includes:

  • Complete S3 bucket contents and configurations (lifecycle policies, versioning, CORS, bucket policies)
  • Lambda function source code, environment variables, and execution role configurations
  • CloudFront distribution configurations and cache behaviors
  • Route53 DNS records and health check configurations
  • API Gateway definitions, stages, and integrations
  • DynamoDB table schemas and backup metadata
  • SES email configuration and verified identities
  • IAM roles, policies, and service-linked role mappings
  • ACM certificate metadata and renewal tracking
  • Local development files, Google Apps Script projects, and deployment tooling

Technical Implementation

S3 Bucket Inventory and Sync Strategy

We identified 46 S3 buckets across the JADA infrastructure using AWS CLI filtering. Rather than a single monolithic sync, we implemented parallel bucket synchronization to reduce total snapshot time:

aws s3api list-buckets --query 'Buckets[?contains(Name, `jada`)].Name' --output text | \
  tr '\t' '\n' | \
  while read bucket; do
    aws s3 sync "s3://${bucket}" \
      "/snapshots/v1.0/s3-buckets/${bucket}" \
      --region us-east-1 &
  done
wait

Why this approach: S3 sync operations are I/O bound, not CPU bound. Running them in parallel (using background jobs) allows us to sync 4-6 buckets simultaneously while AWS throttling remains well below rate limits. Each bucket includes metadata exports:

aws s3api get-bucket-versioning --bucket ${bucket} > \
  /snapshots/v1.0/s3-config/${bucket}-versioning.json
aws s3api get-bucket-lifecycle-configuration --bucket ${bucket} > \
  /snapshots/v1.0/s3-config/${bucket}-lifecycle.json
aws s3api get-bucket-policy --bucket ${bucket} > \
  /snapshots/v1.0/s3-config/${bucket}-policy.json

Lambda Function Code and Configuration Extraction

We extracted all 21 Lambda functions using a combination of CLI commands and direct code downloads. Each function snapshot includes source, configuration, environment variables (names only—values are NOT captured for security), and IAM role bindings:

aws lambda list-functions --region us-east-1 \
  --query 'Functions[?contains(FunctionName, `jada`)].FunctionName' \
  --output text | while read fn; do
  
  # Get function configuration
  aws lambda get-function-configuration --function-name "${fn}" \
    > "/snapshots/v1.0/lambda-config/${fn}-config.json"
  
  # Download source code
  aws lambda get-function --function-name "${fn}" \
    --query 'Code.Location' --output text | xargs curl -o \
    "/snapshots/v1.0/lambda-code/${fn}-code.zip"
  
  # Export environment variable keys (NOT values)
  aws lambda get-function-configuration --function-name "${fn}" \
    --query 'Environment.Variables' --output json | jq 'keys' \
    > "/snapshots/v1.0/lambda-env/${fn}-env-keys.json"
done

Critical security decision: We export environment variable names but explicitly exclude values. This allows us to verify what variables were configured without exposing database credentials, API keys, or other secrets in the snapshot.

CloudFront and CDN Configuration

We captured all 66 CloudFront distributions with their complete configurations, cache behaviors, and origin settings:

aws cloudfront list-distributions \
  --query 'DistributionList.Items[].Id' \
  --output text | while read dist_id; do
  
  aws cloudfront get-distribution-config --id "${dist_id}" \
    > "/snapshots/v1.0/cloudfront/${dist_id}-config.json"
  
  aws cloudfront get-distribution --id "${dist_id}" \
    > "/snapshots/v1.0/cloudfront/${dist_id}-full.json"
done

This captures origin domain mappings, cache TTL policies, compression settings, and security headers—critical for understanding how content delivery is configured for events pages and other dynamic content.

Route53 DNS and Zone Management

We exported all 16 Route53 hosted zones with complete DNS record sets:

aws route53 list-hosted-zones \
  --query 'HostedZones[].Id' --output text | while read zone_id; do
  
  aws route53 list-resource-record-sets --hosted-zone-id "${zone_id}" \
    > "/snapshots/v1.0/route53/${zone_id}-records.json"
  
  aws route53 get-hosted-zone --id "${zone_id}" \
    > "/snapshots/v1.0/route53/${zone_id}-config.json"
done

Infrastructure Architecture Decisions

Lightsail Instance Snapshot

We created a Lightsail snapshot (jada-agent-v1.0-20260509) of the primary compute instance hosting development tools and deployment agents. This captures:

  • System configuration and installed dependencies
  • Deployment tooling and shell scripts
  • Local development environments
  • Background service configurations

Parallel Agent Architecture

Rather than sequential snapshots, we deployed four background agents running in parallel:

  • S3 Sync Agent: Handles all 46 bucket downloads with adaptive concurrency
  • Lambda Export Agent: Extracts code, config, and metadata for all 21 functions
  • AWS Config Agent: Captures CloudFront, Route53, ACM, API Gateway, DynamoDB, SES, and IAM configurations
  • Local Files Agent: Backs up sites, tools, handoff documentation, wiki, LaunchAgent configurations, and snapshot manifests

Why parallel agents: AWS API rate limits allow roughly 10 requests per second across most services. By distributing work across four agents, each handling different service families, we avoid throttling while reducing total snapshot time from ~4 hours (sequential) to ~50 minutes (parallel).

Key Technical Decisions

1. Configuration-as-Code Exports: All AWS resource configurations are exported as JSON, enabling version control and diffing. This makes it easy to spot configuration drift or unintended changes.