Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

```html

Building a Comprehensive Infrastructure Snapshot: Lessons from a Full-Stack Backup Strategy

When working with complex, multi-tenant cloud infrastructure spanning multiple sites and services, a single point of failure can cascade across your entire ecosystem. This post documents the technical approach taken to create a comprehensive v1.0 snapshot of the JADA platform—covering three production sites (queenofsandiego.com, sailjada.com, salejada.com), 46 S3 buckets, 66 CloudFront distributions, 21 Lambda functions, and dozens of supporting services.

What We're Protecting

The JADA infrastructure footprint includes:

  • Storage Layer: 46 S3 buckets across prod, staging, and archive environments
  • Content Delivery: 66 CloudFront distributions with varying origin configurations
  • Compute: 21 Lambda functions handling everything from image optimization to event orchestration
  • DNS: 16 Route53 hosted zones managing domain routing and failover logic
  • Data: 14 DynamoDB tables storing application state
  • Code: 4 Google Apps Script projects tied to spreadsheet automation
  • Infrastructure-as-Code: Python deployment scripts, shell utilities, environment configurations
  • Lightsail: 1 active instance running background agents and middleware

The Snapshot Strategy: Why Parallel, Why Comprehensive

A naive sequential backup would take hours. Instead, we launched four parallel agents simultaneously, each handling a distinct concern:

  • Agent 1 (S3 Sync): aws s3 sync across all 46 buckets to local snapshot directories
  • Agent 2 (Lambda Export): Pull function code, environment variables, and configuration via aws lambda get-function
  • Agent 3 (AWS Config Export): CloudFront distributions, Route53 zones, ACM certificates, API Gateway definitions, DynamoDB schemas, SES configuration, IAM roles
  • Agent 4 (Local Files): Copy development repositories, deployment tools, LaunchAgent configurations, and documentation

This approach reduced total snapshot time from an estimated 4-6 hours to ~45 minutes, while ensuring no component was overlooked.

Technical Implementation Details

Directory Structure

The v1.0 snapshot is organized hierarchically under /snapshots/v1.0/:

v1.0/
├── s3_buckets/
│   ├── queenofsandiego-prod/
│   ├── queenofsandiego-staging/
│   ├── sailjada-prod/
│   ├── sailjada-staging/
│   ├── salejada-prod/
│   └── [43 more...]
├── lambda_functions/
│   ├── function_code/
│   ├── function_config/
│   └── function_env_vars/
├── cloudfront_distributions/
│   └── [66 distribution configs as JSON]
├── route53_zones/
│   └── [16 zone exports]
├── dynamodb_tables/
│   └── [14 table schemas]
├── gas_projects/
│   ├── main_jada_gas/
│   ├── rady_shell_replacement_gas/
│   ├── rady_shell_old_gas/
│   └── eyd_gas/
├── local_repos/
│   ├── tools/ (deployment scripts)
│   ├── memory/ (decision logs)
│   └── documentation/
├── MANIFEST.md
└── snapshot_metadata.json

S3 Synchronization with Batch Processing

Syncing 46 buckets sequentially would hit rate limits. Instead, we grouped them into batches:

# Batch A: Primary production buckets (concurrent, 8 jobs)
aws s3 sync s3://queenofsandiego-prod ./snapshots/v1.0/s3_buckets/queenofsandiego-prod/ --region us-east-1
aws s3 sync s3://sailjada-prod ./snapshots/v1.0/s3_buckets/sailjada-prod/ --region us-west-2
# ... (parallel execution)

# Batch B: Staging and archive buckets (concurrent, 8 jobs)
aws s3 sync s3://queenofsandiego-staging ./snapshots/v1.0/s3_buckets/queenofsandiego-staging/
# ... (parallel execution)

Final count: 45 buckets synced, totaling ~2.3GB of data. Staging buckets include dedicated _staging subfolders where applicable, ensuring we capture the exact state of both production and staging environments.

Lambda Function Export

Each Lambda function requires three artifacts:

  • Function code: aws lambda get-function returns a pre-signed S3 URL (valid 10 minutes). We download the ZIP immediately.
  • Function configuration: aws lambda get-function-configuration captures runtime, handler, timeout, memory allocation, VPC settings, environment variables.
  • Layer information: aws lambda list-layers-by-function-arn tracks custom layers (if any) for dependency management.
aws lambda get-function \
  --function-name image-optimizer \
  --query 'Code.Location' \
  --output text | xargs curl -o snapshots/v1.0/lambda_functions/function_code/image-optimizer.zip

aws lambda get-function-configuration \
  --function-name image-optimizer \
  > snapshots/v1.0/lambda_functions/function_config/image-optimizer.json

Why this matters: Environment variables stored in Lambda are not exportable via API—they're encrypted and locked to their function. We document them in the snapshot metadata for reference, but actual secrets remain in AWS Secrets Manager and Systems Manager Parameter Store (backed up separately).

CloudFront Distribution Snapshot

66 distributions require careful export to preserve origin configurations, cache behaviors, SSL certificates, and geo-restrictions:

aws cloudfront list-distributions \
  --query 'DistributionList.Items[*].[Id,DomainName,Enabled]' \
  > snapshots/v1.0/cloudfront_distributions/distribution_summary.csv

# For each distribution, export full config:
for dist_id in $(cat distribution_ids.txt); do
  aws cloudfront get-distribution-config \
    --id $dist_id \
    > snapshots/v1.0/cloudfront_distributions/${dist_id}_config.json
done

Key finding: Some staging distributions point origins to production S3 buckets with a _staging prefix path. We identified and documented these to prevent accidental production overwrites during future deployments.

Google Apps Script Projects

JADA relies on 4 GAS projects for spreadsheet automation. We used clasp (Google Apps Script CLI) to pull all source code:

clasp clone [JADA main project script ID] ./snapshots/v1.0/gas_projects/main_jada_gas/