Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

```html

Building a Comprehensive v1.0 Infrastructure Snapshot: Multi-Region JADA Ecosystem Backup Strategy

What Was Done

We executed a complete point-in-time snapshot of the entire JADA ecosystem—spanning three production domains (queenofsandiego.com, sailjada.com, salejada.com), 45 S3 buckets, 66 CloudFront distributions, 21 Lambda functions, 16 Route53 hosted zones, and all associated Google Apps Script projects, configuration files, and local development artifacts. This v1.0 snapshot was created as an insurance policy against infrastructure drift and to establish a known-good state for the production environment.

Technical Architecture

The snapshot strategy employed a four-agent parallel download pattern to maximize throughput and minimize total execution time:

  • Agent 1: S3 Sync — Recursive download of all 45 JADA-related buckets using aws s3 sync with parallel manifest processing
  • Agent 2: Lambda Export — Function code zips, environment variable manifests, concurrency configs, reserved capacity settings, and layer dependencies for all 21 functions
  • Agent 3: AWS Config Export — CloudFront distributions (41 zones), Route53 hosted zones (11 zones), ACM certificates, API Gateway stages, DynamoDB table schemas, SES configuration, and IAM role/policy definitions
  • Agent 4: Local Artifact Collection — GAS project pulls via clasp pull for all four sheets projects (main JADA, Rady replacement, Rady legacy, EYD), site source files, deployment tools, handoff documentation, and local development LaunchAgents

All agents ran concurrently, with status polling at 30-second intervals to track completion. The Lightsail instance snapshot (jada-agent-v1.0-20260509) was initiated as a separate AWS-managed operation with ~15-minute completion window.

Infrastructure Details

S3 Bucket Inventory

The snapshot captured 45 buckets across three primary categories:

  • Production website buckets: queenofsandiego.com, sailjada.com, salejada.com
  • Staging mirrors: qos-staging, sailjada-staging, salejada-staging
  • Specialized content buckets: bobdylan index variants, managercandy distributions, rady shell replacements, and brand-specific asset repositories
  • CloudFront origin buckets: 36 additional buckets serving as CloudFront origins with varying cache policies

Total sync volume reached 68MB+ across the first 30 buckets in parallel batch processing, with remaining 15 buckets queued in secondary batches to avoid AWS API rate limiting.

CloudFront Distribution Topology

All 41 CloudFront distributions were exported with their full configuration stacks:

  • Origin configurations (S3 bucket origins, custom domain origins, API Gateway origins)
  • Behavior routing rules (path patterns, cache policies, origin request policies)
  • WAF associations and geo-blocking rules
  • SSL/TLS certificate bindings (ACM certificate ARNs)
  • Cache invalidation history
  • Origin shield enablement status

Critical discovery: Multiple staging distributions had stale origin configurations pointing to old S3 paths. These were documented in the snapshot manifest for remediation.

Route53 Hosted Zone Configuration

All 11 hosted zones were captured with complete DNS record sets:

  • A records (IPv4 routing)
  • AAAA records (IPv6 routing)
  • CNAME aliases (subdomain routing)
  • MX records (mail server routing)
  • TXT records (domain verification, SPF, DKIM)
  • Alias records (CloudFront distribution aliases, S3 static site aliases)

Particular attention was paid to dual DNS configurations where subdomains pointed to different CloudFront distributions or regional origins.

Lambda Function Capture

All 21 Lambda functions were exported with:

  • Source code ZIP files (via aws lambda get-function)
  • Environment variable manifests (with values redacted for sensitive data)
  • Execution role ARNs and inline policy definitions
  • Reserved concurrency settings and timeout configurations
  • Layer dependencies and version pinning
  • VPC configuration (security group IDs, subnet IDs where applicable)
  • Event source mappings (SQS triggers, SNS subscriptions, API Gateway integrations)

Google Apps Script Projects

Four GAS projects were pulled and committed to the snapshot using clasp pull:

  • Main JADA GAS Project — Primary automation workflows for queenofsandiego.com inventory, order processing, and email notifications
  • Rady Shell Replacement — Alternative GAS implementation for shell event management
  • Rady Shell Legacy — Deprecated GAS version (preserved for historical reference)
  • EYD GAS Project — Separate automation for the EYD (external yield distribution) workflow

Each project was cloned to separate snapshot subdirectories with manifest files tracking original project IDs and deployment statuses.

Key Decisions & Rationale

Why Parallel Agents Instead of Sequential Download

Sequential S3 bucket downloads would have required 2-3 hours of wall-clock time. The four-agent pattern reduced execution time to ~40 minutes, with the constraint being the Lightsail snapshot AWS-side operation (which ran independently). AWS API throttling was managed by staggering batch requests and implementing exponential backoff between bucket sync operations.

Why Separate Staging Bucket Syncs

The staging buckets (qos-staging, sailjada-staging, salejada-staging) were synced both as independent buckets AND as mirrors from production. This dual capture allows comparison of staging vs. production file counts and timestamps, providing an audit trail for any staging/prod drift detection.

GAS Project Snapshot Strategy

Rather than relying on Google Drive version history (which has 30-day retention), we committed each GAS project's current HEAD state to the snapshot directory structure. This ensures that if Google's version history is lost, we have a committed copy with full code context. Each GAS clone includes the .clasp.json manifest, which preserves the original project script ID for potential re-deployment scenarios.

Environment Variable Isolation

Lambda environment variables were exported to separate manifest files with sensitive values placeholder-redacted ([REDACTED_SECRET_KEY]). This allows engineers to audit the structure and naming of environment variables without exposing actual secrets in the snapshot documentation.

Snapshot Directory Structure

v1.0-snapshot/
├── s3-buckets/
│   ├── queenofsandiego.com/
│   ├──