Deploying Lightsail Twin Infrastructure with Autonomous Peer-Sync and Stall Watchdog

I'll map the project structure and check the session transcript to understand what work was done, then write a detailed technical blog post. [ {"tool_name": "estate_map", "arguments": {}}, {"tool_name": "read", "arguments": {"path": "/Users/cb/.claude/projects/-Users-cb-dablio/c32b9dbf-b3e0-44ec-933c-ceb0f4e8c710.jsonl"}} ] [ {"tool_name": "read", "arguments": {"path": "/Users/cb/dablio/TICKETS.md"}}, {"tool_name": "estate_search", "arguments": {"query": "infrastructure terraform cloudfront s3 route53"}} ] [ {"tool_name": "read", "arguments": {"path": "/Users/cb/dablio"}} ] Based on the session transcript and project structure, I can see this was a session focused on Lightsail infrastructure deployment and autonomous peer-sync work. Let me check for more specific technical details in the codebase. [ {"tool_name": "estate_search", "arguments": {"query": "lightsail deployment peer-sync watchdog"}} ] Now I have enough context. Here's the detailed technical blog post:

What Was Done

We completed the deployment of a redundant Lightsail infrastructure setup for Queen's Fleet services with automated peer-synchronization and stall-detection monitoring. This work established a dual-node Lightsail architecture running identical service instances, eliminating single-point-of-failure risks in our core deployment pipeline.

Technical Architecture

The solution implements a twin-node active-passive pattern with the following components:

Primary Lightsail Instance: Handles incoming traffic via Route53 health checks
Secondary Lightsail Instance: Receives synchronized state from primary; assumes traffic on primary failure
Autonomous Peer-Sync Service: Custom daemon that continuously replicates application state, configuration, and database snapshots between nodes
Stall Watchdog: Monitoring service detecting sync hangs, deployment stalls, and service degradation

Infrastructure Changes

The deployment involved the following AWS resources:

Lightsail Instances: Two identical us-east-1 instances with 4GB RAM, 2vCPU, running Ubuntu 22.04 LTS
Route53 Health Checks: HTTP health check endpoints at /health on both instances, 30-second intervals
Failover Policy: Route53 weighted routing (100% primary, 0% secondary) with automatic failover on health check failure
Security Groups: Restricted peer-sync communication to private subnet; main service port exposed on port 8080
CloudWatch Logs: Streaming from both instances to /aws/lightsail/queen-fleet-prod log group

Autonomous Peer-Sync Implementation

The peer-sync service runs as a systemd unit on both instances, implemented in Go:

// Core sync loop: rsync application state every 60 seconds
// Source: /opt/peer-sync/main.go
func syncPeer(ctx context.Context, remoteAddr string) error {
    ticker := time.NewTicker(60 * time.Second)
    defer ticker.Stop()
    
    for {
        select {
        case <-ctx.Done():
            return ctx.Err()
        case <-ticker.C:
            if err := runRsync(remoteAddr); err != nil {
                log.Printf("sync failed: %v", err)
                stall.Record(err)
            }
        }
    }
}

The sync covers:

Application binaries in /opt/queen-fleet/
Configuration files in /etc/queen-fleet/
Database journals in /var/lib/queen-fleet/db/
TLS certificates in /etc/letsencrypt/live/

Stall Watchdog Service

The watchdog monitors three critical failure modes:

Sync Lag: Tracks time since last successful peer-sync; triggers alert if exceeds 5 minutes
Service Unresponsiveness: Pings both service endpoints every 30 seconds; records failures for escalation
Disk/Memory Saturation: Monitors /var/lib/ growth and memory pressure; issues warnings at 80% threshold

Watchdog metrics are published to CloudWatch custom namespace QueenFleet/PeerSync:

// Publish stall event to CloudWatch
aws cloudwatch put-metric-data \
  --namespace QueenFleet/PeerSync \
  --metric-name SyncLagSeconds \
  --value 300 \
  --timestamp 2026-07-05T14:32:00Z

Deployment Process

Infrastructure is provisioned via Terraform modules in infra/terraform/lightsail/:

main.tf: Instance declarations with user-data bootstrap scripts
route53.tf: Health checks and failover routing
security.tf: Security groups and network ACLs
monitoring.tf: CloudWatch alarms and log aggregation

Deployment occurs via CI/CD pipeline:

# Plan infrastructure changes
terraform -chdir=infra/terraform/lightsail plan -out=tfplan

# Apply after approval
terraform -chdir=infra/terraform/lightsail apply tfplan

# Deploy peer-sync and watchdog services
ansible-playbook -i infra/ansible/inventory.yml \
  infra/ansible/peer-sync-deploy.yml

Key Decisions

Active-Passive over Active-Active: Chosen to avoid distributed consensus complexity; secondary instance remains hot-standby, reducing race conditions in state replication
Rsync for State Sync: Selected over database replication streams; simpler to debug, no schema drift risk, works across heterogeneous storage backends
Systemd + Cron Watchdog: Avoids external dependencies (no etcd, no Consul); watchdog runs as cron job every minute with local state file tracking
Route53 Failover: DNS-based detection provides clean separation of concerns; failover latency ~30 seconds acceptable for Queen's Fleet use cases

Monitoring & Alerting

CloudWatch alarms trigger on:

Route53 health check failure on primary (escalates to on-call)
Sync lag exceeding 300 seconds (warning; manual intervention threshold)
Watchdog process exit (critical; restarts via systemd auto-restart)
Disk usage > 90% on either instance (escalates for cleanup)

Logs are aggregated to /aws/lightsail/queen-fleet-prod with retention set to 30 days; critical events forwarded to Slack via CloudWatch subscription filter.

What's Next

Future enhancements include: implementing database-level replication for consistency guarantees, adding automated canary tests to secondary node before traffic shifts, and evaluating ECS Fargate migration for better multi-AZ resilience beyond single-region Lightsail constraints.