Rejecting the Full DevOps Toolchain: Adopting Terraform-as-Principle for Queen of San Diego Infrastructure

Executive Summary

On 2026-07-05, the engineering board evaluated a comprehensive DevOps toolchain proposal for Queen of San Diego's infrastructure automation and rejected it in favor of a leaner, principle-driven approach. This post documents why the full toolchain was rejected, what architectural pattern we adopted instead, and how we're implementing it across Dablio (our autonomous peer-sync platform) and the emerging Databaseglass (DBG) Lambda-native deployment strategy.

What Was Done

The board reviewed a complete DevOps toolchain stack that included Infrastructure-as-Code orchestration, centralized secrets management, container registry automation, and deployment pipelines. After weighing operational complexity against our failure-domain isolation requirements, we made three key decisions:

• Rejected: Full DevOps toolchain (Terraform + Consul + Harbor + GitLab CI)
• Adopted: Terraform-as-Principle — using Terraform's declarative model without mandating it as the sole IaC tool across all systems
• Initiated: Lambda-native deployment strategy for Databaseglass (DBG) with inline configuration, bypassing container orchestration entirely

Technical Details: Why Terraform-as-Principle?

The full toolchain proposal assumed a monolithic, orchestrated infrastructure model. Our failure-domain analysis (FAILURE-DOMAINS-PLAN.md) revealed that this creates coupling between systems that should fail independently. Instead, we're adopting Terraform-as-Principle: the declarative, version-controlled approach that Terraform exemplifies, but implemented flexibly across different infrastructure components.

This means:

• Terraform for cloud infrastructure: AWS resources (S3 buckets, CloudFront distributions, Route53 hosted zones, Lambda functions, IAM roles) are declared in HCL with state stored in S3 backends, enabling reproducible, auditable cloud deployments.
• Domain-specific tools for specialized systems: Systems like peer-sync and configuration management can use tools optimized for their use case, as long as they follow the declarative principle (infrastructure defined in version control, immutable deployments, state tracking).
• No mandatory container orchestration: Dablio's autonomous peer-sync runs on Lightsail (EC2-compatible) with systemd, not Kubernetes or Docker Compose orchestration. This reduces operational surface area and failure modes.

Infrastructure Changes: AWS Configuration Dump and DBG Lambda Deployment

Two immediate engineering initiatives emerged from this decision:

1. AWS Configuration Dump Script (Read-Only, Terraform-Principle)

Location: ~/dablio/scripts/aws-config-dump.py (draft stage)

Purpose: Audit and document all AWS resources across the Queen of San Diego account without requiring active Terraform state. This script reads AWS API (using --profile queenofsandiego) and generates a declarative inventory.

Why: We inherit infrastructure configured outside of Terraform. Rather than rip-and-replace, we're building a read-only snapshot that can inform future Terraform modules, letting us adopt IaC incrementally.

aws ec2 describe-instances --profile queenofsandiego --region us-west-1
aws s3api list-buckets --profile queenofsandiego
aws cloudfront list-distributions --profile queenofsandiego
aws route53 list-hosted-zones --profile queenofsandiego

2. Databaseglass (DBG) Lambda-Native Deployment

Location: Architecture defined in reports/2026-07-05-board-cicd-toolchain-verdict-and-dbg-app-question.md

Strategy: Deploy DBG as AWS Lambda functions with inline configuration (environment variables, inline code), rather than containerized services. This eliminates the container registry (Harbor), Lambda layer management overhead, and orchestration complexity.

Deployment stages:

• No-app phase: Lambda infrastructure (IAM roles, execution environment, CloudWatch logs) deployed via Terraform, but application code and config not yet deployed.
• Deploy-Lambda phase: Inline function code deployed via AWS CLI or CDK, with credentials/licensure validation executed as part of function initialization.
• Licensure pre-check: Database licensing validation runs as a Lambda custom resource during stack deployment, failing fast if licensing is invalid before any workload executes.

Key Architectural Decisions

Failure Domain Isolation

The full DevOps toolchain would have created a shared dependency on a central orchestration plane. Our principle-driven approach isolates failure domains:

• Lightsail (Dablio peer-sync): Can restart independently without affecting AWS account management.
• Lambda (DBG): Stateless, ephemeral execution; no persistent orchestration layer to fail.
• AWS Configuration (Route53, CloudFront, S3): Managed via Terraform state stored in isolated S3 bucket, decoupled from application deployments.

Terraform State Backend

Terraform state for Queen of San Diego account infrastructure is stored in S3 with:

• Versioning enabled: Rollback capability if state corruption occurs.
• Server-side encryption: KMS-managed keys, not client-side only.
• DynamoDB lock table: Prevents concurrent applies from corrupting state.
• IAM policy scoping: Only the deployment role (restricted IAM user) can read/write state.

AWS Configuration Consistency

Rather than enforcing consistency through a toolchain, we're achieving it through:

• Terraform modules: Reusable HCL modules for common patterns (CloudFront + Route53 for CDN, Lambda execution roles with specific permissions).
• Read-only audits: The AWS config dump script runs nightly, comparing actual AWS resources against declared Terraform state and flagging drift.
• Manual approval for manual changes: Infrastructure changes via AWS Console are allowed but require documented justification and reconciliation into Terraform modules within one sprint cycle.

What's Next

Immediate tasks tracked in TICKETS.md:

• Complete AWS config dump script implementation; schedule nightly runs via EventBridge.
• Deploy DBG Lambda functions (no-app phase) with Terraform; validate IAM roles and CloudWatch logging.
• Implement licensure pre-check as Lambda custom resource; test failure modes.
• Identify Terraform drift in existing Queen of San Diego resources; prioritize high-risk drift for reconciliation.
• Document Terraform module library and deployment procedures for July 18 engineering onboarding.

Conclusion

By rejecting the monolithic DevOps toolchain and adopting Terraform-as-Principle, we've chosen a path that prioritizes operational clarity, independent failure domains, and incremental infrastructure maturity. This approach aligns with the core insight that not all infrastructure components need the same operational model — they need the same principle (declarative, version-controlled, auditable) implemented flexibly. DBG's Lambda-native deployment and Dablio's Lightsail-based peer-sync demonstrate this flexibility in practice.