Building Visibility into Silent Failures: The Five-Layer ICM Audit Methodology in Action
When systems appear to work but don't, the cost compounds invisibly. Over the past 72 hours, we ran a comprehensive five-layer infrastructure audit across JADA's charter operations, web properties, side ventures, and automation layer. Three critical failures emerged before we even reached strategic planning. This post details what broke, why detection was blind, and how deterministic system design prevents the next surprise.
The Five-Layer Audit Framework
The ICM (Intent-Context-Model) methodology organizes work into five nested layers:
- L0 — Standing Rules: Durable constraints that don't change (e.g., "$1,000/hr charter floor," "never render cropped boat photos").
- L1 — Floorplan: Property/product boundaries and ownership; which system owns what.
- L2 — Context Router: Decision trees and task delegation;
CONTEXT.mdfiles route work to the right handler. - L3 — Factory / Reference Layer: Canonical templates, scripts, and shared code;
~/icloud-jada-ops/as the source of truth. - L4 — Per-Run Product: The deliverable for a specific charter, site, or deployment; ephemeral until committed.
Every subsystem lacking these layers received a CLAUDE.md + CONTEXT.md audit pass that same turn. What emerged were three categories of gaps: operational blindness (systems that should have alerted), broken automation (scripts with missing dependencies), and credential anti-patterns (plaintext secrets in long-term config).
JADA Charter Operations: Three Silent Failures Before July 4
The Immediate Crisis: Tomorrow's charter (Dylan Osborne, 30 guests) exposed a gap in the manifest pipeline:
- Missing USCG Manifest: The folder
~/icloud-jada-ops/passengers/manifests/is empty despite a robustgenerate_manifest.pyscript sitting ready in the codebase. The passenger name list from GetMyBoat was never translated into a manifest document. USCG requirement: two printed copies, one left in the dock stairs before departure. This process was documented in CHARTER-WORKFLOW.md but execution was manual and silently skipped. - Crew Page Rebuild Failure: The automated rebuild script that generates shipcaptaincrew.com crew pages (displaying captain/mate assignments, sail schedules) has been failing for 8 hours. Root cause: the script invokes the AWS CLI with a relative path that assumes
awsis on$PATH; in the cron/Lambda execution context, it is not. The script runs in the default shell without the proper environment bindings. Fix: use absolute path/usr/local/bin/awsor invokesource ~/.zprofilebefore the AWS call. - Payment Ledger Blind Spot: $3,250 of a $3,750 charter fee remains outstanding (only $500 paid). The ledger file
~/icloud-jada-ops/ledger.jsonhas 7 entries but 3 are missing total amounts. Manual Zelle payment watcher (monitoring ~/Library/Containers/com.apple.bird/Data/chat.db) has been blind for 13 days—it requires a Full Disk Access grant that only the user can authorize in macOS Security settings. No alert fires when payments stop arriving.
Other Chronic Failures:
- Unsubscribe Watcher Down 30 Days: The CAN-SPAM compliance tool that strips unsubscribed email addresses from marketing lists has not run since early June. Zero alerts notified ops. Risk: sending to unsubscribed addresses triggers ISP complaints and domain reputation damage.
- Nightly Test Suite Silent Failure: The crew page test gate script (
test_crew_pages.py) has been failing for three weeks—no dashboard, no Slack notification, no email digest. Broken builds are invisible until someone manually runs the script. - Privacy Exposure in Sync: Carole's SMS dumps (~306KB of raw messages) are syncing into
~/icloud-jada-ops/alongside structured CSV data. This represents unencrypted personal conversation history in a shared iCloud workspace; it should be segmented and encrypted or excluded entirely.
What Is Working: The proposals pipeline (Google Sheets → deterministic PDF fill → S3 deploy → link reply to GetMyBoat), CHARTER-WORKFLOW.md decision trees, BSSD blog publisher (zero errors, published today), and the root ICM scaffolding are solid—no failures here.
Infrastructure Layer: Plaintext Credentials and Dead Campaigns
The automation layer audit uncovered two severe patterns:
1. Hardcoded AWS Credentials in LaunchAgents Config: A plaintext AKIA… access key and secret are embedded in the plist file ~/Library/LaunchAgents/com.dangerouscentaur.dev-agent.plist. This is a long-term credential that grants full AWS API access; any process that reads the plist (or any attacker with filesystem access) can assume the identity. The fix is to replace inline credentials with an AWS CLI profile reference (e.g., --profile jada-admin) and store the actual key/secret in ~/.aws/credentials with restrictive file permissions (0600). The key should be rotated immediately and the old key revoked in the AWS IAM console.
2. Email Blast Campaign Went Silent (0 of 3,640 Sent): A July 2 marketing campaign to 3,640 addresses sent exactly zero emails. Post-mortem: the email list contained control characters and whitespace that broke the blast tool's CSV parser; the jq/sed pipeline that cleaned the list before sending did not validate the output. The blast system has no alerting—it ran and completed without reporting success or failure. We discovered this gap only during the audit. Solution: add a post-send validation check that counts sent vs. attempted and pages ops if the success rate drops below 95%.
Key Technical Decisions
- Deterministic Scripts Over Manual Processes: Every repeatable operation (manifest generation, crew page rebuild, email list cleaning, payment reconciliation) should be a committed script in
~/icloud-jada-ops/with inputs and outputs, not a mental workflow. Scripts can be tested, versioned, and monitored; humans cannot scale. - Secrets in Environment, Config in Code: Credentials belong in
~/.aws/credentials(for AWS CLI profiles),~/jada-secrets/creds.txt(for app-specific auth), or environment variables—never in checked-in plist files or shell scripts. The codebase itself should be auditable without exposing a key. - Monitoring as a First-Class Feature: Every cron job, Lambda function, and automated pipeline needs explicit success/failure reporting. No silent failures. Either send to a health dashboard (
status.queenofsandiego.com/crew-health/pattern exists), trigger a Slack webhook, or log to CloudWatch with alarms. - Audit Reproducibility: The five-layer framework itself is testable—if every subsystem has a
CLAUDE.md+CONTEXT.md, the audit can be re-run and compared to a baseline. We should commit the audit findings to~/icloud-jada-ops/AUDIT-2026-07-03.mdas the source of truth for what was broken and what was fixed.
What's Next
The web-properties audit (covering a dozen-plus sites, the CloudFront distribution fleet, Domain DNS, the queenof-cities factory, and deployment tooling) is still in progress. Once complete, we will convene the full audit board—architectural decisions, build sequencing, and strategic prioritization for the three business units (JADA charters, QuickDumpNow trailer rental, and the web-services portfolio).
Immediate actions this week:
- Rotate the plaintext AWS credential from the LaunchAgents plist and replace with profile reference.
- Generate the USCG manifest for Dylan Osborne's charter and test the full print-and-dock workflow.
- Fix the crew page build script AWS CLI PATH and re-enable the nightly test gate with alerting.
- Implement a post-send email validation check (sent count vs. attempted count) with ops paging.
- Segment and encrypt the SMS dumps out of the shared iCloud workspace.
This audit is the first pass of the five-layer framework across all three business lines. More details to follow once the web-properties report lands.