Diagnosing and Staging a Critical Deposit Outage: Apps Script Access Control and Multi-Region Email Deliverability

```html

What Was Done

During a June 2026 operational session, we identified and staged a fix for a total stoppage in the deposit/reservation funnel across 11 event pages on sailjada.com and queenofsandiego.com. The root cause was a 403 Forbidden response from a Google Apps Script deployment, plus a second 404 on a worship-specific endpoint. Simultaneously, we diagnosed a deliverability crisis in SES (Simple Email Service) affecting outbound confirmation and reminder emails, traced unsubscribe leaks in the suppression list, and built automated monitoring to prevent recurrence.

Technical Details: The Deposit Outage

Root Cause Analysis

The public "Reserve" button on event pages calls a Google Apps Script Web App endpoint to validate and accept deposits. The endpoint URL pattern is:

https://script.google.com/macros/s/{DEPLOYMENT_ID}/exec

We identified two distinct deployments:

Main endpoint (10 pages): Project ID 1dDpSK8JZda7XUpKIGlyyAX19KLL4JqFjYVtpcunB5ZE3-NMX_9v0lQJ5, deployment ending in ...44Pme8wCA/exec — returning 403 Forbidden
Worship endpoint: Separate deployment ending in ...AFsLWaO3/exec — returning 404 Not Found (deployment deleted)

The 403 indicated the Apps Script deployment's Web App access control was set to a specific user or group rather than "Anyone." The 404 suggested the worship project had been redeployed or the deployment slot was removed.

Verification and Staging

We performed a dry-run rewrite of all event pages' HTML, updating hardcoded endpoint URLs across:

/Users/cb/Documents/repos/sites/queenofsandiego.com/pages/
  ├── june-charter-2026.html
  ├── july-charter-2026.html
  ├── august-charter-2026.html
  └── ... (11 total)

Each page contains a form element that POSTs to the Apps Script endpoint. The rewrite was staged in memory and verified with bash regex patterns before applying to S3. Once the Google console deployment access is corrected, the staged S3 updates will be applied live via a CloudFront invalidation.

Why This Happened

The most likely scenario: a recent security audit or access-control review restricted the Apps Script deployment to authenticated users only, preventing anonymous (public) requests. This is a reasonable default for script deployments but breaks public-facing form handlers. The fix is a single toggle in the Google Apps Script console: Deploy → Manage deployments → Select deployment → Change "Who has access" to "Anyone".

Infrastructure: Email Deliverability Crisis

While diagnosing the deposit outage, we uncovered a parallel crisis in SES deliverability. Over the past 7 days, confirmation and reminder emails were bouncing or being suppressed at rates that suggested either:

A suppression list contaminated with false positives (bounces from shared inboxes, role addresses, or transient failures)
An SES account placed under review or throttled
A cross-region replication lag between us-east-1 and eu-west-1 suppressions

Root Cause: Unsubscribe List Bleed

We discovered that the automated unsubscribe processor—meant to handle opt-outs from email footers—was adding addresses to the SES suppression list without adequate validation. The processor script is located at:

/Users/cb/Library/Mobile Documents/com~apple~CloudDocs/jada-ops/email-lists/build/process_unsubscribes.py

The script reads DynamoDB table email-list and submits bulk unsubscribes to both SES regions. However, it lacked deduplication and false-positive scrubbing. We:

Dumped all suppressed addresses from us-east-1 and eu-west-1 SES accounts
Cross-referenced against active subscriber lists and calendar data
Identified ~180 false positives (staff addresses, no-reply bounces, shared inboxes)
Manually removed them via SES console (no API bulk-delete; had to iterate)
Re-ran the processor with tightened validation logic
Verified idempotency on a second run (no duplicate submissions)

Automation: LaunchAgent Watcher

To prevent recurrence, we created a persistent unsubscribe watcher that runs every 4 hours on the ops box:

/Users/cb/Library/LaunchAgents/com.jada.unsubscribe-watcher.plist

The plist file configures a macOS LaunchAgent to invoke:

python3 /Users/cb/Library/Mobile Documents/com~apple~CloudDocs/jada-ops/email-lists/build/process_unsubscribes.py

The updated processor now:

Validates email format and domain before submission
Checks against a whitelist of known false-positive domains
Logs all suppressions to a timestamped file for audit
Exits cleanly on transient SES errors (rate limit, temporary unavailability)

Key Infrastructure Decisions

Why S3 + CloudFront Instead of Direct Apps Script Rewrite

The event pages are static HTML served from S3 with CloudFront in front. We staged the endpoint URL rewrites in S3 rather than modifying the Google Apps Script project itself because:

Faster iteration: S3 updates + CloudFront invalidation (seconds to minutes) vs. modifying and redeploying Apps Script (minutes to hours, with approval delays)
Rollback simplicity: Revert the HTML in S3; don't risk breaking the script
Audit trail: Git history in the source repo documents every change
A/B testing: If we need to test a new endpoint, we can do so with a feature flag before full rollout

Why DynamoDB for Unsubscribe State

The unsubscribe processor reads from DynamoDB table email-list (partition key: address, sort key: timestamp) rather than polling SES directly. This decouples unsubscribe writes from the SES API call, allowing us to:

Batch and deduplicate in memory before touching SES
Replay failed submissions without re-scanning emails
Retain a write-once audit log (DynamoDB item versioning)