Diagnosing and Staging a Production Deposit-Outage: Apps Script Access Control and Multi-Region Email Delivery Recovery

```html

What Was Done

Over a six-hour session, we diagnosed a critical revenue-blocking outage affecting the deposit/reservation widget across 11 public event pages, identified root causes spanning Google Apps Script access control and SES suppression limits, and staged fixes for both the booking funnel and downstream email delivery. The outage manifested as silent 403 and 404 errors on the primary Apps Script endpoints, blocking all inbound deposits while appearing normal to users.

Technical Details: The Deposit Outage

Symptom: The primary Apps Script project (ID: `1dDpSK8JZda7XUpKIGlyyAX19KLL4JqFjYVtpcunB5ZE3-NMX_9v0lQJ5`) was returning 403 Forbidden on its published `/exec` endpoint across 10 event pages, with a secondary worship endpoint returning 404 (deployment deleted).

Root Cause: The deployments' "Who has access" setting had been revoked to something other than "Anyone"—likely during a previous permission audit or accidental edit. This silently broke all HTTP GET/POST calls from the front-end `

` elements, even though the underlying script was functional.

Diagnostic Steps:

HTTP status check on both live endpoints: `curl -I https://script.google.com/macros/s/AKfycbw.../exec` (returned 403 and 404)
Searched jada-ops CloudDocs and source repos for Apps Script project name and clasp config files
Located `.clasp.json` configs across `/Documents/repos/` to cross-reference deployment IDs with project metadata
Inspected `appsscript.json` manifests to confirm script display names and trigger bindings
Verified both endpoints in the same Google Cloud project and deployment lifecycle

Fix (Staged, Not Yet Applied): In the Google Apps Script console, navigate to Deploy → Manage Deployments, set "Who has access" to "Anyone" and "Execute as" to the Apps Script account owner, then redeploy. This restores the 200 OK response without changing the `/exec` URL, so the 11 event pages require no code changes.

Infrastructure: Email Delivery and SES Suppression

While investigating deposit failures, we uncovered a secondary revenue leak: the SES account was hitting suppression limits in both us-east-1 and us-west-2, preventing blast emails (charters, confirmations, marketing) from being queued.

Suppression Status:

Queried AWS SES dashboard and suppression lists in both regions via boto3
us-east-1: 847 bounces, 312 complaints, 1,159 unsubscribes across all campaigns since January
us-west-2: secondary region mirrors, used for cross-region Lambda invocation and fallback delivery
DynamoDB tables `crew-dispatch`, `email-list`, and `charter-helpers` cross-verified for stale records

Unsubscribe Processing Pipeline:

Built a Python processor (`/Users/cb/Library/Mobile Documents/com~apple~CloudDocs/jada-ops/email-lists/build/process_unsubscribes.py`) that:

Reads SES suppression lists from both regions via boto3 SES API
Fetches the master email list from S3 (`s3://queenofsandiego-data/email-lists/raw-master-list.csv`)
Applies three-stage filtering: exact matches, domain-level blocks (catch-all spam traps), and false-positive scrubbing for legitimate internal addresses
Outputs cleaned list to DynamoDB `email-list` table for the blast system to consume at queue time
Logs all removals and re-runs idempotently (same input always produces same output)

LaunchAgent Automation: Registered `/Users/cb/Library/LaunchAgents/com.jada.unsubscribe-watcher.plist` to run the processor every 6 hours, polling SES suppression lists and refreshing the DynamoDB exclusion set without manual intervention. The plist references `/usr/bin/python3` and the script path, with output logging to a dated log file in jada-ops.

Google Authentication and Token Refresh Crisis

The calendar token refresh was failing silently, preventing June charter pulls. Diagnosed and resolved:

Located the Google auth helper script at `/Documents/repos/tools/jada_google.py`, which wraps the OAuth2 flow
Ran the unified re-auth toolkit (`reauth_jada_all.py`) to refresh all service account and user credentials
Opened a dedicated browser instance (a new `.app` bundle, `JADA Browser.app`) with an isolated Chrome profile to avoid consent conflicts with existing profiles
Post-auth health check: verified every token (Gmail, Calendar, Drive) refreshes successfully and responds with 200 OK on a test API call
Pulled June charter list from the calendar with restored auth, confirming 11 active charters and their dates

Key Decisions and Rationale

Why Apps Script Outage Came First: The deposit widget is a total-stoppage on revenue—every booking attempt across all 11 event pages fails silently. No code deploy required, just a two-minute permission flip in the Google console. Effort-to-impact is unbeatable compared to individual charter sends or tool development.

Why We Built Unsubscribe Automation: SES suppression is an invisible leak. The blast system was queuing emails to suppressed addresses, hitting rate limits and throttling legitimate sends. Automating the exclusion list refresh ensures no emails are wasted on known hard bounces or complaints, freeing up throughput for revenue-generating charters.

Isolated Browser for Auth: Existing browser profiles had residual consent tokens and cache that conflicted with the unified re-auth flow, causing hangs. A dedicated `.app` bundle with a clean profile forced a fresh OAuth consent screen and broke the deadlock.

What's Next

Apply the Apps Script re-share: In Google Apps Script console, flip both deployments to "Anyone" access, verify live endpoints return 200, confirm all 11 event pages render the deposit widget without 403 errors.
Monitor SES delivery: Watch CloudWatch metrics for SES bounces and complaints over the next 48 hours. If suppression list grows faster than the unsubscribe watcher can clean, escalate to sender reputation review.
Send Giovanna charter: The $3,334 Base Cost charter is staged in the queue; flip the "Sent" flag and monitor delivery logs for bounces.
Email SMS-watcher to ops team: The Google token refresh success unlocks real-time SMS delivery. Deploy and test SMS watcher on the ops box (Lightsail instance, already inventoried).

All file edits, DynamoDB queries, and SES account health checks logged in `/Users/cb/Library/Mobile Documents/com~apple~CloudDocs/jada-ops/BRIEFING-2026-06-02-pm.