Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

```html

Diagnosing and Staging a Google Apps Script Deployment Outage: Deposit Widget Failure Across 11 Event Pages

What Was Done

A critical production incident was identified and diagnosed: the deposit/reservation widget powered by Google Apps Script was returning 403 Forbidden and 404 Not Found errors across all 11 event pages on the sailjada and queenofsandiego domains. This represented a complete funnel blockage for inbound bookings. The root cause was traced to deployment access restrictions and a deleted secondary deployment, and a fix was staged and documented for immediate execution.

The Outage: Scope and Impact

The public-facing "Reserve" widgets on every event page were silently failing. Users clicking to initiate a deposit saw either access denied or a missing endpoint error. Since the failures happened in the browser XHR layer, no alerts fired — the funnel just went dark.

Affected endpoints:

  • Primary: https://script.google.com/macros/s/{EXEC_ID_1}/exec returning 403 Forbidden
  • Worship secondary: https://script.google.com/macros/s/{EXEC_ID_2}/exec returning 404 Not Found
  • Pages affected: All 11 event detail pages across both domains

Why this matters: Every inbound booking attempt was failing without user feedback. The deposit system is the primary revenue funnel; this outage directly blocked cash flow.

Root Cause Analysis

The investigation proceeded in layers:

Step 1: Verify Live Endpoint State

Direct HTTP testing of both Apps Script endpoints confirmed both were down:

curl -i https://script.google.com/macros/s/{EXEC_ID_1}/exec
# Returns: 403 Forbidden

curl -i https://script.google.com/macros/s/{EXEC_ID_2}/exec
# Returns: 404 Not Found

This ruled out DNS, CDN, or application-layer issues; the problem was at the Apps Script deployment level.

Step 2: Locate the Project and Deployment Configuration

Cross-referenced the exec URL against local clasp configs:

  • Searched ~/Documents/repos/ for .clasp.json files containing the exec ID
  • Found the primary project: jada-ops/Apps Script project (ID: 1dDpSK8JZda7XUpKIGlyyAX19KLL4JqFjYVtpcunB5ZE3-NMX_9v0lQJ5)
  • Identified the secondary worship project via appsscript.json display names

Step 3: Determine Cause in Google Console

Logged into the Google Apps Script console for the primary project and inspected deployment settings:

  • Primary deployment: Access set to a restricted group or user. Script was deployed but not callable by "Anyone."
  • Worship deployment: Not found in active deployments — the deployment had been deleted, leaving only the exec URL pointing to a ghost.

Both scenarios explain the dual failure modes: the 403 means the deployment exists but access is denied; the 404 means the deployment was removed entirely.

Staging the Fix

The fix requires two actions in the Google Apps Script console (no code changes, no re-deployment of new logic):

Primary Deployment (403 Fix)

  1. Open script.google.com
  2. Select the project 1dDpSK8JZda7XUpKIGlyyAX19KLL4JqFjYVtpcunB5ZE3-NMX_9v0lQJ5
  3. Navigate to Deploy → Manage Deployments
  4. Click the active deployment (typically the highest version number)
  5. Change:
    • Who has access: Set to Anyone
    • Execute as: Keep as Me (the script owner, typically a service account or your user)
  6. Click Deploy

Key point: This does not create a new deployment. It updates the access control on the existing one. The exec URL remains unchanged; live pages need no edits.

Worship Deployment (404 Fix)

Since the deployment is deleted, two options exist:

  • Option A (if backup exists): Locate a backup of the worship script, re-upload it to Apps Script, and create a new deployment with "Anyone" access.
  • Option B (if no backup): Identify the current reservation logic for worship events, redeploy it as a new Apps Script project, and update the exec URL in the HTML pages calling it.

Option B requires identifying which pages use the worship endpoint and updating /Users/cb/Library/Mobile Documents/com~apple~CloudDocs/jada-ops/ event page templates, then pushing changes to S3.

Why This Architecture Was in Place

Google Apps Script deployments are used here because:

  • Rapid iteration: No build pipeline; changes deploy instantly without Lambda re-packaging or container pushes.
  • Integrated Google Auth: Apps Script natively handles OAuth flows for calendar and Gmail APIs.
  • Cost: Apps Script has a generous free tier for low-volume execution (e.g., deposit initiation).
  • Headless HTTP: The /exec endpoint provides a simple webhook-style interface for browser XHR calls without managing API Gateway routes or custom domains.

The dual-deployment pattern (primary + worship secondary) suggests a refactoring where worship events were moved to a separate Apps Script project, possibly for team access control or audit separation.

Prevention and Monitoring

To prevent recurrence:

  • Deployment access control audit: Add a monthly check that all production Apps Script endpoints have Who has access = Anyone.
  • Synthetic monitoring: Add a CloudWatch or Pingdom health check that hits both exec URLs and alerts if either returns non-2xx. Example: 
    #!/bin/bash
# Check primary endpoint
if ! curl -s -o /dev/null -w "%{http_code}" \
  https://script.google.com/macros/s/{EXEC_ID_1}/exec | grep -q "^2"; then
  # Alert: primary endpoint down
fi
  • Deployment versioning: Document which exec URLs are live and where in the source repos (HTML templates, config files) they are referenced.
  • Access control as code: If Apps Script deployments grow in number, consider codifying deployment creation and access control in a Terraform-style tool or a post-deploy validation script.

What's Next

Once the fix is executed and verified: