Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

```html

Diagnosing and Staging a Critical Deposit System Outage: Apps Script Access Control and Deployment Management

What Happened

During a routine operational review, we discovered that the deposit/reservation widget—deployed across 11 public event pages on sailjada.com and queenofsandiego.com—was returning HTTP 403 (Forbidden) and 404 (Not Found) errors. This meant every inbound booking funnel was silently failing. The root cause: two Google Apps Script deployments had lost their public access configuration, making the /exec endpoints inaccessible to unauthenticated callers.

Root Cause Analysis

We traced the outage to two separate Apps Script projects:

  • Primary endpoint (10 event pages): Project ID 1dDpSK8JZda7XUpKIGlyyAX19KLL4JqFjYVtpcunB5ZE3-NMX_9v0lQJ5, deployment ID ending in ...44Pme8wCA — returning 403 Forbidden
  • Worship endpoint (1 page): Project ID separate, deployment ID ending in ...AFsLWaO3 — returning 404 Not Found (deployment was deleted)

The 403 indicated that the deployment existed but the access control rule had been changed from "Anyone" to a restricted set of users. The 404 indicated the second deployment had been removed entirely, likely during cleanup or a misunderstanding about which version was live.

Technical Investigation Steps

1. Live Endpoint Testing

We verified the current state by making direct HTTP requests to both /exec endpoints: 

curl -I https://script.google.com/macros/s/AKfycbx...44Pme8wCA/exec
# Returns: HTTP 403 Forbidden

curl -I https://script.google.com/macros/s/AKfycbx...AFsLWaO3/exec
# Returns: HTTP 404 Not Found

2. Apps Script Project Inventory

We cross-referenced the Apps Script project IDs across multiple sources:

  • .clasp.json files in the source repository under /Documents/repos/sites/queenofsandiego.com/tools/
  • Notes and deployment records in /Library/Mobile Documents/com~apple~CloudDocs/jada-ops/DEPOSIT-OUTAGE-FIX-2026-06-02.md
  • Google Apps Script console → Project settings → Script ID verification

This confirmed we had the correct project IDs and identified that the live-facing deployment IDs did not match the currently active development deployments.

Infrastructure: Apps Script Deployment Architecture

Google Apps Script deployments are versioned resources. Each deployment gets a unique URL slug at https://script.google.com/macros/s/{DEPLOYMENT_ID}/exec. Key architectural decisions:

  • Separate deployments for production: We maintain distinct deployment IDs for the main deposit flow (10 pages) and the worship-specific flow (1 page). This allows independent rollback and testing without affecting the primary funnel.
  • Access control is per-deployment: Setting a deployment to "Anyone" means any unauthenticated HTTP client can invoke the endpoint. This is required for client-side JavaScript on public web pages to call the Apps Script without OAuth.
  • Execution context: Both deployments are configured to "Execute as Me," meaning they run with the service account credentials of the project owner. This ensures consistent access to the underlying Google Sheets where deposit records are written.
  • URL stability: As long as we redeploy within the same deployment slot, the /exec URL remains unchanged. This means the 11 event pages in production do not need code updates—only the deployment's access control needs to be fixed.

Staging the Fix

The fix for the 403 error is a two-step process within the Google Apps Script console:

  1. Navigate to Deploy → Manage Deployments in the Apps Script editor.
  2. Select the active deployment (ending in ...44Pme8wCA).
  3. Change "Who has access" from the current restricted setting back to "Anyone".
  4. Ensure "Execute as" is set to "Me" (the project owner).
  5. Click Deploy.

For the 404 (deleted worship endpoint), we have two options:

  • Option A: Restore the deployment from the project's version history (if available within Google's retention window).
  • Option B: Create a new deployment of the current code and update the single worship page to call the new endpoint URL.

Option A is preferable because it requires no code changes on the front-end pages. The original deployment URL will be restored without touching any HTML or JavaScript.

Why This Matters: Impact and Prevention

This outage was a total funnel stoppage. While the deposit widget buttons were visible and clickable on all 11 pages, the backend invocation silently failed, meaning:

  • No deposits were being recorded.
  • No confirmation emails were being sent to customers.
  • The team had no visibility into how many booking attempts failed (dark funnel).

The fix is trivial (one console toggle per deployment) but the consequences of not fixing are severe. To prevent recurrence:

  • Monitor deployment health: Add periodic synthetic checks (e.g., CloudWatch scheduled Lambda or Pingdom) that call both /exec endpoints and alert if they return 4xx or 5xx.
  • Document deployment-to-page mappings: Maintain a registry (e.g., in the ops folder) linking each live page to its Apps Script deployment ID. Make this part of the code review checklist.
  • Version control the fix state: Store a deployment-access-status.json file in the repo documenting the required access level for each deployment, so future team members know the expected state.
  • Run smoke tests post-deploy: After any Apps Script deployment, immediately curl both endpoints and verify the response is not 403/404.

What's Next

Once the Apps Script access is restored and both /exec endpoints return 200 OK:

  1. Test the full deposit flow end-to-end on a staging page or directly via curl with sample JSON payload.
  2. Verify that test deposits appear in the linked Google Sheet.
  3. Confirm that confirmation emails are being sent (check SES metrics and sample recipient inbox).
  4. Alert all event coordinators that the reservation system is live again.
  5. Document the incident and the resolution in jada-ops/INCIDENTS/ for future reference.