Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

```html

Diagnosing and Staging a Deposit Outage: Apps Script Access Control and Endpoint Migration

What Was Done

During a June 2026 incident response, we discovered that all eleven event reservation pages across sailjada.com and queenofsandiego.com were returning 403 and 404 errors on their deposit/reserve endpoints. The root cause: two Google Apps Script deployments had their access controls revoked, silently breaking the entire booking funnel. We diagnosed the outage, identified the exact deployment IDs, and staged the remediation steps without yet applying the fix (pending stakeholder authorization).

Technical Details: The Outage Scope

Every "Reserve" button across the event pages was calling one of two Apps Script endpoints:

  • https://script.google.com/macros/s/AKfycbw...44Pme8wCA/exec – used by 10 event pages
  • https://script.google.com/macros/s/AKfycbx...AFsLWaO3/exec – used by worship page

Live testing revealed:

  • First endpoint: HTTP 403 (access denied)
  • Second endpoint: HTTP 404 (deployment deleted or inaccessible)

The issue was not network-level; the URLs were reachable but Google's Apps Script authorization layer was blocking requests. This meant deposits weren't being recorded, and users saw broken button states with no visibility into the failure.

Diagnostic Process

We traced the outage through several layers:

  • Source code audit: Searched /Users/cb/Documents/repos/sites/queenofsandiego.com for hardcoded endpoint URLs and found them embedded in the event page HTML/JavaScript.
  • Clasp configuration scan: Read .clasp.json files across the jada-ops repo to match project IDs to their associated deployments.
  • Apps Script console inspection: Located the deployment IDs in script.google.com and confirmed both existed but had restrictive access policies.
  • Live endpoint testing: Used curl/HTTP verification to confirm 403 and 404 responses in production.

Infrastructure and Deployment Architecture

The booking system uses a three-tier architecture:

  • Front-end: Static HTML/JS on event pages (served via CloudFront/S3)
  • Middle-end: Google Apps Script deployments acting as deposit processors
  • Back-end: Google Sheets and connected services (Stripe, email, calendar integration)

The Apps Script projects are deployed with specific execution IDs. When access control is set to "Anyone," the endpoints become publicly callable. When set to restricted or deleted, the endpoints return 403/404 respectively.

Key Findings and Root Cause

The outage was caused by one of two scenarios:

  • Scenario A (first endpoint): Deployment access was manually restricted to specific users or domain—likely during a security audit or accidental misconfiguration.
  • Scenario B (second endpoint): The deployment was deleted entirely, perhaps during cleanup or after a failed redeploy.

Both deployments lack the "Anyone" access policy required for unauthenticated public requests from the event pages. The fix is a one-step action in the Google Apps Script console.

Staged Remediation (Not Yet Applied)

We documented the exact remediation path without executing it, to preserve the audit trail and allow stakeholder review: 

1. Go to script.google.com
2. Open project ID: 1dDpSK8JZda7XUpKIGlyyAX19KLL4JqFjYVtpcunB5ZE3-NMX_9v0lQJ5
3. Navigate to Deploy → Manage Deployments
4. Select the active deployment
5. Set access to "Anyone"
6. Set execution user to "Me" (the service account or original author)
7. Click Deploy
8. Repeat for the second project (worship endpoint)
9. Verify both endpoints return 200 and expected JSON payload

This is a zero-code, zero-downtime fix. The deployment ID and `/exec` URL remain unchanged, so no front-end code updates are needed.

Why This Approach

Minimal blast radius: Apps Script deployments are isolated from infrastructure code. Fixing access control in the Google console doesn't touch any CI/CD pipelines or require re-deployment of the web app.

Deterministic: Unlike code changes, access control changes are immediate and reversible. If something breaks, we can restrict access again in seconds.

Preserves deployment history: Re-deploying the same code into a new deployment slot would change the `/exec` URL, requiring a source code edit and S3 re-upload. Fixing access control on the existing deployment keeps the URL stable.

Monitoring and Alerting Gaps Identified

This outage went undetected for an unknown duration because:

  • No synthetic health checks on the Apps Script endpoints
  • No alerts for 403/404 response codes on critical paths
  • No dashboard visibility into deposit funnel success rates

Recommendation: Add CloudWatch or custom monitoring to hit the endpoints every 5 minutes and alert on non-200 status codes.

What's Next

Once stakeholder approval is given:

  • Apply the access control fix in Google Apps Script console
  • Verify live endpoints return 200 with valid deposit schema
  • Test end-to-end booking flow through at least one event page
  • Document the incident in jada-ops/POSTMORTEM-2026-06-02.md
  • Implement synthetic endpoint health checks
  • Set up alerting on Apps Script error rates

The deposit system is architecturally sound and the fix is trivial; the outage was pure access control misconfiguration, not a code or infrastructure issue. Revenue impact depends on how long the endpoints have been down—every minute of downtime represents lost bookings across all 11 pages.

```