Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

```html

Diagnosing and Staging a Production Deposit System Outage: Apps Script Access Control and Endpoint Migration

What Was Done

During a June 2026 incident response session, we discovered that the deposit reservation system powering 11 public event pages had entered a complete failure state. The "Reserve" widgets across sailjada.com and queenofsandiego.com were returning HTTP 403 Forbidden and 404 Not Found errors, silently breaking the entire booking funnel. The root cause: two Google Apps Script deployments had incorrect access control settings and one had been deleted entirely. We diagnosed the issue, identified the affected endpoints, and staged a fix requiring a single configuration change in the Google Apps Script console.

The Incident: Scope and Impact

The deposit system consists of two Google Apps Script web app endpoints, each serving different charter types:

  • Primary endpoint (`...44Pme8wCA/exec`): Handles 10 event pages, returning HTTP 403 Forbidden
  • Worship endpoint (`...AFsLWaO3/exec`): Handles 1 specialized page, returning HTTP 404 Not Found (deployment deleted)

All form submissions from the frontend were hitting these endpoints and failing silently—no error messages, just dead buttons. This meant zero visibility into lost bookings; the funnel was dark.

Technical Diagnosis

We began by establishing the actual live HTTP state of both endpoints:

curl -I https://script.google.com/macros/s/.../44Pme8wCA/exec
# Returns: HTTP 403 Forbidden

curl -I https://script.google.com/macros/s/.../AFsLWaO3/exec
# Returns: HTTP 404 Not Found

Next, we traced the endpoint URLs back to their source code. The deposit widget JavaScript is embedded in static HTML files stored in S3 and served through CloudFront. We located the affected pages in the source repository:

  • /Documents/repos/sites/queenofsandiego.com/events/ — contains 11 HTML event pages
  • Each page contains an embedded form with a hardcoded Apps Script endpoint URL
  • Pages sync to S3 bucket queenofsandiego.com via CI/CD
  • CloudFront distribution ID: E... (redacted) serves pages with caching headers

To confirm which Apps Script project contained the broken deployment, we searched the codebase for project identifiers and read the local .clasp.json configuration files:

# In the Apps Script project directory
cat appsscript.json
# Confirmed project display name and deployment metadata

The primary endpoint project ID is 1dDpSK8JZda7XUpKIGlyyAX19KLL4JqFjYVtpcunB5ZE3-NMX_9v0lQJ5. However, the project ID is distinct from the deployment ID—the actual executable endpoint is the /exec URL, which is created during the deploy step.

Root Cause: Access Control Misconfiguration

Google Apps Script web app deployments require explicit access control. The 403 error indicated the deployment existed but was configured with restricted access—likely Execute as Me (the deployment owner) with Who has access = Me only or a specific user/group.

The 404 error on the worship endpoint indicated the deployment had been deleted or never redeployed after a project update.

The Fix: One Configuration Change

To restore service, the deployment access control must be changed to allow public execution. Inside the Google Apps Script console:

  1. Open the affected project in script.google.com
  2. Navigate to Deploy → Manage deployments
  3. Select the live web app deployment
  4. Click the edit icon and set:
    • Who has access: Anyone
    • Execute as: Me (the project owner/service account)
  5. Click Deploy to apply changes

For the worship endpoint (404), if the deployment was deleted, a new deployment must be created by clicking New deployment → Web app → Execute as Me → Who has access: Anyone.

Critically, the /exec URL remains stable across redeployments to the same deployment slot. The frontend code does not require changes if we redeploy into the existing deployment ID.

Why This Happened: Access Control Creep

This class of incident occurs when:

  • Deployment permissions drift: A developer may have restricted access during debugging or security lockdown and forgotten to restore public access
  • Project ownership changes: If the project was transferred to a different Google account, the Execute as context changes, and old deployments may become inaccessible
  • Accidental deletion: Cleanup operations may have removed old deployments without verifying all dependent pages

The symptom (silent 403/404) is especially dangerous because frontend JavaScript has no visibility into why the request failed—it just shows a dead button.

Infrastructure and Dependencies

The full deposit funnel involves multiple layers:

  • Frontend: Static HTML event pages in S3 (s3://queenofsandiego.com/events/)
  • CDN: CloudFront distribution caches pages with 3600s TTL
  • Backend: Google Apps Script web app executes form processing and writes to Google Sheets (data store)
  • Monitoring: HTTP status checks confirm endpoint availability (documented in DEPOSIT-OUTAGE-FIX-2026-06-02.md)

After the fix is deployed, we should verify by hitting the endpoint directly and confirming a 200 response, then invalidating the CloudFront cache for event pages to ensure browsers receive updated JavaScript that can communicate with the restored endpoint.

Key Decisions and Rationale

  • Why check the live endpoint first: The deployment ID alone doesn't confirm the actual HTTP state. Testing both endpoints with curl confirmed the exact nature of each failure (403 vs. 404)
  • Why re-deploy to the same slot: Changing the deployment URL would require updating all 11 event pages in S3, triggering a multi-region deploy. Keeping the same /exec URL avoids this blast radius
  • Why this is revenue-critical: Every booking attempt across 11 pages is currently failing. This is a total stoppage on inbound deposits, not a partial outage. Fixing it unblocks the entire reservation funnel

What's Next

Once the fix is deployed in the Google Apps Script console:

  1. Re-test both endpoints with curl to confirm 200 responses
  2. Invalidate CloudFront cache for /events/* paths to clear any cached error responses
  3. Monitor the Google Apps Script execution logs for any new errors in form processing
  4. Send a test booking through a staging event page to verify end-to-end functionality
  5. Document the