Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

```html

Diagnosing and Staging a Critical Deposit-Processing Outage: Apps Script Access Control & Deployment Strategy

During a routine operational review on June 2, 2026, we discovered that all eleven event-page "Reserve" widgets—the primary booking and deposit collection mechanism across sailjada.com and queenofsandiego.com—were silently failing. This post details the diagnostic process, root cause identification, and the staged remediation strategy we implemented.

The Problem: Silent Booking Funnel Collapse

Every public reservation widget was returning either a 403 Forbidden or 404 Not Found response when users attempted to submit deposit payments. The failure modes were split across two distinct Apps Script deployments:

  • Primary endpoint (10 event pages): https://script.google.com/macros/s/AKfycbzQ...44Pme8wCA/exec — returning 403 Forbidden
  • Secondary endpoint (worship charter page): https://script.google.com/macros/s/AKfycbzM...AFsLWaO3/exec — returning 404 Not Found

The 404 indicated the secondary deployment had been deleted entirely. The 403 suggested the primary deployment existed but lacked public access permissions. Critically, there were no error logs, no alerts, and no user complaints yet—the failures were transparent to both our monitoring and our customers' browsers, likely returning bare HTTP status codes without descriptive error pages.

Root Cause: Access Control & Deployment Lifecycle

Apps Script deployments require explicit access grants. When a deployment is created or updated, the default access level is restricted to the deployment owner. Public HTTP endpoints (used by client-side JavaScript on static event pages) require the deployment to be explicitly set to Who has access = Anyone.

We discovered two separate failures:

  1. Primary deployment (403): The deployment existed but had reverted to owner-only access, likely due to a recent redeployment that didn't preserve the public access setting.
  2. Secondary deployment (404): This deployment had been manually deleted, possibly during cleanup or consolidation work, leaving the worship page pointing to a non-existent endpoint.

Diagnostic Approach: Endpoint Testing & Source Tracing

We traced the failure through multiple layers:

  • Live endpoint testing: Direct HTTP requests to both /exec URLs confirmed the status codes and response bodies.
  • Source repository scanning: Searched /Users/cb/Documents/repos/sites/queenofsandiego.com/ and /Users/cb/Documents/repos/sites/sailjada.com/ for Apps Script project IDs and endpoint URLs across all event page HTML.
  • Google Apps Script project identification: Located project ID 1dDpSK8JZda7XUpKIGlyyAX19KLL4JqFjYVtpcunB5ZE3-NMX_9v0lQJ5 via clasp configuration files and cross-referenced deployment IDs via appsscript.json metadata.
  • Deployment inventory: Listed all active deployments in the Apps Script console and confirmed the secondary deployment no longer existed in the Google Cloud project.

Staged Remediation Strategy

Rather than immediately republish deployments, we staged a fix with verification steps to ensure zero downtime and traceability:

Step 1: Primary Endpoint Access Restoration

For the 403 Forbidden response on the primary endpoint, the remediation is a single console action—no code changes required:

1. Open Google Apps Script console → script.google.com
2. Load project ID: 1dDpSK8JZda7XUpKIGlyyAX19KLL4JqFjYVtpcunB5ZE3-NMX_9v0lQJ5
3. Navigate to Deploy → Manage deployments
4. Select the active deployment (ID: 44Pme8wCA)
5. Edit deployment settings:
   - Who has access: Change to "Anyone"
   - Execute as: Keep as "Me" (service account ownership)
   - New version: Select latest code version
6. Click "Deploy" to apply changes

Why "Execute as Me"? The Apps Script function that processes deposits must maintain owner-level permissions to write to Google Sheets and trigger downstream email/webhook handlers. Public access to the endpoint does not grant callers owner permissions; they inherit the service account's role.

Step 2: Secondary Endpoint Redeployment

For the 404 on the secondary endpoint, we must recreate the deployment:

1. Identify the Apps Script project containing the worship charter handler
2. Ensure the handler function is present and unchanged
3. Create a new deployment via the console:
   - Description: "Worship Charter Deposit Handler (2026-06-02)"
   - Execute as: Service account owner
   - Who has access: Anyone
4. Note the new deployment ID generated
5. Update the worship event page HTML to call the new /exec URL

Why not clone the old deployment? Once a deployment is deleted, the ID is gone. Creating a fresh deployment ensures a clean slate and a single source of truth in the console.

Step 3: Page Rewrite & Verification

After deployment fixes are live, we rewrite all event page HTML to ensure they call the correct endpoints:

  • Files to update: /queenofsandiego.com/events/*.html (10 pages) and /queenofsandiego.com/worship-charters/*.html (1 page) in the source repo and synced to S3.
  • Search pattern: Find all instances of <form action="https://script.google.com/macros/s/AKfyc... and verify the deployment ID matches the newly deployed version.
  • Dry-run validation: Before applying to S3, run a local test form submission against the new endpoints to confirm 200 OK and expected response payload.

Infrastructure & State

The event pages are hosted on S3 behind CloudFront (distribution ID: E2N7Z...) with Route53 alias records pointing to the CloudFront domain. The HTML forms are static; the Apps Script endpoints are the only dynamic backend. By fixing the deployment access control, we restore the deposit funnel without touching S3, CloudFront, or DNS.

Key Decisions

  • No code changes required for the primary outage. This is an access control issue, not a logic bug. Redeploying without fixing permissions would repeat the failure.
  • Staged verification before production sync. We test the new endpoints directly before updating the source repo and syncing to S3, to avoid cascading page caches or CloudFront stale content.
  • Service account execution model. Apps Script functions that write to Sheets or trigger external APIs must execute as the owner, not the caller. This prevents customers from bypassing audit trails or modifying other data.

What's Next

Once the fixes are deployed and verified live