Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

```html

Diagnosing and Staging a Critical Deposit Funnel Outage: Apps Script Access Control and Deployment Recovery

What Was Done

We identified and staged a fix for a complete failure in the deposit collection funnel across 11 public event pages. The root cause: two Google Apps Script deployments were either inaccessible (403 Forbidden) or deleted (404 Not Found), silencing all "Reserve" widget submissions. This post walks through the diagnostic methodology, infrastructure investigation, and the staged remediation waiting for execution.

The Incident: Silent Booking Failures

Every public event page on sailjada.com and queenofsandiego.com serves a "Reserve" booking widget backed by a Google Apps Script HTTP endpoint. When users attempted to submit deposits, the endpoint returned a 403, causing the funnel to fail silently. No errors surfaced in logs; the requests simply died at the Apps Script gateway.

Why this matters: Unlike a visible error page, a 403 on an async request means the frontend never alerts the user. Potential customers hit the Reserve button, see no confirmation, and assume the system is broken—or worse, assume their payment information was lost. Revenue leakage was unquantified but total.

Technical Diagnosis: From Project ID to Live Endpoint

Step 1: Identify the Apps Script Project

We located the project ID 1dDpSK8JZda7XUpKIGlyyAX19KLL4JqFjYVtpcunB5ZE3-NMX_9v0lQJ5 by:

  • Searching jada-ops documentation for Apps Script references
  • Scanning the source repository for .clasp.json clasp configuration files
  • Cross-referencing appsscript.json metadata in the project's cloud store

Step 2: Resolve Deployment vs. Execution Endpoint

A critical distinction: the project ID and the deployment endpoint URL are not interchangeable. The project ID identifies the cloud resource; the deployment endpoint is the publicly callable HTTP gateway. We discovered two active deployments:

  • Main endpoint: https://script.google.com/macros/s/AKfycbw...44Pme8wCA/exec (10 event pages) — returning 403 Forbidden
  • Worship endpoint: https://script.google.com/macros/s/AKfyc...AFsLWaO3/exec (1 page) — returning 404 Not Found

Why the distinction matters: A project can have multiple deployments. Each deployment is a versioned snapshot with its own URL. The 403 suggested the main deployment existed but had access revoked; the 404 suggested the worship deployment was deleted entirely.

Step 3: Live Endpoint Testing

We tested both endpoints directly:

curl -i https://script.google.com/macros/s/AKfycbw...44Pme8wCA/exec
# HTTP/1.1 403 Forbidden

curl -i https://script.google.com/macros/s/AKfyc...AFsLWaO3/exec
# HTTP/1.1 404 Not Found

Confirmed: the production funnel was fully blocked at the Apps Script layer.

Root Cause: Access Control and Deployment State

403 on Main Endpoint

The deployment existed but its access control was set to a restricted scope—likely "Only me" or a specific email whitelist. This is a common misconfiguration when a developer redeploys without explicitly setting the execution policy. The endpoint was reachable by DNS but rejected all callers.

404 on Worship Endpoint

The worship deployment had been deleted entirely, possibly during a cleanup or failed redeployment attempt. The URL remained hardcoded in the page but had nothing to resolve to.

Infrastructure and S3/CloudFront Integration

The 11 event pages are static HTML served from an S3 bucket behind a CloudFront distribution. Each page contains hardcoded JavaScript that targets one of the two Apps Script endpoints:

// In event pages (sailjada.com, queenofsandiego.com)
const DEPOSIT_ENDPOINT = 'https://script.google.com/macros/s/AKfycbw...44Pme8wCA/exec';
const RESERVE_ENDPOINT = 'https://script.google.com/macros/s/AKfyc...AFsLWaO3/exec';

When Apps Script deployments change, these hardcoded URLs must be updated across all pages. Because the endpoints were embedded as constants rather than served from a config API, a deployment ID change would require a rebuild and S3/CloudFront reupload.

Why this architecture was chosen: Direct endpoint embedding eliminates a config service dependency and keeps event pages cacheable at the edge. The tradeoff is that deployment changes require a page rebuild.

Staged Fix and Remediation Path

Immediate Action Required (in Google Apps Script Console):

  1. Navigate to script.google.com and open project 1dDpSK8JZda7XUpKIGlyyAX19KLL4JqFjYVtpcunB5ZE3-NMX_9v0lQJ5
  2. Go to Deploy → Manage deployments
  3. For the main deployment (ID ending ...44Pme8wCA):
    • Change Who has access to "Anyone"
    • Ensure Execute as is set to "Me" (the service account or Apps Script owner)
    • Click Deploy
  4. For the worship endpoint: recreate the deployment from the latest project version, or update the page to use the main endpoint

Expected Outcome: The 403 becomes 200, the 404 is replaced with a valid endpoint. The pages require no code changes; they already target the correct URLs. Cache invalidation via CloudFront invalidation API will push the updated pages to edge nodes within seconds.

Key Decisions and Lessons

  • Why not use a config API? Event pages are static; embedding endpoints keeps them cacheable and dependency-free. For future deployments, consider a lightweight JSON config served from S3, or use environment-specific subdomain patterns if deployment URLs stabilize.
  • Why test live before assuming the fix? A project ID alone doesn't confirm the live state. Deployments can be created, deleted, or have their access revoked without changing the ID. HTTP status codes are ground truth.
  • Why the two-endpoint pattern? The main endpoint serves 10 events; worship is separate, possibly for custom logic or audit separation. If the worship endpoint is deleted, consolidating onto the main endpoint (with proper ACLs) reduces operational surface area.

Preventing Recurrence

Once the immediate outage is resolved:

  • Document the deployment access