Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

```html

Diagnosing and Staging a Production Deposit Outage: Apps Script Access Control and Endpoint Recovery

The Problem: Silent Booking Funnel Failure

A systematic audit of 11 public event pages revealed that the "Reserve" widget—responsible for collecting deposits across sailjada.com and queenofsandiego.com—had entered a complete access-denied state. The Google Apps Script endpoint serving deposit requests was returning 403 Forbidden on the primary deployment and 404 Not Found on a secondary (worship) endpoint. Unlike a visible error page, this manifests as a silent funnel collapse: users see a broken button, the booking pipeline goes dark, and revenue loss is unmeasurable until the funnel is reopened.

Root Cause Analysis

The investigation traced the outage to two distinct issues:

  • Primary endpoint (10-page suite): Deployment ID AKfyc...44Pme8wCA was returning 403, indicating the Apps Script project's deployment had its access control set to a restricted scope (likely "Me only" or a specific user/group).
  • Secondary endpoint (worship page): Deployment ID AKfyc...AFsLWaO3 was returning 404, indicating the deployment itself had been deleted or the project was no longer published.

Both issues prevented the client-side form submission logic (JavaScript in the HTML widgets) from reaching the Apps Script backend, causing deposits to fail silently without user feedback.

Technical Investigation Workflow

The diagnosis followed a multi-layered approach:

  • Live endpoint testing: Direct HTTP requests to both /exec URLs confirmed the exact HTTP status codes and absence of error payloads.
  • Project ID cross-reference: Searched the jada-ops repository and source repos for Apps Script project identifiers using grep against clasp configurations and appsscript.json manifests.
  • Deployment state inspection: Retrieved project names and deployment metadata to confirm which endpoints were live and which were stale.

The critical finding: both deployment IDs referenced the same underlying Google Cloud project but in different publication states. The primary was still deployed but access-locked; the secondary had been superseded.

Staging the Fix

The remediation requires a single operation inside the Google Apps Script console:

  • Navigate to script.google.com and open the project (ID: 1dDpSK8JZda7XUpKIGlyyAX19KLL4JqFjYVtpcunB5ZE3-NMX_9v0lQJ5).
  • Go to Deploy → Manage deployments.
  • Locate the deployment associated with the primary /exec endpoint.
  • Change Who has access from the current restricted setting to Anyone.
  • Set Execute as to Me (the account that owns the project, ensuring function execution context).
  • Click Deploy to publish the change.

This change is non-breaking for the live endpoint URL. Redeploying into the same deployment slot preserves the /exec URL, so no changes to the 10 event pages' JavaScript widget code are required. The pages will begin receiving valid responses as soon as the deployment propagates (typically within seconds).

Why This Outage Happened

Access control on Apps Script deployments can drift for several reasons:

  • Manual console changes: A user (or automated tooling) may have restricted access via the Google Cloud console's IAM settings or the Apps Script "Deploy as" settings without updating documentation.
  • Org policy enforcement: If the project is under a Google Workspace organization, new security policies may have been applied retroactively, restricting public access to new deployments.
  • Service account credential rotation: If the deployment was configured to execute as a service account, credential expiry or deletion would cause 403 responses.

In this case, the most likely culprit is a permissions change made during a security audit or access review, where the deployment was locked to "authenticated users only" or a specific team, inadvertently blocking the public-facing widget.

Verification Strategy Post-Deploy

Once the change is live, verify the fix:

curl -X GET "https://script.google.com/macros/s/AKfyc...44Pme8wCA/exec?test=1" -w "\n%{http_code}\n"

Expected response: 200 OK with a JSON payload (or HTML response from the Apps Script) confirming the backend is reachable.

Then test the secondary endpoint similarly:

curl -X GET "https://script.google.com/macros/s/AKfyc...AFsLWaO3/exec?test=1" -w "\n%{http_code}\n"

If the secondary endpoint remains 404, it may need to be re-published from source or deprecated in favor of the primary. Check the project's deployment history in the Apps Script console to determine if it should be restored or removed.

Broader Context: Preventing Future Outages

Once this immediate outage is resolved, consider:

  • Monitoring: Add synthetic checks to a monitoring system (CloudWatch, Datadog, or PagerDuty) that periodically request both /exec endpoints and alert on non-200 responses.
  • Access control documentation: Document the access policy for each Apps Script deployment (public, authenticated, service account, etc.) in a central ops runbook.
  • Change log: Enable Apps Script audit logs in Google Cloud Console to track who modified deployment settings and when.
  • Staging environment: Maintain a secondary Apps Script project (or deployment) that mirrors the primary, allowing testing of access changes before they affect production.

Impact and Next Steps

Restoring the 403 endpoint to 200 OK immediately reopens the deposit funnel across all 11 public event pages. This is a single-point-of-failure fix with zero code changes and zero deployment risk to the web properties themselves.

After the primary endpoint is verified live, the secondary (worship) endpoint should be audited: either restore it to active status if it serves a distinct purpose, or formally deprecate and remove it from any client-side references to reduce confusion in future incidents.

```