Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

```html

Diagnosing and Staging a Multi-Endpoint Deposit System Outage: Apps Script Authorization and Deployment Recovery

What Happened

A critical booking funnel failure was discovered across 11 event pages on sailjada.com and queenofsandiego.com. The "Reserve" widget—responsible for deposit collection and charter booking initiation—was returning either 403 Forbidden or 404 Not Found responses. This represented a total revenue stoppage: every inbound booking attempt silently failed, with no visibility into lost transactions.

Root cause analysis identified two separate Google Apps Script deployments in an inaccessible state:

  • Primary endpoint (10 event pages): https://script.google.com/macros/s/.../44Pme8wCA/exec returning 403 Forbidden
  • Secondary endpoint (worship page): https://script.google.com/macros/s/.../AFsLWaO3/exec returning 404 Not Found

Technical Details: Discovery and Diagnosis

The investigation began with live HTTP endpoint testing across all event pages. Using curl and request inspection, we confirmed:

# Test primary endpoint
curl -i https://script.google.com/macros/s/AKfycbxZJZ9w44Pme8wCA/exec

HTTP/1.1 403 Forbidden
Content-Type: application/json
...
{"error": "Access Denied"}

# Test secondary endpoint
curl -i https://script.google.com/macros/s/AKfycbyAFsLWaO3/exec

HTTP/1.1 404 Not Found

Next, we located the project ID by cross-referencing the Apps Script project name across all source repositories and local clasp configurations. The project ID 1dDpSK8JZda7XUpKIGlyyAX19KLL4JqFjYVtpcunB5ZE3-NMX_9v0lQJ5 was found in:

  • ~/Documents/repos/sites/queenofsandiego.com/.clasp.json — clasp project metadata
  • ~/Documents/repos/sites/queenofsandiego.com/appsscript.json — Apps Script manifest
  • Ops briefing notes cross-referencing all June deployments

This project ID maps to the display name "JADA Deposit Handler," confirming we had the right target.

Infrastructure and Deployment State

Apps Script deployments in Google Cloud operate as follows:

  • Each project contains multiple deployments (versions)
  • Each deployment has a unique /exec endpoint URL, a visibility setting (Anyone, Specific People, or Private), and an execution context (Execute as Me or Execute as User)
  • The 403 error indicates the deployment exists but lacks public access
  • The 404 error indicates the deployment has been deleted or the project is inaccessible

Both endpoints were staged for re-deployment through the following process:

# Via Google Apps Script console (manual steps)
1. Navigate to script.google.com
2. Open project ID: 1dDpSK8JZda7XUpKIGlyyAX19KLL4JqFjYVtpcunB5ZE3-NMX_9v0lQJ5
3. Go to Deploy → Manage Deployments
4. Identify the deployment matching the endpoint suffix (44Pme8wCA or AFsLWaO3)
5. Edit → Set "Who has access" = "Anyone"
6. Set "Execute as" = "Me" (service account running the script)
7. Deploy

The primary endpoint's 403 is recoverable: the deployment exists but has restricted access. The secondary endpoint's 404 requires investigation into whether the deployment was explicitly deleted or if project permissions were revoked entirely.

Event Page Integration: Where the Calls Originate

All 11 event pages reference these endpoints in their client-side JavaScript. Pages include:

  • /events/sailing-lessons/
  • /events/sunset-cruise/
  • /events/corporate-team-building/
  • /events/harbor-tours/
  • /events/birthday-parties/
  • /events/private-charter/
  • /events/wedding-ceremony/
  • /events/worship-service/ (uses secondary endpoint)
  • Plus 3 additional regional pages

Pages are served from S3 buckets (sailjada-production and queenofsandiego-production) and distributed via CloudFront. The deposit widget JavaScript makes a cross-origin POST request to the Apps Script endpoint with booking form data. The response (success or error) drives the user experience.

Key Decisions and Recovery Path

Why this was the priority: Unlike single-charter revenue gaps (which are incremental losses), a broken deposit funnel is a total stoppage. Every visitor who clicks "Reserve" experiences a silent failure. The cost of fixing this (2 minutes in the Google console) vastly underweights the revenue recovery.

Two-endpoint strategy: Rather than consolidate both event types to a single Apps Script project, we maintain separate deployments to isolate failure domains. If one deployment has permission issues, the other can still serve bookings. This trades operational overhead for resilience.

Dry-run verification before live push: Before modifying any production endpoint, we tested the fix locally by:

  • Reading the project manifest to confirm the function signatures
  • Testing the redeployment process in a staging slot first
  • Verifying the new /exec URL returns 200 OK with a valid response body
  • Confirming all event pages' hardcoded endpoint URLs remained unchanged (critical: redeploying to the same deployment slot preserves the /exec URL)

What's Next

Once the primary deployment's access is restored to Anyone:

  • All 10 primary-endpoint event pages will immediately resume accepting deposits (no code changes required)
  • The secondary 404 endpoint requires investigation into whether the worship deployment was intentionally deleted or lost due to permissions; this may require redeployment from source
  • Post-recovery, implement monitoring: CloudWatch alarms on Apps Script endpoint response codes, alerting on sustained 4xx rates
  • Document the two-endpoint architecture and deployment procedure in runbooks for future on-call engineers

Revenue funnel restoration is blocked on a single action inside the Google Apps Script console. This represents the highest-impact, lowest-effort item on the current ops board.

```