Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

Debugging and Reconciling Duplicate Event Pages: A Multi-System State Synchronization Problem

During a recent development session, we discovered a critical issue in our event management infrastructure: a single booking existed in multiple representations across different systems, with conflicting data and divergent feature completeness. This post documents the discovery process, the root cause analysis, and the decisions made to reconcile state across S3, DynamoDB, CloudFront, and external service providers.

The Problem: One Booking, Multiple Truths

The Jonathan Boatsetter sunset charter (booking ID #xhqgmdh, scheduled for May 30, 2026) existed in at least three conflicting states:

  • S3 page (older): s3://queenofsandiego.com/g/2026-05-30-jonathan-sunset.html — showing 4:30–7:30 PM, created May 9, containing full feature implementation (photo upload, token validation, spam guard, Instagram image pull)
  • S3 page (newer): s3://queenofsandiego.com/g/2026-05-30-jonathan-afternoon.html — showing 4:30–7:30 PM, created May 20, containing a stripped-down 96-line implementation without photo features
  • DynamoDB crew-dispatch table: jada-crew-dispatch with guest_code: S2FJLN — pointing to the afternoon page but storing the corrected start time of 15:30 (3:30 PM)
  • External systems: Boatsetter.com and internal Google Calendar showing 4:30–7:30 PM (stale)

The time mismatch was the smoking gun. The DynamoDB record had been updated to reflect a booking time change to 3:30 PM, but the HTML pages and external systems had not been synchronized.

Root Cause: Agent-on-Agent State Divergence

Timeline reconstruction revealed the issue:

  1. May 9: Original booking at 4:30 PM. Agent A created the feature-rich -sunset.html page with photo upload, token validation, and Instagram integration (all working correctly).
  2. May 20: Booking time changed to 3:30 PM. Agent B updated the DynamoDB crew-dispatch record with the new time (15:30) but apparently did not discover the existing sunset page. Instead, Agent B created a new -afternoon.html page and updated the DynamoDB guest_code pointer to reference it.
  3. Handoff gap: No explicit documentation that these were the same booking. The newer afternoon page was built without the photo-upload features, likely because Agent B was working from a template or prior simpler implementation.
  4. External systems:** No mechanism existed to cascade the May 20 time change from DynamoDB to Boatsetter, Google Calendar, or CloudFront-cached pages.

This is a classic distributed-systems consistency problem: we had eventual consistency (or no consistency mechanism at all) between source-of-truth systems.

Discovery Process: How We Found It

The investigation used a multi-layer approach:


# Step 1: Pull the crew-dispatch DynamoDB record
aws dynamodb get-item \
  --table-name jada-crew-dispatch \
  --key '{"guest_code": {"S": "S2FJLN"}}'

# Result: start_time = 15:30, page_slug = "2026-05-30-jonathan-afternoon"


# Step 2: List all pages for the same date to find duplicates
aws s3 ls s3://queenofsandiego.com/g/ \
  --recursive \
  | grep "2026-05-30"

# Result: Both -sunset.html and -afternoon.html found


# Step 3: Compare file sizes and creation dates
aws s3api head-object \
  --bucket queenofsandiego.com \
  --key "g/2026-05-30-jonathan-sunset.html"

aws s3api head-object \
  --bucket queenofsandiego.com \
  --key "g/2026-05-30-jonathan-afternoon.html"

# Result: sunset is 568 lines (5/9), afternoon is 96 lines (5/20)


# Step 4: Grep both files for time references
grep -i "4:30\|15:30\|3:30" /tmp/2026-05-30-jonathan-sunset.html
grep -i "4:30\|15:30\|3:30" /tmp/2026-05-30-jonathan-afternoon.html

# Result: Both show 4:30 PM in HTML, but DDB shows 15:30 in metadata

Technical Details: The Update Strategy

We implemented a manual but auditable reconciliation rather than automating it, because:

  • The canonical source was ambiguous (which page was actually sent to the guest?)
  • We needed to preserve the feature-rich implementation on the sunset page
  • External systems (Boatsetter, Google Calendar) required manual or API-based updates we control

Changes made:

  1. Update the primary S3 page (2026-05-30-jonathan-sunset.html):
    • Change all time references: 4:30 PM → 3:30 PM, 7:30 PM → 6:30 PM, arrival 4:00 PM → 3:00 PM
    • Update UTC cutoff time: Monday 8:00 PM → Monday 7:00 PM (for server-side validation logic)
    • Preserve all 568 lines of photo-upload, token, and spam-guard logic
    • Upload updated HTML to s3://queenofsandiego.com/g/2026-05-30-jonathan-sunset.html
  2. Invalidate CloudFront distribution for queenofsandiego.com:
    • Path: /g/2026-05-30-jonathan-sunset.html
    • Command: aws cloudfront create-invalidation --distribution-id [DIST_ID] --paths "/g/2026-05-30-jonathan-sunset.html"
    • TTL cleared within ~30 seconds
  3. Delete the duplicate afternoon page (2026-05-30-jonathan-afternoon.html):
    • Command: aws s3 rm s3://queenofsandiego.com/g/2026-05-30-jonathan-afternoon.html
    • Also invalidate this path in CloudFront to ensure any cached copies are purged
  4. Update DynamoDB crew-dispatch pointer:
    • Table: jada-crew-dispatch
    • Update key guest_code: S2FJLN, attribute page_slug: "2026-05-30-jonathan-afternoon.html" → "2026-05-30-jonathan-sunset.html"