Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

Debugging DNS Wildcard Shadowing: How a Single CNAME Record Broke adamcherrycomics Subdomain Resolution

The Problem

The subdomain adamcherrycomics.dangerouscentaur.com stopped responding to requests, returning no DNS resolution despite the underlying CloudFront distribution being healthy and serving content correctly. The issue persisted for hours, and initial diagnostics showed the CloudFront distribution (ID: E2Q4UU71SRNTMB) was returning HTTP 200 responses with valid content when accessed directly with explicit Host headers.

This created a puzzling scenario: the infrastructure was working, but the DNS layer wasn't. The root cause? A subtle DNS record interaction governed by RFC 1034 § 4.3.3 that most developers encounter only once.

Technical Investigation: Peeling Back the Layers

The debugging process followed a methodical top-down approach:

  • Layer 1 – HTTP/HTTPS Response: Verified CloudFront distribution was responding with HTTP 200 and correct content when accessed with curl -H "Host: adamcherrycomics.dangerouscentaur.com"
  • Layer 2 – DNS Resolution: Confirmed DNS lookup for adamcherrycomics.dangerouscentaur.com was failing across multiple resolvers (local resolver, Namecheap authoritative nameservers, and major public DNS services)
  • Layer 3 – Authoritative Nameserver Query: Queried Namecheap's authoritative nameservers directly to examine the live DNS state at the source

The investigation revealed that a wildcard CNAME record existed at the domain level (* CNAME [cloudfront-distribution].cloudfront.net), but DNS queries for adamcherrycomics were not matching this wildcard. Instead, they returned NXDOMAIN (non-existent domain).

Root Cause: DNS Wildcard Shadowing

The issue stemmed from a DNS node shadowing problem. The Namecheap DNS configuration contained:

  • A wildcard CNAME: * CNAME [cloudfront-dist].cloudfront.net (intended to catch all subdomains)
  • An explicit record for www.adamcherrycomics (a CNAME pointing to CloudFront)

When the DNS resolver queried for adamcherrycomics.dangerouscentaur.com, it found that a DNS node existed at the adamcherrycomics label level (created by the www.adamcherrycomics record). According to RFC 1034 section 4.3.3, once a non-wildcard node exists at a given level, wildcard matching does not apply to that level—even if the specific label being queried doesn't have an explicit record.

In other words: the presence of www.adamcherrycomics created a DNS node that prevented adamcherrycomics from matching the wildcard.

The Fix: Adding an Explicit CNAME Record

The solution was to add an explicit CNAME record for adamcherrycomics itself, pointing to the CloudFront distribution. This required:

  1. Fetching current DNS records from Namecheap using their API, capturing all existing records for dangerouscentaur.com
  2. Building the updated record set with the new adamcherrycomics CNAME merged in alongside all existing records
  3. Pushing the updated configuration back to Namecheap using setHosts API call (HostId: 511341200)
  4. Waiting for propagation across Namecheap's authoritative nameserver pool

Verification and Propagation Strategy

After adding the explicit CNAME, verification proceeded in phases:

Phase 1 – Authoritative Nameserver Confirmation:

dig adamcherrycomics.dangerouscentaur.com @ns1.namecheap.com +short
dig adamcherrycomics.dangerouscentaur.com @ns2.namecheap.com +short

This directly queried Namecheap's authoritative nameservers to confirm the record existed before waiting for global propagation.

Phase 2 – Local Resolver Polling: Repeated DNS lookups at 10-second intervals via the local system resolver, detecting when the change propagated through the resolver cache.

Phase 3 – Public Resolver Validation: Checked DNS propagation across Google Public DNS (8.8.8.8), Cloudflare DNS (1.1.1.1), and Quad9 to ensure the change had propagated globally.

Phase 4 – HTTPS Response Polling: Monitored HTTPS responses from the subdomain, waiting for it to respond before declaring the fix complete.

Phase 5 – Content Verification: Final checks using verbose curl to inspect the full HTTP response, headers, and page content to confirm the site was serving the expected Adam Cherry Comics content.

Key Infrastructure Details

  • CloudFront Distribution: E2Q4UU71SRNTMB (verified healthy with direct Host header testing)
  • DNS Provider: Namecheap authoritative nameservers (ns1.namecheap.com, ns2.namecheap.com, etc.)
  • Record Type: CNAME (not A records, because we're pointing to a CloudFront distribution)
  • Parent Domain: dangerouscentaur.com
  • Target Subdomain: adamcherrycomics.dangerouscentaur.com

Why This Matters: RFC 1034 Implications

This issue is a classic example of DNS specification nuance that trips up even experienced engineers. Most developers assume wildcard records work everywhere beneath a domain, but RFC 1034 creates a subtle rule: wildcard expansion only occurs when there is no explicit DNS node at that label level, regardless of whether the specific subdomain being queried has its own record.

In production systems, this means:

  • If you have a wildcard CNAME at *, and you add any subdomain record (like www), you must explicitly create records for any other subdomains you want under that same parent label
  • Wildcard records are not a complete catch-all; they have precise scope limitations defined by the DNS tree structure
  • Testing specific subdomains during development is critical—wildcard behavior differs depending on what other records exist

What's Next

Going forward, the DNS configuration for dangerouscentaur.com now explicitly includes:

  • * CNAME [cloudfront-dist].cloudfront.net (wildcard fallback)
  • www.adamcherrycomics CNAME [cloudfront-dist].cloudfront.net (explicit record)
  • adamcherrycomics