Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

Resolving DNS Wildcard Shadowing: The adamcherrycomics Subdomain Outage

On a recent development session, the adamcherrycomics subdomain at dangerouscentaur.com went down for several hours. What appeared to be a CloudFront distribution failure turned out to be a subtle DNS configuration issue rooted in RFC 1034 wildcard matching semantics. This post details the investigation, root cause analysis, and the fix we implemented.

What Was Done

We diagnosed and resolved a DNS resolution failure for adamcherrycomics.dangerouscentaur.com that was blocking access to content hosted on CloudFront distribution E2Q4UU71SRNTMB. The root cause was DNS wildcard shadowing caused by an existing CNAME record. We fixed it by adding an explicit CNAME record for adamcherrycomics in Namecheap, which took precedence over the wildcard and restored resolution.

Technical Details: Investigation Process

Our investigation followed a systematic top-down approach through the DNS and CDN stack:

  • HTTP Response Check: Initial curl requests to http://adamcherrycomics.dangerouscentaur.com timed out, confirming the service was unreachable.
  • CloudFront Verification: We queried the CloudFront distribution directly using the Host header to bypass DNS. The distribution responded with HTTP 200, confirming the origin and CloudFront configuration were intact.
  • DNS Resolution Testing: We ran DNS lookups against both local resolvers and Namecheap's authoritative nameservers (pdns1 and pdns2). The subdomain failed to resolve at the authoritative level.
  • Namecheap DNS Audit: Fetching current DNS records for dangerouscentaur.com revealed a wildcard CNAME (*) pointing to CloudFront, but no explicit CNAME for the adamcherrycomics subdomain itself.

Root Cause: RFC 1034 Wildcard Shadowing

The DNS configuration had the following records:

*                    CNAME  d111111abcdef8.cloudfront.net
www.adamcherrycomics CNAME  d111111abcdef8.cloudfront.net

According to RFC 1034 § 4.3.3, when a DNS node exists at a level, wildcard entries at that level do not match queries for that node. The www.adamcherrycomics CNAME created a DNS node at adamcherrycomics.dangerouscentaur.com, which shadowed the wildcard entry from matching adamcherrycomics.dangerouscentaur.com. Since there was no explicit CNAME for adamcherrycomics itself, queries returned NXDOMAIN.

This is a common but subtle issue in DNS design: wildcards only match when no other record exists at that label level.

Infrastructure and DNS Changes

We accessed Namecheap's DNS management API to retrieve and modify records for the dangerouscentaur.com domain. The fix involved:

  1. Fetching Existing Records: Retrieved the full DNS record set for dangerouscentaur.com to understand the current state.
  2. Building the Merged Record Set: Constructed a new record set that included:
    • All existing records (wildcard, www.adamcherrycomics, etc.)
    • One new explicit CNAME: adamcherrycomics → d111111abcdef8.cloudfront.net
  3. Applying the Change: Used the Namecheap setHosts API call to write the merged record set atomically. The new record was assigned HostId 511341200.

Verification and Propagation

Post-deployment verification followed this sequence:

$ dig adamcherrycomics.dangerouscentaur.com @ns1.namecheap.com +short
d111111abcdef8.cloudfront.net.

$ dig adamcherrycomics.dangerouscentaur.com @ns2.namecheap.com +short
d111111abcdef8.cloudfront.net.

We polled both Namecheap authoritative nameservers (pdns1 and pdns2) until the new CNAME appeared. Initial checks on pdns1 showed the record in place immediately after the API call. We then polled local resolvers until they returned the CNAME, accounting for TTL-based caching (default Namecheap TTL is typically 1800 seconds, but we waited approximately 30 seconds for resolver propagation).

Finally, we confirmed HTTPS resolution and HTTP response:

$ curl -I https://adamcherrycomics.dangerouscentaur.com/
HTTP/2 200

Key Decisions and Architectural Patterns

Why Explicit CNAME Over Wildcard Adjustment: Rather than removing the wildcard and creating individual CNAMEs for every subdomain, we kept the wildcard in place for flexibility and added the explicit CNAME only where needed. This pattern provides:

  • Least-privilege DNS configuration: only override the wildcard when necessary
  • Future-proofing: new subdomains automatically use the wildcard without DNS changes
  • Reduced API calls: a single CNAME addition rather than wholesale reconfiguration

API-First DNS Management: All changes were made via Namecheap's setHosts API rather than manual UI interactions. This ensures:

  • Auditability: all changes are logged as API transactions
  • Atomicity: the entire record set is updated in one call, avoiding intermediate inconsistencies
  • Reproducibility: the same operation can be scripted and replayed

CloudFront as Origin Validation: By testing CloudFront directly with a Host header override (bypassing DNS), we quickly isolated the issue to DNS rather than the CDN layer. This is a valuable debugging technique when DNS resolution is suspect.

What's Next

This incident reveals a potential monitoring gap. Consider implementing:

  • Synthetic DNS monitoring that regularly resolves all critical subdomains against Namecheap authoritative nameservers
  • CloudFront access logging alerts for HTTP 4xx/5xx spikes from DNS resolution failures
  • DNS record schema validation in a DNS-as-code tool (e.g., Terraform, Pulumi) to catch wildcard shadowing before deployment

For teams managing multiple subdomains with wildcard fallbacks, documenting wildcard interaction patterns in your DNS runbook will help future engineers avoid similar issues.