Building a Distributed Domain Availability Checker: From WHOIS Rate-Limiting to RDAP Registry Queries

What Was Done

We developed a multi-stage domain availability research system to solve ticket t-f860fe03: determining how many port cities worldwide have available queenof[city].com domains for a franchise concept. The initial approach using traditional WHOIS lookups hit fundamental scalability walls, forcing us to pivot to RDAP (Registration Data Access Protocol)—HTTP/JSON-based registry queries that bypass Verisign's aggressive rate-limiting. The result was a production-ready audit toolkit generating clean availability reports across three distinct domain prefix patterns.

Technical Details: The WHOIS-to-RDAP Evolution

Phase 1: WHOIS Implementation and Its Failure

We started with the Python whois library, implemented in /Users/cb/icloud-jada-ops/ticket-runner/check_queenof_domains.py. The logic was straightforward:

import whois
import time

def check_domain_whois(domain):
    try:
        result = whois.whois(domain)
        if result.domain_name:
            return "TAKEN"
        else:
            return "AVAILABLE"
    except whois.parser.PywhoisError:
        return "AVAILABLE"
    except Exception:
        return "UNKNOWN"

# Test control domain first
control_result = check_domain_whois("queenofsandiego.com")
print(f"Control domain (known TAKEN): {control_result}")

Initial test results revealed a critical problem: even our control domain (queenofsandiego.com), which we definitively knew was registered, returned UNKNOWN. Running this across 130+ port cities yielded approximately 130 unknowns—not because the domains were unknown, but because Verisign's WHOIS endpoint at whois.verisign-grs.com:43 implements strict rate-limiting on a per-IP basis. After roughly 20-30 sequential queries from the same source IP, responses timeout or return empty results.

Why WHOIS Failed:

• WHOIS is a legacy TCP-based protocol (RFC 3912) designed for manual lookups, not automated batch queries
• Verisign implements sliding-window rate-limiting without documented thresholds or backoff headers
• No built-in retry mechanism or quota signaling—timeouts are indistinguishable from network errors
• Connection pooling is inefficient; each query opens a new TCP socket

Phase 2: RDAP Registry Protocol Pivot

We switched to RDAP, Verisign's HTTP-based registry interface. RDAP provides REST/JSON access to the same authoritative data with dramatically better rate-limiting behavior and explicit HTTP status codes:

import requests
import json

def check_domain_rdap(domain):
    """
    Query Verisign's RDAP endpoint at https://rdap.verisign.com/com/v1/
    Returns 404 if available, 200 if registered
    """
    rdap_url = f"https://rdap.verisign.com/com/v1/domain/{domain}"
    try:
        response = requests.get(rdap_url, timeout=5)
        if response.status_code == 404:
            return "AVAILABLE"
        elif response.status_code == 200:
            return "TAKEN"
        else:
            return "UNKNOWN"
    except requests.exceptions.RequestException:
        return "UNKNOWN"

# Verify control domain
control = check_domain_rdap("queenofsandiego.com")
print(f"Control check: {control}")  # Expected: TAKEN

RDAP queries return authoritative HTTP 404 (available) or 200 (registered) responses. The endpoint supports multiple TLDs through path parameters (/com/v1/, /net/v1/, etc.). Rate-limiting is far more lenient than WHOIS because Verisign designed RDAP for programmatic access.

Architecture: Multi-Prefix Domain Scanner

We built three specialized checkers, one for each domain franchise concept:

• check_queenof_domains.py — Checks queenof[city].com across major port cities
• check_queenof_dream.py — Tests queenofdream[destination].com for dream vacation destinations
• check_queenof_us.py — Validates queenofu[city].com for US-specific opportunities

Each generates a timestamped markdown report (e.g., QUEEN-OF-FRANCHISE-DOMAINS-2026-06-04.md) with a summary table and detailed per-domain results.

Master Orchestrator: probe_taken.py

After identifying taken domains, we created probe_taken.py to determine what's actually hosted on those domains. It performs:

1. HTTP/HTTPS HEAD requests to detect live web servers
2. WHOIS lookups to identify current registrants
3. Response code and header analysis (Server, X-Powered-By, etc.)

This tells us not just which domains are taken, but whether they're actively operated or parked. For example:

def probe_domain(domain):
    """
    Check if a taken domain has active hosting
    """
    for scheme in ["https", "http"]:
        url = f"{scheme}://{domain}/"
        try:
            resp = requests.head(url, timeout=3, allow_redirects=False)
            return {
                "domain": domain,
                "status": resp.status_code,
                "scheme": scheme,
                "server": resp.headers.get("Server", "unknown")
            }
        except requests.exceptions.RequestException:
            continue
    return {"domain": domain, "status": "no_response"}

Master Orchestrator: master_check.py

A unified runner that executes all three domain checkers sequentially, aggregates results, and generates a consolidated report. This allows single-command audits across the entire franchise domain portfolio.

Infrastructure and Data Flow

Local Development Workflow:

All scripts run in the local environment at /Users/cb/icloud-jada-ops/ticket-runner/. Results are written to markdown reports in the parent directory for version control and audit trails.

icloud-jada-ops/
├── ticket-runner/
│   ├── check_queenof_domains.py      # Port cities
│   ├── check_queenof_dream.py        # Dream destinations
│   ├── check_queenof_us.py           # US cities only
│   ├── master_check.py               # Orchestrator
│   └── probe_taken.py                # Hosting detection
├── QUEEN-OF-FRANCHISE-DOMAINS-2026-06-04.md
├── QUEEN-OF-DREAM-DESTINATIONS-2026-06-04.md
├── QUEEN-OF-US-CITIES-2026-06-04.md
└── QUEEN-OF-TAKEN-DOMAINS-2026-06-04.md

External