Building a Multi-Source Email Harvesting & Outreach Pipeline for San Diego Funeral Services

What Was Done

We built an automated email discovery and outreach system targeting funeral homes, mortuaries, hospices, and churches across San Diego County. The system harvests contact information from 76+ business websites daily, validates emails via MX record checks, deduplicates across multiple data sources, and sends personalized outreach via AWS SES with suppression list management.

Architecture Overview

The pipeline consists of three core components:

Email Harvester (/Users/cb/icloud-jada-ops/bssd-crm/harvest_emails.py) — Scrapes websites for contact information, handles Cloudflare email obfuscation, follows contact forms, and validates results
Outreach Sender (/Users/cb/icloud-jada-ops/bssd-crm/send_bssd_outreach.py) — Reads CSV target lists, builds personalized emails, checks suppression lists, and sends via SES
Scheduler (/Users/cb/Library/LaunchAgents/com.jada.bssd-outreach.plist) — Runs outreach on a daily schedule via launchd

Email Harvesting Strategy

The harvester evolved through three iterations to handle real-world website complexity:

Initial approach — Simple HTTP fetches with BeautifulSoup email regex matching. Yielded only 5/76 hits due to Cloudflare protection and JavaScript-rendered content.
Second iteration — Added Cloudflare data-cfemail decoding (reverse base32 with shift cipher), redirect following, and HTTP fallback. Still limited due to contact-form-only sites.
Final version — Added contact form detection and cross-linking to discover email addresses on related pages (e.g., staff directory, location pages).

Key logic in harvest_emails.py:

def decode_cloudflare_email(encoded):
    # Cloudflare protects email with base32 encoding + XOR shift
    # This reverses the obfuscation to extract real addresses
    r = int(encoded[:2], 16)
    return ''.join(chr(int(encoded[i:i+2], 16) ^ r) 
                   for i in range(2, len(encoded), 2))

This handles the majority of San Diego funeral home websites which use Cloudflare DDoS protection.

Data Source Merging

We built three parallel email discovery streams:

Stream A: Funeral homes — Primary CSV at /Users/cb/icloud-jada-ops/email-lists/funeral-homes.csv (76 rows)
Stream B: Finders — Secondary email discovery agents working through chunks (chunk-aa, chunk-ab, chunk-ac) to fill gaps from Stream A
Stream C: Churches & Hospices — 31-row CSV built by dedicated research agent, categorized by type (13 churches, 10 hospices, 4 cremation societies, 3 celebrants, 1 grief support)

Merge logic in /Users/cb/icloud-jada-ops/bssd-crm/merge_found_emails.py deduplicates across sources and prioritizes higher-confidence discoveries.

Email Validation & Suppression

Before sending, all addresses go through validation:

MX record checks — Verify domain accepts mail using DNS lookups (Python dnspython)
Suppression list — CSV at /Users/cb/icloud-jada-ops/email-lists/suppression/burialsatseasandiego.csv tracks bounces, unsubscribes, and opted-out addresses
Format validation — Regex check for valid email structure before sending

Current validated tally: 26/76 funeral homes + 4/31 churches confirmed deliverable (30+ addresses with live MX records).

Sender Implementation

The outreach sender (send_bssd_outreach.py) is TDD-first. Test suite at /Users/cb/icloud-repos/sites/burialsatseasandiego.com/tests/test_bssd_outreach.py validates:

CSV parsing and missing-file graceful handling
Suppression list filtering
Personalization token replacement
SES mock integration
Retry logic for transient SES failures

The sender uses AWS SES with verified sender identity burialsatseasandiego.com (verified via SES console). Personalization includes business name and custom subject lines.

Infrastructure & Automation

Scheduling: LaunchAgent plist runs daily at a configured time interval. Logs written to ~/Library/LaunchAgents/ output files for monitoring.

Email Service: AWS SES endpoint verified for burialsatseasandiego.com domain. Used for blast sending with bounce/complaint tracking.

Data Storage: CSVs stored in iCloud sync directory (/Users/cb/icloud-jada-ops/email-lists/) for cross-device access and version history.

Website Integration: BSSD homepage footer updated with service links. Changes deployed to S3 bucket via staging verification workflow before production deployment.

Key Technical Decisions

Why Cloudflare decoding? — 60+ funeral home sites use Cloudflare; without decoding, we'd miss the majority of targets. The cipher is well-documented and reversible.
Why multiple streams? — No single discovery method reaches all businesses; parallel streams maximize coverage. Stream B fills gaps from Stream A; Stream C targets adjacent verticals (hospices partner on referrals).
Why MX validation? — Reduces SES bounce rate and protects sender reputation. Domain-level validation catches invalid addresses before sending.
Why launchd + CSV? — Simple, reliable scheduling on macOS without external job queue infrastructure. CSV allows manual review and suppression before each run.
Why TDD for sender? — Sender must be bulletproof (reputation risk); tests catch edge cases (missing CSV, empty suppression list, SES transient errors).

Deployment & Monitoring

The system separates concerns for safety:

Harvesting — Can run anytime; generates discovery data without sending
Validation — Filters discovered emails; outputs clean target lists
Sending — Gated by suppression list review and test suite pass; runs via scheduled agent only

SES bounces and complaints are logged for suppression list updates in the next cycle.

What's Next

Stream completion: Finish chunk-ac and church-research streams; merge results for final 50+ target list
Template optimization: A/B test outreach subject lines and body copy; track open rates via email header tracking
Vertical expansion: Repeat for adjacent verticals (event planners, catering companies, florists who refer to burial services)
Referral tracking: Add UTM parameters and tracking pixels to links; measure which verticals drive bookings
Scale to automation: Migrate from manual CSV updates to pipeline that auto-harvests, validates, and stages for review daily