Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

Building a Two-Way SMS Digest Pipeline: Integrating Twilio, SES, and Local Voice Agent Infrastructure

Over the past development session, we built out a complete SMS digestion and notification system that bridges Twilio's SMS capabilities with our existing voice agent infrastructure and AWS SES for email delivery. This post covers the technical architecture, infrastructure decisions, and command patterns we used to extract, process, and distribute SMS conversation digests.

What Was Done

The core objective was simple: read incoming and outgoing SMS messages from a Twilio-managed phone line (+16199867344, the JADA business line), extract conversation threads by sender, and deliver formatted digests via email to stakeholders. The implementation required three main components:

  • SMS data source discovery and credential management
  • Conversation threading and digest compilation
  • Email delivery via AWS SES

This pattern lets us handle high-volume SMS without requiring manual log review, while keeping the infrastructure lightweight and cost-effective.

Technical Details: Architecture and Implementation

SMS Infrastructure Discovery

The first challenge was locating where SMS data actually lives. The voice agent infrastructure in /var/local/voice_agent/ contains Twilio integration points, but the actual credentials weren't stored in the primary repos.env file. Instead, we found them referenced in:

  • /var/local/voice_agent/phone_agent.py — main Twilio client initialization
  • /var/local/voice_agent/tools/sms_handler.py — SMS send/receive logic
  • .secrets/repos.env — where TWILIO_ACCOUNT_SID and TWILIO_AUTH_TOKEN are actually stored

A key decision here: we opted to read from the local JADA SMS export file rather than making live Twilio API calls on every digest request. This reduces API costs and latency, and gives us a local backup of conversation history. The export file is located at /var/local/voice_agent/data/sms_export.json and contains timestamped message objects with sender phone numbers, message bodies, and delivery status.

Conversation Threading Logic

Raw SMS exports are just flat lists of messages. To create meaningful digests, we needed to:

  1. Group by sender phone number — extract unique phone numbers from the export
  2. Sort chronologically — order messages by timestamp within each thread
  3. Filter by date range — for this digest, we pulled messages from April 25–29 to capture recent activity
  4. Identify context — map phone numbers to human-readable contacts (e.g., "Sergio," "Carole," "Today's hostess")

The command flow looked like this:

grep -r "TWILIO" .secrets/repos.env
find /var/local/voice_agent -name "*.py" | xargs grep -l "sms_handler\|SMS"
cat /var/local/voice_agent/data/sms_export.json | jq '.messages | group_by(.from) | sort_by(.[0].timestamp)'

This gave us conversation threads sorted by most recent activity. We then extracted individual threads by phone number and created human-friendly summaries.

Email Digest Generation and Delivery

Once we had threaded conversations, we compiled them into structured HTML emails and sent them via AWS SES. The decision to use SES instead of a third-party service came down to:

  • Cost — SES is included in the AWS free tier for up to 62,000 emails/month
  • Integration — no new credentials or API keys to manage; we use existing AWS IAM roles
  • Deliverability — SES has strong bounce handling and complaint tracking built in

The digest email command invoked the SES API with:

aws ses send-email \
  --from "noreply@queenofsandiego.com" \
  --to "c.b.ladd@gmail.com" \
  --subject "SMS Digest: Apr 25-29" \
  --html file:///tmp/sms_digest.html

Email templates are stored in /var/local/voice_agent/templates/sms_digest.html and use a simple CSS grid layout to display conversation threads side-by-side, making it easy to scan multiple contacts' messages at once.

Infrastructure and Key Decisions

Data Storage

Rather than querying Twilio's API repeatedly, we maintain a local export:

  • Location: /var/local/voice_agent/data/sms_export.json
  • Refresh strategy: Nightly cron job (see /etc/cron.d/voice_agent_maintenance) that pulls the last 7 days of messages from Twilio and merges them into the local export
  • Retention: 90 days of messages on disk; older records archived to S3

Credential Management

Twilio credentials live in .secrets/repos.env, which is:

  • Never committed to version control
  • Loaded into the voice agent process via systemd EnvironmentFile directive
  • Rotated quarterly via the secret management pipeline in the Lightsail instance

The phone_agent.py module reads these at startup and initializes a persistent Twilio REST client:

twilio_client = Client(
    os.environ['TWILIO_ACCOUNT_SID'],
    os.environ['TWILIO_AUTH_TOKEN']
)

Active Handoffs Tracking

We also read from /var/local/voice_agent/data/active_handoffs.json to understand the current state of ongoing conversations. This file tracks which SMS threads have been escalated to humans, which are resolved, and which are pending. The digest process flags any unresolved threads to ensure nothing slips through.

Command Reference: Pulling and Processing SMS

Here are the core commands used during this session:

# Find Twilio-related files
find /var/local/voice_agent -name "*twilio*" -o -name "*sms*"

# List available tools
ls -la /var/local/voice_agent/tools/

# Check for Twilio credentials
grep -i "TWILIO" .secrets/repos.env

# Extract recent messages from a date range
jq '.messages | map(select(.timestamp >= "2025-04-25" and .timestamp <= "2025-04-29"))' \
  /var/local/voice_agent/data/sms_export.json

# Get conversations sorted by most recent activity
jq '.messages | group_by(.from) | sort_by(max_by(.timestamp).timestamp) | reverse' \
  /var/local/voice_agent/data/sms_export.json

# Extract a single contact's full thread
jq '.messages | map(select(.from == "+16197085371")) | sort_by(.timestamp)' \
  /var/local/voice_agent/data/sms_export.json

# Send digest via SES (wrapper script in /usr/local/bin/send-