Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

```html

Debugging Multi-Layer Email Delivery Failures: Gmail SMTP Relay vs. SES API Authentication

Last week, we discovered that emails sent through a Gmail "Send mail as" alias were failing silently, while the underlying SES infrastructure remained completely healthy. This post walks through the diagnostic process and the two distinct failure modes we encountered.

The Problem: Two Different Failures, One Error Message

Users reported that emails sent from jadasailing@gmail.com using the admin@queenofsandiego.com alias were bouncing with a generic "misconfigured or out of date" error. The initial assumption was that our AWS SES domain configuration was broken. It wasn't — we had two completely separate failures happening simultaneously:

  • Failure 1 (Primary): Gmail's SMTP relay credentials for the alias had expired
  • Failure 2 (Secondary): An earlier email that DID successfully route through SES was rejected downstream by Exchange Online at the recipient domain

Diagnostic Approach: Infrastructure Validation First

Before touching Gmail settings, we needed to verify that our email infrastructure itself was sound. We validated the full SES + DNS stack:

DNS and DKIM Verification

$ aws route53 list-resource-record-sets --hosted-zone-id Z... \
  --query "ResourceRecordSets[?Name=='*._domainkey.queenofsandiego.com.']"

All three DKIM CNAME records were present and pointing to SES-managed DNS. We confirmed:

  • SPF record: v=spf1 include:amazonses.com ~all correctly published
  • DMARC policy: v=DMARC1; p=quarantine; rua=mailto:admin@queenofsandiego.com configured
  • Custom MAIL FROM domain: bounce.queenofsandiego.com with matching SPF and DKIM
  • All three DKIM CNAME records resolving correctly to SES endpoints

SES Identity and Account Status

$ aws ses get-identity-verification-attributes \
  --identities admin@queenofsandiego.com \
  --region us-east-1

The verified identity was active with no issues. We then checked the suppression list:

$ aws sesv2 get-suppressed-destination \
  --email-address pat@steigerwald-dougherty.com \
  --region us-east-1

This revealed the secondary failure: the recipient's email address had been added to our suppression list with a complaint reason — indicating the email DID arrive at their mailbox but was marked as spam/complaint.

Root Cause Analysis: The SMTP Credential Layer

Once we confirmed SES infrastructure was healthy, we focused on the actual SMTP relay that Gmail uses. When you configure a "Send mail as" alias in Gmail with external SMTP, Gmail stores an encrypted username and password pair. For SES SMTP, those credentials are derived from AWS IAM temporary credentials.

Why SES SMTP Credentials Expire

AWS SES SMTP credentials are generated from IAM access keys and follow a specific v4 signature derivation:

# Derived SMTP password format (pseudocode):
# 1. Take IAM secret access key
# 2. Sign "SendRawEmail" string with HMAC-SHA256 using the secret
# 3. Base64 encode the result
# 4. Prepend "SmtpPassword:" + base64

# This means: if the IAM access key rotates, the SMTP password automatically becomes invalid

In our case, we suspected an IAM key rotation had occurred in the jada-ses-smtp user. We verified the current credentials could derive valid SMTP auth:

$ aws iam list-access-keys --user-name jada-ses-smtp

This showed which access keys were active. We then derived fresh SMTP credentials and tested them against the SES SMTP endpoint:

$ openssl s_client -connect email-smtp.us-east-1.amazonaws.com:587 -starttls smtp
# EHLO test
# AUTH LOGIN
# [base64(username)] 
# [base64(derived_password)]

The test succeeded, confirming the current IAM credentials were valid but the ones stored in Gmail were not.

The Secondary Issue: Suppression List and Exchange Online Rejection

While fixing the primary issue, we discovered that at least one prior email had successfully been sent through SES but was rejected by the recipient's mail server. The recipient address was in our us-east-1 suppression list marked with complaint reason code.

$ aws sesv2 list-suppressed-destinations \
  --reason COMPLAINT \
  --region us-east-1 \
  --query "SuppressedDestinationSummaries[?contains(EmailAddress, 'steigerwald')]"

This indicated the email arrived and was manually marked as spam/complaint by the recipient — a separate deliverability issue from the SMTP relay failure.

Infrastructure Decision: Credential Management Pattern

This incident exposed a gap in our credential lifecycle management. We made two architectural decisions:

  • IAM Key Rotation Policy: Implement automated key rotation for the jada-ses-smtp user with a 90-day cycle, with a documented runbook that includes updating both SES SMTP configurations in external tools (Gmail, mail clients, etc.)
  • Monitoring: Add CloudWatch alarms for SES SendRawEmail throttling and bounce/complaint rates, with SNS notifications to engineering@queenofsandiego.com

We stored the credential rotation runbook in the team wiki at /docs/email-infrastructure/ses-smtp-key-rotation.md with exact steps for regenerating credentials in the AWS console and updating them in Gmail's "Send mail as" settings.

What's Next: Preventing Recurrence

Going forward, we're implementing:

  • SES Configuration as Code: Moving identity verification and DKIM setup to CloudFormation templates in the infra/ repository
  • Credential Audit Script: Monthly validation that SMTP credentials in external systems actually work, using a test email to bounce-test@queenofsandiego.com
  • Suppression List Monitoring: Automated exports of the SES suppression list with alerts for new complaints or permanent bounces

The key lesson: email delivery failure can happen at multiple layers (SMTP auth, DNS, ISP reputation, recipient filters). Always validate infrastructure first before assuming configuration is broken.

```