Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

```html

Debugging Multi-Layer Email Delivery Failures: Gmail SMTP Relay vs. Exchange Online Rejection

During a routine email campaign troubleshooting session, we discovered that three outbound emails were failing to deliver. The investigation revealed not one, but two distinct failure modes operating simultaneously—one at the Gmail layer and one at the recipient's mail server. This post walks through the diagnostic methodology and the infrastructure decisions that led us to the root causes.

The Problem Statement

Three emails sent from the JADA system were bouncing back with vague error messages. Initial triage suggested a single point of failure, but deeper inspection revealed a more complex picture involving:

  • Gmail's "Send mail as" alias functionality and its external SMTP relay integration
  • AWS SES SMTP credential lifecycle management
  • SPF/DKIM/DMARC validation at recipient domains
  • Exchange Online bounce handling and suppression list behavior

Failure Mode #1: Stale Gmail SMTP Relay Credentials

The first failure occurred at the Gmail layer. Emails were being composed and sent through jadasailing@gmail.com using Gmail's native "Send mail as" feature to impersonate admin@queenofsandiego.com. Gmail's configuration for this alias included external SMTP relay credentials pointing to AWS SES:

Server: email-smtp.us-east-1.amazonaws.com
Port: 587
Protocol: TLS

The bounce message indicated: "Your message could not be sent because the mail server you specified is misconfigured or out of date." This is Gmail's way of saying it failed to authenticate against the configured SMTP endpoint.

Root Cause Analysis

AWS SES SMTP credentials are derived from IAM user access keys using a region-specific algorithm. When IAM credentials are rotated (either manually or through automated processes), the corresponding SES SMTP credentials become invalid. The Gmail alias was still configured with the old derived credentials, causing authentication to fail at the TLS handshake phase.

The key insight: SES SMTP credentials are not standard username/password pairs stored in AWS. They are computed deterministically from an IAM access key ID and secret access key using an HMAC-SHA256 algorithm that includes the target region. If the underlying IAM credentials change, the derived SMTP password changes.

Verification Steps Performed

We tested the current IAM credentials against the SES SMTP endpoint in us-east-1:

#!/bin/bash
# Derive SES SMTP password from current IAM credentials
# Using AWS SigV4 HMAC-SHA256 algorithm for the target region
# Then attempt SMTP AUTH against email-smtp.us-east-1.amazonaws.com:587

The test confirmed that the current credentials worked, but they differed from what was configured in Gmail. The solution requires updating the Gmail alias settings with freshly derived credentials.

Failure Mode #2: Exchange Online Rejection at Recipient Domain

Parallel to the Gmail relay issue, we discovered that emails that did successfully send through SES were being rejected by Microsoft Exchange Online at the recipient domain steigerwald-dougherty.com.

Email Infrastructure Audit

We performed a comprehensive DNS validation of queenofsandiego.com:

  • SPF Record: Properly configured with SES netblock authorization
  • DKIM Records: All three DKIM CNAME records present and resolvable
  • DMARC Record: Configured with appropriate alignment and reporting
  • Custom MAIL FROM: Configured in SES with verified bounce handling domain

DNS validation confirmed that our sending domain passed all standard authentication checks:

$ dig TXT queenofsandiego.com
$ dig CNAME token1._domainkey.queenofsandiego.com
$ dig CNAME token2._domainkey.queenofsandiego.com
$ dig CNAME token3._domainkey.queenofsandiego.com
$ dig TXT _dmarc.queenofsandiego.com

All records were in place and properly formatted.

SES Suppression List Investigation

We examined the SES suppression list in both us-east-1 and us-west-2 regions. Certain email addresses showed up in the complaint category—meaning they had previously complained to their mailbox provider about emails from our domain. Exchange Online systems typically respect these historical complaint signals.

Specific findings:

  • Checked SES account status and complaint list in us-east-1
  • Verified suppression list size and complaint reason breakdown
  • Cross-referenced bounce dates and complaint timestamps
  • Confirmed that specific test addresses had complaint records dating back several weeks

Exchange Online's aggressive reputation filtering likely flagged our messages based on DKIM/SPF alignment combined with these historical complaint signals. Even though our infrastructure was clean, the sending reputation score was low enough to trigger rejection.

Infrastructure Decisions

Why We Use SES Over Gmail's Built-In SMTP

The architecture currently bifurcates email sending across two systems:

  • Transactional/system emails: AWS SES API (used by Lambda functions in jada-blast-scheduler and jada-blast-sender)
  • User-composed emails: Gmail SMTP relay (via jadasailing@gmail.com)

This separation exists because Gmail's SMTP is more reliable for low-volume, user-initiated sends, while SES scales better for bulk campaign delivery. However, this design creates a credential management burden.

Credential Rotation Policy

We need to implement a scheduled process that:

  1. Regenerates SES SMTP credentials in AWS every 90 days
  2. Automatically updates all configured clients (Gmail aliases, application config files, environment variables)
  3. Tests connectivity before decommissioning old credentials

This should be orchestrated through a Lambda function or Step Functions state machine rather than manual updates.

Remediation Steps

For Failure #1 (Gmail SMTP relay):

  1. Generate new SES SMTP credentials from AWS SES console (SMTP Settings → Create SMTP Credentials)
  2. Update Gmail alias configuration: Settings → Accounts and Import → Send mail as → Edit info
  3. Test the connection through Gmail's verification flow

For Failure #2 (Exchange Online rejection):

  1. Remove addresses from SES suppression list if they were marked as complaints incorrectly
  2. Consider implementing warm-up sends to rebuild sending reputation with Exchange Online
  3. Monitor bounce/complaint metrics through CloudWatch dashboards connected to SES SNS topics

What's Next

We're planning to consolidate email sending onto a single path (SES API) to eliminate the dual-credential management problem. This would involve modifying the Gmail integration to use SES directly rather than Gmail's SMTP relay, which requires changes to the JADA daemon script and potentially the web UI for composing emails.

We're also implementing automated credential rotation and