Diagnosing Multi-Layer Email Delivery Failures: Gmail Relay vs. SES Infrastructure

```html

During a recent incident investigation, we encountered a seemingly simple problem: three emails weren't delivering. What emerged was a textbook example of how email infrastructure failures can mask themselves at different layers of the stack. This post walks through the systematic diagnosis and the architectural decisions that led to the root causes.

The Symptom

Three outbound emails showed delivery failures with the error: misconfigured or out of date. The messages originated from jadasailing@gmail.com but were sent using Gmail's "Send mail as" feature to relay through admin@queenofsandiego.com. The bounces returned to the sending account, not to the admin address—a critical detail that immediately pointed to a Gmail-side failure rather than a downstream issue.

Diagnostic Approach: Layered Infrastructure Analysis

Email delivery involves multiple independent systems, and failures can occur at any layer. We systematically tested each:

AWS SES Infrastructure — Verified domain setup, DKIM/SPF/DMARC records, suppression lists, and account status
Gmail Configuration — Checked the "Send mail as" alias and its SMTP relay settings
Recipient Domain Health — Validated MX records and mail server responsiveness
Credentials and Authentication — Tested SES SMTP credentials derived from current IAM access keys

Root Cause #1: Stale Gmail SMTP Relay Credentials

Gmail's "Send mail as" feature allows sending from an alias by relying on an external SMTP server. In this setup, someone configured the alias to relay through AWS SES using SMTP credentials. The error message and bounce pattern revealed that these credentials had become invalid—either through key rotation or expiration.

How we confirmed this:

The bounce came back to jadasailing@gmail.com, not to admin@queenofsandiego.com. Gmail only bounces to the true sending account when it can't authenticate to the relay server.
We derived fresh SES SMTP credentials from the current IAM access key using the AWS SES SMTP v4 signature algorithm and successfully tested them against the endpoint email-smtp.us-east-1.amazonaws.com:587.
The old credentials stored in Gmail's settings no longer matched the current IAM state.

Why this happened: SES SMTP credentials are derived deterministically from IAM credentials. Any IAM key rotation automatically invalidates previously derived SES SMTP passwords. If the Gmail alias was configured months ago and the underlying IAM key was later rotated (a standard security practice), the alias would silently fail without any notification to the user.

Root Cause #2: Exchange Online Rejection at Recipient Domain

Beyond the Gmail relay failure, we discovered a second, independent delivery failure. Emails sent directly via the SES API (bypassing Gmail) were being rejected by Exchange Online at the recipient domain steigerwald-dougherty.com.

Our analysis confirmed the queenofsandiego.com sending infrastructure was correct:

All three DKIM CNAME records were properly configured in Route53 and resolving
SPF record was set: v=spf1 include:amazonses.com ~all
DMARC policy was configured: v=DMARC1; p=quarantine;
Custom MAIL FROM domain was set to mail.queenofsandiego.com
MX records for queenofsandiego.com were properly configured
No recipients were on the SES suppression list (no bounces, complaints, or permanent failures)
SES account status was healthy with no sending limits or restrictions

The rejection was coming from the recipient's mail server, not from our infrastructure. This required a separate investigation into Exchange Online's acceptance criteria—likely DKIM signature validation, sender reputation, or organization-level security policies.

Technical Details: SES SMTP Credential Derivation

This incident highlighted an often-overlooked detail about SES SMTP credentials. Unlike traditional SMTP servers, SES doesn't store persistent credentials. Instead, credentials are derived on-the-fly from IAM access keys using a deterministic algorithm:

SMTP Username = IAM Access Key ID
SMTP Password = HMAC-SHA256(
    key = "Message",
    data = IAM Secret Access Key
) encoded in Base64

The "Message" value is a literal string used in the HMAC calculation for SES SMTP v4 signatures. Any change to the underlying IAM key invalidates the derived password immediately—there's no grace period or dual-credential support.

Infrastructure Verification Checklist

Here's what we verified programmatically:

SES Domain Identity: admin@queenofsandiego.com — verified and healthy
SES DKIM: All three CNAME records present and resolving in Route53
SES SPF: Correct include directive pointing to amazonses.com
SES DMARC: Policy configured with authentication requirements
SES Custom MAIL FROM: Configured to mail.queenofsandiego.com with bounce handling
SES Account Health: No sending restrictions, daily limit intact, suppression list verified
SES Configuration Sets: Present but not required for basic sending
AWS Region: Primary SES endpoint in us-east-1, with secondary suppression list in us-west-2 for campaign targeting

Key Architectural Decisions

Decision 1: Using AWS SES over Gmail's native SMTP

The architecture relies on SES for bulk and transactional email rather than Gmail's SMTP server. This provides:

Higher sending limits and better scalability
Integration with Lambda functions for triggered sends (via jada-blast-sender)
Suppression list management across regions
Delivery metrics and bounce handling

Gmail's "Send mail as" should only be used for manual, ad-hoc sending—not for production automation.

Decision 2: Derived Credentials Over Persistent SMTP Keys

By using SES's derived credential model, we tie email authentication directly to IAM access key lifecycle. This ensures:

No separate secrets to manage for email sending
Automatic invalidation when IAM keys are rotated
Audit trail through IAM CloudTrail logs

The tradeoff is that any IAM key rotation breaks SMTP relay configurations that were set up with the old credentials.

Remediation Steps

To fix the immediate failures:

Update the Gmail "Send mail as" SMTP configuration with freshly derived SES credentials from the current IAM access key