Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

```html

Debugging Gmail API Scope Issues: OAuth Token Refresh and Local Consent Flow

What Was Done

During a recent development session managing Google API integrations for a Python-based business operations tool, we discovered that a token file lacked the required gmail.compose scope despite previous authentication attempts. The token had been granted gmail.modify but not the compose scope needed to draft messages programmatically. This post documents the diagnostic approach, the re-authentication workflow we built, and the token scope management patterns we implemented.

The Problem: Missing Scopes in Persisted Tokens

The application, reauth_jada_all.py, manages OAuth tokens for Google API access across multiple helper modules (notably jada_google.py). When attempting to compose and send Gmail drafts, API calls returned auth failures despite token refresh succeeding. Investigation revealed that the token file at the unified storage path contained only gmail.modify; the gmail.compose scope was never granted during the original consent flow.

Root cause: The initial OAuth consent request had been configured with incomplete scopes. Token refresh would only return what the persisted consent grant allowed—no amount of re-running refresh() would add new scopes to an existing grant.

Diagnostic Steps and Token Inspection

We took the following approach to understand the token state:

  • Token file enumeration: Located all token files across legacy and unified paths by reading the scope-selection logic in jada_google.py. The tool maintains both legacy token paths (grant-specific) and a unified token location.
  • Scope inspection: Decoded the token files and inspected their scopes claim. Both legacy gmail-full token and the unified token showed incomplete scope coverage.
  • Error classification: Transient SSL errors during refresh were retried with 15-second timeouts; persistent 400 errors on legacy token refresh indicated the token was stale or from a revoked consent grant.
  • Scope coverage mapping: Documented which scopes were required vs. present:
    • Present: gmail.modify, calendar
    • Required for drafting: gmail.compose
    • Required for filtering: gmail.settings.basic

Re-Authentication Workflow

Rather than interactively manage consent screens, we built an automated re-auth flow in reauth_jada_all.py. The workflow:

# Pseudocode of the re-auth flow
1. Start a local HTTP server on localhost (port selected for availability)
2. Construct OAuth authorization URL with required scopes:
   scopes = [
     'https://www.googleapis.com/auth/gmail.compose',
     'https://www.googleapis.com/auth/gmail.modify',
     'https://www.googleapis.com/auth/gmail.settings.basic',
     'https://www.googleapis.com/auth/calendar.readonly'
   ]
3. Open consent URL in browser (Opera in this session)
4. User grants consent in browser; Google redirects to localhost callback
5. Exchange authorization code for refresh token
6. Persist token to unified path
7. Poll token file for presence of gmail.compose scope (2-minute intervals, max 4 hours)

Why local callback server: Avoids out-of-band code exchange, keeps consent flow in the user's browser, and allows immediate detection of successful auth (no polling of external systems).

Why scope polling: Google's token refresh occasionally takes time to propagate the full scope grant. Rather than assume immediate availability, we poll the token file and verify gmail.compose appears before proceeding with operations that depend on it. The 2-minute polling interval balances responsiveness against API quota.

Token File Organization

The application manages tokens across two organizational schemes:

  • Legacy paths: Grant-specific tokens (e.g., ~/.config/gauth/gmail-full, ~/.config/gauth/calendar) from earlier auth sessions.
  • Unified path: A single consolidated token location where all scopes from a fresh auth are written. This simplifies scope discovery and rotation.
  • Selection logic in jada_google.py: The helper module checks for unified token first; falls back to legacy paths if unified is absent. On refresh failure, it attempts refresh on all candidate tokens and updates each (or only the unified, depending on deployment state).

Decoupling scope management from token organization reduces coupling and makes incremental scope addition cleaner: add the scope to the auth request, re-run consent, and the unified token gains it automatically on refresh.

Error Handling and Retry Logic

During development, we encountered two failure modes:

  • Transient SSL/network errors: Token refresh occasionally failed with SSL certificate validation or connection timeouts. These were handled with exponential backoff (15-second retry window) and typically resolved on retry.
  • Persistent 400 errors: Legacy tokens from revoked consent grants returned 400 Bad Request on refresh. These required re-authentication; refresh() alone could not recover them.

The re-auth script distinguishes these by attempting multiple refresh strategies before escalating to user consent request.

Key Architectural Decisions

Why not request all scopes upfront? Consent screens with broad scope lists trigger higher scrutiny in OAuth consent review and complicate testing. We requested scopes incrementally and documented each addition in the auth request, making the purpose of each scope explicit to auditors and developers.

Why async polling rather than blocking wait? The consent flow is synchronous (open browser, user clicks, callback fires). But scope propagation to the token is eventual. Blocking on refresh availability would hang the process; polling with a timeout allows the operator to check back later or handle partial failure gracefully.

Why store secrets in files rather than environment? Refresh tokens are long-lived, operator-owned credentials. File-based storage with restrictive permissions (mode 0600) is more auditable and rotatable than environment variables, which can leak into logs or process inspection tools.

What's Next

Once the scope grant lands on the unified token, the application will:

  • Compose and persist email drafts using the gmail.compose scope
  • Apply filters via gmail.settings.basic to rescue messages from spam folders
  • Reconcile calendar events across multiple data sources

Future refinements will include:

  • Scope versioning: Document required scopes in a manifest file so re-auth can be triggered automatically when new scopes are needed.
  • Token metadata: Store scope grant timestamps and user consent details alongside the token to make rotation and audit trails automatic.
  • Multi-account support: Extend the token selection logic to handle multiple Google accounts and their respective token lifetimes.

This pattern—local callback servers, scope polling, and layered error handling—generalizes to any OAuth 2.0 integration where incremental scope grants and eventual token propagation are in play.

```