Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

```html

Diagnosing and Resolving OAuth Token Failures in Distributed Agent Infrastructure

During a routine health check of the jada-agent orchestrator daemon running on AWS Lightsail (34.239.233.28), we discovered a critical OAuth token expiration affecting the port sheet synchronization pipeline. This post documents the diagnosis methodology, root cause analysis, and the architectural decisions that led to this failure pattern.

What We Found: Token Degradation in Long-Running Services

The jada-agent daemon itself was running healthily—service uptime of 3 days, CPU utilization at 0.65% average with no spikes, and memory consumption well within acceptable bounds (144MB of 914MB available). However, the port_sheet_sync.py script, which runs on a 30-minute interval, had been failing consistently since at least May 13 afternoon UTC with a repeating error:

[port-sheet] token error: HTTP Error 400: Bad Request

This error pattern is the classic signature of an expired or revoked Google OAuth 2.0 refresh token. The sync pipeline was completely blocked, meaning port sheet data was not being synchronized to the backing Google Sheet for approximately 12+ hours.

Technical Diagnosis: How We Isolated the Root Cause

Our investigation followed a structured approach:

  • Service State Verification: Confirmed jada-agent.service was active and running without restart loops or crash patterns, ruling out general daemon failure.
  • Resource Metrics Analysis: Pulled CPU, memory, disk, and network metrics from the Lightsail API for the preceding 2 hours. No resource exhaustion or anomalies were detected.
  • Log Inspection: Reviewed daemon logs and session history, identifying that three agent sessions had run today—two hitting the 30-turn Claude limit (expected for complex tasks) and one completing successfully.
  • OAuth Token Validation: Cross-referenced the port_sheet_sync.py token stored in the environment against Google's OAuth token validation endpoints. The token was indeed expired.

The root cause was straightforward: the Google OAuth token for the service account used by port_sheet_sync.py had reached its expiration time without being refreshed. Unlike user-facing OAuth flows where refresh tokens are automatically cycled, long-running daemon processes require either periodic re-authentication or indefinite-lifetime service account keys.

Architecture Context: Why This Matters

The jada-agent system uses a distributed task orchestration model where:

  • The Lightsail instance runs jada-agent.service, a daemon process that polls a task queue (the progress dashboard) for work.
  • Subsidiary scripts like port_sheet_sync.py run on fixed intervals (30 minutes) to maintain synchronized state with external systems.
  • Each script maintains its own authentication context—in this case, OAuth 2.0 tokens stored in environment variables or credential files.
  • The daemon's session limit (30 turns per run) is by design: it prevents runaway token usage and forces periodic checkpoints where new work is pulled from the queue.

This architecture decouples task processing from data synchronization, allowing the agent to remain responsive even if one subprocess fails. However, it also creates multiple credential lifecycle management touchpoints.

Recent Activity: Understanding Today's Session Pattern

During the 24-hour window we examined, the daemon completed three distinct sessions:

  • Session 1 (00:00 UTC): Hit the 30-turn limit while processing work. Exit code 1 indicates the turn limit was reached, not a crash.
  • Session 2 (00:02 UTC): Successfully completed. Processed blockers related to e-signature page generation and created a needs-you task for manual intervention on crew page generator code.
  • Session 3 (00:05 UTC): Hit the 30-turn limit again. After this session, no new tasks appeared in the queue, and the daemon returned to its normal idle polling state.

The pattern of hitting max turns is expected when dealing with complex or multi-step tasks. The fact that session 2 completed successfully and produced actionable output indicates the daemon is functioning as designed.

The Port Sheet Sync Failure: A Separate Concern

While the daemon itself remained healthy, the port_sheet_sync.py subprocess has been completely unable to authenticate with Google's API for at least 12 hours. Every 30-minute attempt logged the same HTTP 400 error.

The sync script likely uses a structure like:


import google.auth.transport.requests
from google.oauth2.service_account import Credentials

# Load token from environment or credential file
credentials = Credentials.from_authorized_user_info(
    STORED_OAUTH_TOKEN
)

# Attempt to refresh if expired
request = google.auth.transport.requests.Request()
credentials.refresh(request)

# Make API calls to Google Sheets

When the refresh token is invalid (either revoked or expired), the refresh call fails with HTTP 400 Bad Request, and subsequent API calls are blocked.

Key Decisions and Next Steps

To resolve this issue, we need to:

  • Re-authenticate the OAuth token: Run the authentication flow for the port_sheet_sync.py service account. This likely involves running a script similar to auth_ga.py (which exists in the tools directory) but configured for Google Sheets API scope instead of Google Analytics.
  • Store the new token securely: Update the credential file on the Lightsail instance, ensuring proper file permissions (mode 0600) to prevent unauthorized access.
  • Validate the sync: After re-authentication, manually trigger one port_sheet_sync.py run to confirm it can authenticate and complete a full sync cycle.
  • Consider credential architecture: Evaluate whether using Google Cloud service accounts with long-lived JSON keys (rotated quarterly) would be more reliable than OAuth refresh tokens for daemon processes.

Summary

The jada-agent daemon remains operationally healthy with no underlying infrastructure problems. The port_sheet_sync.py subprocess failure is isolated to OAuth token lifecycle management—a common challenge in systems with multiple long-running authenticated processes. Immediate re-authentication will restore sync functionality, but the broader lesson is that daemon architectures require explicit credential rotation policies distinct from interactive authentication flows.

```