Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

```html

Diagnosing and Remediating OAuth Token Failures in Distributed Task Orchestration

During a routine health check of the jada-agent orchestrator daemon running on AWS Lightsail (34.239.233.28), we discovered a critical authentication failure in the port sheet synchronization pipeline. This post details the diagnostic approach, root cause analysis, and remediation strategy for OAuth token lifecycle management in distributed systems.

What Was Done

We performed a comprehensive health audit of the jada-agent daemon service, focusing on:

  • Service status and uptime verification
  • Historical session execution logs and error patterns
  • OAuth token validity for dependent sync processes
  • Resource utilization metrics (CPU, memory, disk)
  • Task queue depth and processing throughput

The audit revealed that while the core daemon remained healthy with 11 days of uptime and normal resource consumption (~0.65% CPU average), the port_sheet_sync.py script—responsible for synchronizing spreadsheet data with Google Sheets—was experiencing cascading authentication failures every 30 minutes.

Technical Details: Root Cause Analysis

The port_sheet_sync.py process, running as a scheduled task on the jada-agent instance, invokes the Google Sheets API using OAuth 2.0 credentials. Every 30-minute sync cycle was failing with:

[port-sheet] token error: HTTP Error 400: Bad Request

This error signature indicates one of three conditions:

  • Token expiration: OAuth access tokens typically expire after 3600 seconds. Refresh tokens should be used to obtain a new access token, but if the refresh token is also expired or revoked, the exchange fails.
  • Token revocation: A manual revocation event occurred in Google's OAuth consent screen or via the Google Admin console.
  • Scope mismatch: The token was issued for a different set of API scopes than what the current script requires.

Given the timestamp pattern (failures "at least since this afternoon"), combined with no recent changes to the sync script's scope requirements, token revocation or expiration of the underlying refresh token is most likely.

Audit Command Examples

The diagnostic session used the following approaches to gather evidence:

# Verify service status and uptime
systemctl status jada-agent.service
journalctl -u jada-agent.service -n 100 --no-pager

# Pull recent sync errors from daemon logs
grep "port_sheet_sync\|port-sheet" /var/log/jada-agent/*.log | tail -50

# Check process resource usage over time
ps aux | grep jada-agent
free -h
df -h /

# Query AWS Lightsail metrics via API (CPU, network, disk)
aws lightsail get-instance-metric-statistics \
  --instance-name jada-agent-primary \
  --metric-name CPUUtilization \
  --statistics Average \
  --period 300 \
  --start-time 2026-05-13T00:00:00Z \
  --end-time 2026-05-13T23:59:59Z

SSH access was obtained via AWS Lightsail's temporary credential API rather than stored keys, reducing key material exposure and enabling auditable access logs.

Infrastructure and Architecture

Daemon Architecture: The jada-agent orchestrator is deployed as a systemd service (jada-agent.service) on a single AWS Lightsail instance running a polling loop with 60-second intervals. The daemon:

  • Monitors a task queue (presumably DynamoDB or similar) for pending work
  • Spawns Claude API sessions with a 30-turn maximum per session to bound token usage and cost
  • Maintains a session quota of 5 concurrent sessions per rolling period
  • Delegates dependent sync tasks (like port_sheet_sync.py) as subprocess invocations

OAuth Token Storage: The Google OAuth credentials for port_sheet_sync.py are persisted in the repos.env file or a dedicated secrets directory. The client_id and client_secret fields exist in the jada token and can be reused for reauthentication, but the refresh_token field has either expired or been revoked.

Sync Process Flow: The port_sheet_sync.py script is invoked on a 30-minute cron schedule or via task queue. It loads OAuth credentials from the stored token, exchanges the refresh token for a new access token, and uses that token to call the Google Sheets API (likely via the google-auth-oauthlib library, which is installed in the environment).

Key Decisions and Trade-offs

Why OAuth Token Management Matters: In distributed task systems, credential rotation and refresh are critical. Unlike monolithic applications where developers manage a single credential lifecycle, orchestrators often delegate work to multiple subprocess scripts, each with their own OAuth flows. A single expired token can silently break entire pipelines without alerting the core daemon—the daemon remains healthy while data synchronization stops.

Why the Daemon Didn't Fail: The jada-agent service itself uses a different authentication mechanism (likely AWS IAM role attached to the Lightsail instance) to interact with task queues and CloudWatch. OAuth tokens for downstream integrations (Google Sheets, etc.) are isolated from the daemon's core health. This is architecturally sound—it prevents a downstream API's authentication failure from cascading to the orchestrator—but it requires explicit monitoring of each subprocess's OAuth status.

Session Limit Behavior: Two of today's three agent runs hit the 30-turn Claude API limit, returning exit code 1. This is intentional rate-limiting, not a crash. The daemon logs it as an error and continues polling, which is correct behavior. However, tasks that exceed 30 turns will remain incomplete and re-queue, potentially creating a backlog if task complexity isn't tuned appropriately.

What's Next: Remediation Path

To restore port sheet synchronization:

  • Re-authenticate Google OAuth: Run the auth_ga.py script (or equivalent OAuth flow script for Sheets) with the dangerouscentaur@gmail.com account to obtain a fresh refresh token. Update the stored credentials in the secrets directory or repos.env.
  • Verify Token Structure: Confirm the new token contains client_id, client_secret, and refresh_token fields before deploying.
  • Validate Script Scope: Ensure port_sheet_sync.py requests the correct Google Sheets API scopes during reauthentication (likely https://www.googleapis.com/auth/spreadsheets).
  • Test Sync Cycle: Manually trigger port_sheet_sync.py to confirm the 30-minute sync succeeds without the HTTP 400 error.
  • Implement Token Expiry Monitoring: Add CloudWatch metrics or daemon logs that alert when OAuth refresh tokens are approaching expiration (e.g., if token metadata includes an issued-at timestamp, warn at 90% of typical refresh window).
  • Document Turn Limits: If task complexity is consistently exceeding 30 turns, adjust the turn limit in the daemon config or break complex tasks into sub-tasks with explicit handoffs.

The daemon itself is oper