Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

```html

Diagnosing and Stabilizing the JADA Agent Daemon: OAuth Token Recovery and Session Management Optimization

During a scheduled health check of the JADA orchestrator daemon running on AWS Lightsail instance 34.239.233.28, we identified a critical OAuth token failure in the port sheet sync pipeline and confirmed the daemon's core stability despite hitting Claude's 30-turn session limits. This post details the diagnostic approach, findings, and remediation strategy.

What Was Done

We conducted a comprehensive health audit of the jada-agent.service systemd unit, including:

  • Service uptime and resource utilization analysis via AWS Lightsail metrics API
  • Real-time daemon logs and process inspection over SSH
  • Session execution pattern analysis over the past 24 hours
  • OAuth token validation for dependent sync scripts
  • Root cause analysis of recurring 30-turn session exits

The diagnosis revealed a healthy daemon with one critical blocker: the Google OAuth token used by port_sheet_sync.py has expired or been revoked, causing all sync operations to fail with HTTP 400 errors every 30 minutes.

Technical Details: Service Health

The jada-agent.service has been running continuously since May 10 with 11 days of upstream instance uptime. Key metrics:

  • CPU utilization: 0.65% average, no spikes detected in the last 2 hours
  • Memory footprint: 144 MB / 914 MB available (15.8% utilization)
  • Disk usage: 6.2 GB / 39 GB (17% used) — adequate headroom for logging
  • Load average: 0.00 — essentially idle between task executions
  • Network status checks: 0 failures in the last 2 hours

This profile is exactly what we expect for an event-driven daemon with a 60-second poll loop. The service is not leaking resources and is responsive to incoming work.

Session Execution Pattern and the 30-Turn Limit

Over the past 24 hours (UTC), the daemon has executed three agent sessions:

  • Session 1 (00:00 UTC): Reached max turns (30) and exited with code 1. This is not a crash—the daemon treats it as expected behavior and continues polling for new tasks.
  • Session 2 (00:02 UTC): Completed successfully without hitting the turn limit. This session processed e-signature link blockers and crew page generator code, generating a "needs-you" task for manual intervention.
  • Session 3 (00:05 UTC): Again hit the 30-turn limit and exited with code 1.

After session 3, the daemon found no new pending tasks and returned to idle state. The previous day's hard stop at midnight UTC (hitting 5/5 session quota before the daily reset) left 3 pending tasks, which cleared at the midnight rollover—confirming the quota system is functioning as designed.

Why the 30-turn exits are not necessarily failures: Claude's session context window has a practical limit. Complex multi-step tasks (like analyzing intricate workflows, debugging nested code blocks, or orchestrating cross-domain changes) can exhaust this window. When the limit is hit, the daemon logs exit code 1, but the task remains in the queue for the next session. This is intentional: it prevents runaway sessions and forces task decomposition. However, if tasks are consistently getting stuck at the boundary, this may indicate they need tighter scope definition or the turn limit itself needs adjustment.

Critical Issue: Google OAuth Token Failure in port_sheet_sync.py

The port sheet sync daemon has been failing every 30 minutes with the following error:

[port-sheet] token error: HTTP Error 400: Bad Request

This indicates that the OAuth 2.0 token stored for the `port_sheet_sync.py` script—which synchronizes booking and scheduling data to a Google Sheet—is either expired or has been revoked by the Google account holder or by OAuth policy.

Impact: Port sheet syncs have not run since at least May 13 afternoon. Any booking data, crew assignments, or scheduling changes made during this window have not been persisted to the canonical Google Sheet.

Root cause: The token was likely issued with a finite lifespan (typically 1 hour for Google OAuth access tokens) and was never refreshed, or the associated Google account revoked the token. Since this is a long-lived daemon process, the token lifecycle must be managed explicitly by refreshing using the stored refresh token, or by re-authenticating.

Infrastructure and Architecture

The JADA ecosystem consists of:

  • Lightsail Instance: 34.239.233.28 running the orchestrator daemon
  • Systemd Service: jada-agent.service configured to auto-restart on failure with exponential backoff
  • Dependent Scripts:
    • port_sheet_sync.py — syncs booking/crew data to Google Sheets (OAuth 2.0 protected)
    • auth_ga.py — handles Google Analytics API authentication
  • Task Queue: Progress dashboard (source TBD from context) — daemon polls every 60 seconds for pending tasks
  • Session Limits: 5 sessions per day (rolling UTC midnight), 30 turns per session

The architecture decouples the orchestrator daemon from dependent external APIs. However, this introduces a token lifecycle management responsibility: each downstream script that uses OAuth must either implement its own refresh logic or rely on the parent daemon to refresh tokens on a schedule.

Key Decisions and Rationale

Why we used Lightsail API + temporary SSH keys instead of stored persistent keys: The persistent jada-key SSH private key was not stored locally in the standard location (~/.ssh/jada-key). Rather than spending time locating a stale key, we used AWS Systems Manager Session Manager to request temporary credentials via the Lightsail API. This approach provides:

  • No persistent key material to manage or rotate on the local machine
  • Automatic expiration (temporary credentials are short-lived)
  • Audit trail via CloudTrail
  • No risk of key compromise if the local machine is exposed

Why we did not immediately fix the OAuth token: Re-authenticating the port sheet sync requires interactive OAuth flow (opening a browser, approving scopes, capturing the auth code). This is environment-dependent and requires coordination with the Google account holder. The diagnostic phase reports the issue; the remediation is a separate ticket.

What's Next

To stabilize the system:

  1. Re-authenticate Google OAuth for port_sheet_sync: Run the credential refresh flow (likely auth_ga.py or equivalent) to obtain fresh OAuth tokens and store them securely in the credentials vault.
  2. Implement token refresh logic: Modify port_sheet_sync.py to detect token expiration and automatically refresh using the refresh token before making API calls. This prevents the 30-minute error cycle.