Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

```html

Diagnosing and Stabilizing the JADA Agent Orchestrator: Daemon Health, OAuth Token Rotation, and Session Limit Management

During a routine health check of our primary agent orchestrator instance (Lightsail, us-east-1), we discovered a functioning daemon with solid uptime but an underlying OAuth credential issue that was silently breaking our port sheet synchronization pipeline. This post walks through the diagnostics, the architecture decisions that enabled remote troubleshooting without stored SSH keys, and the remediation path forward.

What Was Done

We performed a comprehensive health audit of the jada-agent.service daemon running on our primary Lightsail instance, including:

  • Verified service status, uptime (11 days), and resource utilization (CPU, memory, disk)
  • Analyzed three agent sessions executed on 2026-05-13, identifying two that hit the 30-turn Claude limit and one that completed successfully
  • Discovered a persistent Google OAuth token failure in the port_sheet_sync.py script causing 30-minute sync cycles to fail with HTTP 400
  • Confirmed the daemon's task-picking mechanism is working correctly; idle load average reflects normal inter-task behavior
  • Identified the need for OAuth token re-authentication and potential adjustment to agent turn limits for complex tasks

Architecture: SSH Access Without Stored Keys

A key challenge in this engagement was that the private key for the jada-key pair was not stored locally in ~/.ssh/. Rather than searching for a backup key or requesting manual intervention, we leveraged AWS Lightsail's built-in temporary credential API:

aws lightsail get-instance-access-details \
  --instance-name jada-agent-prod \
  --region us-east-1

This call returns a temporary SSH certificate and certificate details that, when paired with the public key infrastructure already in place on the instance, grants time-limited access. We wrote the temporary credentials to a file, set restrictive permissions (chmod 600), performed our diagnostics via SSH, and then immediately rotated (deleted) the temporary files. This approach provides:

  • Zero stored secrets: No persistent private keys in developer machines
  • Audit trail: All access is logged via AWS CloudTrail
  • Time-limited access: Credentials expire after a short window, reducing blast radius
  • Minimal operational overhead: No key management burden for the ops team

Daemon Health: The Numbers

The jada-agent.service is in good operational health:

  • Uptime: 11 days continuous (since ~May 2)
  • CPU utilization: ~0.65% average (60-second polling interval) with no spikes
  • Memory: 144 MB / 914 MB allocated (stable, no leaks observed)
  • Disk: 6.2 GB / 39 GB used (17%, plenty of headroom)
  • Status checks: 0 failures in the last 2 hours (Lightsail native monitoring)
  • Load average: 0.00 between task cycles (expected idle behavior)

Session data from today (UTC) showed three invocations:

  • Session 1 (00:00): Hit the 30-turn limit, exited with code 1
  • Session 2 (00:02): Completed successfully; processed e-signature/crew page blockers and created a needs-you task
  • Session 3 (00:05): Hit the 30-turn limit, exited with code 1

After session 3, no pending tasks remained in the dashboard, and the daemon returned to its idle poll loop. The exit code 1 on max-turn hits is logged as an error in the daemon's journal but does not crash or hang the service—the daemon simply logs and continues. However, complex tasks that require more than 30 turns of reasoning will be incomplete, leaving work for the next invocation.

The OAuth Failure: port_sheet_sync

The most significant issue discovered was a broken Google OAuth credential in the port_sheet_sync.py script. Every 30-minute sync cycle since at least this afternoon has returned:

[port-sheet] token error: HTTP Error 400: Bad Request

This indicates the OAuth token stored for the dangerouscentaur@gmail.com account (which holds our Google Sheets and Analytics properties) has either expired or been revoked. Port sheet synchronization has been non-functional for several hours, meaning any crew or booking data changes have not been reflected in the central data store.

The remediation requires re-authentication. The existing token is stored in the repos.env configuration file on the Lightsail instance. A re-auth flow using auth_ga.py (our custom OAuth handler) will need to be triggered to refresh the token. This script orchestrates the OAuth 2.0 authorization code flow and persists the new credentials securely.

Turn Limit and Task Complexity

Two of today's three sessions hit the 30-turn Claude API limit. This is not a system failure—the daemon is configured to invoke the Claude API with a maximum turn/message count to prevent runaway costs and infinite loops. However, when legitimate work requires more turns than the limit allows, tasks remain incomplete.

The successful session 2 run completed within the turn budget and produced meaningful output (identified blockers in the e-signature and crew page generation logic). Sessions 1 and 3, which hit the limit, likely contained more complex reasoning requirements or were attempting larger code refactors.

Options forward:

  • Increase the turn limit from 30 to 50 or 75 (with corresponding cost/latency trade-offs)
  • Decompose complex tasks into smaller, sequential jobs that each fit within 30 turns
  • Implement a continuation mechanism where a max-turn exit automatically spawns a follow-up task with context from the previous run

Key Decisions Made

Why we used the Lightsail API for SSH instead of searching for the key: Storing SSH private keys on developer machines or in environment files creates credential sprawl and audit complexity. The Lightsail temporary access API is purpose-built for this pattern and integrates directly with IAM and CloudTrail.

Why we focused on the OAuth token first: A broken authentication credential is an operational blocker that silently fails, whereas the turn limit is a design constraint. We prioritized visibility and remediation of the active failure.

Why load average of 0.00 is normal: The daemon runs a 60-second poll loop that checks the task queue. Between executions, the CPU sits idle. A load average near zero indicates the polling mechanism is not consuming resources, which is correct.

What's Next

  • Re-authenticate the Google OAuth token: Invoke auth_ga.py with the dangerouscentaur account to refresh the token and resume port sheet syncs
  • Analyze session 1 and 3 failures: Retrieve the task context and Claude logs to understand why those sessions hit the turn limit; determine if turn limit increase or task decomposition is appropriate