Queen of San Diego — Tech Blog

Under the hood at queenofsandiego.com

```html

Diagnosing and Remediating a Distributed Agent Daemon: OAuth Token Failures and Turn Limits in Production

On May 13, 2026, we conducted a comprehensive health audit of the jada-agent orchestrator daemon running on AWS Lightsail instance 34.239.233.28. The investigation revealed a healthy service tier with one critical OAuth token degradation and a recurring architectural constraint worth documenting. This post covers our diagnostic approach, findings, and remediation strategy.

What Was Done

We performed a multi-layer health check on the jada-agent.service without local SSH key access by:

  • Retrieving temporary SSH credentials via the AWS Lightsail API (avoiding static key management)
  • Collecting daemon logs, service status, system metrics (CPU, memory, disk, status checks), and process activity
  • Analyzing 24-hour session history and task queue behavior
  • Identifying the root cause of recurring port-sheet synchronization failures
  • Assessing the impact of Claude API turn limits on task completion rates

This diagnostic pattern—leveraging temporary credentials over static keys—represents a shift toward ephemeral access in our infrastructure practices.

Technical Details: Service Health

Uptime and Resource Utilization

The jada-agent.service has been active since May 10, 2026—3 days of continuous operation—with 11 days of total instance uptime. The daemon runs a 60-second poll loop that checks the progress dashboard for pending tasks. Key metrics:

  • CPU: 0.65% average (normal for idle polling)
  • Memory: 144 MB / 914 MB (15.8% utilization)
  • Disk: 6.2 GB / 39 GB (17% used)
  • Load average: 0.00 (essentially idle between task invocations)
  • AWS Lightsail status checks: 0 failures in the last 2 hours

These metrics indicate healthy baseline operation. The low CPU during active polling is expected; the daemon spends most cycles sleeping between dashboard queries.

Session and Task Activity

The daemon maintains a 5-session daily quota (rolling window). On May 13 UTC, three sessions were consumed:

  • Session 1 (00:00): Exited with code 1 after hitting the 30-turn Claude API limit
  • Session 2 (00:02): Completed successfully—processed e-signature page blockers and crew page generator tasks, created a downstream needs-you task
  • Session 3 (00:05): Exited with code 1 after hitting the 30-turn limit

After session 3, no new tasks were found in the queue. The daemon idled normally for the remainder of the day. This pattern shows that complex tasks consume the full Claude context window, causing agents to exit before task completion.

Critical Issue: OAuth Token Degradation in port_sheet_sync

Symptom

Every 30-minute execution of port_sheet_sync.py has been failing with consistent HTTP 400 errors:

[port-sheet] token error: HTTP Error 400: Bad Request

The failures began at or before May 13 afternoon UTC. No port sheet syncs have completed since that point.

Root Cause

The Google OAuth token stored for the port-sheet synchronization script has expired or been revoked. This is not a code defect but a credential lifecycle management issue. The token, originally generated via auth_ga.py (the Google Analytics authentication utility), lacks automatic refresh or expiration monitoring.

Why This Matters

Port sheet synchronization is a background task that updates operational tracking sheets via the Google Sheets API. Its failure is silent—the daemon logs the error but continues polling. This can cause cascading data staleness across downstream reporting pipelines that depend on synchronized port data.

Architectural Constraint: Claude API Turn Limits

The daemon orchestrates multi-turn conversations with Claude (30 turns per session). When a task is sufficiently complex, the agent exhausts the turn budget before reaching task completion, forcing an exit with code 1.

This manifested twice on May 13:

  • Sessions 1 and 3: Exited cleanly but incompletely due to context exhaustion
  • Pattern impact: Complex tasks requiring iterative refinement (e.g., code generation with multiple validation loops) tend to hit this limit

The daemon handles this gracefully—it logs the exit state and resumes on the next session—but incomplete tasks remain in the queue, consuming quota inefficiently.

Infrastructure and Deployment Architecture

Lightsail Instance and Access Pattern

The jada-agent orchestrator runs on a dedicated Lightsail instance. During this audit, we avoided relying on static SSH keys (which require local file management and rotation procedures). Instead, we used the AWS Lightsail API endpoint to retrieve temporary SSH credentials, which are ephemeral and tied to IAM roles.

Access pattern used:

# Retrieve temporary key via Lightsail API
aws lightsail get-instance-access-details \
  --instance-name jada-agent-orchestrator \
  --region us-east-1

# SSH with temporary credentials (no static key file needed)
ssh -i /tmp/lightsail_temp_key ec2-user@34.239.233.28

This approach eliminates the need to store private keys on developer machines—credentials are generated on-demand and expire automatically.

Service Configuration

The daemon is managed by systemd as jada-agent.service, configured for auto-restart and persistent logging. The service file invokes a Python polling loop that queries the progress dashboard every 60 seconds for pending tasks.

Key Decisions

1. Ephemeral Credential Access Over Static Keys

Rather than hunting for a locally-stored jada-key private key, we used the Lightsail API to generate temporary credentials. This is more secure (no static keys in the filesystem), more auditable (each access can be logged), and aligns with modern IAM best practices.

2. Holistic Diagnostics: Metrics + Logs + Process State

We pulled CloudWatch metrics, systemd journal logs, and process state (via /proc) in parallel. This multi-dimensional view revealed that the service is healthy at the infrastructure level but has application-level token failures.

3. Distinguishing Between Failures and Constraints

The two max-turns exits (code 1) are not failures in the operational sense—they're architectural constraints. The daemon recovers and continues. However, they represent inefficient quota consumption and warrant investigation into task decomposition or context management.

What's Next

  • OAuth Token Remediation: Re-authenticate the Google OAuth credentials for port-sheet-sync.py. This requires running auth_ga.py with the appropriate service account and storing the refreshed token in the secrets store with appropriate TTL monitoring.
  • Token Lifecycle Monitoring: Implement automated checks for token expiration (e.g., monthly re-validation) rather than reactive diagnosis after