How I Run AI Agents Like an Engineering Review Board

AI Workflow Philosophy

The Engineering Review Board

Three AI agents reviewed the draft of this post before it was published. They agreed on most things and disagreed on three. One missed a fact the others caught. That disagreement is not a bug. It is exactly why I do not trust a single LLM to ship anything that matters.

3 Agents

Reviewing the Draft

1 Human

Making Final Call

~9 sec

PocketOS Deletion

Unreviewed Prod Changes

The Problem

Single Points of Failure

Why relying on a single AI agent for platform engineering is risky.

The Single LLM Risk

One LLM is a Single Point of Failure

A single LLM can produce confident, polished, convincing answers that may still be incomplete, wrong, unsafe, or unrealistic.
It may over-engineer a solution.
It may underestimate production risks or ignore cost tradeoffs.
It might assume staging and production are the same environment.
One perspective equals one set of blind spots.

The Credentials Risk

Broad Access = Incidents

An AI agent with broad credentials is an incident waiting to happen.
AI does not remove the need for engineering discipline; it increases it.
Autonomous agents can execute destructive actions faster than a human can react.
Ambient credentials on local machines are dangerous when an agent "guesses" how to fix an issue.

Case Study

The PocketOS / Railway Incident

A cautionary tale of autonomous agents and system-design failure.

What Happened in April 2026

Reports say PocketOS initially lost access to production data and volume-level backups after an AI agent executed a volumeDelete call in about 9 seconds. The agent was working on a routine staging task when it encountered a credential mismatch, found an ambient Railway API token, and autonomously attempted to fix the issue.

Railway later said it recovered the database and updated its API so volume deletes now follow the same 48-hour soft-delete path as their dashboard.

The real lesson: This wasn't just "AI going rogue." It was a system-design failure involving over-privileged credentials, co-located backups, and missing confirmation gates.

The takeaway: The dangerous part is not AI making a mistake; the dangerous part is giving an autonomous agent broad permissions and destructive access without sufficient guardrails.

🚂 Railway Official Blog ↗ 𝕏 Jer Crane (PocketOS Founder) ↗ 𝕏 Jake Cooper (Railway CEO) ↗

Workflow

The Multi-Agent Review Pattern

I assign roles because role separation forces review.

1. Human Architect: Defines the goal clearly.

2. Planner Agent: Creates the initial architecture.

3. Challenger Agent: Reviews and attacks the architecture.

4. Security/SRE Agents: Check permissions, blast radius, failure modes, and cost.

5. Human Architect: Consolidates feedback and approves the final plan.

6. Coding Agent: Inspects the repo and implements controlled changes.

Frameworks like CrewAI, AutoGen, and LangGraph automate parts of this pattern. I run mine manually because I want full control over which agent sees which context — but the pattern is the same.

When to Use One LLM

Writing a quick bash one-liner
Explaining a specific error message
Drafting a short email
Creating a basic Dockerfile example
Low-risk, isolated tasks

When to Use Multi-Agent Review

Designing system architecture
Reviewing complex DevOps or K8s systems
Security-sensitive infrastructure work
Customer-ready productization
High-stakes production changes

A note on cost: Yes, this approach burns more tokens. I use free tiers where I can and paid tiers for high-stakes work. The cost of one wrong architecture decision on a customer cluster is much higher than a month of API spend across four tools.

Application

The Guardrails

AI agents are infrastructure actors when they have real credentials. They need the same controls we expect from humans.

Least Privilege: Scoped credentials per environment. Read-only by default.

Environment Isolation: No shared production and staging tokens.

Human Approval: Required before any destructive changes or remediation.

Dry-Run First: Preview commands before execution.

Audit Trail: Log every recommendation and command.

Isolated Backups: Backups must live outside the agent's blast radius.

Proof in Practice: The HARP Project

While shaping HARP (Homelab & Hybrid AI Reliability Platform), I used one agent to propose the architecture and another to challenge it. The multi-agent review converged on one specific design decision: no autonomous remediation, ever. The review pushed the design toward read-only evidence packs, mapped runbooks, action tiers, and explicit human approval.

Use AI aggressively, but never blindly.

The final lesson: The future is not "one AI agent does everything." The better model is controlled multi-agent collaboration — one agent plans, another reviews, another checks security, another validates implementation, and the human still owns the final decision.