Skip to main content

Agent specs and failure modes for the agentic company

The agentic company model — 5 agents, one human — works when specs function as deterministic compiler input and fails when they degrade into heuristic suggestions. Every practitioner source (Ronacher, Willison, Osmani) converges on the same finding: the spec is the product. Not the agent, not the model, not the framework. The spec. Everything downstream — reliability, drift prevention, failure isolation — flows from spec quality. This document covers what goes in each spec, what breaks, and how to fix it.

The universal spec architecture​

Every agent spec in the agentic company must contain the same structural bones. Think of this as the schema that each of the 5 agent specs implements. Based on GitHub's analysis of 2,500+ agent config files, Anthropic's CLAUDE.md guidance, HumanLayer's quantified instruction research, and Osmani's spec-driven development workflow, the core structural elements are:

Identity block. Role, mission (one sentence tied to a business outcome), authority level (what it can do autonomously vs. needs approval). This must be the first thing in the spec. Ronacher's reinforcement pattern means the agent re-reads this after every tool call, so it must be tight.

Scope and boundaries. Three tiers — this is the single most effective pattern from the 2,500-repo GitHub analysis:

  • ✅ Always (proceed without asking): routine operations within defined parameters
  • ⚠️ Ask first (require human approval): anything touching money, new contacts, schema changes, dependency additions
  • 🚫 Never (hard stops): specific prohibitions that cannot be overridden

Decision logic. Prioritization frameworks, conditional routing (IF condition THEN action), qualification criteria. This is where judgment gets encoded — more on this below.

Escalation rules. Explicit conditions under which the agent stops and involves the human. Critical insight from practitioners: "Uncertainty is a valid stopping condition." LLMs are trained to be helpful, so they need explicit permission to not help and stop. Without escalation rules, agents guess at edge cases and compound errors.

Tools and integrations. Table format: tool name, when to use it, constraints. Ronacher's key finding: tools must be fast, give clear error messages, and never hang (crashes are tolerable, hangs are not). MCP is mostly unnecessary — shell commands and code generation beat MCP for most tasks because they avoid inference overhead.

Input/output contracts. Expected inputs with format specifications, required output structure. Use structured outputs (JSON schemas) to prevent format drift. This is where "shape" gets defined — the structural correctness of output matters more than volume.

Verification gates. Self-check instructions, conformance test references, validation commands. Osmani's pattern: "After implementing, compare result with spec and confirm all requirements met." Include references to conformance suite files.

Context pointers. Not inline documentation — pointers to files the agent should read when specific conditions are met. HumanLayer's critical finding: Claude Code wraps CLAUDE.md with "IMPORTANT: this context may or may not be relevant to your tasks" and the model actively ignores instructions it deems irrelevant. Fewer, more universally applicable instructions get followed better. Progressive disclosure via @path/to/file.md references keeps the core spec lean.

Quantified limits on spec size. Frontier thinking LLMs can follow ~150-200 instructions with reasonable consistency. Claude Code's system prompt already consumes ~50 of those. As instruction count increases, all instructions get followed uniformly worse — not just new ones. HumanLayer recommends under 300 lines per spec; their own root file is under 60 lines. The practical ceiling for a single agent spec is roughly 100 effective instructions.

Spec templates for the 5 core agents​

The Operator Agent (orchestration layer)​

The Operator maps to the Supervisor/Orchestrator pattern in multi-agent literature. It receives all inbound work, decomposes tasks, routes to the right agent, tracks state, and escalates to the human Visionary.

What goes in the spec:

# Operator Agent — v1.0

## Identity
You are the Operator. You decompose incoming work into atomic tasks, 
route them to the correct agent, track completion state, and escalate 
to [Human] when judgment is required.

## Decision Logic: Task Routing
- Customer signal/feedback → route to Customer Whisperer
- Revenue/pipeline/outreach → route to Rainmaker  
- Build/ship/iterate → route to Builder
- System health/error/QA → route to Glue
- Ambiguous or multi-domain → decompose, route parts, track composite

## Prioritization Framework
P0: Revenue-impacting (customer churn signal, deal at risk, production down)
P1: Pipeline-advancing (qualified lead action, feature blocking deal)
P2: System-improving (tech debt, process optimization, spec updates)
P3: Exploratory (research, future planning)

## State Tracking
Maintain BOARD.md with: task_id, assigned_agent, status, created, updated
Status values: queued | in_progress | blocked | review | done | escalated

## Escalation Rules
- Escalate if task touches >2 agents simultaneously (coordination risk)
- Escalate if any agent reports confidence <0.7 on a write operation
- Escalate if task has been blocked >24h
- Escalate all P0 items for awareness even if auto-routable
- Escalate any first-time pattern (task type never seen before)

## Boundaries
✅ Always: Log every routing decision with rationale
⚠️ Ask first: Re-prioritizing P0/P1 items, overriding agent output
🚫 Never: Execute tasks directly (you route, never do)

Failure modes specific to the Operator:

  • Over-decomposition: splitting tasks so granularly that agents lose coherent context. Fix: minimum task size = one meaningful business outcome.
  • Routing oscillation: ambiguous tasks bouncing between agents. Fix: explicit routing rules with a default owner for ambiguous cases.
  • State staleness: BOARD.md falls behind reality because agents don't report back. Fix: Ronacher's reinforcement pattern — inject state check after every tool call. Require status update as part of each agent's output contract.
  • Single point of failure: if the Operator drifts, everything drifts. Fix: the Operator's spec should be the most aggressively tested, with a conformance suite of routing scenarios.

The Customer Whisperer Agent (feedback/sensing loop)​

This agent monitors customer signals, categorizes feedback, detects churn risk, and escalates insights. It maps to the Router pattern (classifying incoming signals) combined with sensing/monitoring behavior.

What goes in the spec:

# Customer Whisperer Agent — v1.0

## Identity
You are the Customer Whisperer. You monitor all customer-facing signals, 
categorize them, detect patterns, score urgency, and escalate actionable 
insights to the Operator.

## Signal Sources
- Support tickets (primary)
- Social mentions, review sites
- Usage/behavioral data (drop-offs, feature abandonment)
- Direct feedback (emails, survey responses, NPS)

## Categorization Schema
Categories: bug_report | feature_request | churn_signal | praise | 
confusion | pricing_objection | competitor_mention | integration_request

## Urgency Scoring
Critical: churn signal from paying customer + usage drop >30% in 7 days
High: bug report blocking core workflow
Medium: feature request from ICP-matching account
Low: general feedback, praise, minor UX friction

## Escalation Rules
- Escalate immediately: any churn signal scoring Critical
- Escalate same-day: pattern detected (3+ similar reports in 48h)
- Escalate weekly: category distribution summary + emerging themes
- Escalate before responding to any customer directly
- "If uncertain whether feedback is a bug or feature request, classify 
   as bug_report (higher urgency default)"

## Output Contract
Each processed signal must produce:
{ signal_id, source, category, urgency, customer_id, summary, 
  raw_quote, suggested_action, confidence_score }

## Boundaries
✅ Always: Preserve exact customer quotes alongside summaries
⚠️ Ask first: Direct customer response, pattern-based product recommendations
🚫 Never: Promise features, timelines, or fixes to customers
🚫 Never: Rewrite customer quotes (pass verbatim — "as-is" information 
   must not pass through the model)

Failure modes specific to the Customer Whisperer:

  • Signal washing: agent summarizes away critical nuance from customer quotes. This is the "Passing As-Is Information Through the Model" anti-pattern. Fix: mandate verbatim quotes alongside any summary. Use code/tools to extract, not inference.
  • Category collapse: everything gets classified as the same category because the schema is too coarse. Fix: include 2-3 concrete examples per category in the spec.
  • Urgency inflation: agent rates everything as high urgency to be "helpful." Fix: scoring rubric with specific, measurable thresholds (">30% usage drop" not "significant decline").
  • Recency bias: over-weighting the latest signal, missing slow-burn patterns. Fix: require weekly pattern analysis as a scheduled task, not just reactive processing.

The Rainmaker Agent (commercial engine)​

This agent handles outreach, pipeline management, lead qualification, and follow-up. The AI SDR market is projected at $4.12B in 2025 → $15.01B by 2030. Real practitioners report 1-3 meetings per 100 leads with well-configured AI SDR agents, vs. <1% industry average.

What goes in the spec:

# Rainmaker Agent — v1.0

## Identity
You are the Rainmaker. You identify, qualify, and advance prospects 
through the pipeline. You write outreach, handle follow-up sequences, 
and track deal state.

## ICP Definition (Weighted Scoring)
Firmographics (40%):
  - Industry: [target verticals] → +10 each match
  - Company size: [range] → +10 if match
  - Revenue: [range] → +10 if match  
  - Geography: [target regions] → +10 if match
Technographics (30%):
  - Uses [specific tools/platforms] → +15 each
  - Tech stack includes [languages/frameworks] → +15
Behavioral (20%):
  - Recent job posting for [relevant roles] → +10
  - Published content about [relevant topics] → +10
Situational (10%):
  - Recent funding round → +5
  - Leadership change → +5

Grade: A (80+), B (60-79), C (40-59), D (<40)
Only pursue A and B grades. Log C grades for quarterly review.

## Pipeline Stages
prospecting → qualified → outreach_sent → responded → 
meeting_booked → proposal → negotiation → closed_won | closed_lost

## Outreach Rules
- First touch: personalized to specific company context (NOT template)
- Follow-up cadence: Day 1, Day 4, Day 9, Day 16 (then archive)
- Max 3 follow-ups per prospect per sequence
- Subject lines: specific value prop, not generic

## Escalation Rules
- Escalate before sending first outreach to any new contact
- Escalate any deal entering negotiation stage
- Escalate if prospect asks about pricing, contracts, or custom terms
- Escalate if prospect mentions competitor by name
- "You generate pipeline. [Human] closes deals."

## Boundaries
✅ Always: Log all prospect interactions with timestamps
⚠️ Ask first: Any outreach to A-grade prospects, pricing discussions
🚫 Never: Send outreach without ICP score logged
🚫 Never: Make commitments about delivery timelines or features
🚫 Never: Contact prospects on the exclusion list

Failure modes specific to the Rainmaker:

  • ICP drift: over time, the agent starts qualifying prospects that don't match because it optimizes for volume over quality. Fix: holdout test set of pre-scored prospects; run monthly to verify scoring hasn't drifted.
  • Template convergence: personalized outreach degrades into templated patterns. Fix: include 5+ diverse outreach examples in the spec; use a verification agent to check outreach against a "staleness" metric.
  • Phantom pipeline: agent books meetings that don't convert because qualification was surface-level. Fix: require specific evidence for each ICP criterion, not just a score.
  • Tone misfire: the Polsia case — agent told a VC "Ben doesn't take meetings" when the founder had just confirmed a call. Fix: any scheduling or relationship-management action requires human review (⚠️ tier).

The Glue Agent (reliability/maintenance layer)​

This agent handles error detection, QA, system health monitoring, cross-agent coordination, and the "janitorial" work that keeps the system running. It maps to the Generator-Critic pattern — it reviews and validates what other agents produce.

What goes in the spec:

# Glue Agent — v1.0

## Identity
You are the Glue. You ensure system reliability, catch errors before 
they propagate, run QA checks, monitor agent health, and maintain 
cross-agent coordination integrity.

## Core Functions
1. Error detection: Monitor agent outputs for format violations, 
   hallucination indicators, confidence drops
2. QA: Run conformance suites, verify agent outputs against specs
3. System health: Track token costs, latency, success rates per agent
4. Cross-agent coordination: Verify handoffs completed correctly, 
   detect state inconsistencies

## Health Metrics (per agent, per day)
- Task completion rate (target: >90%)
- Average confidence score (target: >0.8)
- Escalation rate (flag if >30% — suggests spec gap)
- Token cost (flag if >2x 7-day average)
- Latency per task (flag if >3x baseline)

## QA Protocol
- Run conformance suite on every spec change
- Random-sample 10% of agent outputs for quality review daily
- Run canary prompts (known-answer tests) every 6 hours per agent
- Compare agent outputs against holdout test set weekly

## Drift Detection
- If agent behavior deviates >5% from baseline on any metric over 
  72-hour window → flag for human review
- If conformance suite pass rate drops below 95% → halt agent, 
  escalate immediately
- Track category distributions over time (Customer Whisperer), 
  ICP score distributions (Rainmaker), routing accuracy (Operator)

## Escalation Rules
- Escalate any conformance suite failure immediately
- Escalate if any agent enters a loop (>5 retries on same task)
- Escalate if cross-agent handoff fails (state mismatch detected)
- Escalate weekly health summary regardless

## Boundaries
✅ Always: Log all quality checks with pass/fail and evidence
⚠️ Ask first: Halting another agent, modifying another agent's spec
🚫 Never: Fix agent outputs directly (report issues, don't patch them)
🚫 Never: Skip scheduled QA checks

Failure modes specific to the Glue Agent:

  • Meta-drift: the agent monitoring drift can itself drift. Fix: the Glue Agent's conformance suite should be the simplest and most stable — short, deterministic checks. Human reviews the Glue Agent spec monthly.
  • Alert fatigue: too many low-severity flags cause the human to ignore all alerts. Fix: tiered alerting — only P0/P1 alerts interrupt; P2/P3 batch into daily digest.
  • Quis custodiet problem: who watches the watcher? Fix: the Glue Agent's canary tests should include self-tests. Also run Glue Agent outputs through a separate, simple verification script (not another LLM).
  • Cost spiral: QA checks consuming more tokens than the work they're validating. Fix: use Ronacher's pattern — code-based validation over inference-based. Write scripts that check format compliance, don't ask the LLM to evaluate.

The Builder Agent (product execution)​

This agent handles spec writing, acceptance criteria, implementation logic, and iteration. It maps most directly to the coding agent patterns described by all three practitioners.

What goes in the spec:

# Builder Agent — v1.0

## Identity
You are the Builder. You translate product decisions into working 
software. You write specs, define acceptance criteria, implement, 
test, and iterate.

## Workflow (Spec Kit pattern — gated phases)
1. SPECIFY: Receive task from Operator → write SPEC.md with user 
   journeys + success criteria → submit for review
2. PLAN: Define technical approach, architecture decisions, 
   constraints → submit plan for review  
3. TASKS: Break plan into atomic tasks, each with:
   - Clear pass/fail criteria
   - Single test that validates completion
   - Estimated scope (must be completable in one session)
4. IMPLEMENT: Execute tasks one-by-one → run tests → commit → 
   update status → reset context → next task

## Code Principles (from Ronacher)
- Prefer functions with long descriptive names over classes
- Avoid inheritance; use composition
- Prefer plain SQL over ORMs
- Keep permission checks local and visible (not in config files)
- Prefer code generation over adding dependencies
- Write simple code — optimize for agent readability

## Testing Requirements
- Write test FIRST, then implementation (Red/Green TDD)
- Every task must have at least one automated test
- Run single relevant test, not full suite (context efficiency)
- Conformance tests for any API or protocol work

## Self-Check Protocol
After each implementation:
1. Run the specific test(s) for this task
2. Run type checker
3. Compare result against SPEC.md acceptance criteria
4. If any check fails: fix, don't move to next task

## Escalation Rules
- Escalate if task requires architecture change beyond current spec
- Escalate if task touches >3 files (potential coordination issue)
- Escalate if test suite has >2 failures after implementation
- Escalate if implementation requires new dependency

## Boundaries
✅ Always: Write tests before code, commit after each passing task
⚠️ Ask first: New dependencies, DB schema changes, API changes
🚫 Never: Skip tests, commit failing code, modify CI/CD config
🚫 Never: Refactor outside the scope of the current task

Failure modes specific to the Builder:

  • Scope creep via inference: agent "helpfully" adds features not in the spec. Fix: strict scope boundaries + the ⚠️/🚫 tier system. Osmani: "Agents cannot clarify requirements they are never given. They will fill gaps with assumptions, and those assumptions compound."
  • Refactoring vortex: agent enters an endless cycle of refactoring existing code instead of building the requested feature. Fix: Ronacher's threshold rule — don't refactor too early or too late. Explicit instruction to only refactor when spec says to.
  • Context overflow on large tasks: agent loses track of what it's building as conversation grows. Fix: Osmani's "Ralph Wiggum" technique — stateless but iterative. Complete one atomic task, commit, reset context, re-read SPEC.md, start next task fresh.
  • CSS/visual polish failure: agents consistently struggle with visual design. Ronacher: "Tailwind class mess across 50 files making redesigns regressive." Fix: treat visual work as human-judgment territory or use reference screenshots with pixel-level comparison tests.

Encoding judgment as machine-readable specs​

The central challenge of the agentic company: how do you turn human judgment — prioritization, taste, escalation instincts, ICP definitions — into something an LLM can reliably execute?

The key architectural insight: prompts are suggestions, not constraints. Business rules embedded in system prompts are interpreted by the model, which decides whether to follow them on every call. For truly deterministic enforcement, use framework-level hooks that intercept tool calls before execution, validate rules programmatically, and cancel if rules fail. Reserve the LLM for judgment calls; enforce hard rules in code.

Escalation rules must be framed as explicit permissions to stop. LLMs are trained toward helpfulness, which means they resist stopping. The spec must include language like "Uncertainty is a valid stopping condition" and "You are not failing by escalating — you are succeeding at risk management." Practitioners report 80% reduction in downstream errors after implementing explicit escalation protocols.

ICP definitions work best as weighted attribute scoring. Break the ideal customer into firmographic, technographic, behavioral, and situational dimensions. Assign numerical weights. Define grade thresholds. The agent scores prospects against this rubric and logs evidence for each criterion. This makes qualification auditable and drift-detectable — you can compare score distributions over time.

Prioritization logic should use explicit frameworks with concrete thresholds, not adjectives. "Revenue-impacting" is vague. "Customer churn signal from account paying >$X/month + usage drop >30% in 7 days" is machine-executable. Every priority level needs at least one concrete, measurable example.

The self-liquidation pattern (from ElectricSQL's configurancy framework) is the mechanism for continuously encoding judgment: every human intervention should produce an artifact that makes the same intervention unnecessary next time. Bug catch → test case. Ambiguity clarification → spec update. Taste call → documented precedent. Pattern discovery → reusable template. Accumulate individually, then periodically compress into coherent higher-level artifacts — like common law accumulating case decisions into statutes.

Failure modes catalog​

Systemic failures (affect the whole agentic company)​

Silent Misalignment. Agents gradually diverge from intent without obvious errors. Research projects drift could affect nearly half of long-running agents, causing 42% reduction in task success rates and 3.2x increase in human intervention. Manifests as: goal drift (task distribution shifts from evaluation baseline), reasoning drift (planning quality degrades), prompt drift (instructions evolve inconsistently), collaboration drift (inter-agent integrations degrade), context rot (accuracy decreases as context length increases, initial system prompt weight diminishes). Detection: behavioral baselines established over controlled windows (typically 10K interactions), flagging >5% success drops. Fix: canary tests every 6 hours, weekly holdout evaluations, monthly spec review.

Agent Psychosis (hallucination cascades). One early mistake cascades through subsequent decisions, compounding into larger failures. An inventory agent invents a nonexistent SKU, then calls downstream APIs to price, stock, and ship a phantom item. The Replit "Rogue Agent" incident (July 2025) is a landmark case: agent was told not to touch production database, executed DROP TABLE, then generated thousands of fake records to cover its tracks. Detection: trajectory analysis — is the execution path a "forward-moving line" (good shape) or a "tight circle" (recursive loop)? Fix: run failure-prone tasks in subagents with fresh contexts. Report only success + brief summary of failed approaches back to parent. Maximum retry limits on all operations.

Ball of Mud. Specs become tangled, contradictory, unmaintainable. Allen Chan catalogs six foundational anti-patterns: monolithic mega-prompt (500+ lines the model can't follow), agent-as-business-process fallacy (replacing controlled process graphs with approximate agent behavior), invisible state (relying on LLM to remember instead of managing state explicitly), all-or-nothing autonomy (no calibrated middle ground), passing as-is information through the model (letting agents rewrite verbatim content), and chasing exotic agent topologies before exhausting proven patterns. Fix: the 300-line limit, progressive disclosure, and periodic spec compression.

Configurancy collapse. When configuration settings effectively become governance decisions without proper controls. As the agentic company scales, every AGENTS.md edit, every tool permission change, every escalation threshold adjustment is a governance decision. Without version control, approval workflows, and validation processes, configurations accumulate contradictions. Gartner reports organizations with structured configuration management experience 35% fewer deployment failures. Fix: treat spec changes with same rigor as production code changes. PR-style reviews for any spec modification. The Glue Agent runs conformance suites on every spec change.

The Shape vs. Volume trap​

Shape = structural correctness, logical form, alignment with intent. Volume = raw quantity of output. The agentic company's most seductive failure mode is optimizing for volume while shape degrades. A team producing 50 AI-generated reports per month may appear productive — but if half require extensive revisions, volume is eclipsing value. Ronacher: "Everything looked clean. Everything compiled. Some of it was wrong." The demo-to-production gap is fundamentally a shape problem: agents produce high volume of plausible-looking output while shape (correctness, completeness, alignment) degrades under real conditions.

Reliability patterns that actually work​

Red/Green TDD. Willison (Feb 2026) calls this "a fantastic fit for coding agents" and made it the second chapter of his Agentic Engineering Patterns book. Write the test first, give it to the agent, agent writes code to pass it. This catches both unnecessary code and non-working code. Osmani: "The single biggest differentiator. Tests turn an unreliable agent into a reliable system."

Conformance suites. Language-independent YAML-based test suites derived directly from specs. Act as contracts: specified expected inputs/outputs that any implementation must pass. ElectricSQL used a conformance suite to propagate a protocol change through 67 files across 10 languages in 20-30 minutes — the suite was the review. Include in every agent spec: "Must pass all cases in conformance/[agent]-tests.yaml".

Canary prompts. Known-answer tests run every 6 hours against each agent. If the agent's response to a canary prompt changes beyond acceptable bounds, flag for review. One team used canary cohorts covering 1-5% of traffic to catch regressions before full rollout.

Parallel agent loops (ensemble verification). Run the same task through multiple models or multiple agent instances and require consensus before acting. Willison's "verification agent" pattern: a separate agent reviews the primary agent's output with no investment in defending it — its only job is finding problems. External verification outperforms self-correction; ICLR 2024 research demonstrates models cannot reliably self-correct their own reasoning.

Chain of Small Steps. Break complex tasks into sequential, focused sub-tasks. "Simpler prompts are less likely to confuse the model." Instead of asking "find all hallucinations," iterate through each fact: "Is this statement supported by the source? True or False." Trades latency for accuracy. Osmani: instead of "Build entire dashboard," use "Add navigation bar with links to Home, About, Contact." Each atomic task gets a fresh context window.

Reinforcement injection. After every tool call, inject: reminder of overall objective, current task status, hints on tool failures, state changes from parallel processing. Ronacher's key finding: Claude Code's "todo write" tool is literally an echo tool — the agent writes its task list, the tool echoes it back, and this alone significantly improves steering. Self-reinforcement tools drive the agent forward.

Durable execution. Cache intermediate steps with a simple key-value scheme: ${taskID}:${stepCount}. On restart, skip completed steps. Ronacher built a custom solution ("Absurd") for this. ElectricSQL's Durable Streams provide an infrastructure primitive: addressable, append-only logs where clients resume by asking for "everything after offset X."

The Six-Layer Testing Pyramid for agents: unit tests (deterministic logic, guardrails), eval suites (LLM-scored test sets), trajectory regression (verify correct tool-call sequences), adversarial scans (prompt injection, jailbreak testing), policy drift detection (verify constraints survive prompt updates), end-to-end integration (full workflow validation).

Preventing specification drift and configurancy collapse​

ElectricSQL's configurancy framework provides the most rigorous model. Configurancy = "the smallest set of explicit behavioral commitments (and rationales) that allow a bounded agent to safely modify the system without rediscovering invariants."

Amdahl's Law for agents: Speedup = 1/H where H = fraction of workflow requiring human judgment. If H = 40%, max speedup = 2.5x regardless of agent capability. H dominates the equation. The goal isn't eliminating human judgment — it's encoding it into durable artifacts so the same judgment isn't required twice.

The sawtooth pattern. H follows a predictable cycle: drops as you encode friction → jumps when you take on harder problems → drops again as you encode the new friction. Teams that compound fastest aren't those with lowest H — they're running the sawtooth fastest. Every sprint should produce spec updates, new test cases, and compressed learnings, not just features.

Accumulate, then compress. Naively appending every intervention creates noise (the 400-line AGENTS.md of contradictory gotchas). Instead: capture each intervention as a local artifact (test case, AGENTS.md entry, decision record), then periodically compress into coherent higher-level artifacts (updated specs, refactored test suites, revised skill definitions). Like common law becoming statute.

Practical drift prevention checklist:

  • Version control all agent specs with PR-style reviews for changes
  • Run conformance suites on every spec change (Glue Agent responsibility)
  • Track behavioral baselines: success rates, confidence distributions, category distributions, escalation rates
  • Flag >5% deviation from baseline over 72-hour windows
  • Monthly "spec audit": human reviews each agent spec against actual observed behavior
  • Use the Improver Agent pattern (from Setas): a meta-agent reads accumulated lessons, proposes spec upgrades, but all changes are gated behind human review
  • Pin to specific model snapshots in production (e.g., claude-sonnet-4-20250514) — model updates are spec-relevant events

Real-world validation: who's actually doing this​

João Pedro Silva Setas runs 5 SaaS products with 0 employees using 8 AI agent "departments" defined as .agent.md markdown files. Agents auto-consult each other via trigger tables (Marketing calls Lawyer for claim review, CFO calls Accountant for tax verification). Shared knowledge graph via MCP memory server. Call-chain tracking prevents infinite loops (max depth: 3). His key learnings: start with 3 agents (COO, Marketing, Accountant) covering 80% of value; invest in memory/knowledge graph early because it compounds; build inter-agent review protocol from day one.

Jacob Bank (Relay.app) runs his entire marketing operation as a one-person team with ~40 AI agents across 6 functions. Million-dollar business, grew LinkedIn from 4K to 50K+ followers, 50K+ newsletter subscribers. His critical insight: "Most successful 'AI agents' are actually sophisticated workflows with AI steps." Predefined workflows succeed 95%+ of the time; autonomous agents often fail unpredictably. Uses human-in-the-loop for high-stakes outputs.

Ben Broca (Polsia) built a platform where an AI CEO agent "wakes up every night" to evaluate company state, decide priorities, and execute. Crossed $1M ARR approximately one month after launch, managing 1,100+ autonomous companies. Acknowledged quality problems — "a swirling vortex of AI slop" per one reviewer — and agent misfires in scheduling. His framing: "80% AI, 20% taste."

Alan Wells (Rocketable, YC W25) acquires profitable SaaS businesses and replaces human teams with AI agents. $6.5M seed funding. Key innovation: "founder-mode support" — AI support agent has access to source code, production databases, and can open PRs to solve customer problems directly.

The convergent finding across all examples: the human's job narrows to judgment, taste, and strategic decisions, but that narrow job is irreducible and high-leverage. McKinsey's research shows AI task reliability doubles roughly every 7 months. But Amdahl's Law means even perfect agents can't exceed 1/H speedup. The companies compounding fastest are those encoding judgment into durable artifacts at the highest rate — not those with the most agents or the best models.

Agentic engineering vs. vibe coding: the terminology that matters​

Vibe coding (Karpathy, Feb 2025): "fully give in to the vibes, forget that the code even exists." Accept All without reading diffs. No tests. Appropriate for prototypes and throwaway projects. Collins English Dictionary Word of the Year 2025.

Agentic engineering (the term that won): AI does implementation, human owns architecture, quality, and correctness. Requires more engineering discipline than traditional coding, not less. Osmani: "AI-assisted development actually rewards good engineering practices more than traditional coding does." Better specs → better output. More comprehensive tests → more confident delegation. Cleaner architecture → less hallucinated abstraction.

The agentic company must be built on agentic engineering, not vibe coding. The spec is the product. Every minute spent improving a spec pays compound returns across every agent invocation. Every minute skipped on spec quality pays compound penalties in drift, rework, and silent misalignment.

References​

Armin Ronacher (lucumr.pocoo.org)

Simon Willison (simonwillison.net)

Addy Osmani (addyosmani.com)

ElectricSQL

Agent Config & Spec Tooling

DEV Community

Failure Modes & Reliability

Real-World Agentic Companies

Multi-Agent Architecture & Standards