I run something different: a fixed harness called Flavor Neo. It is not a model or a framework. It is a loop structure that turns "the strongest model I have access to" into a leader that plans, delegates, and judges, while cheaper models do the scanning. It lives as a single SKILL.md file you drop into your agent's skills directory. No dependencies, no runtime, no daemon; just a protocol your agent follows when you say "flavor-neo" or "neo audit."

The problem it solves

Here's what happens when you throw a big task at a single agent:

1. Token waste on grunt work. Your 5 USD/MTok Opus model spends 80% of its context scanning files for stale dates and mismatched names. A 1 USD/MTok Haiku or Gemini Flash model does that work equally well.

2. Self-grading. The agent implements a fix, then evaluates its own fix. It shares its own blind spots. If it misunderstood the requirement, it will misunderstand the check too.

3. Runaway loops. Without a hard stop, a revise cycle burns tokens indefinitely. I once watched 40 USD disappear on a loop thrashing between two conflicting fixes.

Flavor Neo addresses all three by separating who does what and who checks whom.

The shape of the loop

The leader spends premium tokens on judgment: planning, delegation, review, deciding what "done" means. Workers spend cheap tokens on volume: scanning files for contradictions, running searches, reporting what they find. The verifier matters more than any single worker. It is the one role that is not allowed to grade its own homework.

Why a different model family for the verifier

Same-model verification has correlated blind spots. If Claude misunderstands a requirement during implementation, Claude-as-verifier will likely misunderstand it the same way. Routing the verifier through GPT-5.4 (or Gemini, or whatever is available that is not the worker's model) breaks this correlation. The verifier gets only the goal contract and the candidate result, never the implementer's reasoning, so it evaluates the output, not the rationale.

The five rules

Rule 1: don't use the full loop by default

Before dispatching workers, I ask whether the task needs Flavor Neo at all:

• Under 30 minutes, one file, low ambiguity → just do it directly. No orchestration needed.

• Simple implementation but correctness matters → skip the workers, but still run an independent verifier as a one-off check.

• Need a specialist tool/model but no real parallelism → one worker plus a verifier, but this is lightweight delegation, not the full loop.

• 2-3 genuinely independent scan lanes, or the task is long-horizon and unattended → this is where Flavor Neo earns its keep. The full loop with 2-4 parallel workers, a collator, and a cross-model verifier.

The overhead of coordination (context duplication, summarization loss, merge conflicts) is real. If setting up lanes costs more than doing the task directly, don't fan out. Most requests don't need the full machine. About 70% of my daily work stays below the threshold. The full Flavor Neo loop triggers maybe twice a week.

Rule 2: every run starts with a goal contract

No contract, no verification target. Every dispatch states:

• The outcome: what specific thing should be true when this is done.

• The definition of done: not "looks good" but "file X contains Y, test Z passes, URL responds with status 200."

• The proof requirement: a test result, a URL, a screenshot, a diff. Something inspectable, not a claim. Workers cannot self-report success; they must produce an artifact the verifier can check.

• The brake: max rounds, and an instruction to report a blocker honestly rather than invent success.

This is the contract that flows through the entire loop. The verifier checks against the contract, not against vibes.

Rule 3: lanes tile the goal completely, with zero overlap

If two workers touch the same files, that's not parallelism; it's a merge conflict waiting to happen. Lanes are split by deliverable (an inspectable artifact with a done-state), never by activity like "research" or "improve," because only a deliverable can carry a proof requirement.

In practice, I split by domain:

• Architecture / Auth lane: scans API contracts, auth flows, schema mismatches across src/api/, src/auth/, openapi.yaml

• Dates / Ownership / Blockers lane: scans for stale dates, wrong owners, resolved blockers still listed in README.md, CODEOWNERS, project docs

• Docs / Tests / Staleness lane: scans for outdated docs, dead tests, claims that contradict code in docs/, test/, inline comments

Each worker gets its file list, its focus domain, and its output format. Workers report findings as structured items with severity (HIGH / MEDIUM / LOW). They do not fix anything; report only. The strong model does the fixing.

Rule 4: the verifier is a different model, fresh context, no self-grading

The single most important rule in the whole system. An implementer that also verifies its own work shares its own blind spots. The verifier gets:

• The original goal contract

• The candidate result (the actual files, post-fix)

• Nothing else: no worker reasoning, no collator notes

It returns exactly one of:

C-01: PASS
C-02: FAIL — date in README still says 2025, file shows 2026 in git log
C-03: PASS
...
NEW FINDINGS: Section 4.2 claims "no Docker required" but Dockerfile exists at root
VERDICT: REVISE (delta packet: fix C-02, address new finding)

The verifier's output format is rigid on purpose. PASS or REVISE with specific findings. "Looks mostly good" is a non-answer that lets problems through.

Rule 5: a hard brake, always

Feedback loops without damping oscillate or run forever. The skill specifies one hard constraint: ≤3 rounds. If the verifier returns REVISE three times, the loop stops and reports what's unresolved. Most runs converge in 1-2 rounds. After the brake trips, unresolved items go into the commit message's "Still open" section for human review; they are not silently dropped.

Without a brake, a revise loop is a machine that bills you silently while never converging. I learned this the expensive way.

A real example: documentation audit

Last month I ran Flavor Neo against a project with ~40 markdown files that had drifted over six months of active development. Architecture docs referenced services that had been renamed. Dates were wrong. The CODEOWNERS file listed people who had left.

Setup (Leader, ~2 minutes):

Split into 3 lanes. Gave each worker its file list and scan criteria. Total leader tokens: ~3,000.

Workers (parallel, ~4 minutes):

Three explore agents ran simultaneously on Claude Haiku 4.5 (1 USD/MTok input, 5 USD/MTok output). Each scanned its domain and reported structured findings:

• Worker A (architecture lane, 12 files): 8 issues found, ~4,000 input tokens + ~1,200 output tokens

• Worker B (dates/ownership lane, 15 files): 12 issues found, ~5,500 input + ~1,800 output

• Worker C (docs/tests lane, 13 files): 6 issues found, ~4,500 input + ~1,000 output

Representative findings:

FINDING: docs/architecture.md §3 says "auth-service handles login"
  but src/gateway-v2/routes/auth.ts has owned this since commit a3f2e1 (March)
  Severity: HIGH

FINDING: CODEOWNERS line 14 lists @cheneva for /src/pipeline/
  but cheneva's last commit was November; @ucalyptus owns this now
  Severity: HIGH

FINDING: README.md "Last updated: 2025-11" but 38 files changed since then
  Severity: MEDIUM

Total worker cost: ~14,000 input tokens × 1 USD/MTok + ~4,000 output tokens × 5 USD/MTok = ~0.03 USD.

Collation (Leader, ~3 minutes):

Deduplicated: 5 findings were flagged by multiple workers. Prioritized: 14 HIGH, 4 MEDIUM, 3 LOW. Applied fixes to all HIGH and MEDIUM items. Skipped LOW. Leader token cost at Opus rates (5 USD/MTok input, 25 USD/MTok output): ~4,000 input + ~3,000 output = ~0.10 USD.

Verification (GPT-5.4, ~5 minutes):

Verifier confirmed 16 fixes, caught 2 that introduced new inconsistencies (I had updated a service name in the architecture doc but missed a reference in the onboarding guide), and found 1 issue all three workers had missed. Returned REVISE with a 3-item delta packet. Verifier cost at GPT-5.4 rates (2.50 USD/MTok input, 15 USD/MTok output): ~6,000 input + ~1,500 output = ~0.04 USD.

Round 2 (~3 minutes):

Applied the delta. Verifier returned PASS. Round 2 cost: ~0.03 USD.

Total wall-clock: ~17 minutes. Total cost: ~0.20 USD.

For comparison, running the same task through a single Opus context window would require loading all 40 files (~200k tokens of corpus) plus prompts and output, costing roughly 1-2 USD at Opus rates. The single-model approach also risks missing cross-references between files that fall outside the active attention window, which is exactly what happened with the onboarding guide reference the verifier caught.

The actual savings depend heavily on the ratio of scannable grunt work to judgment-required decisions. In this case, most of the work was scanning (workers did it cheaply), and the leader only touched deduplication and fix application. On a task with less parallelizable scanning, the savings would be smaller.

How to trigger it

Drop SKILL.md into .claude/skills/flavor-neo/ (or wherever your agent reads skills from). Then say:

flavor-neo: audit all markdown files in docs/ for contradictions against the codebase

The leader (your strong model) reads the skill, plans the lanes, dispatches workers, collates, and sends to the verifier. You don't manually configure the loop; the skill protocol handles the structure. You provide the scope and the goal contract.

If you only have access to one model family (say, only Claude), the skill specifies that the verifier should be a different model family than the workers. In practice, if cross-family access is not available, running the verifier as a separate agent with completely fresh context (no shared conversation history) still catches many issues that self-review misses, though correlated blind spots remain a risk. The key constraint is that the verifier never sees the workers' reasoning.

Where this breaks

The happy path above is real, but so are these failure modes:

Workers producing garbage. Cheap models sometimes hallucinate contradictions that don't exist, especially when scanning code they don't fully understand. The collation step catches most of these (the leader is a strong model and can distinguish real findings from noise), but if all three workers hallucinate the same non-issue, it passes through. Mitigation: the verifier reads the actual files, not the worker reports, so fabricated findings get caught at the gate.

Verifier leniency. A verifier that rubber-stamps fixes is worse than no verifier at all, because it creates false confidence. This happens most often when the verifier's prompt is too loose or when the goal contract is vague. Mitigation: the goal contract must specify concrete, checkable criteria. "Improve the docs" is not a contract. "Every service name in docs/ matches the directory name in src/" is.

Lanes that aren't independent. If Worker A's lane overlaps with Worker B's, the collator has to reconcile conflicting findings about the same file. This is coordination overhead that the pattern is supposed to avoid. Mitigation: split lanes by deliverable, not by activity, and verify that file lists don't overlap before dispatching.

The strong model failing at collation. The leader is responsible for deduplication, prioritization, and fix quality. If the leader makes a bad fix, the verifier should catch it, but if both the leader and verifier miss the same thing, it ships broken. This is the residual risk the pattern cannot eliminate. It reduces the probability but does not reach zero.

The commit trail

Every Flavor Neo run produces a commit with a structured message:

flavor-neo batch 1: resolve 18 contradictions (C-01–C-18), GPT-5.4 verified

3 worker lanes (Claude Haiku: arch, dates, docs) + collator (Opus 4.6) + verifier (GPT-5.4).
Round 1: 16 fixes passed, verifier caught 2 + 1 new → REVISE.
Round 2: all 3 delta fixes passed → PASS.

Resolved:
- C-01: docs/architecture.md — updated service name from old-api to gateway-v2
- C-02: README.md — corrected last-updated date from 2025-11 to 2026-06
...

Still open: C-19 (LOW) — comment in test/helpers.ts references removed fixture

This matters for auditability. Six months from now, someone looking at git log can see exactly what was checked, by which models, how many rounds it took, and what was left open.

What this actually buys you

• Token cost drops significantly because only the leader (the expensive model) touches every decision; the bulk of scanning happens on models that cost 5x less per token. The exact savings depend on how much of the task is parallelizable scanning vs. judgment. In the documentation audit above, the ratio was roughly 10:1 in favor of scanning, so the Flavor Neo run cost ~0.20 USD versus ~1-2 USD for a single Opus pass. On a task that is mostly judgment, the savings would be smaller.

• Correctness goes up because nothing ships on the implementer's say-so. An independent, fresh-context check gates every result. Cross-model verification catches correlated blind spots that same-model self-review misses.

• It scales to long, unattended work because the brake and the goal contract mean it can run without you watching, and it fails loudly instead of quietly drifting.

• The audit trail is automatic. Every run documents what was found, what was fixed, what was verified, and what remains open.

When not to use it

• Under 3 files. Just do it yourself. The coordination overhead exceeds the benefit.

• Code changes with test suites. Tests are the verifier. You don't need a second verification loop when npm test already gates correctness.

• Simple single-file edits. A leader-only pass is fine. Don't bring the full loop to fix a typo.

The ideas behind it

None of this is exotic. It's five well-known principles applied to agents instead of teams and machines:

1. Brooks's Law: adding people to a late project makes it later, because communication overhead grows faster than throughput. The same principle applies here: more workers means more coordination, so cap them at 2-4 where diminishing returns set in.

2. NASA WBS 100% rule: child scopes must sum to exactly the parent, no gaps, no overlap. That's Rule 3.

3. Verification ≠ validation (systems engineering): verification asks "did we build it right?" (does the output meet the spec); validation asks "did we build the right thing?" (does it meet the user's actual need). In this loop, unit tests handle basic verification. The independent verifier checking against the goal contract adds a second layer of verification from a fresh perspective, catching specification-level errors the implementer's own tests miss. That's Rule 4.

4. Donella Meadows on balancing feedback: in Thinking in Systems, Meadows shows that feedback loops with delays and insufficient damping overshoot or oscillate. The hard brake (≤3 rounds) is the damping mechanism. Without it, the revise loop can thrash indefinitely between competing fixes. That's Rule 5.

5. The "Fable as leader" pattern (Machina / @EXM7777): use the strongest available model as orchestrator, not as laborer. The insight that leadership and labor want different cost profiles.

The skill is open source, a single markdown file you drop into .claude/skills/:

https://github.com/ucalyptus/skills

There's a lesson that keeps getting relearned by every generation of SaaS founders who decide to go upmarket. It's not technical. It's not about product-market fit. It's about something far more uncomfortable: enterprises are built for legibility, not velocity. And that's a feature, not a bug.

If you're building AI-native software and you think the hardest part is the model, the UX, or the demo — you haven't started yet.

Legibility Is the Product

When a Fortune 500 insurer processes a claim, seventeen people might touch it. Not because they're inefficient. Because each touchpoint is an audit surface. Every handoff is a control. Every approval is a record that a regulator can subpoena in three years when something goes wrong.

Large organizations in financial services, insurance, and compliance don't move slowly because they're stupid. They move slowly because they've been burned — by regulators, by lawsuits, by operational failures that cost billions. The bureaucracy isn't rot. It's scar tissue. It's institutional memory encoded in process.

When you walk into a room and pitch your beautiful AI workflow that "eliminates manual steps," you're not offering liberation. You're offering illegibility. You're asking them to collapse audit trails. To trust a black box. To explain to their regulator why a model made a decision that used to have a human's signature on it.

And they will smile politely and never call you back.

"We Have SOC 2 Type II" Is Not a Sentence

I need founders to hear this clearly: saying "we have SOC 2 Type II" to a Chief Risk Officer at a top-10 bank is like saying "I have a driver's license" when someone asks if you can fly a 747.

SOC 2 is table stakes. It's the minimum evidence that you're not storing passwords in plaintext. It tells an enterprise nothing about:

• How your model handles PII in inference

• What happens when your vendor (the model provider) changes their data retention policy

• Whether your system can produce an explainable decision trail for a regulator

• How you handle cross-border data residency

• What your incident response looks like when (not if) something goes sideways

• Whether you've even heard of their specific regulatory framework (OSFI, MAS, PRA, NYDFS, DORA — pick your acronym)

The naiveté isn't charming. It signals that you haven't done the work to understand their world. And in their world, a vendor who doesn't understand compliance is a vendor who will create compliance problems.

The Timeline Is the Territory

Here's what actually happens when an enterprise says "we're interested":

Quarter 1: Discovery & Evaluation. Your champion (a mid-level director who saw your demo at a conference) is socializing the idea internally. You're not in a sales cycle. You're in a political cycle. They're figuring out who needs to say yes, who might say no, and what existing vendor relationships you'd threaten.

Quarter 2: Proof of Concept. You've been invited to a POC. Congratulations. This is not a sale. This is a test — and not just of your product. They're testing whether you can operate inside their constraints. Can you work in their environment? Can you use their identity provider? Can you handle their data classification requirements? Can your team sit through a two-hour architecture review without getting defensive?

Quarter 3: Security & Information Risk. Now the real fun begins. Their InfoSec team will send you a 400-question security questionnaire. Their third-party risk team will want to understand your supply chain. Their data privacy office will want a Data Protection Impact Assessment. If you're using a foundation model, they'll want to know the model provider's data handling, training data provenance, and subprocessor list. This is not bureaucratic theater. This is them doing their job.

Quarter 4: Procurement & Legal. You've passed security review. Now legal wants to negotiate the MSA. They'll redline your limitation of liability. They'll want uncapped indemnity for data breaches. They'll want audit rights. They'll want termination for convenience. Your "standard contract" will be unrecognizable by the time it's signed.

Quarter 5+: Production (Maybe). If everything goes well — if your champion hasn't left the company, if their budget hasn't been frozen, if a regulator hasn't issued new guidance that changes the risk calculus — you might go live.

That's 12-15 months from first conversation to first dollar of ARR. Often longer.

The Mental Preparation Nobody Talks About

The bitter lesson isn't that enterprise sales is slow. Every blog post tells you that. The bitter lesson is what it does to your team when you're not prepared for it.

Your engineers will be demoralized. They built something extraordinary, and it's sitting in a staging environment for six months while a compliance team reviews documentation. Your salespeople will churn. They came from PLG companies where closed-won meant a credit card form. Your board will get nervous. "Why isn't the pipeline converting?"

You need to prepare your team psychologically for this:

Hire people who've done it before. Not "enterprise sales experience" on a resume. People who've personally sat through a regulator meeting. People who know what a TPRM framework looks like from the inside.

Build compliance into the product, not around it. Explainability, audit logging, data lineage, role-based access with granular permissions — these aren't features. They're prerequisites.

Respect the process. When an enterprise asks you to fill out their security questionnaire for the third time because a different business unit is evaluating you, don't complain. Fill it out. Again. Cheerfully. This is the game.

Budget for it. Enterprise sales isn't just expensive in time. It's expensive in people. Solutions engineers, security architects, GRC analysts, legal counsel — these are the roles that close enterprise deals, not SDRs.

The Founders Who Win

The founders who actually succeed in enterprise AI aren't the ones with the best models. They're the ones who internalized something deeply uncomfortable:

The customer's caution is rational.

They're not Luddites. They're not "behind." They're managing risk across thousands of employees, millions of customers, and regulatory bodies that can shut them down. When they ask you to jump through hoops, they're not being difficult. They're asking you to prove that you won't be the reason they end up in a consent order.

The startups that win are the ones that stop treating enterprise requirements as obstacles and start treating them as the product specification. The compliance framework isn't in the way — it is the way.

The Actual Bitter Lesson

Rich Sutton's original "Bitter Lesson" in AI research was about how general methods that leverage computation always beat clever hand-engineered approaches. The analogy for enterprise sales is this:

There is no shortcut that leverages your way past institutional trust-building.

No amount of product brilliance, no demo magic, no "AI-native" pixie dust will compress a regulated enterprise's evaluation process. You cannot growth-hack your way into a bank's production environment. You cannot "move fast and break things" when the things you'd break are protected by federal law.

The founders who accept this — who build their companies, their cultures, and their products around this reality from day one — are the ones who'll still be here in five years, running the infrastructure that actually powers enterprise AI.

Everyone else will pivot to SMB, sell to their board that "enterprise wasn't the right fit," and wonder why the market didn't reward their technical brilliance.

The market did reward technical brilliance. It just also required patience, humility, and respect for the systems that keep the world running.

That's the bitter lesson. Swallow it early.

This isn't a loose analogy. It's the same cybernetic architecture — one that predates both K8s and software entirely.

The Pattern Is 250 Years Old

In 1788, James Watt attached a centrifugal governor to a steam engine. Before this, workers manually adjusted valves to stabilize engine speed. The governor changed the job: sense rotational speed, compare to target, adjust the valve, repeat. The worker's role shifted from turning valves to designing and tuning the governor.

In 1948, Norbert Wiener formalized this as cybernetics: any system that senses, compares against a reference signal, and acts to reduce error. Biological, mechanical, or computational — the structure is identical.

In 2014, Kubernetes applied it to infrastructure: declare desired state in YAML, let controllers continuously reconcile reality toward it. The engineer's role shifted from restarting services to writing manifests.

In 2025, the AI community applies it to cognition: declare a goal condition, let an agent loop sense, diff, and act until converged. The engineer's role shifts from writing code to designing the harness.

Each era raises the abstraction. The pattern never changes.

The K8s Reconciliation Pattern

Every Kubernetes controller follows this cycle:

1. Desired State — a YAML manifest declaring what should exist

2. Observe — the controller watches current state via the API server

3. Diff — compare desired vs actual

4. Act — apply the minimal change to close the gap

5. Repeat — re-enter the loop, forever

The controller never executes a procedural script. It doesn't know "how" to get from A to B in advance. It only knows how to make the next incremental correction. If a pod dies, the loop notices and recreates it. If config drifts, the loop notices and reconciles. Self-healing is not a feature — it's an emergent property of the loop structure.

There's a reason some of the best AI coders doing agent-loop work since early 2025 are former Kubernetes engineers. They've internalized this pattern at the infrastructure level and simply moved it up a layer.

Loop Engineering Is the Same Architecture

Replace "pod" with "codebase" and "YAML manifest" with "user intent":

1. Goal — a task spec, acceptance criteria, or desired end-state

2. Sense — read the environment (code, tests, logs, browser state)

3. Plan — diff the goal against current reality

4. Execute — make one incremental change (tool call, code edit, test run)

5. Repeat — re-enter the loop until the goal condition is met

The agent doesn't follow a pre-planned script. It continuously reconciles toward the goal. If a test fails, the loop notices and fixes. If the environment changes, the loop adapts. Self-healing emerges from the structure, not from special-case error handling.

Design Principles That Transfer

When you see Loop Engineering as K8s-style orchestration, a decade of production learnings transfers directly:

Declarative over imperative. K8s succeeded because operators declare what, not how. The best agent loops work the same way — "all tests pass and the feature matches this spec" beats a 47-step procedural checklist.

Idempotent actions. K8s controllers can safely re-run without side effects. Agent loops need the same property. This is why "edit this file to contain X" works better than "append Y to this file" — the first is convergent, the second drifts on re-execution.

Observe before acting. K8s never assumes state — it reads current reality first. Agents that skip observation (generating code without reading what exists) produce drift, conflicts, and wasted work.

Small diffs, high frequency. A K8s controller doesn't rewrite the entire cluster on each pass. It makes the minimum viable change. Agent loops that generate entire files in one shot are the equivalent of kubectl delete all && kubectl apply — technically works, but fragile and wasteful.

Crash-only design. K8s controllers are designed to be killed and restarted at any point. The loop picks up where it left off because state is externalized. The Fresh-Clone Loop embodies this: clone the repo into a clean environment, follow the README exactly, record every gap, fix the docs, discard the environment, start from a fresh clone. Repeat until a brand-new environment goes from clone to running in one uninterrupted pass. The agent is disposable; the externalized state (the repo, the README) is what persists.

The Control Plane Layer

K8s didn't just give us controllers — it gave us a control plane: etcd for state, the API server for observation, RBAC for access control, admission webhooks for policy gates.

The equivalent is emerging in agent infrastructure:

• etcd → context stores, memory systems, task boards, filesystem

• API Server → tool interfaces, MCP servers, browser harnesses

• Controllers → agent loops, reconciliation cycles

• Admission Webhooks → human-in-the-loop gates, approval flows before merge

• RBAC → sandboxed execution, permission scoping, scoped file access

• CRDs → custom tool schemas, skill definitions

• Scheduler → router/orchestrator deciding which model handles which subtask

When someone builds a "software factory" with a taskboard feeding agent loops, they're building a control plane. When someone adds approval gates before an agent can merge code, they're implementing admission webhooks. When someone uses Infrastructure-as-Actors with durable objects managing sandboxes — single-writer, stateful, durable, with scheduled alarms for teardown — they're building the K8s scheduler for AI workloads.

The vocabulary is different. The architecture is identical.

What K8s Teaches Us About Failure Modes

K8s has 10+ years of production learnings about what goes wrong in reconciliation loops. Every one of these failure modes is now appearing in agent loops:

Thrashing. A controller that makes changes too aggressively can oscillate — create 10 pods, kill 8, create 6, kill 4. Agent loops that over-correct on each iteration exhibit the same behavior: fix a test, break two others, fix those, break the original. The fix in both cases: dampen the response, make smaller changes, add backoff.

Stale reads. If the controller reads cached state instead of current state, it acts on lies. Agents that rely on stale context windows (long conversations with outdated code state) make the same mistake. The fix: fresh-clone loops. Start from a clean read of reality every iteration. Don't trust memory.

Infinite reconciliation. Sometimes the desired state is unreachable. A K8s controller trying to schedule a pod with 1TB RAM on 64GB nodes will loop forever. Agent loops given impossible specs do the same. The fix: maximum iteration counts, cost budgets, human-in-the-loop circuit breakers, and "this goal is unreachable" detection.

Conflicting controllers. Two K8s controllers fighting over the same resource create chaos. Two agent loops editing the same file without coordination produce merge conflicts and reverted fixes. The fix: clear ownership boundaries. K8s uses labels and selectors. Agent infrastructure needs task-scoping — one loop owns one bounded resource set.

Unobservable state. A controller that can't read a resource's actual state can't reconcile it. Agents without proper sensing (no test feedback, no browser state, no lint output) can't converge either. The fix: invest in observability before investing in more sophisticated planning.

The Hill Climbing Machine

The pattern extends one level up. Individual agent loops reconcile code toward a goal. But who reconciles the agent itself toward better performance?

The answer is the same architecture, applied recursively:

1. Encode domain expertise into a v0 agent and ship it

2. Observe agent behavior — collect traces at scale

3. Mine traces for failure patterns

4. Turn failures into evals you can hill-climb against

5. Improve the agent (harness engineering → fine-tuning → more harness engineering)

6. Repeat

This is the outer reconciliation loop. The "desired state" is agent performance on your eval suite. The "observation" is trace data. The "action" is agent improvement. It's K8s all the way up.

The Practitioner's Takeaway

If you're building agent loops today:

• Study K8s controller patterns. The controller-runtime source is a masterclass in reconciliation design.

• Make your loops declarative. Define the exit condition, not the path.

• Externalize state aggressively. The agent should be killable and restartable without losing progress.

• Add circuit breakers. Maximum iterations, cost budgets, human gates.

• Keep actions idempotent. If the loop re-executes a step, nothing should break.

• Observe before every action. Never assume the world hasn't changed since your last read.

• Dampen corrections. One small change per iteration. Verify before making the next.

• Scope ownership. One loop, one bounded resource set. No shared-state free-for-alls.

Loop Engineering isn't new. It's Kubernetes for cognition. It's Wiener's cybernetics for code. It's Watt's governor for software systems. The sooner we recognize that, the sooner we stop reinventing the same failure modes that infrastructure engineers — and mechanical engineers before them — solved decades ago.

Resources

• Harness Engineering Is Cybernetics — @odysseus0z

• From Executor to Steersman — Zach's Compass

• Kubernetes Design Principles

• Beyond the Prompt: What Loop Engineering Actually Means

• Control Theory and Feedback Loops for Self-Healing Systems

• Loop Library — Curated Agent Loops

• Infrastructure-as-Actors

• The Fresh-Clone Loop

• Building the Hill Climbing Machine

The entire agent evaluation loop is now automatable. Production traces can be collected automatically. Failure clusters can be detected automatically. Representative datasets can be generated automatically. Evals can be run automatically. Candidate changes can be shipped automatically when the metrics are green.

That is the dream version of continual improvement. It is also how agent slop gets manufactured at scale.

Not obviously broken output. Worse: subtly misaligned output. Agents optimizing against an evaluation suite that was correct six weeks ago, before the product changed, before users found new paths through the system, before your own opinion about quality sharpened.

The dashboards stay green. The agent feels worse. Users notice before the eval suite does.

The Loop Can Be Automated. The Target Cannot.

Rich Sutton's bitter lesson was that general methods plus compute beat hand-built human knowledge again and again.

There is a related, inverted lesson forming in agent engineering:

You can automate the labor of evaluation. You cannot safely automate the judgment that defines what the evaluation should mean.

The distinction matters because most agent teams eventually build a loop like this:

• collect production traces

• sample or cluster failures

• generate or refresh datasets

• run evals

• propose prompt, tool, or routing changes

• deploy when the suite improves

Each step is real work. Each step is increasingly easy to give to agents. That part is good.

The failure mode appears when the loop is treated as closed. Once nobody is reading traces with taste and domain context, the system starts optimizing against a stale picture of the world.

A Concrete Failure Pattern

Imagine a commercial insurance carrier using an AI triage agent for first-notice-of-loss intake. The agent reads incoming claim documents, extracts facts, classifies complexity, and routes each claim to the right adjuster pool.

The team has done the responsible engineering work. Traces flow into observability. A CI pipeline runs evals against thousands of historical claims. Routing accuracy stays above 90 percent. Leadership sees a stable metric and moves on.

Then the team closes the loop. Every week an orchestration agent pulls traces, flags low-confidence cases, generates synthetic augmentation data, refreshes the eval suite, and deploys changes when accuracy holds.

For months, nothing looks wrong.

The problem is not in the automation. The automation does exactly what it was asked to do. The problem is that the business changed. The carrier expanded into new states. The new jurisdictions had different litigation patterns, claim timelines, and compliance obligations. The original eval dataset did not represent those cases. The loop kept improving performance against yesterday's understanding.

In production, the agent quietly over-routes a class of coastal commercial property claims to a general adjusting pool instead of a specialist pool. The claims still move. Nothing explodes. But reserves get worse, letters go out inconsistently, and compliance risk accumulates.

No eval catches it because no eval was measuring it.

That is the dangerous version of agent slop: not a hallucination in a demo, but a system that becomes increasingly confident while drifting away from the work it is supposed to do.

Your Mental Model Is The Target Function

Your evaluation suite is not the source of truth. It is a frozen artifact of what you believed mattered when you wrote it.

Your actual target function is your evolving mental model of quality:

• what a good answer feels like

• which failure modes matter now

• where users are getting confused

• which edge cases became common

• which risks are no longer theoretical

• what your product is becoming

That model is contextual. It changes. It includes taste, judgment, domain knowledge, product direction, and the weird texture of real user behavior.

The moment you stop reading traces, you stop updating the model. The agent keeps optimizing anyway.

This is how teams end up measuring the wrong thing with impressive precision.

What Should Stay Human

Most of the loop should go to agents. The mechanical work is exactly where agents are useful.

Give agents dataset generation. Give them eval execution. Give them known failure-mode detection. Give them report generation. Give them deployment proposals when metrics improve.

But keep these steps human:

• read a random sample of production traces

• form opinions about what good looks like

• leave feedback in language the system can learn from

• update eval criteria as the product, users, and risks change

Do not only inspect the traces your evals flagged. Those are the failures you already knew how to ask for. Random sampling is how you discover the failures your current vocabulary cannot yet name.

The Cheapest Control Is Taste

The practical fix is usually smaller than teams expect.

Put a domain expert, product owner, or senior engineer on a weekly trace-review rotation. Sample a small number of production runs. Read them directly. Mark what feels wrong. Write down why. Turn recurring observations into eval criteria.

That may be two hours a week. It may become 45 minutes once the system learns what kinds of ambiguity to surface.

The point is not manual review as bureaucracy. The point is manual review as gradient.

Your feedback is the signal the rest of the loop needs. Without it, agents can still generate datasets, run evals, and ship changes. They just do it against an increasingly stale definition of quality.

As the easy failures get automated away, what remains is opinionated nuance. That is where the product becomes yours. Two teams can use the same model, the same tracing stack, the same eval runner, and the same deployment pipeline. The difference is the quality of the human taste being injected into the loop.

The Operating Principle

Automate the labor. Do not automate the judgment.

That means:

• agents can execute the loop

• humans must revise the target

• evals should be treated as living product artifacts

• trace review should be scheduled work, not emergency work

• green metrics should never replace direct contact with production behavior

The more precisely you can articulate what good looks like, the more safely you can hand off. But the act of deciding what good looks like remains load-bearing.

You are the eval. Your taste, your domain judgment, your reading of real traces, and your willingness to update the suite are the target function.

If you automate that away, the optimization has nowhere good to go.

The fast loop belongs to the agent: observe, decide, act, verify, repeat. The slower loop belongs to the human: set intent, define guardrails, evaluate outcomes, and change the system when the pattern of results says it should change.

When people say "the agent did the work," what they usually mean is that the agent handled the short-cycle execution path. The human still shaped the mission, constrained the operating envelope, and decided whether the results were acceptable. That is not removal. That is promotion.

The old mental model was simple: human thinks, human acts, software assists.

The emerging model is different: human defines the frame, agent runs inside it, and software records enough state for the system to be inspectable and correctable.

That changes the role of human judgment. It matters less in the moment-to-moment mechanics of execution and more in the structure around execution.

The new human job is not to click faster than the machine. It is to decide:

• What outcome actually matters

• What constraints cannot be violated

• What tradeoffs are acceptable

• What signals mean the system is drifting

• When the frame itself needs to change

This is a more strategic role, but it is also a more demanding one. A human in the strategic loop cannot hide behind activity. The system exposes whether the goals, thresholds, and review criteria were well chosen.

That is why the future of agents is not just about autonomy. It is about governance with shorter feedback cycles.

An agent can iterate through a task dozens or hundreds of times faster than a person. It can retry, branch, compare, and recover without waiting for a human to press the next button. But that speed is only useful if someone has defined the objective and the stopping conditions well enough for the fast loop to stay productive.

Without that frame, autonomy degrades into expensive motion. The agent looks busy. Logs fill up. Tools fire. Tokens burn. But the system is not converging on anything worth keeping.

The practical implication is that good human operators will start to look less like operators and more like system designers.

They will spend more time on:

• Intent definition

• Policy and permissions

• Success metrics

• Review checkpoints

• Exception routing

• Post-run diagnosis

And less time on:

• Manual execution

• Repetitive coordination

• Low-level tool invocation

• Short-horizon status checking

This is not a hand-wavy organizational shift. It maps cleanly to the engineering structure of an agent system.

At the execution layer, the agent runs a tight control loop. It reads the current state, chooses an action, calls a tool, checks the result, updates the state, and continues.

Above that sits a control plane shaped by humans: goals, policies, task boundaries, escalation rules, budgets, and evaluation criteria.

Above that sits an even slower loop: review the outcomes across many runs, identify patterns of failure or waste, and redesign the frame.

In other words, humans move from the transaction layer to the architecture layer.

That does not make humans less important. It makes their mistakes more consequential and their judgment more leveraged.

A weak worker can be corrected one task at a time. A weak frame gets multiplied across every autonomous run.

That is the real shift agents introduce. They amplify not only execution, but also whatever upstream judgment shaped the execution.

The organizations that benefit most from agents will not be the ones that simply automate the most steps. They will be the ones that become good at designing loops.

They will know how to:

• Give agents clear intent without over-specifying every action

• Set hard guardrails without freezing the system

• Review outputs at the right altitude

• Turn repeated failures into better policies

• Decide when a human must step back into the fast loop

This also explains why "human in the loop" is now too vague a phrase. The real question is: which loop?

Are humans approving every action in the fast path? Are they only reviewing samples after execution? Are they setting policy weekly and auditing exceptions daily? Are they redesigning the workflow monthly based on aggregate outcomes?

Those are very different systems, and they produce very different economics.

The companies that win with agents will treat loop design as a core capability. They will explicitly decide what runs at machine speed, what gets escalated, what gets logged, what gets measured, and what gets changed when the results stop matching intent.

That is the future of agents.

Not humans removed from the loop.

Humans promoted to a slower, higher-leverage one.

The Problem With Conventional Agent Frameworks

Most agent frameworks are LLM-first: the conversation loop is the core, tools are attached to it, rules layer on top, and logging is bolted on at the end for observability. State is persisted as retrievable "memory."

Figure 1. Conventional LLM-centric stacks treat logging as a secondary concern. In a log-centric design, the append-only event log becomes the source of truth, the working graph is derived from it, and the LLM is just one behavior among many.

Consequences of this design:

• State is opaque: working memory lives in conversation context, not a queryable store.

• Replaying a run is hard: there is no source of truth; logs are side-effects, not the primary record.

• Forking is expensive: you must re-run from scratch because there is no checkpoint mechanism.

• Coordination between agents is explicit: agents pass messages directly instead of reacting to a shared world state.

• Observability is retroactive: tracing gets added after bugs appear.

The Core Inversion: Log-First Architecture

The core move is to invert the stack. Make the append-only event log the source of truth. Everything else — the working graph, the agent's beliefs, and task state — becomes a deterministic projection of that log.

The architecture shift is simple but profound: instead of treating the LLM loop as the runtime and storage as a helper, the event log becomes the runtime backbone. The live graph is derived from that log, and behaviors operate against the graph rather than owning the whole system.

The log is not observability. The log IS the agent.

A Concrete Runtime Shape

To make the idea concrete, think in terms of a runtime with three layers: authoritative events, projected graph state, and reactive behaviors.

Three-Layer Model

Figure 2. A three-layer runtime separates source-of-truth events, projected state, and reactive behaviors. Relation behaviors are first-class, and every behavior writes back to the shared event log.

Relation Behaviors — Edges as First-Class Citizens

Traditional agent graphs put all logic in nodes. A stronger design puts semantic logic on edges:

• A task can depend_on another task

• A task or belief can contradict an existing belief

• Evidence can support or weaken a claim

• Blocking, dependency, and contradiction logic live in the relationship itself

When the contradicts relation fires, a behavior automatically triggers — no explicit coordination code needed.

What the Log Enables: Four Superpowers

Deterministic Replay

Figure 3. Once execution history is stored as events, replay becomes deterministic and branch exploration becomes cheap: reuse the shared prefix, pay only for divergent work, and compare outcomes directly.

Any run can be reconstructed exactly from its log. This means:

• Full audit trail from goal to individual model call

• Debugging without re-running expensive LLM calls

• Compliance/reproducibility for regulated domains

Efficient Forking

Fork at any historical event. The shared prefix replays from cache — no redundant LLM calls for the common history. Branch, diff, compare outputs.

Complete Causal Lineage

Every event knows what caused it:

Goal: "Analyze market"
  └─ spawned Task: "Research competitors"
       └─ LLM call: "List top 5 competitors"
            └─ Tool call: web_search("competitor analysis 2026")
                 └─ Evidence node: search result
                      └─ Behavior: "contradicts" edge fired
                           └─ LLM call: "Revise belief about market stability"

No more "why did the agent do that?" — the log has full provenance.

Implicit Coordination (No A2A Protocol Needed)

Agents coordinate by reacting to the graph, not by calling each other. No explicit A2A protocol. No DAG. No workflow engine.

Deep Dive — Runtime Design Details

The Event Schema

Every event in the log carries:

• `id`:id:** unique identifier.

• `type`:type:** event category such as task.created, llm.response, or tool.called.

• `payload`:payload:** event-specific data.

• `actor`:actor:** which behavior emitted the event.

• `causality`:causality:** the triggering event or events.

• `timestamp`:timestamp:** wall-clock time of emission.

The causality chain is what makes full lineage possible — every event knows its parent.

Handling LLM Non-Determinism

The hardest problem in event-sourced agents is that LLMs are not pure functions. The practical answer is content-addressed caching keyed on a normalized hash of the request such as system message, model parameters, and output schema. On replay, a matching hash serves the cached response from the log instead of making a fresh API call.

Two replay modes:

• Permissive — serves cached responses, allows new calls for edited prompts (default)

• Strict — validates byte-for-byte reproducibility; flags the first offending event if anything diverges

Forking Economics

The cost savings from cached prefix replay are concrete:

200-step run, fork at step 150:
  ├─ Steps 0–149  →  replay from cache  →  $0, ~0ms per step
  └─ Steps 150–200 →  execute normally  →  full cost

Without forking: pay 200 steps × N variants
With forking:    pay 50 steps × N variants + 150 steps once

Fork lineage is verified by literal shared event IDs — not a copy, the same events. The structural diff then shows exactly which graph objects, relations, and patches differ between branches.

The Determinism Contract

Behaviors must not:

• Call random() directly (use event-record entropy instead)

• Read wall-clock time directly (use event timestamp)

• Generate fresh UUIDs (obtain from event records)

• Perform I/O outside framework primitives

• Depend on mutable global state

The runtime does not statically enforce this. Violations surface at replay as a divergence error pinned to the first mismatched event. For LLM calls, the contract applies to replay (via cache), not initial execution.

What a Real Run Looks Like

In a realistic agent workflow, the event stream gets large quickly. A run can look like this:

671 events
 93 objects (3 companies, 24 questions, 25 claims, 3 memos)
 76 relations
103 model calls
 48 tool calls
  0 lines of orchestration code

Coordination was entirely reactive — no explicit scheduling, no DAG, no workflow engine.

Lineage is the deliverable — every claim links back to its behavior, triggering event, and specific model request. The causal chain from goal to output is reconstructable from the log alone.

Comparison to Memory-Layer Approaches

Memory-layer systems treat memory as a derived layer atop the agent. A log-centric design inverts that relationship: the log is primary, and memory in the form of a graph is a projection of it. That means you can reconstruct any past memory state, not just the current one.

Known Limitations

This architecture still has real limitations:

• Reactive cascade risk: behaviors can trigger loops, so runs need budgets such as event caps, cost limits, and recursion depth.

• Log scaling: replay cost is linear, and the system does not yet have mature compaction or checkpointing for million-event runs.

• Schema evolution: graph schema changes require migration tooling.

• Side-effect replay: external writes and emails happen only on first run; only responses replay cleanly.

• No empirical benchmarks: the contribution is architectural rather than a head-to-head performance comparison.

Where Akka Fits — Filling the Production Gap

If you want to take this architecture into production, Akka is one of the clearest reference points for the surrounding infrastructure. It gives you battle-tested event sourcing, clustering, sharding, projections, and recovery semantics without asking you to invent those mechanisms from scratch.

Conceptual Mapping

The easiest way to read the mapping is this: the agent runtime contributes the agent-native abstractions, while Akka contributes the production-grade event-sourcing primitives. The concepts line up surprisingly cleanly.

What Akka Brings That A Minimal Agent Runtime Doesn't Have Yet

• Log scaling: Akka Persistence snapshots let you snapshot state every N events and replay only the tail.

• Schema evolution: Akka Persistence event adapters can transform older event formats on read.

• Distributed multi-agent coordination: Akka Cluster Sharding routes behaviors to stable nodes and keeps state alive across failures.

• External side-effect deduplication: Akka's at-least-once delivery model pairs with idempotent receivers so tools do not get re-executed incorrectly on recovery.

• Backpressure in reactive cascades: Akka Streams gives you typed, backpressured pipelines so behaviors cannot flood one another.

• Observability and metrics: Akka Telemetry provides message-rate, mailbox, and journal-latency visibility out of the box.

How a Production Log-Centric Agent Would Look With Akka

At production scale, you would expect persistent behavior actors writing to an event journal, projections building graph views from that journal, snapshots reducing replay cost, and a dedicated layer wrapping LLM calls as event-producing behaviors.

The Shared Philosophical DNA

Both this style of agent runtime and Akka are grounded in the same insight that the distributed systems community learned in the 2010s:

Mutable shared state is the enemy. Immutable events are the foundation.

Akka's EventSourcedBehavior (in Akka Typed) is nearly a direct implementation of a behavior attached to a projected log:

// Akka EventSourcedBehavior ≈ log-projected behavior
EventSourcedBehavior[Command, Event, State](
  persistenceId = PersistenceId("agent", agentId),
  emptyState   = AgentState.empty,
  commandHandler = (state, cmd) => Effect.persist(toEvent(cmd)),
  eventHandler   = (state, evt) => state.applyEvent(evt)   // deterministic projection
)

The difference is scope. Akka is a general distributed computing toolkit. An LLM-native runtime adds content-addressed LLM caching, a causality-aware event schema, fork-and-diff workflows, and graph projections typed around agent concepts such as beliefs, tasks, evidence, and relations.

When to Use Each

The practical split is straightforward:

• Use a purpose-built log-centric agent runtime when you want the architecture directly, fast iteration, and agent-native concepts out of the box.

• Use Akka when you already live in the JVM world and need battle-tested event sourcing, clustering, sharding, snapshots, and durable distributed runtime behavior.

• Use a hybrid approach when you want the causal and replay model of a log-centric agent runtime but need the surrounding production infrastructure that mature event-sourcing stacks already solved.

Log-Centric vs Conventional — Side-by-Side

The core difference is not UI, prompt style, or framework ergonomics. It is where truth lives and what you can do once that truth is durable.

• Source of truth: conventional systems rely on LLM conversation context; log-centric systems rely on an append-only event log.

• State: conventional state is opaque and in-context; log-centric state is explicit and queryable through the graph.

• Replay: conventional systems re-run from scratch; log-centric systems replay deterministically from the log.

• Forking: conventional branching is expensive; log-centric branching is cheap because the shared prefix is cached.

• Agent coordination: conventional systems depend on explicit messages or A2A protocols; log-centric systems coordinate implicitly through graph reactivity.

• Observability: conventional observability is bolted on; in log-centric systems the record is intrinsic.

• LLM role: conventional systems make the LLM the loop; log-centric systems treat it as one behavior among many.

• Long-running agents: conventional systems hit context-window limits; log-centric systems persist durable graph state over time.

Where This Fits in the Agent Landscape

This model sits between classic orchestration frameworks and full distributed-systems thinking. It borrows from event sourcing, stream processing, Redux-style projection, and actor systems, then applies those ideas to LLM agents in a way most agent stacks still do not.

Resources

• arXiv paper

• Active Graph site

• Active Graph GitHub

• Active Graph docs

• Tweet — paper announcement

• Tweet — better visualization

• Active Graph open source tweet

• Akka Persistence docs

• Akka Projection docs

Most teams I talk to treat evals like a test suite. You build the agent, write some tests, watch it pass them, ship it. Done.

That model is wrong — and it's why so many agents plateau after the first sprint.

Evals are not a scoreboard. They're a training signal. Your agent will hill-climb on whatever you give it to optimize for. If your evals don't encode the real behavior you want — not the proxy, not the offline approximation, the actual thing your users need — you'll ship an agent that passes your tests and fails your users.

Here's what I've learned building agents in production, and what the field is converging on in 2026.

1. Evals are the substrate

"Evals are the substrate that determines what your agent does in production. They're training data for agents because we literally fit our agent to pass Evals via hill-climbing algorithms and human edits to pass failure modes. Once you get your agent into users hands, the eval generation loop compounds."

The word "substrate" is doing real work here. Not tests. Not benchmarks. Substrate — the medium in which something grows.

Your agent grows to fit its evals. Every prompt edit, every tool tweak, every model swap — you measure the result against your evals and keep what passes. That's hill-climbing. It's happening whether you've named it or not.

The question isn't whether you're hill-climbing. It's whether you're climbing the right hill.

I've built systems where the eval was a holdout dataset — historical labels, frozen in time. We got very good at predicting the past. The metric looked great. Production results were flat. The hill we climbed wasn't the hill that mattered.

The gap between "passes offline evals" and "works in production" is exactly the gap between the frozen labels you have and the live behavior you actually care about. A user who asks a follow-up because the first answer was wrong — that's signal. A document that fails to extract correctly — signal. A recommendation that gets ignored — signal. None of those were in my eval set at first.

The moment you wire those production signals back into your evals, the hill changes shape. And the agent starts improving at what you actually care about.

2. Hill-climbing is the mechanism

"This harness optimization process is becoming much more agent driven with humans reviewing and curating evals/rewards to hill climb on. Evals are a moat and thus data to produce evals is a moat. Especially true for vertical agent companies. This is because agents can fit to most Eval sets today."

Here's the operational picture: you have an agent, you have a harness — the runtime, tools, prompts, skill definitions. You run the agent against your eval set. Some pass, some fail.

For failures, you do one of three things:

1. Edit the prompt

2. Change a tool definition

3. Add the failure as a new eval case

Then you run again. That loop — run → fail → edit → run — is hill-climbing. The harness is the surface you're optimizing over.

What I've found is that curation is the hardest part. It's easy to write a hundred evals. It's hard to write evals that are:

• Representative of what users actually encounter (not what you imagine they encounter)

• Sensitive enough to distinguish genuinely good responses from superficially good ones

• Robust enough not to be gamed by surface changes

In document processing systems I've worked on, the best evals came from the failure queue — messages that failed in production, got stuck, or produced wrong output. Those aren't hypothetical edge cases. They're real ones. Each is a new data point about exactly where the agent breaks.

That failure queue is your eval generator, if you treat it as one.

3. The compound loop

"new goal unlocked: grind so everyone forgets what a harness is and they just tell us what their agent needs to do, and we just chef it all together for them (harness + evals + envs + infra + observability + self-improvement + good-vibes)"

The vision here is compounding. Not just "evals improve the agent" — evals improve the agent → the improved agent reaches users → users surface new failure modes → those become new evals → the agent improves again.

Once this loop is running, it's hard to stop. Early on you're writing evals manually, dogfooding your own product to find where it breaks. Later, production data does most of the discovery for you. The best teams I've seen have an almost-automated pipeline from "something went wrong in prod" to "there's a new eval case in the backlog."

The key enablers:

Observability: you have to be able to see failures. Structured logging with correlation IDs, run IDs, case IDs. If you can't query "show me all the cases where the agent's output was flagged," you can't close the loop.

Categorization: not all failures are equal. Some are model failures. Some are tool failures. Some are eval failures (the eval expected the wrong answer). Knowing which is which tells you where to intervene.

Low friction to add evals: if adding a new eval case takes an hour, you won't do it for every production failure. If it takes a minute, you will. This is infrastructure, not process.

The "chef it all together" framing is the right end state: a team that describes what the agent should do, and has infrastructure that measures whether it does that thing, surfaces the gaps, and closes them. The eval harness is the product, not the afterthought.

4. Gall's Law: earn your complexity

"Gall's law: 'a complex system that works is invariably found to have evolved from a simple system that worked.' Also very relevant to agent systems. Most teams are trying to jump straight to autonomous complexity before they have evals, observability, or feedback loops in place. 2026 is the year of evals."

The agent teams I've seen struggle have almost always skipped the observable-simple-system phase. They built an orchestrator with five subagents, a memory layer, and tool use on day one. When it breaks — and it breaks — there's no way to tell which part failed. The eval is "the user seemed unhappy." That's not a training signal; it's a symptom.

My rule: before you add complexity, make sure you can see what's happening. Traffic interception, request logging, structured output validation, human review queues — pick your poison, but have something. A simple agent with good observability will outrun a complex agent with none. You can improve what you can measure. You can only guess at what you can't.

The practical implication: your first eval should be manual. A human reviewing the agent's outputs and labeling them good or bad is a valid eval harness. It's slow, it doesn't scale, but it forces you to articulate what "good" looks like. That articulation is the hard part. Everything else is tooling.

Complexity earns its place when the simple system is working and you know exactly which constraint it's hitting. Not before.

5. The frozen benchmark trap

"A benchmark is a frozen view of a solvable world. An RL environment is the same world made interactive. A harness is the runtime that lets agents act in it. A recipe is a compressed solution trajectory. A data generator is the world sampled at scale."

This mental map is the clearest framing I've seen of where evals need to go.

Most agent teams are at level 1: frozen benchmark. A test set, a score, a ship decision. The problem is that this test set was assembled from a distribution of inputs that existed at a point in time, labeled by people who had to make assumptions about what the right answer was. It doesn't capture the full distribution of what the agent will actually encounter. It captures what someone had time to label.

The unlock is level 2: interactive environment. Instead of "predict the right answer for these labeled inputs," it's "act in this environment and we'll observe what happens." The eval stops being a static check and starts being a simulation of real usage. Failures are now observable behaviors, not output mismatches.

From there, the natural progression:

• Harness: you build the runtime around the environment. Tools, skills, APIs, prompts — the harness is the surface the agent operates on, and the surface you optimize over.

• Recipe: accumulated knowledge compressed into structure. A wireframe template, a skill definition, a system prompt that encodes what good behavior looks like. Recipes are what you get when a domain expert and an agent engineer work together long enough.

• Data generator: at scale, the world generates evals for you. Production traffic, user interactions, edge cases you didn't anticipate — they surface continuously and become the raw material for the next round of eval curation.

Most teams in 2026 are at level 1. The best teams are building toward level 5. The distance between them is compounding every month.

6. Evals as measurement infrastructure

"Evals, benchmarks, and RL environments. Measurement infrastructure that decides whether agents are safe to ship."

"Measurement infrastructure" — not tests, not QA, infrastructure — is the right framing.

Infrastructure implies:

• Built once, used many times

• Maintained and improved as the system evolves

• A prerequisite for everything that depends on it

• Skimping on it creates debt that compounds

When I look back at agent systems I've built that stayed healthy versus ones that rotted quickly, the difference is almost always measurement infrastructure. Systems with eval harnesses improved iteratively. Systems without froze at their initial performance level because there was no way to know if a change was an improvement or a regression.

The "moat" framing is real. If your eval set encodes all the good behavior your agent needs to exhibit — edge cases, difficult inputs, real failure modes from production — that set is genuinely hard to replicate. It took real production data and real failure events to build. A competitor starting from scratch doesn't have that signal. They have benchmarks. You have signal.

This is why measurement infrastructure compounds in value over time, not just in the short run.

7. The human layer: who actually does this work

"Companies need help figuring out which models will work best for their workflows, they need extensive evals setup often, they need change management support for workflows, they need to get their data setup for the agents, and constant tuning of the agentic system for their process."

The eval harness doesn't build itself. Someone has to:

• Watch production for failures

• Translate failures into eval cases

• Run the hill-climbing loop

• Know when the metric is being gamed versus genuinely improving

• Change the eval when the eval is wrong

That's a human job. And it's one of the most technically demanding jobs in AI right now — precisely because it sits at the intersection of domain knowledge, engineering, and judgment about what actually matters.

The best AI teams I've worked alongside treat eval curation as a primary engineering discipline. Not something that happens after the "real work" is done. The eval harness is the real work. It's the system that makes the agent improvable. Everything else — the model choice, the infrastructure, the UI — is replaceable. The evals, and the process for generating them, aren't.

The practical implication: whoever is closest to production failures should have a direct line to your eval set. Whether that's a data scientist triaging a failure queue, a product manager reviewing user feedback, or an engineer watching structured logs — the signal is there. The question is whether you've built the pipeline to capture it, and whether the person who sees the failure can turn it into a new eval case in under five minutes.

If not, that's the first thing to fix.

Close the loop

The teams winning on agents in 2026 aren't the ones with the best models. Models are increasingly commoditized. They're the teams with the best eval infrastructure — the ones who've turned production failures into training signal, moved from frozen benchmarks to interactive environments, and made "add an eval" a reflex rather than a project.

Hill-climbing never stops. The question is whether you're climbing the right hill, and whether your instrumentation is sensitive enough to tell.

Start with one eval. Make it manual. Make it honest. Wire it to something real — an actual failure mode, an actual user scenario your agent encountered yesterday. Then close the loop.

The compounding starts when you do.

Over the past year, I've shipped several agents into production at the insurance company where I work. These aren't chatbots. They're systems that autonomously process documents, query internal databases, draft correspondence, escalate edge cases, and hand off to humans — across hours, sometimes days. Here's what I learned.

Why "Long-Running" Is a Different Problem

Most agent tutorials show you something that completes in under 30 seconds. Prompt in, tool calls, response out. Clean.

Real enterprise workflows don't look like that. My agents need to:

• Wait for a human to upload a document before continuing

• Pause mid-task because a downstream API is throttled

• Resume after a weekend without losing context

• Survive infrastructure restarts, deployments, and the occasional Kubernetes eviction

The moment your agent spans more than one process lifetime, you've entered a different design space. You need persistent state, explicit checkpointing, and failure recovery — not just a retry decorator.

Architecture: The Three Layers

After a lot of iteration, I landed on a three-layer architecture.

Layer 1: The Orchestrator

This is the durable brain. It knows what the agent is supposed to accomplish, what steps have been completed, and what comes next. It lives in the database, not in memory. Every time the agent advances a step, the orchestrator writes a checkpoint before doing anything else.

I model each agent run as a state machine. States are explicit: WAITING_FOR_DOCUMENT, EXTRACTING_DATA, AWAITING_HUMAN_REVIEW, COMPLETE, FAILED. Transitions are logged. You can reconstruct exactly where any agent run is — and was — at any point.

Layer 2: The Worker

Workers are stateless. They pick up a task from a queue, execute one step of the state machine, write results back to the orchestrator, and die. If a worker crashes mid-step, the orchestrator detects it via heartbeat timeout and reassigns the task. No task is lost because no task lives only in the worker's memory.

This is the biggest mindset shift from building "chatbots": treat every agent step as a unit of work that can fail and be retried independently.

Layer 3: The Human-in-the-Loop Interface

Some steps require a human. I don't fight this — I model it explicitly. When the agent reaches a step that needs human input, it writes a task to a work queue and suspends itself. A human completes the task (via a simple UI). The agent is notified and resumes.

This means agents can be "paused" for days waiting on a human without holding any compute. The orchestrator just has a row in the database marked WAITING_FOR_HUMAN.

Checkpointing: The Thing That Actually Saves You

I checkpoint aggressively. Before every tool call. Before every LLM call. After every meaningful piece of work.

Here's the pattern I follow:

1. Write intent before acting. Before calling an external API, write "I am about to call X with these parameters" to the checkpoint store. If the process crashes, you know what was about to happen.

1. Write result after completing. After the call succeeds, update the checkpoint to "X returned Y." Now recovery is: did we complete this step? If yes, skip it. If no, re-execute.

1. Make tool calls idempotent. This isn't always possible, but where it is, do it. Generate a UUID per intent, pass it as an idempotency key to external APIs, and you can safely retry without duplicating effects.

The checkpoint store is just a Postgres table. Nothing exotic. Each row has: run_id, step_name, status, input_hash, output, created_at. The agent loads its checkpoint on startup and skips any steps already marked complete.

Observability: What I Wish I'd Built Earlier

When an agent run fails at step 7 of 12, three days into its execution, you need to be able to answer: What did it do? What did it see? Why did it make that decision?

I now instrument every agent run with:

• Structured event logs: Every LLM call, every tool invocation, every state transition. Logged as structured JSON with run_id, step, timestamp, input, output, latency_ms.

• Token usage tracking: LLM costs compound fast when you have hundreds of parallel agent runs. I log input and output tokens per call and alert when a run exceeds 2× the expected token budget.

• A trace viewer: I built a simple internal UI that lets me pull up any agent run by ID and see its full timeline. This was a weekend project that has saved me countless hours of log-diving.

• Anomaly alerts: If a run has been in WAITING_FOR_HUMAN for more than 48 hours, we alert. If a run exceeds its expected duration by 3×, we alert. These are just cron jobs querying the orchestrator table.

The key insight: treat your agent runs like distributed transactions. The tooling exists for observing distributed systems — adapt it.

Handling Failures Gracefully

Agents fail. LLMs hallucinate. APIs time out. Documents are malformed. I've accepted this and designed for it.

Retry with backoff on transient failures. Network errors, rate limits, and temporary API outages are transient. I retry these with exponential backoff, up to a maximum. After max retries, the step fails and the run enters FAILED_RETRIABLE state.

Escalate on semantic failures. If the LLM returns something structurally valid but semantically wrong (extracted a date that doesn't exist, classified a document into the wrong category), I don't silently swallow it. I escalate to human review. The agent writes a note explaining what it saw and why it's uncertain, and a human resolves it.

Dead-letter runs for catastrophic failures. If a run hits an unrecoverable error, it goes to a dead-letter queue. A human triages it. We track dead-letter rate as a KPI. If it spikes, something is systematically wrong and we investigate immediately.

Never let a run disappear silently. This sounds obvious but it's easy to miss. An agent that crashes without updating its status looks identical to an agent that's still running. I use heartbeats: every worker pings the orchestrator every 30 seconds. If a heartbeat is missed for 2 minutes, the run is marked as SUSPECTED_DEAD and reassigned.

Lessons From Production

Lesson 1: The context window is not your memory. Early on, I passed the entire history of a run into every LLM call. Costs exploded. Latency ballooned. Now I summarize completed steps into a compact "run summary" and only pass recent context in full. The agent doesn't need to re-read everything it's done — just know what it's already accomplished.

Lesson 2: Humans are part of the system. I used to think of human-in-the-loop as a failure mode — something to minimize. Now I treat it as a first-class system component. Some decisions should go to humans. The agent's job is to make those handoffs clean and well-documented, not to eliminate them.

Lesson 3: Idempotency is load-bearing. Every time I skipped making a tool idempotent because "it's unlikely to be retried," it got retried, and something bad happened. It's always worth the 10 extra minutes.

Lesson 4: Schema-validate every LLM output. Every LLM call in my agents returns structured output validated against a schema at runtime. If the schema check fails, the step fails — loudly, with a detailed error — rather than propagating garbage downstream.

Lesson 5: Separate the LLM from the decision. The LLM suggests. The orchestrator decides. I don't let LLM output directly trigger irreversible actions. There's always a thin validation and approval layer between "the model said X" and "the system did X."

The Payoff

Running agents in this environment is genuinely hard. The failure modes are subtle, the stakes are real (insurance decisions affect people's lives), and the organizational context is conservative by design.

But it's also deeply rewarding. The agents I've shipped process hundreds of cases a week that would otherwise queue for days. They catch document inconsistencies humans miss at 2am. They make the humans who do review cases faster and more confident because the boring legwork is already done.

The patterns here — durable orchestration, aggressive checkpointing, explicit human handoff, observable runs — aren't specific to insurance. They apply anywhere agents need to survive contact with the real world.

Build for failure from day one. Your future self will thank you.

The arc of browser plug-ins — from power-user toy, to security liability, to the quiet backbone of the agentic web.

There's a moment in late 2024 that I keep coming back to. Anthropic ships Computer Use — a way for Claude to take screenshots and control a mouse cursor. The cleanest path to doing this in a browser isn't a new protocol or a custom runtime. It's a Chrome extension.

Something that had been declared dead a hundred times.

Act I: The Golden Age (2008–2014)

Firefox invented the modern browser extension. Not just as a feature — as a philosophy. A browser was an OS-in-waiting, and anyone could extend it. Greasemonkey let you rewrite any webpage with a few lines of JavaScript. AdBlock Plus blocked every ad on the internet. Firebug built the first real devtools before browsers had them.

When Chrome launched in 2008, Google took the model and made it faster, more sandboxed, and distributable via the Web Store. By 2013, there were tens of thousands of extensions. The category had its own celebrity developers, acquisition stories, and venture bets.

This era's defining trait: extensions as consumer products. You installed them for yourself. Password managers, tab managers, grammar checkers, ad blockers. The browser was a platform and extensions were apps.

Act II: The Long Decline (2015–2021)

Then came the rot.

It arrived slowly — a few extensions caught data harvesting in 2017, then a few more. Between 2024 and 2026, coordinated malware campaigns would go on to affect over 8.8 million users across Chrome, Edge, and Firefox. The Web Store turned out to be a decent distribution channel for malware. Extensions requesting <all_urls> permission got acquired for five figures, had their update payload quietly replaced, and started exfiltrating browsing history to ad brokers.

Google's response was Manifest V3, announced in 2018, which became one of the most contentious spec debates in browser history. Replacing persistent background pages with short-lived service workers. Replacing the webRequest blocking API that ad blockers depended on with a declarative declarativeNetRequest API capped on rules.

The ad blocker developers were furious. uBlock Origin's developer wrote long posts explaining exactly why the new model was insufficient. The EFF weighed in. Users signed petitions.

The vibe around extensions shifted entirely. Security-conscious companies started blocking extensions in managed Chrome deployments. The narrative calcified: extensions are a supply-chain risk, a performance tax, a privacy gamble. Between 2018 and 2021, if you worked in developer tools or security tooling, extensions were something you used reluctantly — not a platform you built for.

Act III: The Resurgence (2022–Present)

Four things happened nearly simultaneously, and together they changed the calculus entirely.

1. Computer Use Agents

In October 2024, Anthropic launched Computer Use — a capability that lets Claude take screenshots and control a computer. OpenAI followed with Operator in January 2025, powered by a model they called the Computer-Using Agent (CUA).

The performance numbers were striking. The open-source Browser Use framework hit 89% on WebVoyager (a standardized web task benchmark), compared to 87% for OpenAI Operator and 56% for Anthropic's initial Computer Use release. These numbers moved fast — by mid-2025, Anthropic's computer use tooling was shipping production-ready headers across Claude Opus 4.7 and Sonnet 4.6.

In August 2025, Anthropic launched Claude for Chrome — a Chrome extension that gives Claude a persistent sidecar in the browser, with permission to take actions on the user's behalf. Rolling out to 1,000 subscribers on the Max plan as a research preview. Google launched Gemini integrations with Chrome. Perplexity launched its own AI browser, Comet. OpenAI merged Operator directly into ChatGPT as "agent mode" in July 2025.

The browser extension is the agent's body. It has the right permissions — <all_urls>, content script injection, synthetic event dispatch — and it's already trusted by the browser's security model in a way an external process is not.

Extensions didn't become popular again because they got better. They became necessary because the use case required them.

2. Remote CDP and Browser Harness Tooling

The Chrome DevTools Protocol — the WebSocket API that Chrome exposes when launched with --remote-debugging-port — has existed since 2011. For years it was mostly used by Puppeteer and later Playwright for test automation.

What changed was the architecture around it. A new pattern emerged: a lightweight daemon process holds the CDP connection to an already-running Chrome, and scripts send one-shot JSON commands over a Unix socket. No Playwright overhead, no Node.js runtime, no process spawning — just a direct line to the user's real browser, with their cookies and authenticated sessions.

browser-harness <<'PY'
new_tab("https://example.com")
wait_for_load()
print(page_info())
PY

This made browser automation feel like a UNIX tool. First navigation is new_tab() not goto() — because you're attaching to the user's live browser, not a clean test instance.

Cloud providers took the same insight and productized it at scale. Browserbase, founded in 2024, raised $67.5M total — including a $40M Series B led by Notable Capital in June 2025, valuing them at $300M. By 2025 they were running 50 million browser sessions per year, with customers including Airtable, Instacart, Notion, Stripe, Perplexity, and Vercel. Forbes named them to their Next Billion-Dollar Startups list.

The pattern is the same everywhere: get a cdpUrl, resolve the WebSocket endpoint via /json/version, drive the remote Chrome instance. The browser becomes stateless infrastructure you rent by the session.

3. WebGPU-Based In-Browser LLMs

This one is weirder and arguably bigger in the long run.

WebGPU shipped in Chrome 113 in May 2023. Unlike WebGL, it's a proper compute API — shaders, compute pipelines, buffer access. For ML inference, this means matrix multiplications at speeds that benchmark at roughly 80% of native Metal or CUDA throughput. On an Apple M3 Max, WebLLM runs Llama 3.1 8B at 4-bit quantization at ~41 tokens per second — about 80% of native MLC-LLM performance. Phi 3.5 Mini hits 71 tokens per second. On discrete NVIDIA hardware, WebGPU is 10–15× faster than WASM for token generation.

WebLLM, MediaPipe LLM Inference, and Transformers.js proved that Gemma 2B, Phi-3-mini, and Qwen-1.5B could run entirely in the browser — no server, no API key, no round-trip latency. The practical sweet spot is 1B–3B parameter models at 4-bit quantization for reliable cross-device performance.

Where do extensions re-enter? An extension can load a model once into the background service worker and expose it via message passing to every tab. This is meaningfully better than each page re-downloading and re-initializing a 2GB model file. The extension becomes a local model host.

The privacy angle is also genuinely new. An extension running Gemma via WebGPU processes data that never leaves the machine — legally and technically. No server to subpoena. No request to intercept. For health data, legal documents, personal finance: this is a capability class that didn't exist three years ago.

4. Manifest V3 Actually Landed (And It's Fine)

Here's the part that surprises people: after all the drama, Manifest V3 turned out to be largely fine for the new use cases.

Chrome 139, released July 2025, completed the MV3 transition by fully removing MV2 extensions. By August 2025, 73.4% of actively maintained extensions had migrated. More telling: 90% of new extension uploads are already in MV3. The developer community absorbed the change.

For agent workloads specifically, the MV3 constraints are non-issues or improvements. Event-driven service workers rather than persistent background pages? Better for agent tasks that should only run when triggered. The controversial declarativeNetRequest cap? Irrelevant to screenshot capture, DOM injection, or synthetic input dispatch.

The ad blocker developers had a real grievance — for ad blockers. For computer use agents, browser harnesses, and in-browser LLMs, MV3 is not a blocker. The new use cases were inadvertently designed around the new constraints.

What This Actually Means

The first wave of browser extensions was consumer software. The second wave is infrastructure.

A consumer extension is something you choose to install for yourself. Infrastructure is something an application depends on — something that has to be there, permissioned correctly, composing with other systems.

The AI extensions market reflects this shift. There were 238 AI-powered Chrome extensions with meaningful user bases in 2025. By early 2026, that number was 442 — an 85.7% year-over-year increase, with 115.5 million combined downloads. The market was valued at roughly $2.3 billion in 2025, with projections ranging from $8.2 billion to $17.5 billion by the early 2030s.

The technical primitives that make this possible — CDP, content scripts, native messaging, WebGPU — aren't new. They've been there for years. What changed is that we found use cases that need exactly those capabilities and nothing simpler.

Where It Goes

Extension-as-agent-runtime becomes a standard pattern. Just as every serious product eventually shipped a mobile app, every AI agent product involving the web will ship an extension. Not because it's trendy, but because the alternative — screen capture plus pixel-coordinate clicking — is brittle, slow, and has no access to the DOM or authenticated session state.

In-browser model inference becomes a tier, not a novelty. As models shrink and WebGPU matures, the question won't be "can you run a model in the browser" but "which tier do you want — local, edge, or cloud?" Extensions that manage this routing will be the abstraction layer.

CDP browser farms consolidate around a few infrastructure players. Browserbase's $300M valuation after 16 months of existence is a signal. The pattern is too clean not to become commodity infrastructure. Three or four players will own this the way AWS owns EC2.

The security concerns didn't go away. The same properties that make extensions powerful for agents — <all_urls> permission, access to every page, background persistence — are the properties that made them dangerous before. Security researchers have already flagged that agentic browser extensions are prompt-injection surfaces: a malicious page can try to hijack the agent's actions by embedding instructions in its content. Simon Willison has argued that the entire concept of an agentic browser extension "is fatally flawed and cannot be built safely." That's probably too strong — but it's not wrong that the security model is unsolved.

Extensions had a decade as consumer software, a half-decade as a liability, and are now becoming infrastructure for the agentic web. The category didn't change. The demand did.

The browser turns out to be the last-mile interface for AI agents — not because it was designed for that, but because it's where humans already are, where the data already lives, and where the sessions are already authenticated. The extension is the seam between the model and that world.

They were never really dead. They were just waiting for something to actually need them.

Building in this space — browser agents, CDP tooling, WebGPU inference? I'd love to hear what you're working on.

What I've learned building LLM agents that actually run in production

There's a gap between the demo-tier agent tutorials you'll find online and what it actually takes to ship an LLM agent inside a large organization. The demos show you a loop. The enterprise asks: what happens when the LLM is slow? What happens when your tool returns garbage? What happens when the user asks a follow-up question and the agent has forgotten everything?

Over the past year I've been building agentic AI systems in a regulated financial services environment. The patterns below aren't academic — each one solved a real failure mode I ran into. I've stripped out all the proprietary details, but the engineering is real.

The Enterprise Constraint Set

Before diving into patterns, it helps to understand what's different in enterprise:

• Latency budgets are tight. Users in ops or analytics workflows won't wait 8 seconds for a response.

• Cost per query matters at scale. An extra LLM call per message × 10,000 queries/month adds up fast.

• Auditability is non-negotiable. In regulated industries, you need to be able to explain why the agent used a particular tool or returned a particular answer.

• Data lives behind access controls. You can't just "give the agent everything" — tool access has to be scoped to what the user is allowed to see.

These constraints shape every pattern below.

Pattern 1: Hybrid Intent Classification (Model Router)

Problem: You need to know what the user is asking before you can pick the right tools. But calling an LLM for every classification is expensive and slow.

Solution: Two-stage classification — regex heuristics first, LLM fallback only when heuristics fail.

def classify_intent(query: str) -> dict:
    # Stage 1: fast heuristics (< 1ms)
    result = _classify_by_heuristics(query)
    if result is not None:
        return result

    # Stage 2: LLM fallback (~400–600ms)
    return _classify_by_llm(query, llm)

def _classify_by_heuristics(query: str) -> Optional[str]:
    query_lower = query.lower()
    if CLINIC_PATTERNS.search(query_lower):
        return {"intent": "clinic", "entity": _extract_clinic_name(query)}
    if POSTAL_PATTERNS.search(query_lower):
        return {"intent": "postal", "entity": _extract_postal_code(query)}
    if DOCTOR_MENTION.search(query):
        return {"intent": "name", "entity": _extract_person_name(query)}
    return None  # signal: fall through to LLM

In practice: ~80% of queries are handled by heuristics. The LLM only fires for ambiguous cases. This keeps median latency under 100ms while preserving accuracy for edge cases.

The LLM classifier returns structured JSON:

prompt = """You are an intent classifier. Return ONLY valid JSON:
{"intents": [{"type": "<name|clinic|postal|unknown>", "object": "<value_or_null>"}]}

Query: {query}"""

Forcing JSON output makes the downstream routing deterministic. No parsing surprises.

Pattern 2: Dynamic Tool Selection

Problem: Giving the agent access to every tool at all times is wasteful and confusing. An agent with 8 tools available will sometimes use the wrong one.

Solution: Only enable tools that are relevant to the detected intent.

# Base set: always enabled
enabled_tools = [tools["analyze_data"]]

# Entity-specific tools: only when we have a named entity
has_entity = any(
    i["type"] in ["name", "clinic", "postal"]
    for i in classified_intents
)

if has_entity:
    enabled_tools.extend([
        tools["search_registry"],
        tools["get_records"],
        tools["aggregate_monthly"],
        tools["analyze_activity"],
        tools["get_summary_stats"],
    ])

agent = Agent(instructions=SYSTEM_PROMPT, tools=enabled_tools, model=model)

Why it matters: Fewer tools = smaller decision space = faster tool selection + fewer wrong-tool errors. It also gives you implicit routing — the agent physically cannot call a tool you haven't enabled.

Pattern 3: Multi-Algorithm Entity Search (Search Agent)

Problem: Users don't spell consistently. "Dr. Johnson" vs "Johnson, Sarah" vs "Dr. Sara Jonhson" are all the same person.

Solution: Three-layer matching — exact, fuzzy (Levenshtein), n-gram (Jaccard). Apply all three; keep results that pass any one threshold.

def _fuzz_score(x: str, y: str) -> float:
    """Levenshtein-based partial ratio."""
    return 1.0 if x == y else fuzz.partial_ratio(x, y) / 100.0

def _ngram_sim(x: str, y: str) -> float:
    """Jaccard similarity on token sets."""
    set_x, set_y = set(x.split()), set(y.split())
    inter = set_x & set_y
    union = set_x | set_y
    return len(inter) / len(union) if union else 0.0

# Apply both scores, pass if either exceeds threshold
df["fuzz"] = df["NAME"].apply(lambda n: _fuzz_score(n.upper(), query_upper))
df["ngram"] = df["NAME"].apply(lambda n: _ngram_sim(n.upper(), query_upper))

matches = df[
    (df["NAME"].str.upper() == query_upper) |   # exact
    (df["fuzz"] >= 0.6) |                        # fuzzy
    (df["ngram"] >= 0.5)                         # n-gram
]

Why both fuzzy and n-gram?

• Fuzzy catches typos: "Jonhson" → "Johnson"

• N-gram catches reorderings: "Sarah Johnson" → "Johnson Sarah"

• Combined: you get recall without sacrificing precision

Pattern 4: Layered Fallback

Problem: Production systems fail. The LLM times out. The database is slow. The SDK isn't installed in some environment.

Solution: Multiple independent fallback layers, each one degrading gracefully to the next.

# Layer 1: SDK availability
try:
    from agents import Agent, Runner, function_tool
    SDK_AVAILABLE = True
except ImportError:
    SDK_AVAILABLE = False

if not SDK_AVAILABLE:
    return "Agent SDK not available. Please install openai-agents."

# Layer 2: Classification fallback (heuristic → LLM → default)
intent = _classify_by_heuristics(query)
if intent is None:
    try:
        intent = _classify_by_llm(query, llm)
    except RateLimitError:
        intent = _classify_by_llm(query, llm)  # single retry
    except Exception:
        intent = {"type": "unknown", "object": None}  # safe default

# Layer 3: Tool execution fallback
try:
    result = run_tool(params)
except Exception as e:
    return json.dumps({"error": str(e), "success": False})

Key principle: Every layer has exactly one job. The SDK check doesn't know about rate limits. The rate limit handler doesn't know about tool failures. Each layer handles exactly one class of failure and passes everything else down.

Pattern 5: Session Memory and Context Reuse (Feedback Loop)

Problem: Users ask follow-up questions. "What about that second provider?" doesn't make sense without memory of the previous turn.

Solution: SQLite-backed session memory + explicit context reuse for ambiguous follow-ups.

# On session start
session_id = user_session.get("id")
memory = SQLiteSession(session_id=session_id, db_path=":memory:")
user_session.set("agent_memory", memory)

# On each message
result = await Runner.run(
    agent,
    input=query,
    session=memory  # agent sees full conversation history
)

# Context reuse for unknown intents (follow-up questions)
if any(i["type"] == "unknown" for i in intents):
    previous_tools = user_session.get("previous_enabled_tools")
    if previous_tools:
        # "What about that other provider?" → reuse last search context
        enabled_tools = previous_tools

The context reuse heuristic is simple: if the intent is "unknown" (no entities detected), the user is probably asking about something already established in the conversation. Reuse the tool set from the previous turn instead of starting fresh.

Pattern 6: In-Memory Data Caching

Problem: Loading a large DataFrame from a data lake on every query adds 2+ seconds of latency.

Solution: Lazy-load into a module-level cache. First call pays the load cost; every subsequent call uses the in-memory copy.

_registry_df: Optional[pd.DataFrame] = None
_records_df: Optional[pd.DataFrame] = None

def _load_registry() -> pd.DataFrame:
    global _registry_df
    if _registry_df is None:
        dt = DeltaTable(REGISTRY_PATH)
        _registry_df = dt.to_pandas()
    return _registry_df

Result: First query: ~2s. Subsequent queries: ~50ms. 40× improvement for a few lines of code.

Watch out for: Memory pressure if DataFrames are very large, and stale data if the underlying source updates. For most analytics use cases, session-lifetime caching is a reasonable trade-off.

Pattern 7: Selective Result Limiting

Problem: Returning 50,000 rows to the LLM context window is expensive, slow, and usually useless.

Solution: Return summaries and limited result sets by default. Only go wide when the user explicitly asks.

@function_tool
def get_records(entity_ids: list[str]) -> str:
    """Get records for the given entity IDs."""
    result = fetch_records(entity_ids, limit=100)  # never return everything
    return json.dumps(result, default=str)

@function_tool
def get_summary_stats(entity_ids: list[str]) -> str:
    """Get summary statistics: first/last date, total amounts, counts."""
    # Returns 5 numbers instead of 5000 rows
    result = compute_summaries(entity_ids)
    return json.dumps(result, default=str)

Design your tool set so the agent reaches for summaries first and raw records only when it genuinely needs them. This is a prompt engineering + tool design problem together — your tool descriptions need to clearly signal when to use which.

By the Numbers

Two patterns that pay for themselves immediately — the classification split and the caching improvement:

The caching win is almost embarrassingly simple. The classification split is worth thinking about carefully upfront — the 80% heuristic coverage number will vary by domain, but even 50% coverage halves your classification cost.

How These Patterns Combine

These patterns aren't independent — they're designed to compose:

Pattern 1 (Model Router)
    → classifies intent
    → feeds Pattern 2 (Dynamic Tool Selection)
        → scopes agent's decision space
        → Pattern 3 (Search Agent) handles entity lookup
        → Pattern 7 (Result Limiting) keeps context lean
            → Pattern 5 (Feedback Loop) handles follow-ups
                → Pattern 4 (Fallback) catches everything that breaks
                    → Pattern 6 (Caching) keeps it fast

The compounding effect: classification is fast (Pattern 1), so tool selection is accurate (Pattern 2), so the agent stays focused, so responses are faster and cheaper, so you can afford the LLM fallback for genuinely hard cases.

What I'd Apply First

If you're building an enterprise agent today and you can only pick three:

1. Hybrid intent classification — the cost/latency savings pay for everything else.

2. Dynamic tool selection — directly reduces agent confusion in multi-tool systems.

3. Layered fallback — production systems fail; don't let one failure mode take down the whole chain.

The memory and caching patterns matter a lot once you're at scale, but the first three are table stakes for any agent you're running in front of real users.

These patterns aren't specific to any one framework — I've used them with OpenAI Agents SDK, LangChain, and raw API calls. The underlying logic transfers.

If you're building something similar or have patterns that worked for you, I'd love to hear about it in the comments.

Forward Deployed Engineers are the hottest job in tech right now.

OpenAI just launched its own FDE agency. Anthropic is hiring them at scale. Palantir built its entire go-to-market on them. Stripe created a new role — the "Forward Deployed AI Accelerator" — embedding AI-native engineers directly inside their marketing teams, assigned to cohorts of 20 people each, building custom tools alongside them until every person is self-sufficient. Box's CEO wrote publicly that building AI agents for internal functions is "a highly technical job, very much akin to a forward deployed engineer."

The external FDE market is exploding because someone has to do the hard part of deploying AI inside real organizations — understanding business processes end to end, wiring up the right models, setting up evals, managing workflow change, and tuning agentic systems continuously until they actually work.

Here's the thing: your company almost certainly won't bring one of these people in for your internal data team. The going rate is reportedly $10K/day for the big-name providers. And even if they did — it wouldn't be enough.

The CEO of Box said it clearly: "External FDEs, in my opinion, will not make your company an AI-first company. You can have the sleekest multi-agent orchestrations and still have the majority of your employee base hating AI, avoiding AI, and distrusting leadership decisions on it."

Stripe understood this when they designed their accelerator role. They built it specifically because "most employees won't upskill themselves. They'll need someone who is embedded within their teams to build alongside them."

That someone can be you — from the inside, with years of organizational context, at a fraction of the friction.

That's the Internal FDE. And if you're a data scientist or AI engineer inside a large enterprise, this is the most important role you're not being asked to play.

Why Your Work Disappears Without This Role

Let me describe something that happens constantly in enterprise data teams.

A team of smart people spends six months building a genuinely excellent model. It performs well on holdout. The business case is solid. The model goes into production.

And then... it gets used by three analysts, never seen by the decision-makers it was built for, and quietly deprecated when a new VP comes in with different priorities.

The code was fine. The data science was fine. The problem was that nobody translated it.

Nobody walked the floors. Nobody told the story in language that connected to the business team's quarterly priorities. Nobody made the downstream teams feel like partners instead of an upstream data source. Nobody built the internal credibility that makes a model a business asset instead of a technical artifact.

This is the FDE gap.

Deploying agents — or any complex AI capability — is fundamentally harder than deploying software. Software generally works the same way every time. With agents, you're deploying the equivalent of work output inside the enterprise. The business expects tasks solved nearly end-to-end. That means someone has to deeply understand the business process, handle model selection, set up evals, manage workflow change, get the data right, and tune the system continuously. It's not a side project. It's a mission-critical engineering role.

In the external FDE model, a vendor does this for you — at enormous cost, with no lasting institutional knowledge. In the internal model, it's you. With years of organizational context they could never have.

What an Internal FDE Actually Does

You are the connective tissue between deep technical work and business reality. That means four things, specifically.

You are a translator. When you talk to the underwriting team, you don't lead with AUC scores. You lead with "here's what this means for your loss ratio." When you're presenting to the executive leadership team, you don't lead with model architecture. You lead with the business decision this changes. Translation is not dumbing down — it's respecting your audience enough to meet them in their world.

You are a scout. Before anyone on your team builds anything, you've already had the conversations. You know what the business unit is worried about this quarter. You know where the downstream teams are frustrated. You know which operations lead would be an incredible champion if someone just brought them a relevant prototype. You don't wait for a project brief to discover business needs. You accumulate that intelligence continuously.

You are a credibility builder. Technical credibility comes from doing good work. Business credibility comes from showing up consistently, being honest when something doesn't work, and making people's lives easier before you ask them for anything. The Internal FDE is known before any project starts. When a new initiative needs a data partner, your team's name comes up because you've been in those rooms.

You are an amplifier. Great work that nobody knows about is just expensive hobby time. The Internal FDE makes sure that when the team delivers something valuable, the right people understand what it does, how it was built, and what's possible next. Not with self-promotion, but with clear, consistent communication that connects outputs to outcomes.

Your Unfair Advantage

External FDEs bring technical depth and pattern recognition from deploying similar systems across many companies. That's genuinely valuable.

But they lack something you already have: years of accumulated context about how this specific organization thinks, decides, and resists change.

There's a reason the best specialist consultants are almost always people who spent a decade inside the industry before going independent. If you tear your ACL, you want to go to an ACL surgeon — someone who has seen this specific failure mode hundreds of times in this specific context, not a generalist learning on your dime.

You are the ACL surgeon for your company's data problems. You know the internal politics. You know why the last initiative failed. You know which VP will champion something and which one will quietly kill it. You know which downstream team has the best data hygiene and which one will fight every integration request.

External FDEs have to earn that context from scratch, and they're on the clock while they do it. You already have it. The question is whether you're using it.

Lead With Beliefs, Not Features

The most common mistake in internal technical communication: leading with the work instead of the so-what.

"We've built a gradient boosted model with 87 features and a cross-validated AUC of 0.84" is accurate. It also doesn't start a conversation with anyone who isn't already thinking about the same technical problem you are.

Compare it to a belief statement: "I believe our models will only drive value if the business team owns the output, not the AI team." That tells your stakeholders immediately where you stand and why. It gives them something to react to, agree with, or push back on. It starts a real conversation.

This is what the best FDEs do with their clients — they don't open with product specs, they open with a diagnosis of the problem and a clear point of view on what matters. The same applies internally.

Before any major presentation or communication, ask yourself: what's the belief I want this audience to leave with? Work backwards from that.

Building Your Internal Audience

The first people who engage with your work set the tone for everyone else.

If the first reaction to your model readout is a VP saying "this is exactly what I've been asking for," everyone else in the room recalibrates. If the first reaction is "I don't understand what this means," you're spending the rest of the meeting recovering.

This means you need to do pre-work. Not to game the room, but to make sure the first engagers are informed engagers. Before any major internal presentation:

Talk to one or two stakeholders in advance. Not to get their approval, but to understand their current mental model and make sure your framing lands. Ask them what question they'd most want this work to answer. Then answer that question first.

Find your internal champion before you need one. In every business unit that interacts with your team, there's usually someone who "gets it" — who thinks quantitatively, who's frustrated with the status quo, who would love to have a data partner. That person is your ally. Invest in the relationship before you have a project that needs them.

Make your first engagement easy. Leave stakeholders with something clear to do or respond to. Not "let us know if you have questions" — but "we're going to pilot this with two underwriters in Q3 — can you connect us with the right people on your team by end of next week?"

The Communication Types That Build Credibility

Different types of internal communication build different kinds of trust. The most effective Internal FDEs cycle through all four.

Thought leadership: Share a perspective on where AI and data are headed in your specific domain — pricing, risk, claims, whatever your function is. You have a unique vantage point. Nobody else at your company sits at the intersection of the data and the business problem you're closest to. That vantage point is valuable, but only if you share it. Write a short internal piece. Send it to three people you respect. If it resonates, expand it.

Personal stories: The most underused communication type inside enterprise data teams. "Here's what I thought when we started this project, here's what we found, here's what we were wrong about" is more compelling than any polished methodology deck. It's also more credible. Anyone can dress up a success. The willingness to narrate the failure modes, the pivots, the things that surprised you — that's what makes people trust your judgment on the next project.

How-to guides: Technical teams build institutional knowledge that lives in people's heads and disappears when they leave. The Internal FDE turns that knowledge into artifacts. A one-pager on how to use your team's outputs. A short guide on how to read a lift chart for a non-data audience. Documentation that enables the business team to get value from your work without needing you in the room every time. This is leverage.

Predictions: "Here's what I think is going to matter in our space in the next twelve months" is a powerful thing to say internally, especially if you're right more often than not. It positions you as forward-looking, gives leadership something to react to, and often turns into a real project when someone says "actually, we've been thinking about that too."

The Practical Mechanics

Here's what actually building an internal FDE practice looks like, week to week.

Have one non-project coffee per week. One conversation with someone outside your immediate team that isn't about a current project. Just: what are you working on, what's keeping you up, what do you wish was different? This is how you build your scout intelligence. Over a year, these conversations compound into a map of the organization that nobody else has.

Write one short internal thing per month. It doesn't have to be long. A one-page take on something interesting you've learned. A short summary of a paper that has implications for your work. A retrospective on a project that didn't go as expected. Share it with a small list of people who would genuinely find it interesting. Do not mass-blast to the whole data org. Quality of audience matters more than quantity.

Create one reusable artifact per quarter. The Internal FDE leaves infrastructure behind. What would help the business teams work better with data, even when you're not in the room? A template. A guide. A decision framework. A cheat sheet. Something that gives you leverage without requiring your continuous presence.

Show up before you're needed. The worst time to build relationships with a business unit is when you need something from them on a deadline. Show up at their all-hands when you're invited. Ask to sit in on a team meeting just to understand their workflow better. This is not political maneuvering. It's the basics of being a good colleague in a large organization where teams default to siloing.

The Objection: "I'm a Data Scientist, Not a Communicator"

I've heard this from genuinely excellent technical people, and I want to push back on it.

Communication isn't a personality trait. It's a skill, and like any skill, it improves with practice and feedback. The data scientists who are most effective in enterprise settings aren't necessarily the most naturally outgoing people. They're the people who've built the habit of translating their work into business language, and who do it consistently.

There's also a structural argument here. In a large company, the value of technical work is not determined by the quality of the technical work. It's determined by whether decision-makers trust the team that produced it, understand what it does, and see a path to acting on it. That trust is built through human interaction, not through model cards.

You can be the best data scientist in the building, and if nobody above the technical level knows what you're working on or why it matters, you will have less impact than someone half as technically skilled who communicates well. That's not cynical — it's just how organizations work.

The Internal FDE role is not about becoming a politician or a salesperson. It's about deciding that you care enough about the impact of your work to do the last mile that makes it real.

What You're Really Building

The frustrating thing about doing excellent technical work in a large organization is that excellence is necessary but not sufficient. A model that lives in a notebook, or gets used by three people, or gets deprecated when priorities shift — that model didn't matter, regardless of how technically sound it was.

The Internal FDE closes the gap between good work and real impact. It's the part of the job that nobody assigns you, that doesn't show up in your performance criteria, and that most technical people quietly resent having to do.

But it's also the part that determines whether your work changes anything.

Companies are now paying $10K a day to bring in external people to do this. They're creating new job titles for it. They're embedding engineers inside teams specifically to solve the change management and translation problem that technical teams consistently fail to solve on their own.

You can be that person from the inside. With more context, more continuity, and a fraction of the friction.

That's worth building.

If you're building an AI or data practice inside a large enterprise and want to think through what the Internal FDE role looks like on your specific team, I'd love to hear what you're running into. Drop a reply or send a message.

Neither works if you actually want to own your model's behaviour.

This post is a system design walkthrough of an LLM Studio — the software infrastructure that lets a small team run the full model-improvement loop end to end. Think of it like a HelloInterview deep dive, but the system being designed is your own fine-tuning platform.

What Problem Are We Actually Solving?

The core loop of model ownership looks like this: you have a task, you have data, you want a model that gets better at that task over time. The challenge isn't the ML math — libraries handle that. The challenge is the software infrastructure around it.

You need to:

• Ingest and version training data (including synthetically generated data)

• Run fine-tuning jobs without blocking everything else

• Serve the resulting models without redeploying when new checkpoints land

• Evaluate models against each other without writing one-off scripts

• Let non-ML engineers trigger experiments through a UI

A notebook can do one of these. A studio does all of them, reliably, repeatedly.

High-Level Architecture

The studio is six loosely coupled services. Each has a single responsibility, its own database schema, and communicates over HTTP.

The two most important separations:

Finetuning and inference have different lifecycles. Training is GPU-hungry, bursty, and failure-tolerant — if a job crashes, retry it. Inference is always-on and latency-sensitive — if it crashes, users notice. Coupling them in one process means a bad training job can kill your serving layer.

The data service is its own thing. Synthetic data generation is slow (minutes per batch), LLM-heavy, and produces intermediate artefacts that need versioning. Baking it into the API would turn every data request into a blocking call.

Deep Dive: The Finetuning Service

This is the most complex service. Its job is to take a submitted training request and reliably produce a fine-tuned model checkpoint.

Job Scheduling

The service uses APScheduler with a background scheduler running three recurring jobs:

scheduler.add_job(finetuning_scheduler.run_fine_tuning_jobs, "interval", seconds=N)
scheduler.add_job(finetuning_scheduler.check_running_jobs,    "interval", seconds=N)
scheduler.add_job(finetuning_scheduler.check_jobs_waiting_for_checkpointing, "interval", seconds=N)

Why a polling scheduler instead of a task queue? Because fine-tuning jobs are long-running (minutes to hours) and stateful. Celery is designed for short tasks. APScheduler lets the service own the full job lifecycle — queuing, running, checkpointing, failure — without fighting a task framework's assumptions.

The Training Runners

Two runners live inside the finetuning service: one for SFT (Supervised Fine-Tuning) and one for GRPO (the reinforcement learning variant). They're separate processes, not just functions — the runner is launched as a subprocess with explicit hyperparameter flags. This means:

• A runner crash doesn't take down the service

• You can run multiple jobs in parallel on separate GPUs

• Runner code can be updated without restarting the scheduler

Checkpoints get written to a shared filesystem path and the service marks the job as CHECKPOINTING before promoting the adapter to the model registry.

Deep Dive: The Inference Service

The inference service exposes an OpenAI-compatible chat completions endpoint. The model underneath can be Ollama (local dev) or vLLM (production), but the contract to callers doesn't change.

POST /inference/v1/chat/completions
{ "model": "my-finetuned-adapter-v3", "messages": [...] }

Why Ollama Locally, vLLM in Prod?

Ollama is trivially easy to set up on a laptop — ollama pull llama3 and you're serving. But Ollama's throughput is single-request, which is fine for development and the model playground, but not for evaluation runs that fire 200 completions in parallel.

vLLM uses continuous batching — incoming requests share KV cache and GPU compute. For the GRPO training loop, which needs to generate 8 completions per prompt across thousands of prompts, vLLM is 5–10x faster. The service switches backends by environment variable; no code changes needed.

For burst production workloads, the same service can be pointed at a Modal-hosted endpoint — GPU spins up on demand, billing stops when idle.

Deep Dive: The Data Service (`svc`)

The data service handles everything that touches training data: ingestion, synthetic generation, augmentation, curation, and evaluation. It's the most Celery-heavy service in the stack.

The Synthetic Data Pipeline

The pipeline is async by design. A generation request comes in via the API, gets dispatched to a Celery worker, and the frontend polls for progress. The worker:

1. Ingests source documents (PDFs, plain text, YouTube transcripts via the API)

2. Chunks the content into token-bounded windows

3. Calls the LLM gateway to generate pairs (QA, CoT, or summaries)

4. Streams completed pairs back to the dataset in real-time

5. Marks the task complete and updates the dataset snapshot

The three generation modes matter architecturally, not just academically:

• QA pairs are cheap and fast to generate, good for instruction-following baselines

• Chain of Thought pairs cost ~3x more tokens but produce dramatically better structured-output models — the model learns to reason before answering

• Summaries are useful when your inference task involves compression, not extraction

Evaluation

The evaluation service can run multiple models against a snapshot in parallel and score them using either an LLM-as-judge rubric or a bring-your-own-evaluator endpoint. Results stream back per-datapoint, so you see the leaderboard filling in live rather than waiting for a batch job to complete.

Deep Dive: The Gateway

Every LLM call in the entire system — synthetic data generation, evaluation, GRPO reward scoring, the model playground — goes through a single gateway layer. This is not optional architecture astronautics. It pays for itself immediately.

What you get for free:

• Cost visibility — which service is spending what, on which model

• Routing — fall back from GPT-4o to Sonnet if one provider is down

• Caching — identical prompts during eval runs don't get billed twice

• Rate limit management — the gateway absorbs burst traffic so individual services don't need to implement retry logic

The gateway is a thin sidecar — one Docker container, no business logic. Portkey, LiteLLM, or a self-hosted proxy all work. The key is that it's a hard architectural boundary: no service calls an LLM provider directly.

Deep Dive: The API and Frontend

The api service is the single front door. It handles:

• Auth — API key verification for all service-to-service calls

• Job dispatch — accepts training requests and forwards them to the finetuning service

• Model registry — tracks which checkpoints exist and their metadata

• Dataset management — versions snapshots, stores split configurations

The frontend is SvelteKit. The key UI surfaces are: dataset view (upload, inspect, generate), experiment configurator (pick base model + hyperparameters), job monitor (live training logs), and the model playground (side-by-side completion comparison).

Local vs Production

Running locally and running in production are architecturally identical — same six services, same HTTP contracts. The only differences are the backing implementations:

The local stack boots with a single docker compose up. Each service gets its own container, Redis runs as a sidecar, and Ollama handles inference. Switching to production means swapping environment variables — no code changes.

Key Design Decisions

Why not a monolith? A monolith works until you need to scale the finetuning service to GPU machines while keeping the API on cheap CPU containers. Service separation lets each component scale independently and fail independently.

Why polling for job state instead of webhooks? Training jobs run for minutes to hours. Webhook delivery has a real failure surface over that timeframe. The frontend polls /api/jobs/:id every few seconds — simple, robust, no webhook infrastructure needed.

Why Celery for data tasks but APScheduler for training jobs? Celery is optimised for many short tasks with a shared worker pool. Training jobs are one long task per GPU — APScheduler's interval-based polling and explicit state machine fits better.

Why OpenAI-compatible inference endpoints? Every evaluation library, every agent framework, every LLM client already speaks this protocol. Wrapping your fine-tuned model in a compatible endpoint means zero integration cost downstream.

What Breaks at Scale

A few things that look fine in development but become problems in production:

Shared filesystem for checkpoints. Locally, all services can read the same path. In production, you need an object store (S3, Azure Blob) as the checkpoint backend, with the finetuning and inference services both mounting it. This is a one-line config change if you design for it upfront.

Celery worker saturation. A single LLM call for synthetic data generation can take 10–30 seconds. If you have 10 concurrent generation tasks each firing 100 LLM calls, your worker pool fills up fast. Separate worker queues for fast tasks (evaluation) and slow tasks (generation) early.

No rate limiting on the evaluation service. Running evals against all models in your registry against a large dataset can fire thousands of LLM calls in minutes. The gateway handles rate limits, but the evaluation service should have a concurrency cap per eval job.

SGLang vs vLLM: Which One Goes in the Inference Service?

The studio's inference service is deliberately abstracted behind an environment variable. Locally it points at Ollama. In production, you pick an engine. And in 2026, that means choosing between vLLM and SGLang — two projects that started as competitors but have now specialised into different niches.

The 2024 narrative of "SGLang is 3× faster" no longer holds. vLLM's V1 rewrite (January 2025) pulled the two within 3–5% on dense 70B models at typical concurrency. But diverge from that workload profile and the gap opens sharply in one direction or the other.

The Core Architectural Difference

The engines diverge fundamentally on how they store KV cache.

vLLM uses PagedAttention — fixed-size physical blocks, hash-based per-block sharing, similar to OS memory paging. Predictable, well-understood, maps cleanly to most hardware.

SGLang uses RadixAttention — the entire live KV cache is a compressed prefix trie. New requests walk the tree from root; matched nodes mean their KV state is already on-GPU. When multiple requests share a system prompt, a retrieved document, or a conversation history, RadixAttention shares that prefix automatically across tenants with no configuration. The cache-aware router biases routing toward whichever worker holds the warmest matching prefix.

This isn't a marginal implementation detail. On agent loops, RAG pipelines, and multi-turn chat — anywhere prefixes repeat across requests — RadixAttention is the correct data structure and PagedAttention is the wrong one. The papers report up to 6.4× speedup on those workloads.

Where Each Wins

The MoE picture is stark. On GPT-OSS-120B with a tuned configuration, vLLM hit 6,095 tok/s versus SGLang's 1,814 — a 3.4× lead. DeepSeek V3/R1 runs in the other direction: SGLang's 96-H100 production deployment matches DeepSeek's own infrastructure throughput at roughly $0.20 per million output tokens, about one-fifth the cost of the public API. The DeepSeek GitHub repo officially recommends SGLang.

vLLM's durable advantage is breadth. It runs on NVIDIA, AMD, Google TPU, AWS Inferentia, Intel Gaudi, Apple Silicon, and IBM Z mainframes. SGLang covers most of those but TPU support lags. If you might multi-cloud, vLLM is the safer choice. It also covers ~200 model architectures — encoder-decoder, Mamba, Whisper, T5 — that SGLang doesn't support at all.

The Production Decision

The default for a generalist team is start with vLLM. Then add SGLang as a second inference backend behind the gateway for the specific workloads where it wins: prefix-heavy pipelines, DeepSeek/MoE serving, structured output at scale, or RL training rollouts.

Both engines expose identical /v1/chat/completions semantics. The application-layer switching cost is genuinely near zero. Run both, route by workload type, measure on real traffic before committing to one.

The deeper shift is that in 2026 you're not picking an inference engine — you're picking a stack. Dynamo or llm-d on top, vLLM or SGLang underneath, with the gateway routing between them. The orchestration layer is increasingly engine-agnostic. What remains different — RadixAttention's tree, vLLM's hardware breadth, SGLang's DeepSeek depth — is exactly what makes the choice still matter.

The architecture is deliberately boring. Six services, HTTP, Postgres, Redis, one LLM gateway. The interesting engineering is in the details: the job state machine, the streaming data pipeline, the dual-backend inference layer. Those are where the real decisions live.

If you're building something like this or have questions about any of the design choices, I'd love to hear from you.

Here's the current list, with what each one does and why I built it.

LinkNotes

What it is: A private, password-protected link vault with a CRUD API. Each link has a title, URL, tags, source, and a notes field.

Why I built it: Most bookmark managers optimize for quantity. Pocket, Raindrop, browser bookmarks — they're all great at storing links and terrible at storing why you saved them. Three months later you open a bookmark and have no idea what it was for.

LinkNotes forces a notes field. "Why does this matter?" is a required part of saving. The source field tracks where it came from. Tags let me filter by project or topic. And because it exposes a simple API, I can add links from scripts, automations, or other tools without touching the UI.

Built on Cloudflare Workers + D1. The whole thing is a single Worker — no external services, no database bills.

Tree Chat

What it is: A chat interface where conversations branch into trees. Use /branch [prompt] at any point to fork from the current message and explore a different direction, preserving the original thread.

Why I built it: Standard chat UIs have a fundamental problem: they're linear. If you're exploring a problem with an LLM — comparing two architectural approaches, exploring different framings of a question, trying a prompt variation — you have to either scroll back and lose your place, or start a new conversation and lose the context.

A tree solves this. Each branch inherits the context up to the fork point. You can explore three different answers to the same question simultaneously, navigate back to any node, and keep the whole exploration in one place.

Built on Cloudflare Pages with a D1 database for conversation persistence. Each node stores its parent ID and the full message content, and the tree is reconstructed on load.

TabReplay

What it is: Upload a file of URLs, and TabReplay lets you review each one with a live preview and triage them — keep, read now, or discard.

Why I built it: Tab hoarding is a real problem. I'll open 40 tabs in a research session and then not close Chrome for a week because I'm afraid of losing something. The real answer isn't better tab management — it's committing to a decision about each tab before closing.

TabReplay makes that decision-making fast. Export your tabs as a list of URLs (most browsers support this), drop the file in, and review each one with the actual page loaded alongside it. It uses Playwright under the hood to visit and render each URL, so you see what's actually on the page, not just the title.

HAR Vision

What it is: Drop a .har file — your browser's network log — and get instant visual analysis, timeline view, and request filtering. Runs entirely in your browser; nothing leaves your machine.

Why I built it: HAR files are invaluable for debugging, reverse engineering APIs, or understanding why a page is slow. But the raw JSON is unreadable, and the only decent viewer (Chrome DevTools) requires you to reproduce the session live. If you captured a HAR from a user's session or a CI test run, you're stuck parsing JSON by hand.

HAR Vision loads the file locally and gives you a filterable timeline — filter by domain, method, status code, or response time. Useful for finding the slow request in a 300-request page load, or spotting which API endpoints a web app calls.

ClipShot

What it is: Drop a video file, scrub through a Premiere Pro-style thumbnail timeline, and capture frames at native resolution. Export individually or as a ZIP.

Why I built it: I record a lot of screen recordings for documentation and demos. Extracting a good frame usually meant: open the video in QuickTime, scrub manually, take a screenshot, crop. ClipShot turns this into: drop the video, click the frame, done.

The whole pipeline runs in the browser using the HTML5 <video> element and Canvas API — no upload, no server, no waiting for processing. You capture exactly what you see in the timeline at native resolution, then export individual frames or bulk-download as a ZIP.

GridSplitter

What it is: Upload a sprite sheet, contact sheet, or photo grid — GridSplitter auto-detects the rows and columns using computer vision and splits it into individual files for download.

Why I built it: I was building a small game prototype and needed to extract individual frames from sprite sheets I found online. Every existing tool either required you to manually specify the grid dimensions or was a desktop app. GridSplitter uses edge detection to find the grid structure automatically — you don't count rows or columns, you just upload and download.

The CV pipeline runs in the browser using a WASM port of OpenCV. Processing happens entirely client-side, so there's no file size limit and no privacy concern with uploading assets.

Healpix

What it is: A browser-based photo healing tool. Paint over blemishes, dust spots, watermarks, or unwanted objects with a brush — the inpainting engine fills them in seamlessly.

Why I built it: The Photoshop healing brush is one of those tools that feels like magic the first time you use it. But Photoshop is $55/month and overkill if that's all you need. Healpix does one thing — heal — using the same underlying inpainting algorithms (Telea and Navier-Stokes), in the browser, with no install.

Paint the region you want removed, release the mouse, and the engine fills it in. Adjustable brush size, full undo/redo history, and export when done.

Watermark Remover

What it is: Upload a PDF, set of images, or MP4 video — configure the watermark region — and it strips the watermark using CV inpainting. Processes PDF pages in batch and video frame-by-frame.

Why I built it: Most watermark remover tools online are either broken, behind a paywall, or cap file sizes aggressively. This one uses the same Telea and Navier-Stokes inpainting algorithms that Healpix uses, but targets a fixed region across many frames — which is exactly how most watermarks work (static, bottom-right corner).

The video processing pipeline extracts frames at 1fps, inpaints each frame, then re-encodes the video with ffmpeg. Slow, but it works on arbitrarily long videos.

Swap Fitter

What it is: Two-step image compositing. Upload a base image, draw a selection over the region you want to replace, then upload a second image to fill that region. Runs client-side.

Why I built it: I kept needing to answer "what would this look like if..." questions — swapping a UI component in a screenshot, replacing a label on a product photo, mocking up a design variant. Opening Figma or Photoshop for a two-minute task is too much friction. Swap Fitter is just the compositing step, nothing else.

Step 1: upload base image, draw selection. Step 2: upload replacement, which gets scaled to fit the selection. Export. Done.

AutoCarousel

What it is: Upload images in order, set the slide duration (how long each image stays on screen), and export as an MP4 video file.

Why I built it: Social media carousels — especially on LinkedIn and Instagram — get more reach than static images. But converting a set of slides or screenshots into a video carousel usually involves either a design tool with video export (slow) or a social media scheduler (locked in). AutoCarousel just turns images into a video, no account required.

Upload your images in order, drag to reorder if needed, set 3–5 seconds per slide, and export. That's it.

Video Looper

What it is: Drop a short video clip and an audio track — the tool loops the video to exactly match the audio's duration and exports a single combined file.

Why I built it: A common content format is a looping visual over a music track — ambience videos, demo backgrounds, lo-fi aesthetic clips. The problem: your video might be 4 seconds and your audio 3 minutes. Manually looping in a video editor means dragging the same clip 45 times.

Video Looper calculates exactly how many loops are needed, handles the partial last loop cleanly, and muxes the audio track in. Upload both files, click create, done.

The pattern I've noticed across all of these: the tools I actually reach for are the ones that do exactly one thing and require zero configuration. No accounts, no settings screens, no onboarding. You understand them in ten seconds or you close the tab.

That's the design constraint I try to build to. Most of these took a few hours each. The hard part was figuring out what to leave out.

Claude Code has a Telegram plugin for exactly this. You run claude --channels plugin:telegram@claude-plugins-official, pair your account, and suddenly you can message Claude from anywhere.

The problem: it kept dying.

Crash it once, it's gone. Reboot your Mac, it's gone. And since it runs as a foreground process, the moment you close the terminal, it's gone.

So I built a watchdog.

The constraint that changes everything

Claude Code is a TUI app — it needs a real terminal (a TTY) to run. You can't just start it as a background daemon directly under launchd. It fails silently.

The fix: tmux. Run Claude inside a tmux session, and launchd manages the watchdog that watches that session.

launchd → watchdog.sh → tmux session → claude --channels

This is the core architecture. Everything else is consequence.

Three layers of self-healing

I didn't want to ever think about this again, so I designed for complete autonomy:

Layer 1 — Process: watchdog.sh loops every 60 seconds and runs three checks:

1. Is the tmux session alive?

2. Is the Claude PID alive inside it?

3. Does the Telegram Bot API respond to a getMe call?

If any check fails, it kills the stale session and relaunches.

Layer 2 — Watchdog: The watchdog itself runs under launchd with KeepAlive: true. If watchdog.sh crashes or exits for any reason, launchd restarts it within ~10 seconds.

Layer 3 — Machine: The launchd agent has RunAtLoad: true. Reboot your Mac, log in, and within 60 seconds Claude is back on Telegram. No manual steps.

The only failures that still require human intervention: OAuth token expiry and bot token revocation. Everything else recovers automatically in 10–120 seconds.

Two bugs I hit that weren't obvious

Bug 1: Don't use `getUpdates` as your health check.getUpdates as your health check.**

My first instinct for the Telegram health check was to call getUpdates — if it returns updates, the bot is alive. Seems reasonable.

Wrong. Telegram only allows one long-poll consumer per bot token. The Claude plugin already holds that connection. When my watchdog called getUpdates, Telegram rejected it with a conflict error — and worse, it kicked the plugin's own connection, causing exactly the failure I was trying to detect.

The fix: getMe instead. It's a read-only endpoint that doesn't compete with the plugin's polling loop.

Bug 2: `pgrep` is dangerous when you're running multiple bots.pgrep is dangerous when you're running multiple bots.**

The original version cleaned up stale processes with:

pkill -f "claude.*channels.*telegram"

This killed every Claude channels process on the machine — including a completely separate bot (cc-sdas) that had nothing to do with this watchdog.

The fix: track only your own PID. The launcher writes its process ID to a file. The watchdog reads that file and only ever kills that specific PID. No pattern matching, no collateral damage.

Exponential backoff

If something is fundamentally broken — say, OAuth expired — you don't want the watchdog hammering restarts every 60 seconds forever. So failures back off:

• 1st failure → restart, wait 30s

• 2nd failure → restart, wait 60s

• 3rd+ failure → restart, wait 120s (capped)

• On recovery → reset to normal interval

This prevents restart storms while still recovering reasonably fast when the underlying issue is fixed.

What I'd do differently

One thing I didn't solve: Claude shows a "Do you trust this project?" prompt on first launch in a new tmux session, because the working directory defaults to /. It waits for Enter — silently blocking startup. The workaround is tmux send-keys -t claude-telegram Enter, but the real fix is launching from ~ or pre-trusting the directory.

Also worth noting: this whole setup assumes your Mac stays on and logged in. The battery-dying scenario is genuinely unrecoverable without physical intervention. If you want true 24/7 availability, you'd need a VPS instead of a local Mac. For my use case (it's my own machine), this is good enough.

The repo

Everything is at github.com/ucalyptus/claude-tmux-watchdog — the launcher, watchdog, health check script, and launchd plist. Should work on any Mac running Claude Code with the Telegram plugin.

If you're running Claude Code and want it reachable from your phone without babysitting it, this is the setup.

A world where you had months to spare. A world where hiring a lawyer for a ₹50,000 dispute made economic sense. A world where physically showing up to a courthouse was the only way to be heard.

That world is gone. But the system remains.

The math doesn't work

Here's the absurdity: in India, the average civil case takes 3–4 years to resolve. The average cost of litigation — factoring in lawyer fees, court fees, and lost time — often exceeds the disputed amount itself.

Which means most people don't pursue it. They eat the loss, swallow the injustice, and move on.

This isn't a gap in the system. It's the system working exactly as designed — for people with resources, not for everyone.

What if disputes just... resolved?

That's the question behind Sulajh (Hindi: सुलझ — to untangle, to resolve).

Sulajh is an online dispute resolution platform. You file a claim, the other party responds, a neutral reviews both sides with AI assistance, and you reach a settlement — all online, often in days.

No courtrooms. No travel. No months of waiting.

The process has four steps:

1. File a Claim — describe the dispute, upload your evidence

2. Respondent is Notified — the other party gets a chance to respond

3. Mediation & Review — a case manager and a neutral review both sides; AI flags key points and identifies common ground

4. Resolution — a fair settlement or a binding decision

That's it.

What makes AI useful here

Dispute resolution is fundamentally a reasoning problem. You have two versions of events, a set of documents, and a question: what's fair?

AI doesn't decide that. But it does something valuable: it reads everything. It doesn't miss the clause buried on page 7. It doesn't favor the party who talks more confidently. It doesn't get tired.

The AI in Sulajh acts as a research layer for the neutral — surfacing relevant precedents, summarizing documents, flagging inconsistencies, and identifying where the parties actually agree (which is usually more than either side admits).

The human neutral still makes the call. But they make it with better information, faster.

Who is this for?

• A freelancer chasing an unpaid invoice

• A tenant disputing a wrongful deduction from their security deposit

• A small business with a supplier who delivered defective goods

• A consumer whose warranty claim was rejected without reason

These are disputes that never reach court — not because they're not valid, but because the economics don't make sense. Sulajh makes them resolvable.

The broader picture

Online dispute resolution isn't new. eBay was quietly resolving 60 million disputes a year through automated mediation long before anyone was calling it ODR. PayPal, Airbnb, Uber — every platform-scale company built informal dispute systems because the alternative was chaos.

What's new is making this available outside of platforms. For disputes that happen in the real world, between real people, without a platform intermediary to absorb the coordination cost.

That's the hard part. And that's what Sulajh is trying to solve.

The legal system will not reform itself fast enough. But technology can route around it — not by replacing justice, but by making it accessible.

Sulajh is early. But the direction is right.

If you're dealing with a dispute — commercial, consumer, or otherwise — try Sulajh. Or just reply to this email. I'm curious what kinds of disputes feel most underserved to you. Sulajh. Or just reply to this email. I'm curious what kinds of disputes feel most underserved to you.