How Should Startups Architect AI Agent Systems?

Start with one agent embedded in your product as a feature: a single model, a short explicit tool list, permission boundaries around anything irreversible, a human gate, and logging from day one. Add memory, orchestration and multi-agent patterns only when customer growth forces them. The most common failure I see is a startup building an agent platform before it has an agent that works. This is what I tell founders, based on running five agents in my own repos.

Founders ask me some version of "should we build a multi-agent system?" before they have shipped a single agent. I understand the impulse. The demos are spectacular. But I run a team of AI agents at Pairty, I write about how that team works, and the honest answer is that almost everyone should start two rungs lower than they want to.

One model, a handful of tools you can list on an index card, a system prompt in version control, and a named owner on your team. It lives in your product like any other feature. Most of the "we need agents" conversations I have as a fractional CTO end here, and they should. Move up only when one agent's context or tool list has grown past what one person can hold in their head, or when two distinct workflows are fighting over one prompt.

Deterministic code owns control flow. Agents own the work inside each step. This is where my own practice sits. Our founding engineer, an OpenClaw agent we call web3jason, picks up a Trello card, gets a plan approved by four AI stakeholder personas (CTO, Quant Researcher, Head of Engineering, Director of Product Design), executes with tests first, opens a PR, and posts a Maestro recording of the feature to Slack. Only after all four personas approve may execution begin. That sequence is code and checklists, not a model deciding what happens next. The orchestration is boring on purpose. Stay here until you have evidence, from your own logs, that agent-to-agent negotiation beats code-owned control flow for your workload.

Useful for a narrow set of problems: broad research sweeps, adversarial verification, work that is naturally parallel and cheap to throw away. It is also the most common way I watch startups spend a seed round on tokens. I wrote after Ai4 that benchmarks don't ship. Neither do demo architectures. A Level 3 system nobody can debug is worse than a Level 1 system everybody understands.

Model choice. I pick per task, not per company. A frontier model for judgment-heavy steps and a small, fast one for mechanical steps. I revisit this quarterly because the ground moves under it.

Tools and permissions. Every tool is enumerated. No agent gets a general-purpose shell where a scoped API will do. Anything irreversible sits behind a gate, and at Pairty the gated list is written down: auth and Privy, swap logic, wallet operations, production endpoints, bonding-curve parameters, CI secrets. Anthropic's engineering notes on harnesses for long-running agents and harness design are the best public writing I have found on why this layer, not the model, is where most of the engineering lives.

Context and memory. I start with no memory beyond the task. Retrieval comes when users repeat themselves. Durable memory comes only when the product's value depends on it, because everything an agent remembers is also something you now have to protect.

Human gates. Scaled to blast radius. Reversible internal work is agent-autonomous. Anything customer-visible gets human review. Money, auth and data deletion stay human-only. The code version of this policy is its own post: Should AI agents write production code?

Observability and evals. Every tool call and every token is logged from the first day, because I cannot improve an agent I cannot replay. Before a workflow scales, it gets evals. Ten hand-checked cases beat none, and I have never regretted writing them early.

These are my observations from a five-agent team, not a benchmark.

All coding work runs through Claude Code on a flat subscription. CI fixes run in GitHub Actions. Pay-per-token API calls are reserved for product features, each with metering and a budget. That split means the engineering side of the bill is predictable and the product side is measurable per workflow.

Two things I have learned watching it. Agent cost scales with the work, roughly linearly, with none of the management overhead that comes with human headcount. That cuts both ways: a well-priced workflow is a gift, and a mispriced one loses money on every run without ever complaining. And the expensive failure is rarely one big job. It is an agent quietly retrying a failing task in a loop. Retry budgets live in the orchestration layer in my systems, not in hope.

Every guardrail above exists because something got through. These are from the known-issues list we published with our operating prompts, so they are real.

Four files disagreed about our app version: app.json, package.json, Xcode and a hard-coded value in CI. That is a tool-boundary problem. An agent told to bump the version would have picked one. We made package.json the single source of truth.

Our Qodo autofix workflow listened for a workflow name that did not exist, so it never fired. Green checks with nothing behind them. That is an observability problem, and the reason I read what tests assert.

A deploy workflow echoed a production secret into its logs. That is a permissions problem, and it is why CI secrets are on the locked list.

Jest hung on open handles from Apollo and Privy and had been "fixed" with a three-minute kill. That is a retry-budget problem wearing a test runner's clothes.

And the one that taught me the most, from what OpenClaw is becoming: password managers assume a human is present. When a desktop vault locks at 2am, a human reaches for a fingerprint. An agent hits a wall. The fix was not to make the agent better at pretending to be me. It was to give each agent its own scoped, short-lived credentials so no agent ever sees a vault.

Don't build the multi-agent demo for the investor deck and run single-agent economics underneath it. Don't hand an agent a shell and a browser and call it a tool design. Don't ship without a way to stop every agent in one action. Don't store everything the agent sees and discover the compliance problem later. And don't skip evals because "we review everything." Review is a person. It does not scale. Evals do.

Once agents transact with the outside world, the question becomes which agent, operated by whom, allowed to do what. That is an identity problem, and API keys are not the answer. It is verifiable credentials and delegation, the same problem I worked on at Cerebrum and again with CredZK. I wrote about where the identity industry is heading on this in my notes from Identity Week America 2026.

Single agent or multi-agent to start? Single, as a product feature. Multi-agent when your own logs show code-owned orchestration is the bottleneck.

How do I keep costs under control? Log tokens per task from day one and price each workflow. Cost problems are almost always architecture problems.

Which framework? The one your team can debug. Tool boundaries, logging and evals are the parts you own regardless of framework, and they are the parts that matter.

Start at the bottom of the ladder. The view from Level 1 is better than most people expect.