Highlights

• The highest-scoring toolkit was not from OpenAI or Anthropic, but the more useful finding is that no single framework won. The best choice changed with the task.
• There is a real split between frameworks that are good at following instructions and frameworks that are good at reasoning through problems. Testing on one task type would have hidden this completely. We designed three.
• The AI agent SDK you pick today could quietly lock you into a provider ecosystem. Switching later is possible, but it costs you tracing, cost visibility, and eval tooling that tends to be tightly coupled to whoever built the framework.

We spent a few weeks doing something most teams skip: we actually built the thing before picking a framework.

Most AI agent framework comparisons out there are based on docs, demos, and vibes. Ours is based on building the same agent across six frameworks, once in each toolkit, and running the same set of tasks across all of them. This is what we found.

One thing to keep in mind throughout: this evaluation reflects the frameworks as we tested them in July 2026, and this space moves fast, so specific versions and capabilities will have changed since. These scores also come from the specific assistant we built and the three tasks we tested. They are not absolute rankings of the tools. A different product, with different needs, could shift them, so read this as a guide for our kind of work rather than a universal verdict. 

Why we did this

The toolkit we standardise on becomes load-bearing infrastructure for most of our future agent work. Getting it wrong means rebuilding later. That made it worth a few weeks of real work rather than a quick read.

Two things drove the decision to build instead of read.

First, every framework looks good in a pitch. The only way to actually judge one is to put it under the same conditions as the others and see where it struggles. Second, a toolkit can be strong on fixed routines and completely fall apart on open-ended work, or the reverse. Testing on a single task type gives you a distorted picture. So we designed three.

What we built

We built a daily assistant for developers. It connects to Jira, GitHub, Slack, Gmail and Calendar, pulls from all five, and links related items. A Jira ticket to its pull request to the Slack thread where it got discussed.

The assistant helps with standup preparation, status updates, priority triage and meeting prep. It holds a normal back-and-forth conversation. It backs every answer with real data from the connected tools rather than guessing. It checks with the developer before doing anything that cannot be undone, posting a message or closing a ticket, and hands off when it is unsure or lacks access.

The same agent, replicated across six frameworks: Claude Agent SDK, OpenAI Agents SDK, Mastra, Agno, DeepAgents and Hermes.

How we tested it

We put each version of the agent through three kinds of task, because this is how real work tends to show up.

Fixed steps. A set procedure that must run in the same order every time. We used daily standup preparation, ending with a post that waits for human approval before it goes anywhere.

Known scope. A specific question with a knowable answer: "What is the status of PROJ-1234 and is there a pull request open for it?"

Open-ended. A vague request with no single right answer: "I am overcommitted this week, help me prioritise."

To keep the comparison clean, we held everything constant except the framework. Same Claude model, same shared tool layer, same frozen sample data. What we measured was the framework itself, not the plumbing or the data. Each task ran several times per toolkit so one lucky or unlucky run could not decide the result.

How we scored each toolkit

Every toolkit was rated across the same dimensions:

• Build effort and developer experience
• Primitive fit — tools, subagents, memory, guardrails, human approval, streaming
• Determinism and control
• Observability and evaluation
• Cost and tokens
• Model portability and lock-in
• Open source, self-hosting and data residency
• Production readiness
• Language and runtime fit
• Reliability across runs

The results

One thing to keep in mind: these scores come from the specific assistant we built and the three tasks we ran. They are not absolute rankings. A different product, with different constraints, could shift them considerably. Read this as a guide for our kind of work, not a universal verdict.

What we actually learned

ㅤ

There is no single winner.

That sounds like a cop-out. It is not. The honest conclusion from six real builds is that the best AI agent SDK depends entirely on what you are building. Frameworks that excel at fixed, ordered procedures tend to be weaker at open-ended reasoning, and vice versa.

ㅤ

Workflow engines matter more than we expected.

Agno and Mastra both have real workflow engines that enforce step order rather than relying on the model to follow instructions. For fixed-step tasks — standup prep, automated reporting, approval-gated actions — this was a meaningful advantage. The code-first SDKs are more flexible, but that flexibility comes with less predictability on structured tasks.

ㅤ

Self-hosting changes the calculus entirely.

Hermes came out at the top of our table, but a big part of that score comes from data residency and the ability to run the whole pipeline, including the model, on infrastructure you control. If that is not a constraint for your work, the gap narrows considerably.

ㅤ

Model portability is a hidden cost.

The Claude Agent SDK and OpenAI Agents SDK both work best when you stay in their respective ecosystems. Switching later is not impossible, but it will cost you, particularly around tracing, cost visibility and eval tooling that is often coupled to the provider. Worth thinking about before you commit.

ㅤ

Observability is the thing teams underestimate.

On paper, every framework claims some level of observability. In practice, the depth varies a lot. DeepAgents had the weakest story here, which made diagnosing failures during testing much slower than it needed to be.

Recommendations by lane

Since there is no universal answer, here is how we would match tool to work:

ㅤ

For flagship engineering and SDLC work: The Claude Agent SDK maps most closely to how our playbook already operates. Primitive fit is strong, inline cost visibility is genuinely useful, and if you are already Anthropic-first, the tradeoffs are manageable.

ㅤ

For agents that must follow a set procedure reliably: Agno is the strongest fit. Its workflow engine guarantees step order rather than hoping the model follows instructions. For TypeScript and Node teams, Mastra is the equivalent, with one caveat worth knowing: it is young and fast-moving, and it had a resolved npm supply-chain incident in mid-2026, so pin your dependencies and keep a security review cadence rather than installing and forgetting.

ㅤ

For self-hosted or residency-sensitive work, and for scheduled or operational assistants: Hermes stands apart. It can run the whole pipeline, including the model, on infrastructure you control. That distinction matters for certain clients and certain data.

ㅤ

One practical note: keep any choice reversible behind a thin abstraction layer. Different parts of a product may end up on different frameworks, and the landscape is moving quickly enough that your answer in six months may not be the same as today.

What comes next

Each toolkit has its own detailed write-up with the full dimension-by-dimension breakdown and how it handled each of the three task modes. Those are linked from our internal report.

If you are evaluating AI agent frameworks for your own stack, the most useful thing we can tell you is this: pick one task that represents your hardest real-world use case, build it, and see where the framework makes you fight it. The answer will be more useful than anything in the documentation.

FAQs

ㅤ

1. Which AI agent framework should I start with if I am just beginning to build agents?

Start with the SDK that matches the model you are already using — Claude Agent SDK for Anthropic, OpenAI Agents SDK for OpenAI. Both keep the surface area small. Pick the one that lets you ship something in a day and learn from real use.

ㅤ

2. Does the choice of framework affect how much I pay in API costs?

The model provider sets the per-token rate, but the framework shapes how many tokens you spend. Two things drive this. First, token efficiency: some frameworks add overhead through verbose prompts or unnecessary tool calls. Second, and easier to miss, the architecture a framework pushes you toward: a multi-agent pattern that fans out to several specialists rebuilds shared context on every call, which ran up prompt-cache-write costs in our testing faster than a single-agent design would. So built-in cost visibility matters, because the structure you choose can cost more than the base rate suggests. Hermes is the exception to all of this: it can run the model locally, which changes the cost structure entirely. 

ㅤ

3. Can I switch frameworks later without rebuilding everything?

Yes, but the switching cost is rarely the agent logic — it is the tracing, evals and cost dashboards that tend to be tightly coupled to the framework. Keep your tool definitions behind a thin abstraction from day one and the switch stays manageable. 

ㅤ

4. What is the difference between a workflow engine and a code-first SDK, and why does it matter?

A workflow engine enforces step order at runtime. A code-first SDK gives you primitives and lets the model decide the sequence. Workflow engines are more reliable for fixed procedures; code-first SDKs handle open-ended reasoning better. Agno and Mastra are workflow-engine-first. Claude Agent SDK and OpenAI Agents SDK are code-first. 

ㅤ

5. How do I know if my agent needs human-in-the-loop support?

If it can take any action that is hard to reverse — posting a message, closing a ticket, sending an email — it needs approval gates. The question is how the framework handles it. Some make it a first-class primitive. Others leave it to you, which means more surface area for it to break.