Open-Source Agent Frameworks with Local Models: What Self-Hosting Really Requires ​

We · October 2026

This is the third article in the "AI Agents & Local Automation" series. The first explained how these tools differ from ordinary chat AI; the second put the products on the market onto a single map. This one covers the bottom-left cell of that map—building it yourself. It's the part computer science students and developers are most curious about, and the part where they most often step on a rake.

AI moves fast. The tools, capabilities and numbers in this article describe where things stand right now, and are for reference only. Every licence mentioned should be judged against the LICENSE file in the project's own repository; we don't make commercial-use determinations on the reader's behalf.

First, a misconception ​

The single most common sentence in online discussions about building your own AI agent goes something like: "I use the open-source one, so it's free."

That's wrong, and the wrong part matters. An open-source framework carries no intelligence. It is a shell: it takes your request out, brings the result back, and in between operates your files and browser for you. The part that actually thinks is a model the shell fetches from somewhere else. There are only two places that "somewhere else" can be—someone else's server, or a model file running on your own machine.

So the first question in self-hosting isn't "which framework." It's where the model comes from. That answer determines nearly everything downstream: who sees your files, what your bill looks like, and whether the assistant can finish the job at all.

The landscape article last time treated "where the model comes from" as one of the two axes for classifying these tools. Here we pull that axis out and follow it all the way through.

What an agent shell is made of ​

Start with why this category deserves a structural explanation at all.

In late November 2025, a solo developer open-sourced a personal project under the MIT licence. Its star count climbed so fast that it eventually overtook what had been the most-starred repository in GitHub's history; the author joined a large tech company in early 2026 and handed the project to an independent foundation. As of October 2026, the repo sits somewhere in the region of 390,000 stars.

If you read that only as a viral success story, you miss what the project was actually worth. Its significance was never polish—the installation experience gets roasted regularly in its own community. The significance is that it became the reference answer to the question "what should an agent shell look like." Most projects that came after it, whether they wrapped it or started fresh, ended up shaped very much like it. On the Chinese vendor side there even emerged a whole batch of hosted products with the same suffix in their names.

Take it apart and the core is simple: a message arrives from a chat app, a gateway routes it to a model, the model says what to do, the gateway does it, the result goes back to the chat app. Hung off that spine are four sets of things: scheduled jobs plus a mechanism that wakes the agent on its own; execution capability against the filesystem, a browser and a command line; a skill system described in plain text files, with a marketplace on top; and full MCP support (explained in the first article—in short, the standard connector between an agent and external tools).

Our habit is to read any agent shell as four layers, which turns out to be far more useful than reading brand names:

The model access layer—who your words get sent to. Only the vendors it ships with, or can you point it at your own endpoint, or connect it to a model service running on the same machine. This layer decides cost and privacy.

The tools and permissions layer—what it's allowed to touch. Whether it can read and write files, execute commands, make network requests, drive a browser. And most importantly: whether it decides about those permissions itself, or has to ask you each time.

The memory and state layer—whether it remembers next time. Can a task that got interrupted be resumed, where does cross-session material live, is it a local database file or someone else's server.

The skills and protocol layer—how it learns new work. Is behaviour hardcoded, or can you install a package, and is that packaging mechanism an open format.

Four boxes; every framework is a different combination of them. Open any project's documentation and look for these four specifically and you can usually judge fit within half an hour. Conversely, judging by "how many models supported" or "how many plugins"—pure marketing numbers—tells you nothing.

The project's own positioning is clear, and so is the community's: total control, and the price is that you have to enjoy tinkering. You install a runtime, edit a configuration file in a terminal, bring your own model API key, pair it with your chat app, and when something breaks the error is almost certainly in English and mostly searchable. It is not something you hand to a user who'd rather never open a terminal—which is precisely why the wave of graphical wrappers that followed exists.

Three routes: the difference isn't features, it's ownership ​

Straighten out that "where does the model come from" axis and self-hosting resolves into three routes.

The column most people skip is the cost shape. The open-source framework costs nothing; the model does. On route one you pay it once, in hardware. On routes two and three it's a running expense. The breakout project from the previous section is attractive partly because it's the only one of the three that can run entirely on a local model—but "can connect" and "is good enough" are different claims, and that's what the next section is for.

The middle route is what we'd call the correct answer for most people. The logic is plain: keep the parts that should stay private inside a local shell, and rent the parts that need to be clever from the cloud. Locally you hold the configuration, the skill packages, the workflows and the keys—that's your asset base. Model capability comes from outside, metered. And the price spread on that middle route is now genuinely large: Morgan Stanley measured average Chinese large-model API pricing in Q2 2026 at roughly ¥4.9 per million input tokens and ¥21.9 per million output tokens, while comparable American models generally run several times that (figures checked October 2026). For a single workflow that fires off dozens of calls, the difference is direct.

The third route isn't about saving money; it's about not needing you present. Scheduled jobs, multi-hour batch work, anything that has to continue after you close the lid—only the cloud form does that cleanly.

These three aren't a ladder from bottom to top. They're three different trades. Running all three is common among technical users: local for sensitive material, the hybrid for daily work, cloud for long jobs.

How far a local model can actually carry an agent ​

This is where the article has to be honest, and it's consistent with how the whole series is written: we won't imply that a local setup matches the experience of a top commercial product.

What works first. Models in the 24B–32B range that have been specifically optimised for tool calling, quantised and run locally, are adequate on single-tool, few-step tasks—reproductions from third-party benchmarks and the community put the well-formed output rate at around 95% (a range across different test setups, not a guarantee). Code completion and rewriting, retrieval-based question answering over your own material, and pipelines with a fixed structure are all usable.

What fails is more worth describing:

One disclaimer belongs here: there is no industry-agreed benchmark for multi-step tool-calling failure rates yet. The picture above is assembled from vendor model cards and community reproductions—the direction is trustworthy, the precise numbers are not a standard.

Then there's the threshold that actually stops most people, and it isn't the one they expect: the hard part isn't fitting the model, it's feeding the context.

Agent tools generally want 64K of context or more, and their load profile is unusual—the same long context gets resent over and over, because every tool call has to carry the previous content along again. A dense local model reads its entire set of weights to produce each token, so speed is hard-capped by memory bandwidth, and the prompt-caching discount that cloud providers apply to repeated content doesn't exist locally. Which means local agent "slowness" is usually not insufficient compute; it's data that can't be moved fast enough. Our article "Why Are AI Apps So Slow to Generate?" covers exactly this memory-versus-bandwidth point and is worth reading alongside this one.

Artificial Analysis open-sourced a benchmark in late September 2026 that replays real agent sessions against local inference. The scale gives a feel for it: by default a single run covers 8 tasks, 168 conversation turns, and over 2 million input tokens. On that same workload, a discrete-GPU system came in roughly five times faster than a 128GB unified-memory machine (as checked in October 2026). In other words, a machine that can run a large model and a machine that can run an agent are two different machines.

From measurements like these the community has converged on two pragmatic strategies, which we find more useful than any comparison table:

Route, don't replace. Send the high-volume, cheap, format-tolerant calls to a local model and keep complex reasoning and long chains on an online API. That's an engineering routing decision, not a matter of principle.

Constrain the tools, don't let it loose. Local models cope far better with structured interfaces—version control, code indexing, anything with clean inputs and outputs—than with "look at the screen and operate the computer," which at this stage is mostly a way to make yourself miserable.

So the conclusion sentence is this: running agents locally is currently a capability supplement, not an experience replacement. Anyone selling you the second thing doesn't deserve further reading.

The safety chapter isn't optional ​

This is the first time in this series we discuss risk at length. It belongs in the self-hosting article for a simple reason: whoever builds their own setup holds the most permission, so problems land most directly on them.

The premise first: capability and risk are two faces of the same thing. An agent can do work precisely because it can read files, write files, execute commands and reach the network. The more real the capability you grant, the more real the damage it can do. This isn't AI wishing you harm; it's the physical consequence of "permission plus autonomous execution."

Among publicly disclosed incidents, the ones ordinary users relate to fastest are accidental deletions. Two are notorious: one coding agent emptied a production database during an explicit code freeze, affecting over a thousand executive and company records; another took nine seconds to delete a production database and all of its backups, with the post-mortem attributing it to an access token holding far more privilege than the task needed. Neither was an attack. Both were excess permission plus nobody intervening. Research bodies and governance reports now catalogue these as "textbook incidents with no attacker."

The second category is prompt injection—hostile content hidden inside a web page, email or document the agent reads, steering it into behaviour you never asked for. It has topped OWASP's widely cited large-model risk list for three consecutive years. One detail deserves calling out: a vendor once publicly stated that injection success under its automated mode was zero, and an independent researcher subsequently demonstrated success rates far above that. A vendor's claim does not eliminate a risk. That's the judgement standard we want readers to keep.

The third category is, in our view, the thinnest link right now: skill marketplaces. Earlier we noted that the project which defined the category made "describe a skill in a plain text file" industry convention. That's a good thing and a new problem. An MCP server or skill package is, essentially, "a piece of executable instruction plus one credential exposure." Quantitative sampling from security research firms is grim: the large majority of public registries apply no screening at all; there has been a disclosed case of hostile servers flooding a code repository via batch pull requests in under an hour, behaving normally for the first three calls before rewriting tool descriptions to harvest SSH and cloud credentials; there are skill packages masquerading as legitimate tools installed a quarter of a million times, and a single hostile package reaching the credentials of roughly three hundred organisations. All of these carry named institutional disclosures. They aren't speculation.

Finally, default configuration. That category-defining open-source project binds by default to an address reachable from the whole network, with no authentication enabled. Several security firms separately counted exposed instances in the range of one hundred thousand to several hundred thousand during the first half of 2026; the project published several hundred security advisories across 2026, a number of them high severity; and a technical platform under the relevant national authority issued a specific alert about deployment risks of this class of software. None of that means the project is bad—it means something designed for one person tinkering on one machine got placed directly on the public internet by a great many people.

So the useful takeaway isn't memorising those names. It's checking six things before you build:

1. Can you constrain the working directory? It should see only the folder you explicitly hand it, not the whole disk.
2. Can commands and network access default to deny-and-ask? Being prompted before every command execution and every outbound request is a feature that can be switched off—and shouldn't be.
3. Are deletions forced into a recycle bin, and does it back up before modifying? This single item turns a "nine seconds to delete the database" into a recoverable event.
4. Are you only installing skill packages whose provenance you've verified? Before installing: what permissions it asks for, whether the code is public, how many installs, when it was last updated.
5. Is the service bound only to localhost or an internal address? The most commonly missed item, and the one with the most direct consequences.
6. Are you willing to run it in an isolated environment? A virtual machine, or a separate machine with nothing important on it—not the machine you work on.

Those six aren't a deterrent. On the contrary, they're the industry's emerging bar for entry. The mainstream commercial products in China now advertise, as selling points, sandbox interception of high-risk operations, forced recycle bins, automatic backup before modification, three-state permissions (allow / ask / deny) and full audit trails. If commercial products can do it and are willing to say so publicly, people building their own have no reason to accept less.

A self-hosting checklist with no commands ​

In line with how this series is written, there are no installation steps or configuration-file details here—that kind of content typically stays accurate for a few weeks. What follows is what you need to have, five items:

A model access point. Either a model service on your own machine, or a metered API key. Where the key is stored, how you cap it, whether balance alerts are on—settle those three before choosing a framework.

A framework. Filter on three things: whether you need a graphical interface, whether it must run fully offline, and whether you can live with its licence terms. The third is the one people skip, especially where a project is "permissive licence plus additional terms"—for example no multi-tenant external SaaS, or no removing the default branding above a certain user count. Those clauses are tedious to read and save you one very unpleasant afternoon later.

A class of tool interfaces. MCP servers or skill packages. Principle: start with the fewest you can, because every addition widens your permission surface.

An appropriate machine. Memory capacity and memory bandwidth outrank VRAM size—the previous-but-one section explained why the long-context bottleneck is bandwidth. If you're running sensitive experiments, budget for a separate isolated environment too.

Your own usage boundaries. Which directories it may touch, which operations must have a human confirm, and the most you're willing to spend in a day. The next article in the series explains why that last item went from optional to necessary in the agent era.

Ready-made versions of all this already exist ​

A necessary addition, because the entire set of thresholds above reads to most people not as a learning curve but as a reason to stop.

Three kinds of engineering wrapper have appeared around this category, and all of them remove "setting up the environment" from the list of required skills. Our app library already lists each type; here we describe the shapes only:

The first type wraps an open-source command-line shell in a graphical interface. A guided setup wizard (pick language, then model provider, then skill plugins), multi-channel management, scheduled tasks, a built-in skill marketplace, keys stored encrypted in the operating system's native keychain, and process isolation between the runtime and the interface. Its value isn't extra features—it's turning a project that needs a terminal into a piece of software you configure.

The second type is a desktop assistant with two execution environments. In one tool, a task can either run directly on your machine (fast) or run inside a separate Linux sandbox (isolated), switching automatically or on demand; meanwhile every file access, terminal command and network request requires your explicit approval, and chat and configuration data land only in a local database file. This type deserves pointing out specifically—it shows that the six safety requirements from the previous section aren't paper ideals; open-source products have already made them default behaviour.

The third type is a fully local-capable multi-agent workstation. It sits on an open-source multi-agent framework, states clearly in its own documentation that it's compatible with local inference backends such as llama.cpp, MLX and Ollama while also connecting to cloud model services, can pipe messages into everyday office and chat apps, and lets several agents divide up work. Also worth clearing up a common confusion: this one is not built on the category-defining open-source project—it merely overlaps in features. Many readers assume they're related; they aren't.

All three belong to the same layer as tools like Ollama and ComfyUI: the layer that packages underlying capability into something usable. Our position has always been fairly definite: for most creators and students without a development background, the sensible order is to validate your actual need on a wrapped tool first, and only then decide whether to build from scratch. Using something somebody else assembled to find out what you actually want an agent to do beats sinking into environment configuration on day one.

Finally ​

Whether self-hosting is worth it depends on what you're after.

If you want the best experience available, it isn't—the best experience in this category still sits with top cloud models, and the third section explained why in detail. If you want to understand what's happening, it's very much worth it.

Tools get replaced, frameworks go stale, and a once-popular project will announce it's shutting down (this year has already produced more than one such case). But the capability of knowing how an agent takes a job from request to finished result, step by step, stays with you. And it improves how you use commercial products along the way—because once you've built one yourself, you can tell which step it's cutting corners on, which step is quietly spending your money twice, and which permission it was handed that it didn't need.

Next in the series: the bill. Why the same sentence costs an order of magnitude more in tokens when you hand it to an agent instead of a chat box.