Guide · Self-hosted
The self-hosted agent stack: running AI agents on your own infrastructure
The loudest demand in the agent ecosystem isn’t a feature — it’s a location. Teams want agents that run on their machines, with their data, under their rules. Of the 247 records in this index, 212 are open source, and most of them can run entirely on hardware you control. This guide is the honest map of that territory: when self-hosting is worth it, and the stack layer by layer when it is.
Every hosted agent product carries the same three asterisks: your data lives on someone else’s infrastructure, your costs scale with someone else’s pricing page, and your capabilities change on someone else’s schedule. For plenty of teams those are acceptable — hosted tools are faster to adopt and someone else carries the pager. But if you’re reading this, at least one of those asterisks probably itches: a legal team that asked where the documents go, a usage bill that crossed a line, or an integration a SaaS product will never build. Self-hosting is the answer to a specific itch, not an ideology — and the stack you need depends on which itch it is.
One framing before the layers: “self-hosted” is not one decision but two. You can self-host the application — the agent platform, the workflows, the documents — while still calling a hosted model API. Or you can go all the way and run the model locally too. The first is an afternoon; the second is an infrastructure commitment. Most teams should start with the first and let their data classification, not their enthusiasm, decide about the second.
The core decision: what are you actually protecting?
Self-hosting has four honest motives, and each implies a different amount of stack. Data control — documents, credentials, customer records stay home — is satisfied by self-hosting the application and being deliberate about which model API sees what. Cost at scale — heavy, steady usage where per-seat or per-request pricing inverts — usually points the same way, plus a hard look at model spend. Control — pinned versions, deep internal integrations, patched behavior — is the open-source dividend and needs no GPU. Only compliance — data that legally may not leave — forces the full commitment: local models on local hardware.
Name your motive first. Teams that skip this step buy GPUs to solve a problem a Docker Compose file would have solved, or — worse — run a “private” deployment that quietly ships every prompt to a hosted API they never audited.
The layers
Five layers, from the thing your team touches to the thing that pages you at night.
1 · The agent platform
This is the biggest choice, and the index has real depth here. For workflow-shaped automation with AI steps, n8n is the most-adopted self-hostable engine in the entire index, with activepieces as the lighter alternative. For building agents and RAG apps behind one visual roof, difyis the broadest platform. For “chat with our documents” without ceremony, anything-llm and khoj are the fastest paths, with MaxKB in the same family. For a general autonomous workhorse, AutoGPT and goose run on your machines; for code-first teams composing their own, langchain, crewAI, and agnoare the frameworks you’ll reach for. All of these are open source; every one can run where your data lives.
— pick by the job, not the star count — a knowledge-Q&A team on an autonomous-agent platform (or vice versa) fights the tool daily, and the migration later costs more than the evaluation would have.
2 · The model question
Application self-hosted, model hosted: the pragmatic default — you keep documents and orchestration home and pay per token for frontier quality. Application and model both local: open-weight models behind an inference server on your own GPUs — the standard engines here (Ollama, vLLM, llama.cpp) aren’t agent products and so aren’t in our index, but they’re the field’s well-trodden picks. The decision rubric is your data classification: regulated or contractually-bound data forces local inference; everything else is a cost-and-quality trade you should make per use case. Local models have closed much of the gap for extraction, classification, and internal Q&A; they still trail for the hardest reasoning — which is why hybrid routing (local for the routine, API for the hard) is the pattern that keeps showing up.
— this is the layer where “self-hosted” quietly becomes marketing — audit which model API your deployment actually calls before telling anyone the data stays home.
3 · Safe execution
Agents run code, browse, and call tools — on your infrastructure, with your credentials. Give that work a blast radius. daytona exists for exactly this: secure, isolated infrastructure for running AI-generated code. Coding agents like openinterpreter and gemini-cli execute locally by design — run them in containers with scoped credentials rather than your shell account. Browser work (browser-use) deserves its own browser profile with nothing valuable logged in. The rule that generalizes: every capability an agent has is a capability an attacker can rent through a crafted input.
— an agent with unsandboxed code execution and production credentials is one prompt injection away from being your most efficient insider threat.
4 · Data, retrieval, and memory
The reason most teams self-host is that the data is sensitive — which makes the data layer the point. firecrawl and Scrapling are the self-hostable ingestion picks for web content; ragflow brings deep document parsing to your own hardware; mem0 is the in-index persistent-memory layer when the assistant should remember users across sessions. For the retrieval architecture itself — chunking, vector stores, reranking — the RAG stack guide applies unchanged; self-hosting moves the machines, not the physics.
— retrieval quality — chunking, parsing, reranking — decides whether the private assistant is actually good; privacy just decides where it runs. The RAG guide's lessons all still apply at home.
5 · Operations
Docker Compose on one box is the honest starting point for everything above, and genuinely sufficient for most internal deployments. What earns your attention instead of cluster architecture: pinned versions with a monthly upgrade slot (this ecosystem moves fast, and staleness compounds), backups of the databases under your platform — tested by restoring once — egress rules so the agent’s network access is as scoped as its credentials, and a log you’ll actually read when a run goes weird. Graduate to orchestration when load or availability demands it, and treat that as a milestone, not a starting requirement.
— the deployment that works is the one with boring ops — a documented compose file, scheduled upgrades, and backups you have actually restored once.
The mistakes that sink self-hosted deployments
- Self-hosting the app but not the model. If your self-hosted agent still calls a hosted LLM API, your data leaves the building on every request. That can be a fine trade — most teams make it — but make it consciously, not by default.
- Confusing “open source” with “free to run.” The license is free; the GPU, the ops time, the upgrades, and the 2 a.m. restart are not. Price your own hours into the comparison before calling self-hosting the cheap option.
- Running agents with production credentials on day one. An agent that can execute code or call tools will eventually do something surprising. Sandboxed execution and scoped, revocable credentials are the price of admission, not an upgrade.
- Skipping upgrades. The agent ecosystem moves monthly, and a self-hosted deployment that falls a year behind becomes unupgradeable in practice. Schedule updates like any other dependency, or the fork you never wanted chooses you.
- Kubernetes before product-market fit. One box with Docker Compose runs almost every tool in this guide well enough to learn whether the agent is useful. Add orchestration when load demands it, not because a tutorial did.
- No egress rules. The whole point was data control — then the agent gets unrestricted outbound network access and posts your context wherever a tool call points it. Restrict egress the way you restricted ingress.
Which should you pick?
- → anything-llm or khoj on one box, hosted model API, done in an afternoon.
- → n8n self-hosted, or activepieces if you want lighter.
- → dify, with ragflow when the documents are the hard part.
- → any of the above plus open-weight models behind a local inference server, real GPUs, and a security review of every tool the agent can touch.
- → langchain, crewAI, or agno, with daytona underneath anything that executes code.
Frequently asked
- Why self-host an AI agent at all?
- Four honest reasons: data control (documents, credentials, and conversations never leave your infrastructure), cost at scale (hosted per-seat or per-request pricing inverts once usage is heavy and steady), control (pin versions, patch behavior, integrate with internal systems no SaaS will touch), and compliance (some data simply may not leave). If none of those apply to you yet, hosted tools are usually the faster path — self-host when one of the four becomes real, not for its own sake.
- Do I also need to self-host the model?
- It's a separate decision. Most self-hosted agent deployments still call a hosted model API — the workflow engine, documents, and orchestration stay home while prompts go out. Full sovereignty means running open-weight models on your own GPUs behind an inference server, which is a real infrastructure commitment: it buys total data control and predictable marginal cost, and it costs you hardware, model-quality trade-offs, and another system to operate. Decide based on what your data actually is — regulated data pushes you to local models; internal tooling usually doesn't need them.
- What does self-hosting actually cost?
- For app-layer self-hosting: a modest VPS or one server, plus whatever hosted model API you call, plus the real line item — your time for setup, upgrades, and incidents. For full local inference add GPU hardware or GPU-cloud rental, which dwarfs the rest. The honest comparison isn't 'free vs $30 a seat' — it's 'ops hours plus infrastructure vs subscription,' and the crossover typically arrives with steady team-wide usage, not with the first user.
- What's the minimum viable self-hosted setup?
- One Linux box, Docker Compose, one platform from this guide, and a hosted model API key. That combination gets a team a private agent workbench in an afternoon and teaches you what you'd actually need at the next stage. Resist adding a vector database, an orchestrator, and a GPU cluster until the simple version has proven the use case.
- Is a self-hosted agent secure by default?
- No — it's private by default, which is different. You've removed the SaaS vendor from your threat model and added yourself as the security team. The specific new risks are agent-shaped: code execution needs sandboxing, tool credentials need scoping, and outbound network access needs rules, because an agent that can browse and call tools is an exfiltration path if compromised by a prompt-injection or a bad dependency. Treat the agent like an untrusted junior employee with a fast keyboard.
Primary sources
The official documentation for the tools this guide recommends:
- n8n — official docs, including the self-hosting guide.
- Dify — official site and self-hosted deployment docs.
- AnythingLLM — official site for the desktop and self-hosted versions.
- Daytona — official docs for sandboxed AI-code execution.
More stack guides
RAG
How to build a RAG knowledge assistant
Platform or compose-it-yourself, then the seven layers — ingestion, chunking, vector store, retrieval, orchestration, memory, and eval.
Voice
How to build a voice AI agent
Latency is the tyrant. Platform vs compose, then telephony, speech-to-text, the LLM, text-to-speech, turn-taking, and testing.
Coding
How to build a coding agent
Adopt an agent or build your own harness, then the layers — the agent loop, code context, safe execution, the model, memory, and review.
Browser
How to build a browser-automation agent
Perception is the hard part, not clicking. DOM vs vision vs VLA, then the runtime, the driver, the model, planning, and reliability.
Autonomous
How to build an autonomous agent
Autonomy is a control problem, not a capability problem. Adopt, build, or hand-roll — then framework, tools, memory, execution, orchestration, and governance.
MCP
How to build an MCP server
The protocol is trivial; tool design is the craft. Wrap, build, or aggregate — then SDK and transport, tool design, resources, reliability, and distribution.
Frameworks
How to choose an agent framework
Do you need one at all? The five archetypes — code-first, role-based, typed, TypeScript-native, visual — and the criteria that actually predict regret.
Workflow
How to build AI workflow automation
Most “agents” should be workflows with LLM steps. Triggers, the engine, the LLM step, approval gates, reliability — and when to graduate to an agent loop.