I’ve been increasingly concerned about the economics of SaaS AI providers (e.g. OpenAI & Anthropic) especially when combined with the dependency they foster. It seems to me that we are headed towards a future of AI lock-in and rising costs. Anthropic shifted their enterprise plan over to token based consumption cost models and I anticipate most other vendors are going to follow in that direction. It seems obvious to me that flat-rate plans are economically unsustainable for AI SaaS vendors as they push for more adoption and deeper adoption.
With that in mind, I want to start exploring what an alternative to vendor lock-in and ever rising costs could look like. I am testing the open source offerings in this space and finding out how compelling an open source AI technology stack can be. I will also be exploring how a hybrid self-hosted and commercial technology stack can function and why you might want to do that. I will be documenting these efforts over a series of posts in the coming weeks and each post will be a deeper dive on a particular part of the stack. I don’t have a lot of preconceived notions of how well this stack will work but I’m exploring this from a place of curiosity and learning. I have the advantage of extensive home lab experience and the experience of running open source software as infrastructure for the company I work for.
Let’s talk a bit about the stack I intend to explore. This list will likely not be exhaustive as I will likely find additional rabbit holes to fall into as I explore and learn more about this. But for organizational purposes, the upcoming posts will likely each be focused on one of these topics:
LiteLLM - A sort of AI middleware. It allows multiple AI backends to be centralized into a single compatible API. We’ll consider the utility of seamlessly being able to switch users between Anthropic or OpenAI or Gemini or Ollama.
OpenWebUI and LibreChat - We’ll explore the chat UI that most users would be interacting with on a daily basis. I’ll compare and contrast with commercial offerings like Claude and ChatGPT.
OpenCode - Developers love Claude Code and I want to kick the tires on the open source equivalent. From what I understand, Anthropic’s models aren’t exactly at the top of the leaderboard when it comes to AI coding so lets look at the utility of an agent that can switch models whenever we want to.
Ollama - I’ll run open weight models on commodity hardware in my basement. I’m curious what we can do without exotic hardware.Mobile Apps - We’ll take a look at the mobile app ecosystem for open source tools and see how they stack up against the commercial offerings.
MCP Server Gateways - I’ll build my own MCP server gateway. We’ll talk about strategies for consolidating MCP connections and enabling transparency and eliminating redundancy.
OpenTelemetry - We’ll talk about enabling observability in an AI tool stack so we can see what is happening with routing decisions, backend latency and token usage.
Edge Inference - I want to look at the current state of edge inference and whether we can use edge inference to intelligently route tasks to different AI backends based on complexity.
Agentic AI - I will explore extending our AI stack by integrating self-hosted n8n to enable agentic AI capabilities.
Most importantly, I won’t be working with any particularly exotic hardware for any of this. I run a mini-pc with a Ryzen CPU running Proxmox for my VM’s and I have a MacBook Pro for running AI models locally. I’m setting out to validate whether you can do meaningful local inference with hardware that should be well within the means of most businesses that might want to explore this operating model. And for when we run into capacity barriers, I will develop strategies to thoughtfully and economically integrate frontier models into the self-hosted AI stack. At some point, I may look at expanding to a Mac Studio or a Dell Pro Max with GB10 but that’s just speculation at this point.
I suspect that self-hosted AI infrastructure is far more usable and cost effective than one might initially suspect. I am set on exploring this concept further and documenting where this concept holds up and where it still falls short of frontier model vendors.
I’ll be back soon with a deeper dive on LiteLLM and then we’ll talk more about open weight models after that.