I've done a fair amount of research since I published my initial post on a self-hosted AI infrastructure and some of my ideas about how to structure this have evolved. I'd like to provide an update on what my target is at this moment.
Since a picture is worth a thousand words, I'll start with a diagram.
1000 Words of Explanation
LiteLLM is still at the center of my architecture. As outlined in my prior post, I have found it to be a capable and feature-rich platform to stitch all this together. Plus it seems to be under pretty active development as there have been frequent updates and new features being added.
I was running into limitations using my MacBook Pro. Firstly, there was just an issue of availability because I also want to use it as a laptop frequently and so there was always some barrier to using it to serve models for lab work. So I purchased a Strix Halo system that will be dedicated to serving models. I also purchased a Mac Mini and I will be using it to run a small LLM that will be used as a complexity classifier for queries. More on that will be coming soon.
I've also settled into a more specific suite of tools that I want to explore more.
- Perplexica - Supposedly comparable to Perplexity. It's an AI powered research tool.
- Open Notebook - I was looking for something similar to NotebookLM and this is what surfaced.
- OpenCode - This is an open source agentic coding harness. Similar to Claude Code or Codex.
- OpenClaw - This is an agentic AI platform that became incredibly popular this year.
- OpenWebUI - This is a more traditional chat interface similar to ChatGPT.
And I have decided to add some adjacent infrastructure pieces.
- Headroom - This is a query compression tool that supposedly can have a pretty dramatic effect on token usage.
- Qdrant - This is a popular vector store database. I would like to learn more about how RAG works and this is a necessary component.
- SearXNG - This is a self-hosted metasearch engine. I am using it to expose a search tool to harnesses I am using.
Just in case you are counting, I did not hit my 1000 word target on the details of my planned architecture. I will be posting about each one as I set them up and integrate so there will be lots more interesting posts coming up.