I wanted to add search providers to my LiteLLM config and it ended up being a bit more interesting than expected for a couple of reasons. First, the deployment went very smoothly and then did not work at all. After a lot of troubleshooting and a significant deviation from a…
I have been dual-booting my MacBook Pro and using that to run Ollama and serve models for the past couple of months but that quickly became an annoying usage pattern. I was not able to effectively pop online for a few minutes to work on a new tool integration…
I've done a fair amount of research since I published my initial post on a self-hosted AI infrastructure and some of my ideas about how to structure this have evolved. I'd like to provide an update on what my target is at this moment. Since…
Our first post in my self-hosted AI series is about LiteLLM (https://www.litellm.ai/). LiteLLM is an open source AI gateway. It enables you to aggregate multiple AI vendors and expose them in a single OpenAI API compatible endpoint. All of your clients can plugin to that single…
I’ve been increasingly concerned about the economics of SaaS AI providers (e.g. OpenAI & Anthropic) especially when combined with the dependency they foster. It seems to me that we are headed towards a future of AI lock-in and rising costs. Anthropic shifted their enterprise plan over to…