IMO it's all about privacy. Perhaps also availability if the main LLM providers start pulling shenanigans but it seems like that's not going to be a huge problem with how many big players are in the space.
I think a great use case for this would be in a company that doesn't want all of their employees sending LLM queries about what they're working on outside the company. Buy one or two of these and give everybody a client to connect to it and hey presto you've got a secure private LLM everybody in the company can use while keeping data private.
I’ll add to this that while I couldn’t care less about open AI seeing my general coding questions, I wouldn’t run actual important data through ChatGPT.
With a local model, I could toss anything in there. Database query outputs, private keys, stuff like that. This’ll probably become more relevant as we give LLM’s broader use over certain systems.
Like right now I still mostly just type or paste stuff into ChatGPT. But what about when I have a little database copilot that needs to read query results, and maybe even run its own subset of queries like schema checks? Or some open source computer-use type thingy needs to click around in all sorts of places I don’t want openAI going, like my .env or my bash profile? That’s the kinda thing I’d only use a local model for
I think a great use case for this would be in a company that doesn't want all of their employees sending LLM queries about what they're working on outside the company. Buy one or two of these and give everybody a client to connect to it and hey presto you've got a secure private LLM everybody in the company can use while keeping data private.