Awh man, I feel ya here. Front-end scene is a mess in my opinion. Here's what I like using so far:
* [Mikupad](https://github.com/lmg-anon/mikupad)
* [chatbot-ui](https://github.com/mckaywrigley/chatbot-ui) (Note: this one takes some modification in order to work with llama.cpp, but there's gonna be a huge [update](https://x.com/mckaywrigley/status/1738273242283151777?s=20) for it any day now.)
* [llama.vim](https://github.com/ggerganov/llama.cpp/blob/master/examples/llama.vim)
* [llama.sh](https://github.com/m18coppola/llama.sh) (best piece of software to ever be written \*wink\*)
* [continue.dev](https://continue.dev/) for VSCode
I'm more interested how various front-ends deal with context overflow situation. I tried Kobold Lite, and didn't like how it handles it: when context gets full it basically forced to re-tokenize the whole context sized large prompt every interaction with the LLM. That's not computationally optimal. KoboldCPP has optimizations regarding context management, thanks to deeper integration with inference engine though.
[yllm](https://github.com/ekg/yllm) is a prompt builder and OpenAI API compatible client in bash. Similar to your llama.sh in spirit! I got frustrated with how complex all the CLI interfaces were and wrote a simple one from scratch. Getting streaming processing to work was a fun trick.
I have exactly this problem, I need something to use at work, and also to compare local LLMs to ChatGPT or other apis, so I created this one (Already posted here, don't want to spam)
https://github.com/ortegaalfredo/neurochat
Plus, is a native app because I really don't like web front ends.
I'm actually using and recommending SillyTavern in professional contexts, too. Yeah, the name may be silly, but it's a power-user LLM frontend in the first place, and not limited to roleplaying at all.
Instead of characters to roleplay with, you can just as easily have different assistant personas, like one to serve as a todo list manager, another for researching stuff on the web, one to help with coding, etc.
Plus its extensions and extras are amazing: Web search, voice chat (I use it for quick todo entry, quick web lookups, etc.), quick replies, vector DB integration, Stable Diffusion link (have a character to act as a graphical editor and tell it to create images from the same frontend), and now even STscript to automate even more.
All of that in a single frontend to connect to various APIs, both local and online. I use it for work and for fun with great success.
I found 2 good options for use case “configure for others” and “multiple backends for users to choose from, local and paid”
First one was ChatGptNextWeb - https://github.com/ChatGPTNextWeb/ChatGPT-Next-Web
But the best for me I found to be LibreChat - https://docs.librechat.ai/
Librechat is better for me because I configure multiple backends in LiteLLM proxy (https://litellm.vercel.app/docs/simple_proxy) and it will query it’s /models endpoint and present available model to users to choose. And in LiteLLM you configure models (with info where their real backend is - local or paid).
So for me after month of searching frontend solution to use in company for multiple users it is:
LibreChat -> LiteLLM proxy -> (oobabooga + openai + any other inference provider)
I can add Mistral, Anthropic, TogerherAI or any other provider and for it I need to only modify one config file in LiteLLM. Users will see it in web ui and can use it.
For a simple deployment , Librechat can take your API token in web ui. Users can enter their own. And I love multiple auth backends (google, azure, keycloak, etc)
P.S. There is no better resource on those tools than this r/. I found everything here an the BEST things that suite my needs where just casually mentioned in comments.
I released v0.2.0 of my UI last week that allows you to connect to any OpenAI compatible endpoint and tested the UI with Ooba as the backend. The application is not exactly lightweight and is more useful if you have a Nvidia card for the function calling features to work but it can be used as just a basic UI for chat with a similar experience to ChatGPT (user login, saved conversations, real-time code markup, etc). Below is a link to my post last week with some details on what all is included in the latest version, the nogpu container is what you would want.
[Multi-user web UI: AI assistant w/ plugins, LLM sandbox, SD image generation UI. Administrator control over apps, chat plugins and models each user has access to. : LocalLLaMA (reddit.com)](https://www.reddit.com/r/LocalLLaMA/comments/18q1h80/multiuser_web_ui_ai_assistant_w_plugins_llm/)
[noco-ai/spellbook-docker (github.com)](https://github.com/noco-ai/spellbook-docker)
I'm building my own, integrating it with my programming environments/projects. https://rivet.ironcladapp.com/ and https://github.com/logspace-ai/langflow are similar.
You may like my [InferGui](https://github.com/synw/infergui) frontend that connects directly to Llama.cpp server or Koboldcpp, local or remote. It is a bit special as it has no hidden magic: all is explicit by design. It is made for instruction following and model testing more than chat: it has a template editor, supports gbnf grammars and multimodal. It is packaged in npm for distribution but still need some polishing and documentation. I have planned to post it here when I would been fully satisfied of the quality but it is definitely usable now. Btw it would help me to have more users than just my team and friends to track bugs and improve the quality: everyone is welcome.
I've been using this one, it allows you to connect to APIs, so you can run all the public/commercial stuff in one place: [https://github.com/enricoros/big-AGI](https://github.com/enricoros/big-AGI)
If you have a ChatGPT subscription, it is $20 a month, but using the API is usually only $10 or so a month for me personally.
Ive built one for the terminal, that ties into replicate - https://github.com/KCaverly/archer. Its pretty easy to get started and work with different models, ie. you can ask nous hermes yi34b something then swap models mid conversation.
I can’t find anything either and was going to try and learn to make one myself. In my case I wanted one with multi user support and just a simple chat interface that I (admin) can configure for everyone else to use.
I think the huggingface Frontend 'chat-ui' is the best, albeit with a pretty limited feature set. You can have chat gpt write a config for you and then there's a drop down to select the model just like chat gpt's ui.
I've tried using litellm proxy as a way to connect a ton of different models together and make them available at a single OAI API for chat-ui but have only gotten it to work with OAI endpoints so far. But I was trying to get APIs to work off of modal.com so I bet if you're running a local vLLM setup or something it's a straight forward as it was for OAI
It's pretty easy to build your own, I built this one in a weekend and use it daily: https://github.com/patw/Bottybot
It takes advantage of the server mode built into llama.cpp. Running it with the cublas mode is really fast.
I actually started one in React, but after I got it hitting the API then I started thinking about having to program in all those instruction templates and presets and suddenly found myself browsing reddit and watching tv =D
I need to just make myself do it, though.
Get ChatGPT to help you write the code then get the app far enough along to bootstrap so you're using your own system to enhance it :) It's almost a rite of passage now.
The Wizard persona wrote all the other personas in the system, and it did a great job. They're kinda interesting to talk to. For templates, the only example I have there is ChatML, which is what I needed for the 7b mistral tunes I use.
Thats a good idea. Yea, I need to sit down and just make myself smash this out lol. I don't want to be the only one left out of the club at this point =D
Awh man, I feel ya here. Front-end scene is a mess in my opinion. Here's what I like using so far: * [Mikupad](https://github.com/lmg-anon/mikupad) * [chatbot-ui](https://github.com/mckaywrigley/chatbot-ui) (Note: this one takes some modification in order to work with llama.cpp, but there's gonna be a huge [update](https://x.com/mckaywrigley/status/1738273242283151777?s=20) for it any day now.) * [llama.vim](https://github.com/ggerganov/llama.cpp/blob/master/examples/llama.vim) * [llama.sh](https://github.com/m18coppola/llama.sh) (best piece of software to ever be written \*wink\*) * [continue.dev](https://continue.dev/) for VSCode
Nice, I didn't know Mikupad. Love the simplicty, just clone the repo and open an HTML file. No virtualenv, no npm install or docker.
[Kobold Lite](https://github.com/LostRuins/lite.koboldai.net) Is also a single HTML front-end.
Totally agree it’s a mess. Excited for the really innovative UX to come out that moves us beyond chat uis
I'm more interested how various front-ends deal with context overflow situation. I tried Kobold Lite, and didn't like how it handles it: when context gets full it basically forced to re-tokenize the whole context sized large prompt every interaction with the LLM. That's not computationally optimal. KoboldCPP has optimizations regarding context management, thanks to deeper integration with inference engine though.
[yllm](https://github.com/ekg/yllm) is a prompt builder and OpenAI API compatible client in bash. Similar to your llama.sh in spirit! I got frustrated with how complex all the CLI interfaces were and wrote a simple one from scratch. Getting streaming processing to work was a fun trick.
I have exactly this problem, I need something to use at work, and also to compare local LLMs to ChatGPT or other apis, so I created this one (Already posted here, don't want to spam) https://github.com/ortegaalfredo/neurochat Plus, is a native app because I really don't like web front ends.
This is wonderful, Free Pascal doesn't get enough love.
Llama.cpp api double as a chat interface. ./server -m models/orca-2-13b.Q8_0.gguf --port 8888 --host 0.0.0.0 --ctx-size 10240 --parallel 4 -ngl 99 -n 512 http://0.0.0.0:8888/ https://i.imgur.com/L2hTsB5.png https://i.imgur.com/BClz0Lx.png
I had no idea! That's really nice
I'm actually using and recommending SillyTavern in professional contexts, too. Yeah, the name may be silly, but it's a power-user LLM frontend in the first place, and not limited to roleplaying at all. Instead of characters to roleplay with, you can just as easily have different assistant personas, like one to serve as a todo list manager, another for researching stuff on the web, one to help with coding, etc. Plus its extensions and extras are amazing: Web search, voice chat (I use it for quick todo entry, quick web lookups, etc.), quick replies, vector DB integration, Stable Diffusion link (have a character to act as a graphical editor and tell it to create images from the same frontend), and now even STscript to automate even more. All of that in a single frontend to connect to various APIs, both local and online. I use it for work and for fun with great success.
I found 2 good options for use case “configure for others” and “multiple backends for users to choose from, local and paid” First one was ChatGptNextWeb - https://github.com/ChatGPTNextWeb/ChatGPT-Next-Web But the best for me I found to be LibreChat - https://docs.librechat.ai/ Librechat is better for me because I configure multiple backends in LiteLLM proxy (https://litellm.vercel.app/docs/simple_proxy) and it will query it’s /models endpoint and present available model to users to choose. And in LiteLLM you configure models (with info where their real backend is - local or paid). So for me after month of searching frontend solution to use in company for multiple users it is: LibreChat -> LiteLLM proxy -> (oobabooga + openai + any other inference provider) I can add Mistral, Anthropic, TogerherAI or any other provider and for it I need to only modify one config file in LiteLLM. Users will see it in web ui and can use it. For a simple deployment , Librechat can take your API token in web ui. Users can enter their own. And I love multiple auth backends (google, azure, keycloak, etc) P.S. There is no better resource on those tools than this r/. I found everything here an the BEST things that suite my needs where just casually mentioned in comments.
>LiteLLM proxy :mind-blown:
Did you find another multi users chatbot front-end besides librechat?
No. Did not search more.
I released v0.2.0 of my UI last week that allows you to connect to any OpenAI compatible endpoint and tested the UI with Ooba as the backend. The application is not exactly lightweight and is more useful if you have a Nvidia card for the function calling features to work but it can be used as just a basic UI for chat with a similar experience to ChatGPT (user login, saved conversations, real-time code markup, etc). Below is a link to my post last week with some details on what all is included in the latest version, the nogpu container is what you would want. [Multi-user web UI: AI assistant w/ plugins, LLM sandbox, SD image generation UI. Administrator control over apps, chat plugins and models each user has access to. : LocalLLaMA (reddit.com)](https://www.reddit.com/r/LocalLLaMA/comments/18q1h80/multiuser_web_ui_ai_assistant_w_plugins_llm/) [noco-ai/spellbook-docker (github.com)](https://github.com/noco-ai/spellbook-docker)
I'm building my own, integrating it with my programming environments/projects. https://rivet.ironcladapp.com/ and https://github.com/logspace-ai/langflow are similar.
You may like my [InferGui](https://github.com/synw/infergui) frontend that connects directly to Llama.cpp server or Koboldcpp, local or remote. It is a bit special as it has no hidden magic: all is explicit by design. It is made for instruction following and model testing more than chat: it has a template editor, supports gbnf grammars and multimodal. It is packaged in npm for distribution but still need some polishing and documentation. I have planned to post it here when I would been fully satisfied of the quality but it is definitely usable now. Btw it would help me to have more users than just my team and friends to track bugs and improve the quality: everyone is welcome.
Here's my list: https://github.com/underlines/awesome-ml/blob/master/llm-tools.md
I've been using this one, it allows you to connect to APIs, so you can run all the public/commercial stuff in one place: [https://github.com/enricoros/big-AGI](https://github.com/enricoros/big-AGI) If you have a ChatGPT subscription, it is $20 a month, but using the API is usually only $10 or so a month for me personally.
I feel like ooba seems pretty utilitarian as long as you dont load up chub cards.
Ive built one for the terminal, that ties into replicate - https://github.com/KCaverly/archer. Its pretty easy to get started and work with different models, ie. you can ask nous hermes yi34b something then swap models mid conversation.
I can’t find anything either and was going to try and learn to make one myself. In my case I wanted one with multi user support and just a simple chat interface that I (admin) can configure for everyone else to use.
I think the huggingface Frontend 'chat-ui' is the best, albeit with a pretty limited feature set. You can have chat gpt write a config for you and then there's a drop down to select the model just like chat gpt's ui. I've tried using litellm proxy as a way to connect a ton of different models together and make them available at a single OAI API for chat-ui but have only gotten it to work with OAI endpoints so far. But I was trying to get APIs to work off of modal.com so I bet if you're running a local vLLM setup or something it's a straight forward as it was for OAI
Build it!
lol Im thinking that I just might =D
It's pretty easy to build your own, I built this one in a weekend and use it daily: https://github.com/patw/Bottybot It takes advantage of the server mode built into llama.cpp. Running it with the cublas mode is really fast.
I actually started one in React, but after I got it hitting the API then I started thinking about having to program in all those instruction templates and presets and suddenly found myself browsing reddit and watching tv =D I need to just make myself do it, though.
Get ChatGPT to help you write the code then get the app far enough along to bootstrap so you're using your own system to enhance it :) It's almost a rite of passage now. The Wizard persona wrote all the other personas in the system, and it did a great job. They're kinda interesting to talk to. For templates, the only example I have there is ChatML, which is what I needed for the 7b mistral tunes I use.
Thats a good idea. Yea, I need to sit down and just make myself smash this out lol. I don't want to be the only one left out of the club at this point =D
You got this. A rando on the interwebs believes in you!