Use local models instead of API-GPT

[AestheticMayhem]

You guys think this is in reach? Hackathon...

[rkeshwani]

This is tricky because training a local model is resource intensive and all of the ones out there are no where close to gpt-4's quality.

[AestheticMayhem]

Thank you very much, im waiting for your version patiently, if you can share it here than when you have finished, so we get updated on your progress

[AestheticMayhem]

Awesome dude, thank you, amazing

[HWiese1980]

I've created a fork, pushed my changes and created a Draft-PR (hoping that it's okay to do that). Here you go: #3032

It's not finished yet.

Regarding 30B models: I have not managed yet to get them running. It seems like I have too little RAM (32GB).

[Giorgiosambere]

I think this is the lines to backdoor and hook up in it... Here is the current implementation of the env , language_model_util in the main files of the auto-gpt repository script folder, including the changes made. However, I need to integrate the local host from the language model output program file.

I've integrated Oobabooga text-generation-ui API in this function. It triggers the API but the model (GPT-4 Alpaca 13B 4bit 128g) fails miserably understanding what it is supposed to do. I'm going to try this again with a 30B GPT-4 Alpaca model, but I'm not particularly optimistic...

this already exists https://github.com/Josh-XT/Agent-LLM/blob/main/provider/oobabooga.py if you can make a PR on this one it would be good because it also can support addins unlike AutoGPT

it does have some problems, however

\oobabooga.py", line 25, in instruct
    return response.json()['data'][0].replace("\n", "\n")

edit: this may also be good for running 30B since it supports llama.cpp and koboldcpp.exe endpoints (much easier to get RAM mode running than ooba)

404 is there any alternative? i really want to run this

[cat6666666]

how to edit .env and llm_untils.py? could somebody post a step by step list to help me set down all files, that i can use local LLM models which is loading by LM Studio? Thx!

[determo13]

Someone ask Auto-Chatgpt to rewrite Auto-chatgpt so it can work with local vicuna or alpaca models :) done

[sollxy]

tried it but with without gtp4 the code review function doesnt work :)

[BryanOgd]

is there a standard api format for parsing input and output of any NLP (large or small) model? - [GPT-NeoX-20B] for instance as well.

[endolith]

I tried to get Auto-GPT to improve Auto-GPT, but it's currently too buggy to fix its own bugs. :/

[GoZippy]

I tried to get Auto-GPT to improve Auto-GPT, but it's currently too buggy to fix its own bugs. :/

same here - but this time I had it research how to handle python errors using github examples and then write a launcher for autogpt python scripts... working on a new branch now.

[AestheticMayhem]

I tried to get Auto-GPT to improve Auto-GPT, but it's currently too buggy to fix its own bugs. :/

same here - but this time I had it research how to handle python errors using github examples and then write a launcher for autogpt python scripts... working on a new branch now.

thanks for sharing

[GoMightyAlgorythmGo]

You guys think this is in reach? Hackathon...

-Better but more ressource intensive: https://www.youtube.com/watch?v=jb4r1CL2tcc&ab_channel=MartinThissen
-Probably good + low ram: https://www.youtube.com/watch?v=dF2eu-C87Pk&list=PLNfIIU7EN4iO8GQ_3Jsq5aoIHYbnDQXg8&index=14&ab_channel=PromptEngineering
-Can confirm works, also maybe lower ram: https://www.youtube.com/watch?v=bGcrTGsSNaY&list=PLNfIIU7EN4iO8GQ_3Jsq5aoIHYbnDQXg8&index=22&ab_channel=MatthewBerman
-Koala: a chatbot fine-tuned from LLaMA on user-shared conversations and open-source datasets. This one performs similarly to Vicuna. seems to be also open source and legal because no openAI data (not sure tho)

Virtual memory you dont need 30gb

Model comparisons: https://chat.lmsys.org/

[marinohardin]

Dumb question, I haven't used it yet; does Vicuna run purely on CPU? No CUDA / MPS / GPU acceleration?

[alreadydone]

https://github.com/ggerganov/llama.cpp#description

[karingijs2727]

Maybe pythia models are good for it?

[GoMightyAlgorythmGo]

There is/coming "RedPjama" language model / 1.2 Trillion free token dataset to download, heard about it from yt video on 19.04.2023

[HWiese1980]

Dumb question, I haven't used it yet; does Vicuna run purely on CPU? No CUDA / MPS / GPU acceleration?

You can use them on CPU, but you can also utilize CUDA acceleration. They are already amazingly good for conversational tasks. I've been playing around with oobabooga/text-generation-ui during the last couple days. It's amazing what this thing can already do locally. There are plenty of freely available models on HuggingFace. I've played with Vicuna 13B and GPT4 Alpaca 13B, both in their 4bit variant. They work like a charm. However, I've only tried some conversation so far. I have not seriously delved into code generation territory yet.

For CUDA acceleration you'll need a beefy GPU though. Upper class GPUs are capable of running the 13B models. My 3090Ti also runs 30B models, but they are noticeably slower than the 13B models. For Auto-GPT I would try it again with a 30B model as here speed isn't that much of the essence as it is in a simulated conversation.

[SlistInc]

Please consider that there is active work on getting plugins to work, especially by @BillSchumacher who is making good headway there and also managed to hook up autoGPT to Vicuna (I think). The idea is that such a connection to a local LLM is then made through plugins. This will avoid bloating of autoGPT but also allow the community to participate much more easily by just focusing on plugins.

[AestheticMayhem]

impressive, thanks man

[karingijs2727]

I have a question: are the models need to be fine tuned like a chatbot or just normal models are better for this task?

[AestheticMayhem]

models should be instruction finetuned to comprehend better, thats why gpt 3.5 and 4 are still at the top, but OpenAI revealed a promising model, we just need the link between autogpt and the local llm as api, i still couldnt get my head around it, im a novice in programming, even with the help of chatgpt, i would love to see an integration of the gpt4all v2 model, because the vicuna model needs more than 8GVram

[Kuchiriel]

I think if we imitate the OpenAI REST API and use it as a middleware between oogabooga text gen webui, we can just point the auto-gpt to the api replica. Just an idea, I am not doing this. If this happens to come into the main branch, we can use it by changing the param in the .env file to the name of the local llm, and the system understand it and point to a local REST connected to the ooogabooga.

[GoZippy]

Id rather just add another layer of options... Select local LLM or remote LLM and define a common set of functions that then go thru a transaction manager for whichever LLM selected that translates requests and formats it for whichever API or local service is selected for that specific Agent... Allow user to install and add credentials for any of them... Connector for oobabooga would be nice... Needs some sort of translation later though... I'd propose adding translation layer and normalizing agentgpt data layers internally so it can be translated and handled in layer for selected third party or local service...

[endolith]

Doesn't this already exist?

https://github.com/abetlen/llama-cpp-python
https://github.com/keldenl/gpt-llama.cpp

[Anonym0us33]

https://github.com/LostRuins/koboldcpp

[lc0rp]

Duplicate of #395