Auto-GPT - Benefits of a fully local instance

[SlistInc]

I wanted to ask the community what you would think of an Auto-GPT that could run locally. Currently I have the feeling that we are using a lot of external services including OpenAI (of course), ElevenLabs, Pinecone.

I personally think it would be beneficial to be able to run it locally for a variety of reasons:

• Cost: some people simply cannot afford to pay for the API's and accesses in various services
• Privacy: some people don't like the idea that OpenAI is able to see all our queries
• Quality: if we are able to fine tune some models for auto-GPT we may actually improve the output performance
• Politics: some countries (e.g. Italy) are blocking access to chatGPT. Being able to run it locally will surely help some legitimate users in restrictive governments
• Corporate influence: we don't know how the answers from OpenAI models are influenced by corporate decisions

I think we should strive for a fully local Auto-GPT. What do you think? Is this something you feel is important too?

[franklin050187]

would it be possible to use GPT4ALL instead of OpenAi ?

[SlistInc]

I think there are a few interesting alternatives such as the OpenAssistant project, Vicuna or GPT4LL. It may be possible to fine-tune a model for various tasks (e.g. one specialising in code, one in research, one in planning etc).

[innermost47]

I am very interested in it ! I'm in

[Subcode]

Maybe ChatGLM is also an option?
https://github.com/THUDM/ChatGLM-6B

[zajerao]

• I would like to add to this the suggestion that perhaps we can have a distro or live DVD or USB bootable image for auto-GPT, so it can download all those python versions libs, dependencies etc w/o conflicting with the rest of the machine, which in my case gave the macbook an indigestion.
• Perhaps it is even too soon for me to speak as I have not even been able to see auto-gpt run properly yet - it crashes and does not actually write an output to disk.
• Main reasons from that list I subscribe to: Privacy, Stability, Eco-friendliness. Privacy at the top - while corporations will soon have a secure direct VPN line with the Azure data centres, we will be left under attack from token thieves, impersonators and memory cache scanners.
  As with any such proposal it poses some problems that need pondering:
• If the online first version is not stable enough and quite early days, who is going to create -and even more importantly maintain and update- a version we can use on a local machine?
• Meanwhile I found this https://towardsai.net/p/machine-learning/llama-gpt4all-simplified-local-chatgpt which seems to give hope for a gpt4all?

[nickwhartonkc]

Second that. From: franklin050187 ***@***.***> Sent: Friday, April 7, 2023 10:58 AM To: Torantulino/Auto-GPT ***@***.***> Cc: Subscribed ***@***.***> Subject: Re: [Torantulino/Auto-GPT] Auto-GPT - Benefits of a fully local instance (Discussion #395) would it be possible to use GPT4ALL instead of OpenAi ? - Reply to this email directly, view it on GitHub<#395 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AUHVVE2PIKCHNORWO5KQYFDXAA2QTANCNFSM6AAAAAAWWUYLO4>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.******@***.***>>

[zatt]

Looks like someone is working on this here: #25
Also, fully-offline Auto-GPT: #348

[MchLrnX]

I’d love to have an option to combine multiple models or select from them.

[fblissjr]

Agree. Incorporating langchain might make this easier for LLMProvider?

[BigShadex]

Just my 2 cents...

But how difficult would it be to have it running through the normal GPT, instead of the API ..

I personally have a paid account which gives you unlimited gpt 3.5 and 20 something messages every 3-4 hours in GPT4 as most people here are probably aware of...

I'm just thinking... It might seem like abit of an old fashioned way of doing it.. But instead of running it through the API which uses tokens.. Is it possible to have it run a script which opens up a browser ( or for people running it locally a similar function ) and use it like normal?

I haven't even played around with it yet, i have only seen videos.. but the whole idea of burning through tokens and what not isn't overly appealing when im already paying for gpt 4.

If i get some time in the next few days i might see what i can come up with

[zatt]

This would be great. I’m sure that soon local fine-tuned LLMs will be able to outperform even gpt-4 on specific tasks with the right data.

Just like how community-made Stable Diffusion local models are far superior to Dall-E 2. One could even create a LLM LoRA for X task then a different LoRA for another task for more custom results.

Longer-term it would be great to allow people to opt-in sharing chat data in Auto-GPT (also ChatGPT and Oobabooga too) it could create a constant flywheel to generally improve open source local models overall.

[SlistInc]

I generally think that local models would give AutoGPT a boost, simply because it would be so much easier, more accessible (and generally cheaper) to try, test, develop and simply explore the possibilities and limitations.

[Subcode]

If you get it working through the webversion let me know, would also be awesome for people who use the free version. Might be required to build in a kind of "Wait until i can use more requests". But its preferabele over spending a lot of money on tokens. Slower but free is good.

[PHONGTHIEUHON]

Have thought about this. If this happen, we will get no rate limit (as long as 1 question asked per time) and still have gpt4.
Above that, it is free within 20$ per month.
How is your progress, can I participate?

[Subcode]

ChatGPT web version has a rate limit. about 100 msg an hour or so which you would easily reach with autoGPT

[pmb2]

Check out the issue I have posted here #461, newer more powerful, yet still light weight and nearly free models are out now.
I talk about Vicuna, but that was hours ago who knows what is out now
Ive heard about open assistant but ti think we are doing better here.
We need a jira board.

[Slamsneider]

An Auto-Gpt running, only using my 13B model would be nice. Not very powerful but as I see it it could do the trick enabling me to use natural language communicating with my AI to do advanced reasoning, planning and execution of my tasks. Adding code ad hoc. Simple tasks albeit useful.

diving into the code...

[lUdyk]

I wanted to ask the community what you would think of an Auto-GPT that could run locally. Currently I have the feeling that we are using a lot of external services including OpenAI (of course), ElevenLabs, Pinecone.

I personally think it would be beneficial to be able to run it locally for a variety of reasons:

• Cost: some people simply cannot afford to pay for the API's and accesses in various services
• Privacy: some people don't like the idea that OpenAI is able to see all our queries
• Quality: if we are able to fine tune some models for auto-GPT we may actually improve the output performance
• Politics: some countries (e.g. Italy) are blocking access to chatGPT. Being able to run it locally will surely help some legitimate users in restrictive governments
• Corporate influence: we don't know how the answers from OpenAI models are influenced by corporate decisions

I think we should strive for a fully local Auto-GPT. What do you think? Is this something you feel is important too?

change the gpt4 for open asistans model that is released on April 15 that does not have to use any api key that can be run locally ?

[SlistInc]

I think the Open Assistant model will be interesting to watch. They are collecting a lot of community curated training data and may thus be actually quite good (I saw it has quite some coding related chat-data).

[aliasfoxkde]

My vote would be to integrate with oobabooga/text-generation-webui which can easily manage various AI models on your machine locally. And with the advancements of vicuna, LoRA, 4-bit quantized, and cpp models what's possible right now is amazing. It's been stated objectively, though I don't know if there is a true way to be objective in this manner, that Vicuna has reached 90% of GPT4 (and others say at least the level of 3.5).

[DaveChini]

I see this messed up all the time. Vicuna is 90% of chatGpt so 3.5, though it has been the most impressive "open source" model that I've tried. The biggest benefit to running a local model is it can be finetuned on autogpt's task/goal input json output. Meaning it would be more reliable and generally free to use.

[aliasfoxkde]

Agreed, but I think the biggest benefit for me is being able run Auto-GPT and other auto-prompts non-stop without the need to use or pay for credits. At the very least for testing.

[Aaronminer1]

we could have the local agen communicat with another gpt model running on the same machine or on a second machine
Title: Building a Locally Hosted GPT-Neo Chatbot Accessible Over a Network

Objective: The goal of this project is to create a locally hosted GPT-Neo chatbot that can be accessed by another program running on a different system within the same Wi-Fi network. This will replace the current dependency on OpenAI's API, allowing the chatbot to be used without the need for an API key and internet access to OpenAI's servers.

Project Workflow:

Set up the local environment

Install necessary Python libraries, including Transformers and Flask.
Download and configure the GPT-Neo model.
Develop a Flask web application

Create a Flask app with a single endpoint (e.g., /generate) to handle chatbot requests.
Implement the chatbot functionality using the GPT-Neo model, taking input text and returning generated responses.
Host the Flask app on the local system

Run the Flask app on the local machine, making it accessible over the network using the machine's local IP address.
Modify the program running on the other system

Update the program to send requests to the locally hosted GPT-Neo model instead of using the OpenAI API.
Test and troubleshoot

Ensure that the other system can successfully send requests to the locally hosted GPT-Neo chatbot and receive accurate responses.
Document the project

Create a detailed project description, including the objective, workflow, and any necessary instructions for setting up and using the chatbot.
By following this workflow, you will replace the dependency on OpenAI's API with a locally hosted GPT-Neo model that can be accessed by another system on the same Wi-Fi network. This will provide a more efficient and cost-effective solution for using the GPT-Neo chatbot within your local environment.

If you decide to host both projects on the same machine, the workflow will become more straightforward. You can directly integrate the GPT-Neo model into the other program without needing to create a separate web service using Flask. Here's an updated workflow for this scenario:

Set up the local environment

Install necessary Python libraries, including Transformers.
Download and configure the GPT-Neo model.
Modify the program running on the same system

Update the program to incorporate the GPT-Neo model directly instead of making API calls to OpenAI.
Replace the API call code with the code that uses the GPT-Neo model to generate responses based on the input text.
Test and troubleshoot

Ensure that the program can successfully use the locally hosted GPT-Neo model and receive accurate responses.
Document the project

Create a detailed project description, including the objective, workflow, and any necessary instructions for setting up and using the integrated GPT-Neo model.
By hosting both projects on the same machine and directly integrating the GPT-Neo model into the other program, you eliminate the need for a separate web service and simplify the overall architecture. This approach will provide a more efficient solution for using the GPT-Neo chatbot within your local environment.

[Aaronminer1]

When you start the agent you could have it ask if you want to use local resources or enable api recourses

[zatt]

Just sharing this in case you haven't seen it: https://github.com/hyperonym/basaran

"Basaran is an open-source alternative to the OpenAI text completion API. It provides a compatible streaming API for your Hugging Face Transformers-based text generation models."

[CodR-2]

Hi @zatt Have you gotten basaran to work with it? I keep getting this error: openai.error.AuthenticationError: Incorrect API key provided: http://1*******************ions. You can find your API key at https://platform.openai.com/account/api-keys.

[CodR-2]

I used this as the api key: http://127.0.0.1/v1/completions and I get the error still.

[wiseman-timelord]

Its been done already with AutoLLM : https://github.com/kabeer11000/autollm
I have been playing with it in my fork of AutoLLM, I think its based on AutoGPT 0.1.0, it wont work correctly for file operations, but the code for the LLM works, at least for specific few models, such as, "ggml-vicuna-13b-1.1-q4_1.bin" and "gpt4-x-alpaca-13b-native-ggml-model-q4_0.bin".

My conclusion was the best configuration of LLMs would be something like default to Vicuna 8-bit, so we can get the 4096k tokens, and something like codegen 7b for specifically writing code, maybe GPT4-X-Alpaca for brainstorming/creative writing, specfic models for specific purposes with a default model.

[CodR-2]

Hi, do you have a tutorial I can follow? Preferably one that uses docker? Thanks for chiming in.

[mxcl]

Project looks dead, which is a shame. Did it evolve to something else?

[wiseman-timelord]

They forked it off Llama.Cpp instead of AutoGPT, prob not best. But the code is still good for up to AutoGPT 1.2, and I think AutoLLM is based on AutoGPT 1.0.

For the best models see here, I am currently interested in using the OpenCodeInterpreter model with AutoGPT, as it has chat and high programming scores in 7B.

[lc0rp]

Merging this with #34