Feeding all the data it needs to complete a task

[lduperval]

Hi,

I want AutoGPT to use files I have created as input for its task. I have ingested the data and in my goals, I tell it not to go to the Internet to find more information. In short, I want it to take the information from an interview transcript and write an article from it.

However, when I run the task with the goals, it keeps saying it needs the transcript, but I don't know how to provide it.

So now, it's stuck in a do_nothing loop.

Any tips?

NEXT ACTION:  COMMAND = do_nothing ARGUMENTS = {}
SYSTEM:  Command do_nothing returned: No action performed.
 THOUGHTS:  I need to receive the transcript of the interview in order to determine the main points discussed. I will ask the user for the transcript.
REASONING:  Without the transcript, I cannot determine the main points discussed in the interview.
PLAN:
-  Ask the user for the transcript of the interview.
-  Once I have the transcript, I can determine the main points discussed and begin writing the article.
CRITICISM:  I need to be more proactive in asking for information when I don't have enough context to make a decision.
NEXT ACTION:  COMMAND = do_nothing ARGUMENTS = {}

Thanks,

L

[bjornfix]

Current version is only capable of reading .txt files. So the text you want it to read has to be in that format.

Here is how I do it:

Manually make two folders in the directory auto_gpt_workspace.

Call the first one transcript. Put the text files you want it to read inside that directory. Add a line at the end of each file saying what to read next. I would write "Read 2.txt next" on a line by itself after the content.

Make another one called article.

Set Goal 1 like this:

The directory transcripts is the working directory for you and your team from now on. All documents regarding this project can be found in the directory transcripts. Read the content of all the files in that directory before proceeding with any other goals. The content in those documents can NOT be ignored. Start reading 1.txt first. At the end of that document you will be instructed what to do next.

Goal 2 will then be: Write an article only based on what you read. You are not allowed to use internet or Google to gather more information. You are not allowed to start any agents for this task. Save the article you wrote to the folder article. Name it article.txt

Give it the role Article Writer

[bjornfix]

It's not easy to wrap your head around these concepts. I never though (as having background as a programmer and SEO guy) to sit and think about role play to get a piece of software to behave in a certain way.

But, here we are :)

[cooleydw494]

honestly I tried a version of your method that doesn't include specifically telling the AI to read the files. In my mind, being reminded that it knows about a topic already should be enough if pre-seeding is working correctly. If pre-seeding works, the point is that you don't have to have the AI read all those files after you start. I did everything else the way you did, but honestly it doesn't seem to have an impact. While your method seems like it would get the AI to ingest the files, wouldn't it read them one by one just like it would if you told it to do that without using pre-seeding? I feel like there's something about this I'm just not getting.

[cooleydw494]

btw I had Chat GPT 4 write a script that converts an entire directory to a new directory with headers explaining what the file represents and footers that do what you did (say what file should be read next). It is written in my case to always use the README as the entry point (the first file to read in other words). It is written with several specific elements of my own use case in mind, but it should be extremely helpful to anyone with a little coding knowledge. If anybody less technical wants me to help them tool it for their use-case I'm happy to do so.

The script is intended to be run in the autogpt_workspace directory, and targets an existing directory there and copy it into a new directory with nothing but text versions. I've specified the type of file extensions I'm looking for and other things. I had Chat GPT add comments that are helpful to anyone who wants to tweak the script.

import os
import shutil

def main():
    # Modify the source directory path according to your needs
    source_dir = './VirtueMaster'

    # Modify the output directory path according to your needs
    output_dir = './VirtueMasterTxt'

    # Modify the file types you want to process
    file_types = ('.md', '.js', '.ts', '.json', '.sql')

    # Initialize the variable for the README.md file
    readme_file = None

    # Check if the output directory exists, if not, create it
    if not os.path.exists(output_dir):
        os.makedirs(output_dir)

    # Walk through the source directory to find files with the specified file types
    for root, dirs, files in os.walk(source_dir):
        for file in files:
            if file.endswith(file_types) and file != 'README.md':
                # Create the input and output file paths
                input_file_path = os.path.join(root, file)
                relative_path = os.path.relpath(input_file_path, source_dir)
                new_root = root.replace(source_dir, output_dir)
                output_file_path = os.path.join(new_root, f'{os.path.splitext(file)[0]}.{os.path.splitext(file)[1][1:]}.txt')

                # Create the output directory if it doesn't exist
                if not os.path.exists(new_root):
                    os.makedirs(new_root)

                # Read the input file and write the output file with custom text
                with open(input_file_path, 'r', encoding='ISO-8859-1') as infile, open(output_file_path, 'w', encoding='ISO-8859-1') as outfile:
                    # Change the message here if needed
                    outfile.write(f'This is a txt representation of the VirtueMaster file located at {relative_path}\n\n')
                    outfile.write(infile.read())

            # Process the README.md file separately
            elif file == 'README.md':
                readme_input_file_path = os.path.join(root, file)
                readme_relative_path = os.path.relpath(readme_input_file_path, source_dir)

    # Convert the README.md file to a txt file
    if readme_file:
        readme_new_root = readme_input_file_path.replace(source_dir, output_dir)
        readme_output_file_path = os.path.join(os.path.dirname(readme_new_root), f'{os.path.splitext("README.md")[0]}.{os.path.splitext("README.md")[1][1:]}.txt')
        with open(readme_input_file_path, 'r', encoding='ISO-8859-1') as infile, open(readme_output_file_path, 'w', encoding='ISO-8859-1') as outfile:
            # Change the message here if needed
            outfile.write(f'This is a txt representation of the VirtueMaster file located at {readme_relative_path}\n\n')
            outfile.write(infile.read())

if __name__ == '__main__':
    main()

just put this in a file called convert_dir_to_txt.py in the autogpt_workspace directory and run python ./convert_dir_to_txt.py. You may need to make it executable, which on Mac OS would be chmod +x convert_dir_to_txt.py.

[angelaiba99]

Thank you for sharing this approach! I would like to do something similar, but I am concerned about the token limit. How big is your directory? Have you ever combined this approach with semantic search to solve for this?

[cooleydw494]

@angelaiba99 I've actually stopped using AutoGPT mostly and have been developing a framework, starting from a collection of scripts like this. At the moment I've gotten away from the ingestion topic and have been focusing a lot on leveraging the power of really strategic one-shot prompts with a lot of custom logic for mass file operations. You might be into it, although the documentation needs updating (the code itself is well-documented already tho).

[savethehousemouse]

chatgpt offers the magic of user input. A real world example:
I: "you are a experienced carpenter. I want to build a table. Ask me as many questions for the task as you need. when you have all necessary information, provide the tools, the amount of material and a step-by-step guide to build the table with the tools and material that you calculated."
CGPT: "Great! I'd be happy to help you build a table. I'll need to ask you a few questions to gather all the necessary information:

1. What are the desired dimensions (length, width, and height) for your table?
2. What...
   Please answer these questions, and then I can provide you with a list of tools, materials, and a step-by-step guide to build your table."
   I: "1: 80 cm x 120 cm, 2: ..."
   You get the drift. (full protocol in text file) I have not been able to get aGPT to prompt similarly. Is it because I don't understand how or it's not implemented (yet??)?

chatgpt-dialog.txt