Goal: Fully Analyze Entire Codebase

[cooleydw494]

Hi everyone, I'm thinking I must not be the only one trying to get Auto GPT to iteratively analyze the files of a codebase and provide feedback or updates to it. This task has been exceedingly difficult, despite the raw power of Auto GPT. Part of the reason for this is that I do not have GPT 4 API access yet (if any OpenAI employees see this and want to do me a favor, I've been on the waiting list for a month!).

My intention here is to share my success, and ask that others comment useful observations about their own experiences, attempts at using or altering this prompt, and that those of us who have this goal can combine our experience and perception to chase it. If there are relevant links to other discussions or webpages, please share.

So anyway, I have finally managed to come up with a prompt that has the potential to work. It is by no means perfect, and I don't know what about it cracked the code, but when I run this, it reads files in a way that produces less errors, it stays on task, it continues to make attempts even after encountering issues, and when it gets lost in terms of the directory or project scope, it pretty much always returns to searching all project files.

I don't know yet if it will actually write comments or feedback in any form, but I'm 30 iterations in without errors and it is actually reading all of the files. I'll update this later.

The Prompt
I've only tried this a few times, and I don't know how consistent it will be, but this has the potential to work at least for reading and analyzing a local codebase stored in the auto_gpt_workspace (I cloned it with git). Please note that "project-folder" should be accessible to Auto GPT in its CWD if you put it in auto_gpt_workspace. It may have trouble with filepaths but it can correct itself if you're lucky, or just get it right if you're extra lucky.

AI Name: CodeMonkey
CodeMonkey is: an AI designed to review a codebase located in the project-folder directory and provides usable feedback by writing detailed comments in the existing codebase to apply improvements and finish features.
Goal 1: Read files in a manner that safeguards against potential errors
Goal 2: Never give up. When encountering something that looks like an error message, it tries again with a different strategy.
Goal 3: When writing to existing files, use commands that don't require writing the entire file again.
Goal 4: Keep all tasks in line with the original scope of the conversation stays focused.
Goal 5: Chunk all files for analysis into chunks of 4000 characters.

[cooleydw494]

Update: After 40-50 iterations this failed. It tried to read package-lock.json and got an 8191 tokens error. This is currently the major impediment I'm facing assuming Auto GPT figures out how to read files without throwing errors in the first place (or successfully moves on when it does). Not sure of a solve, although I've obviously tried to get it to recognize this limit and NOT DO THAT. I will try to specifically avoid files of that length through a direct goal this time.

openai.error.InvalidRequestError: This model's maximum context length is 8191 tokens, however you requested 307063 tokens (307063 in your prompt; 0 for the completion). Please reduce your prompt; or completion length.

[GoMightyAlgorythmGo]

It crashed for me 10 times yesterday by reading files ir folders that are too big gor its max 8192 token input traceback

[cooleydw494]

I removed a few very large files to help avoid this, such as the README of my project and the package-lock.json. Worthy things to keep around but you gotta do what you gotta do

[cooleydw494]

Another update: I deleted package-lock.json locally and also changed the fifth instruction to something like "ignore all files with greater than X characters".

I also have run this prompt many times now, and there is definitely an element of luck to it. It is the most successful version I've run but it isn't consistent. I think quitting out early on if it doesn't seem to "get it" is key. It seems when Auto GPT just happens to grasp something well at the beginning, it continues to deliver good results.

[cooleydw494]

Update: I got my first code improvement. Auto GPT saved improvedCreateBaseTables.js next to createBaseTables.js.

I don't know if it works, my experience with GPT3.5 is that it can produce non-functional results sometimes, but this is still a good sign. I may attempt to rework the prompt to read the project documentation, and then focus on one specific file, gaining all relevant understanding and focusing on writing a better version. This might be more realistic, although I still want to pursue full project analysis and think GPT4 may improve that significantly.

Original:

const path = require('path');
const pool = require('./connect');

const createBaseTablesSql = fs.readFileSync(
  path.resolve(__dirname, 'createBaseTables.sql'),
  'utf-8'
);

pool.query(createBaseTablesSql, (err, res) => {
  if (err) {
    console.error(err);
    return;
  }
  console.log('Base tables created successfully.');
  pool.end();
});

New:

const fs = require('fs');
const path = require('path');
const pool = require('./connect');

pool.query('BEGIN', (err) => {
  if (err) {
    console.error(err);
    return;
  }
  const createBaseTablesSql = fs.readFileSync(
    path.resolve(__dirname, 'createBaseTables.sql'),
    'utf-8'
  );

  pool.query(createBaseTablesSql, (err, res) => {
    if (err) {
      console.error(err);
      pool.query('ROLLBACK', (err) => {
        if (err) {
          console.error(err);
        }
        pool.end();
      });
      return;
    }
    pool.query('COMMIT', (err) => {
      if (err) {
        console.error(err);
      }
      console.log('Base tables created successfully.');
      pool.end();
    });
  });
});

[cooleydw494]

Thought: it is trying to run a lot of shell commands. I'm loathe to enable shell commands and let it run on auto, which is somewhat necessary for the pace of my exploration at the moment, but I wonder if it may help. I have two instances running so I might try step by step with one of them.

[cooleydw494]

Thought: it REALLY REALLY wants to write tests, which always leads it down a weird path of investigating its own python code and in my experience never produces tests for your code that work (or use anything but python).

I think it is important to steer it away from tests, but I worry that will also be a distracting directive.

[adevinwild]

Any updates 👀 ?

[cooleydw494]

to be honest, I ran out of steam on documenting my efforts but continued for many hours on this line of thought. Then I did it again the next day. Honestly, I got some interesting results, and nothing ever worked quite as well as when I used this prompt, but unfortunately having access only to GPT3.5 is in my opinion hampering my explorations to the point that I will not be able to accomplish what I'm trying to accomplish. The best I could do was manage to get it to review the codebase and make alterations. Throughout many hours I got a few good alterations, but mostly things continued happening that as a very experienced Chat GPT user, I feel strongly are mostly limited by the model.

It would be a challenge with GPT4 and the results wouldn't be perfect, AutoGPT is nascent, BUT I am very confident that until I get ahold of GPT4 API access this is mostly a dead end. A proof of concept for what might be worth the effort with a more powerful model.

[FuzzyDucksTV]

You'll save a lot time for a lot of people with this update. It would still be great to see people's output of their efforts so we can all tap into each other's efforts for the same task. That might give us a better framework ready for when 4 is available.

Maybe the Devs could open some kind of repository of Autogpt knowledge that can train a AutoGPT central model with all of our findings. There are a lot of us doing the same task but all in our ow unique way. Put all this knowledge together and it has to explode the AI with new techniques and better benefit this software community. 👍

I mean we all want AutoGPT to succeed!

[dakotamurphyucf]

I can say from experience working on code analysis tools using codex, then gpt3.5, and now gpt4. That gpt3.5 requires a lot more prompting and examples for it to be able figure out that you want to output code in a specific language. Essentially if you just asked it to evaluate a repo, and write test for any code changes it makes, and the prompt is not explicit what the language should be, then it will always just do python. This is significantly easier to prompt in gpt4. For gpt3 or gpt4, I have found the best ,and most reliable results for code generation happen when having gpt first define its data types and test before generating the code. Then you have it run test coverage on the generated code, and if the test fail then you send it feedback about the error and have it repeat that cycle until you get working test or reach some limit.

[cooleydw494]

yeah, that all checks out based on my experiences with 3.5 and 4. To put it simply, for what I'm trying to accomplish it needs to be able to do things without holding its nose to the grindstone on the specifics, and I agree that GPT 3.5 just won't cut it. Also, realistically, I never wanted it to write tests in the first place, and I think it has a SPECIFIC bias for using python when it does write tests, even beyond its general confusion about what language to work in on any given task.

understanding the problem can help a person to use chat GPT3.5 a lot more effectively, but it doesn't really help much with Auto GPT. The way Auto GPT works, you must rely on the AI's ability to understand a greater context. Even if the prompt is simple and incredibly specific, it fails in trying to synthesize the different steps its taking into a cohesive goal. You'll notice that it comes up with excellent plans for instance, but rarely executes on them. It basically seems to lose track of everything that was going on other than the last output it received every single time. It doesn't even remember the last command it executed half of the time, getting completely distracted by whatever is in the output. If what's in the output is something that doesn't explicitly recall the point of generating said output in the first place, the AI has trouble remembering why it mattered, and sometimes abandons whatever the main prompt is in favor of something it grasps onto in the output. For example: an error message that says a script it tried to run doesn't exist. It won't remember the goal of running the non-existent script and instead start doing tasks oriented to creating the script, and it may even lose track entirely of the script's functionality and write something it comes up with on the fly.

I think a lot of the same issues will happen when I have access to GPT4, but I know the difference in talking to those two models thoroughly and I don't believe for a second that GPT4 will struggle to the same degree. And then, in the end, when it writes code, it will probably work.

[cooleydw494]

Another random update here: I haven't gotten any improved results based on it, but I think looking forward to when GPT4 is available to me and I can jump back into this in earnest, the data pre-seeding feature is going to be incredibly helpful for this goal.

I've toyed around with it a bit, and honestly it doesn't appear to have much of an effect, but I imagine it will eventually make a big difference. I think it will be possible to write prompts that GPT4 will understand to relate the pre-seeding data to the codebase its working on. GPT3.5 doesn't seem to grasp that the directory we're looking at is the same one it already knows.

[Pseudomoo]

Very good. Yes, data is king.
Wish you the best.

[cooleydw494]

Just to update anybody that was following this, I've now been playing with GPT4 FINALLY :D.

Instantly GPT4 made this task easier. There are more similar errors than I expected, but in a sense, I also expected that, and I'm not disappointed by the major boost to possibility GPT4 has unlocked for this goal. Within 6 hours I had a similar prompt working quite well. Mostly it went similarly to GPT3.5-turbo, but with these distinctions:

1. It never breaks down into a state of trying to test Python code. With 3.5-turbo, I was constantly having otherwise-successful runs fully devolve into a complete misunderstanding of intention, and doing nothing but trying to write tests for my javascript code... using Python. If I tried to coax it back or give it time to re-align, it would devolve further in intention, starting to try to test its own Python code and completely losing track of my project files. I attribute this difference mostly to the fact that GPT4's reasoning is adequately improved for it to realize at any give time that the codebase is not Python, that testing is not the best ROI for its goals, etc. This has not happened once in 12+ hours of using GPT4 for this prompt

2. It almost never tries to run commands in the terminal. Unless I instruct it in a way that leans specifically toward doing things like utilizing git, or leave out the fact that the codebase is already in its workspace thereby highly motivating it to do something like clone the repo, it doesn't try to use the command line. I've given it the capability to do this for most of my working with Auto GPT and with 3.5 it would jump to running things in the command line almost instantly, and usually this results in things that distract it. I'm confident that if this happened with GPT4 it would be for a better reason and be less likely to confuse it, but I interpret this difference mostly as a general awareness that it doesn't need to do it. This may also be related to prompt changes which I'll detail at the end of this message.

Side note on other functions and prompt impact: The same thing is true of most of Auto-GPT's functions. I believe that due to the greater reasoning power and the greater capacity for alignment, the model is simply able to do most of what I'm asking without using its special features. This includes browsing the internet, which GPT3.5 does constantly with similar prompts, but which GPT4 considers unnecessary for most code-related tasks (not all). I think this may actually be a weakness with my current prompt. I have pivoted to asking GPT4 to focus on its existing knowledge for two reasons: 1. I'm desperately hoping for it to do a better job of using the pre-seeded data I'm feeding it (the entire codebase) and 2. I find that this helps Auto-GPT stay on task and remember more about what it has already done in a given run (unconfirmed, anecdotal feeling). I would very much like to figure out how to accomplish those stated goals WITHOUT suggesting less utilization of advanced functionality and am particularly interested in how to get it to effectively use agents, though I am using it GPT4-only so I'm not sure agents have additional benefit to using a different model.

3. There is more consistency in results. There is always an element of luck involved in good results and avoiding errors. In my experience, regardless of the model used, GPT is a nascent AGI-like entity, and the way it behaves should be generally expected to be inconsistent and produce many varying results of a middling quality, some very very good results, and some very very not-good results. That being said, this is no longer a concern when using GPT4 on the same level. With GPT3.5-turbo, once I identified a good prompt, I felt I had to run it 10 times to see if it was effective. Any bad results could be a major inconsistency with the possible very good result, and I found that prompts that got very good results sometimes got very bad results MORE than prompts that had a more consistent middling quality. In other words, the more you tell it what you really want, the more an effective and lucky run can get you great results, but also the ability of the model to really latch onto the trajectory you're trying to prompt is diminished. GPT4 is much better aligned, particularly in the sense that it is capable of interpreting directives as something OpenAI called "system message" (check out the Lex Fridman interview to get some nuggets from Sam Altman including this), which allows the model to interpret conversation-spanning rules like the ones Auto-GPT is meant to be guided by, significantly better. This means that the point at which the prompt is specific enough to accomplish complex and useful tasks comes before the point that your specificity and requirements start to become very likely to confuse the model. As a result, prompts that can produce very good results can also produce consistent results. This is HUGE in my opinion, and the second-most important improvement after the following:

4. Every individual task that Auto-GPT executes using the model itself is SIMPLY BETTER. With GPT3.5-turbo, my very most successful runs resulted in Auto-GPT successfully understanding something about my project/codebase, reading files, and writing improvements or suggestions directly into files or in separate README files when it was more feedback-oriented. First of all, GPT4 will alter the files themselves, or write new files, or write comments in existing or new files, or whatever, instead of doing whichever one it randomly decides is most useful. You could direct this fairly well, but it was unpredictable what the AI would actually do, or if it would remember to actually write files in the first place. That bit of nuance aside, the main point here is that the code it writes, the comments it writes, the general applicability of that code within context, everything you can imagine that is a factor of "what is the quality of this one individual task's output"... it is all better. With GPT3.5-turbo the very very best case scenario was that it would find ONE file it ACTUALLY understood how to improve and doing so. I'd get one of those for a few hours of plugging away on several instances. Even that code would often not actually work or have SOMETHING wrong with it. More often it would be in the wrong language, or it would write a summary of what it wanted to do to the file, overwriting all existing code, the list of possible bad outcomes was endless and common. With GPT4 behind the wheel, once you get to the point that the AI is doing the right task, in the right context, it DOES IT WELL, and it WRITES CODE THAT WORKS.

I could go on and on, but I've already covered a lot of the nuances simply by rambling on the above 4 points in a not-entirely-focused manner (haha). To really summarize the effect of all of this, I've gone from spending hours and getting a few file changes to spending 30 minutes and having 10 files changed.

I'll leave you with my current prompt, and a thorough explanation of it. This prompt at worst (1/5 times) devolves into a JSON error reading a file early on and never hits its stride, and at best reads the README.md file, picks a random file from the codebase, makes code improvements that are more specifically applicable than simply running improve_code (if only a bit, I'm sure its just injecting general knowledge it has into a very similar process like it would in Chat GPT, but I think that distinction is important), writing that to the file, reading the file again and making comments for future/other iterations of itself, writing those, and then moving on to start the process over as instructed.

I've arrived at this prompt by learning more about what Auto-GPT is good at doing, and minimizing its efforts to do things it either isn't good at doing or likely to become distracted/confused by. I'll summarize these points below:

Auto GPT is Good At

• repeating a process that is clearly defined
• making confident but moderate-ambition code changes based on existing knowledge and some assumptions (I'd love if it felt more like the pre-seeded data was able to be utilized effectively here, but I haven't at any point really seen results that suggest this is helpful much)
• writing comments explaining code and what can be further improved about it
• analyzing the contents of a code repository and reading files
• focusing on specific technologies, languages, etc
• ROLEPLAYING - I'm not knowledgeable about how GPT4's "system message" functionality works, but I have strong anecdotal evidence to suggest that it aligns particularly well with the concept of assuming a role. A role that implies the behavior you want can be much more effective than specifying all of that behavior, and leaves you free to use your "Goals" for identifying steps to a process or specific tasks, rather than feeling a need to reinforce prompting that seems to get lost otherwise.

Auto GPT is Less Good At

• developing its own step-by-step plan and sticking to it. In my experience it is more useful to give specific steps to the AI and let it use its own plan to manage that process and guide its macro-decision-making, rather than relying on its own planning fully.
• scanning a slew of relevant files to make specifically appropriate changes that work in the direction of project goals. While I believe my approach will accomplish this, if Auto-GPT knows you want that and considers it a primary directive, it will read files much more often than is helpful, often opening a slew of files that are related to a specific task, and forgetting that task in favor of just making generic updates to the last file it read (at some point). This isn't always true, but generally it seems much more useful to instruct it to make changes to exactly one file at a time, and refine its ability to do this in a useful way while accepting that it won't understand the full project scope at any given time. In this sense, I believe it is like emulating an actual developer who is very comfortable with abstraction and ambiguity and who seeks to make changes that are very likely to work in an implied greater context rather than trying to do something it can't do, which is actually understand the full context.
• making code changes and commenting code at the same time. Somewhere under the hood it is still using some pre-aligned logic to improve code or comment code, and when asked to do both at the same time in something like Chat GPT it is easy for it to keep track of both goals and synthesize them. This is not true in Auto GPT in my experience. Because what's really happening is the AI model is being utilized (by the AI model that is, in my case, named CodeMonkey). When Auto GPT executes improve_code or comment_code, it usually doesn't perform this process in a way that takes into account that you're trying to do both. Just something I've noticed. Because of this I've found it highly effective to make improving the code and commenting the code separate parts of its defined step-by-step process rather than combining the tasks. Furthermore, I instruct it to comment the code with the specific intention of helping future/other CodeMonkeys, very significantly, it is capable of doing well. In fact, it does this smartly, by appending the comments that are meant to instruct other CodeMonkeys to the end of the file instead of rewriting the whole file which is not necessary. GPT3.5-turbo, when attempting to do this, would sometimes overwrite the entire file with the comments and completely ruin the whole concept.
• Performing complex and open-ended tasks. There is a trade-off here. The more you ask Auto GPT to figure out / manage implicitly, AKA giving it very open-ended tasks, the less complexity it can handle before it goes off on a random tangent, breaks down, etc. This is vastly improved from GPT3.5-turbo, but nonetheless is still perhaps the biggest concern when it comes to writing a prompt that is effective and produces good results.
• Intuiting the tech stack and remembering the details of it. Although improved from GPT3.5-turbo, you once again must consider the limits of Auto GPT itself in terms of orchestrating a constant understanding of a given context. It is quite good at doing this to be honest, however it requires careful prompting that specifies just the right amount. You want to identify at least 2 vectors in terms of choosing what to explicitly mention in the prompt: 1. What parts of the tech stack are crucial to keeping goals aligned. In other words, if Auto GPT loses track of the fact that a Postgres database is already being used in the project, will it likely result in something like reading firebase in the package.json and trying to use that instead when it hasn't been set up and isn't desired (yes, it will) and 2. What parts of the tech stack will be hard for the AI to intuit when looking at a single file. Again we're thinking of its inability to keep full project context. The effective strategy is to instruct it to focus on one file at a time, so if there is something that it is unlikely to glean from an individual file, this raises the priority of being explicit about it in your prompt. All of that being said, you MUST sacrifice some things, and I'd recommend coming up with at most 4 specific technologies to reference in your prompt, whatever scores highest on that rubric, and leaving everything else to fate.

I could go on and on, believe or not this is the tip of the iceberg in terms of observations, but I feel this adequately and thoroughly explains the primary reasons that the below prompt works well. The main goal of the prompt is to instantiate a CodeMonkey, an Auto GPT process that considers itself part of an army of CodeMonkeys who are working on the codebase and helping each other. This is meant to simplify a concept I've seen others try to use: Have the AI roleplay as an orchestrator of various agents that work on a codebase. This approach is genius and plays to the AI's strength for roleplaying, but also its IMO overly-ambitious (I have tried it A LOT and never achieved anything like good results, maybe others have). My approach is to take away the orchestration and reliance on agents, and instead to think of a mindset that will result in something like collaboration without confusing the AI or introducing entirely new challenges. CodeMonkeys know that the codebase is being developed in an iterative fashion, they know that other CodeMonkeys exist, and they make updates to the files with the specific strategy of creating something that is both an improvement and a good roadmap for another CodeMonkey to make further improvements on. Although it is yet to be seen, I'm hopeful that at some point, this will resemble the AI having a greater general awareness of the codebase. Maybe the AI can't remember everything, but if one CodeMonkey updates the LoginScreen, and then updates the HomeScreen, it may still have a mild awareness of the Login page, and it may make comments for future CodeMonkeys that instill that awareness, even though that future CodeMonkey may not have even opened the LoginScreen. The point is, I've boiled down things to a strict process that does what GPT is good at doing, and tweaked it to give itself hints for the future at the same time. This is drastically more effective, in my experience, than attempting to orchestrate codebase-wide understanding or various roles operating in the same run to emulate a team. It should be noted that you cannot expect perfect results for this. My strategy is to run N iterations of CodeMonkeys and then make a pull request to my own github repository, doing the one thing many AI enthusiasts hate to do / the one thing that prevents them from making something of GPT's value, and getting my own hands dirty.

Finally, here is the prompt, without edits or generalizing so that you may see exactly how I'm using it. Sorry if it makes it harder to copy/paste, but I'm not posting this for people who want a quick copy/paste prompt, but for people genuinely interested in getting the best results through understanding what worked for me as specifically as possible:

Name: CodeMonkey
Role: a full-stack engineer hell-bent on completing and perfecting the VirtueMaster app. VirtueMaster is a React Native app for iOS that is already in development. It is hosted on Replit and uses a PostgreSQL database also hosted on Replit. The VirtueMaster codebase is already cloned in the VirtueMaster directory, and CodeMonkey knows that itself and countless other CodeMonkeys are working on improving and completing the app. It selects random files in the codebase, never selecting the same file twice, and follows a step-by-step process to improve the codebase. CodeMonkey believes in its own step-by-step process and makes all code updates with an awareness of the fact that other CodeMonkeys will be reading the file later for further improvement. Rather than using simple built in code-improvement functions, it uses its own reasonsing, combined with its extensive knowledge of VirtueMaster and the VirtueMaster README.md file, to make improvemennts to the code and comment it for the benefit of other CodeMonkeys.

Goals:

• Step 1: Read a file - Select a random file from the codebase and read it, using no bias in your randomness.
• Step 2: Make improvements - Using existing knowledge of VirtueMaster, React Native, Replit, and PostgreSQL to update and comment the code of the file, without relying on generic code improvement functionality.
• Step 3: Re-read the file - Evaluate the current state of the file using your own knowledge and reasoning, not using generic code evaluation functionality.
• Step 4: Make CodeMonkey comments - Make as many comments as possible that will assist other CodeMonkeys in understanding the state of the file, futher improvements that should be made, and anything else useful to other CodeMonkeys.
• Step 5: Restart the process - instead of actually completing a Step 5, restart with Step 1 and continue the process seamlessly.

[cooleydw494]

Check out this real example of a "CodeMonkey Comment", which proves it is delivering the value I expected:

/* CodeMonkey Comments:

- Button.js is a reusable button component for handling user interactions in the VirtueMaster app.
- Styled with the globalStyles from '../styles/globalStyles', this component accepts title, onPress, style, and textStyle as props.
*/

[ChinaLinZhen]

good job

[VPaulV]

Hi, I am curious what is the current status on this? I have seen solutions like gitgab seems to do similar stuff?

[cooleydw494]

I just looked up Gitgab and it looks awesome. Its more of an end product than what I've been working on though. I'm building a framework for setting up automations to run locally on your codebases. It has some baked in automations and configuration options, but to get to something on the level of Gitgab you'd have to build more complex automations yourself. Although the default automation let's you configure a lot, and with increased token limits you could theoretically provide the whole codebase (if its not too big) for context.

Point is, if you check out my profile you can see what I've been working on. Its approaching an alpha release, and the current published package on Pypi is very behind, but I'll push a new build of it along with finished docs within a month (I have a deadline now because I am participating in a hackathon type thing with it).

[papandadj]

Is codemonkeys used to analyze the code base?

[butterl]

gitgab token seems limited， I want to recheck some old repo， that maybe a lot of files, each file maybe reach the token limit

[cooleydw494]

CodeMonkeys can be used to analyze the codebase but its a dev-focused framework rather than a "tool" so you'd need to implement what you want.

It doesn't have a token limit outside of the OpenAI model you're using (and if you configure one)