How to Parse PDF Documents?

[tchrapko]

I have a slew of technical documents that I would like to have ChatGPT ingest so that I can ask questions about the content. In another forum someone said that this was possible, saying:

for that I would take all of your documents (chunk them up so each chunked document is less than some max token limit). Then you can get text embeddings for all of the docs. Once you have the embeddings, you can add them to a vector db (along with the raw text) and then you can use the vector db to grab relevant information to add to the context. (You embed a search-query/prompt and the most similar/relevant documents will be fetched from the vector db)

I've gotten Auto-GPT to run successfully, but don't know how to go about using it to facilitate this process. Any ideas?

[codenan42]

I'm not an expert but you can go to the folder where you install Auto-GPT, inside the folder look for another folder called "auto_gpt_workspace". In there you can place the pdf you want it to parse and remember the file name for e.g. readthis.pdf. Then in your prompt in Auto-GPT somewhere in the goal or the role you can just mention it to read the file readthis.pdf and have it summarize it or whatever according to your needs.

[tchrapko]

I put a file into the workspace and the system does attempt to access it, but returns the following:

SYSTEM: Command read_file returned: Error: [Errno 2] No such file or directory: 'auto_gpt_workspace\AdvancedCapture.pdf'

The file is in that location, and the name is exactly correct. Not sure why there are two backslashes before the filename. I also tried to explicitly ask it to remove the extra slash. No joy.

[Keithsguest]

No, no, no I'm on Linux and I fix it but straight up putting the entire directory not just the gpt_workspace

for example, you need to search for the file home/Kenny/Auto-GPT/auto_gpt_workspace/AdvancedCapture.pdf

[mso9999]

I also want to do this, got the same error and maybe got past it by creating another folder auto_gpt_workspace inside the original one (it seemed to add an extra path), but it doesnt read the pdf: Command read_file returned: Error: 'utf-8' codec can't decode byte 0xe2 in position 11: invalid continuation byte

[hjmaier]

Same problem here. I guess we need a library to read PDFs

[MemornSun]

same problem too

[tchrapko]

I haven't had a chance to try this, but there is a section of the README.md on pre-seeding: "Memory pre-seeding allows you to ingest files into memory and pre-seed it before running Auto-GPT." Sounds like a great place to start. \-------- Original Message --------

On Apr 18, 2023, 6:29 PM, hjmaier < ***@***.***> wrote: Same problem here. I guess we need a library to read PDFs — Reply to this email directly, [view it on GitHub][], or [unsubscribe][]. You are receiving this because you authored the thread.![AKODW4JLVXOIQJJJ3MFWOILXB4PWNA5CNFSM6AAAAAAWW2RYJ2WGG33NNVSW45C7OR4XAZNRIRUXGY3VONZWS33OINXW23LFNZ2KUY3PNVWWK3TUL5UWJTQAKZEBM.gif][]Message ID: ***@***.***>

[view it on GitHub]: #409 (reply in thread) [unsubscribe]: https://github.com/notifications/unsubscribe-auth/AKODW4IHMW7XBSCVIJUZLD3XB4PWNANCNFSM6AAAAAAWW2RYJY [AKODW4JLVXOIQJJJ3MFWOILXB4PWNA5CNFSM6AAAAAAWW2RYJ2WGG33NNVSW45C7OR4XAZNRIRUXGY3VONZWS33OINXW23LFNZ2KUY3PNVWWK3TUL5UWJTQAKZEBM.gif]: https://github.com/notifications/beacon/AKODW4JLVXOIQJJJ3MFWOILXB4PWNA5CNFSM6AAAAAAWW2RYJ2WGG33NNVSW45C7OR4XAZNRIRUXGY3VONZWS33OINXW23LFNZ2KUY3PNVWWK3TUL5UWJTQAKZEBM.gif

[hjmaier]

I've tried it. Did not work. It was reading about 190 bytes and stopped.

[tommy-qichang]

Same here. Maybe we need to convert the pdf to txt?

[tchrapko]

I'm just shootin the shit here, but did I imagine reading somewhere that the 3.5 model has to handle documents, or is that a thing? Sent from Proton Mail mobile \-------- Original Message --------

On Apr 18, 2023, 9:05 PM, hjmaier < ***@***.***> wrote: I've tried it. Did not work. It was reading about 190 bytes and stopped. — Reply to this email directly, [view it on GitHub][], or [unsubscribe][]. You are receiving this because you authored the thread.![AKODW4JZYALXWBC4TC6XXJTXB5CAJA5CNFSM6AAAAAAWW2RYJ2WGG33NNVSW45C7OR4XAZNRIRUXGY3VONZWS33OINXW23LFNZ2KUY3PNVWWK3TUL5UWJTQAKZFTU.gif][]Message ID: ***@***.***>

[view it on GitHub]: #409 (reply in thread) [unsubscribe]: https://github.com/notifications/unsubscribe-auth/AKODW4IKY4XLORTHEHBC2BTXB5CAJANCNFSM6AAAAAAWW2RYJY [AKODW4JZYALXWBC4TC6XXJTXB5CAJA5CNFSM6AAAAAAWW2RYJ2WGG33NNVSW45C7OR4XAZNRIRUXGY3VONZWS33OINXW23LFNZ2KUY3PNVWWK3TUL5UWJTQAKZFTU.gif]: https://github.com/notifications/beacon/AKODW4JZYALXWBC4TC6XXJTXB5CAJA5CNFSM6AAAAAAWW2RYJ2WGG33NNVSW45C7OR4XAZNRIRUXGY3VONZWS33OINXW23LFNZ2KUY3PNVWWK3TUL5UWJTQAKZFTU.gif

[Cypherthegreat]

Good evening everyone so to ingest you need to do the following, I managed to get it to work if you have the files as .txt files in utf8 format/encoding

The DIR path is relative to the auto_gpt_workspace directory, so python data_ingestion.py --dir . --init will ingest everything in auto_gpt_workspace directory.

(https://github.com/Significant-Gravitas/Auto-GPT#-memory-pre-seeding) ~ source is there

I Hope that helps

[RohollhSH]

I explained to him where to find the PDFs and asked me to put them in auto_gpt_workspace, now he tries to use "read_file" command with mode: rb, but gives me this error:
Command read_file returned: Error: 'utf-8' codec can't decode byte 0xf6 in position 10: invalid start byte.
The PDFs have a lot of drawings and charts probably I need to explain to him to study them as Images.

[Cypherthegreat]

Hi
You cannot put a PDF into it it must be a .txt file with utf8
use a pdf to txt converter such as this one https://www.zamzar.com/convert/pdf-to-txt/ (i have no idea if they save the data you send to this service so please do your own research :P)
Thanks
Cypher

[magamagaman1]

I installed PyPDF2 and have been trying to get it to use it, but it won't

[RohollhSH]

yeah for me he was trying to install PyPDF2 too but wasn't successful. so I installed it myself and told him but still wasn't able to use it, I think it was an Import library problem and I wasn't smart enough to help him

[m0zzaR]

I installed PyPDF2 and have been trying to get it to use it, but it won't

PyPDF2 is depreciated, you can see this if you look in the library. I use pdfminer instead.

[tchrapko]

Sent from Proton Mail mobile I hate to give up on Auto-GPT, but this process worked really well and handled my documents in native PDF. That is, I didn't have to convert them myself. https://youtu.be/TLf90ipMzfE \-------- Original Message --------

On Apr 24, 2023, 9:08 AM, magamagaman1 < ***@***.***> wrote: I installed PyPDF2 and have been trying to get it to use it, but it won't — Reply to this email directly, [view it on GitHub][], or [unsubscribe][]. You are receiving this because you authored the thread.![AKODW4MCIRADIM4BMVQNMDLXC2CNZA5CNFSM6AAAAAAWW2RYJ2WGG33NNVSW45C7OR4XAZNRIRUXGY3VONZWS33OINXW23LFNZ2KUY3PNVWWK3TUL5UWJTQAK4OH4.gif][]Message ID: ***@***.***>

[view it on GitHub]: #409 (reply in thread) [unsubscribe]: https://github.com/notifications/unsubscribe-auth/AKODW4MNK5SCNJPSWU4BD73XC2CNZANCNFSM6AAAAAAWW2RYJY [AKODW4MCIRADIM4BMVQNMDLXC2CNZA5CNFSM6AAAAAAWW2RYJ2WGG33NNVSW45C7OR4XAZNRIRUXGY3VONZWS33OINXW23LFNZ2KUY3PNVWWK3TUL5UWJTQAK4OH4.gif]: https://github.com/notifications/beacon/AKODW4MCIRADIM4BMVQNMDLXC2CNZA5CNFSM6AAAAAAWW2RYJ2WGG33NNVSW45C7OR4XAZNRIRUXGY3VONZWS33OINXW23LFNZ2KUY3PNVWWK3TUL5UWJTQAK4OH4.gif

[MikkelAJ92]

Im not sure how my agent got there, but it managed to download the pdf and i currently making a python script using the library BeautifulSoup to attempt to solve it... I have no idea where this is going!

[Rakly3]

Need to export the contents of the pdf file to a simple UTF-8 file. Split the document into pieces so they are not more than 3000 tokens large. (AutoGPT crashes trying to read larger files.)

I actual got AutoGPT write me the code to do it even!

# splits utf-8 files into smaller digestable chunks for AutoGPT. Currently set at '3000 OpenAI ChatGPT tokens'. Last filesplit might not be correctly split! 
# Input_file and output_file have to be in the same directory of the script, or the full path of the file if it's in a different folder.

import os

# Define the maximum number of tokens in each file
MAX_TOKENS_PER_FILE = 3000

# Define the input and output file paths
input_file_path = 'input_file'
output_file_prefix = 'output_file'

# Load the input file
with open(input_file_path, 'r', encoding='utf-8') as f:
 input_text = f.read()

# Split the input text into tokens
input_tokens = input_text.split()

# Calculate the number of output files required
num_files = int(len(input_tokens) / MAX_TOKENS_PER_FILE)
if len(input_tokens) % MAX_TOKENS_PER_FILE != 0:
 num_files += 1

# Write the output files
start_index = 0
for i in range(num_files):
 output_file_path = f'{output_file_prefix}_{i}.txt'
 with open(output_file_path, 'w', encoding='utf-8') as f:
  f.write(' '.join(input_tokens[start_index:start_index + MAX_TOKENS_PER_FILE]))
 start_index += MAX_TOKENS_PER_FILE

print('Input file split into smaller files.')

Currently I'm experimenting with getting AutoGPT to use it's local memory from small files as source. data_ingestion.py sadly does not work at this time. The memory gets wiped. Even if you ingest it after launching AutoGPT. The cool thing however is that it only got the code right after I also told it to use data_ingestion.py as an example to base it on.

One prompt (1 goal) also seems to work better than making a list. Just like you would use normal ChatGPT. Which isn't surprising as it will then send everything to the same agent instead of using different ones for each step.

[arrfonseca]

SO where do I insert this piece of code?

[m0zzaR]

SO where do I insert this piece of code?

This is a python file and it is most likely in the AutoGPT workspace. You can ask AutoGPT to run said python file. (Haven't tested it so just going off of what I presume)

[ouzmno]

So can I understand that as a Python file, as long as I got all the necessary modules imported in that Python file, auto-gpt can use the py file to generate pdf whenever she needs?

[ouzmno]

SO where do I insert this piece of code?

This is a python file and it is most likely in the AutoGPT workspace. You can ask AutoGPT to run said python file. (Haven't tested it so just going off of what I presume)