Feature Request / Discussion: Find the correct information from memory

[gingters]

Hi. I was looking into SK 1.0(.1) and for that I am examining Chat Copilot.

I understand that this is a demo and some sort of reference implementation for SK, but based on the data we have it's frankly unusable as a chat with your document / data tool or even demo, and I want to elaborate, why this is the case.

My setup: I use Qdrant as a vector store and I use OpenAI gpt-3.5 and the OpenAI embeddings ada002.

I have a (German) word document. This contains the statues of a sports club, internal rules, data protection declaration, the current elected club board, information about club membership fees etc. I uploaded this as a global document.

Now when I ask questions about the club, i.e. I want all information that stated in the first paragraph of the statues (name of the club, official abbreviation, location, at what court it's registered, fiscal year definition) All is contained in that document, but I never get a correct answer. No matter how I rephrase my question and what I ask additionally.

I figured it might be that the document isn't found and tried to debug into the code.
I added additional logging output into the SemanticMemoryRetriever to figure out what happens.

First, I talk to the chat bot in German. It extracts the intent (in english) and searches with that. Of course it's not ideal to search through German content with an english query.

Therefore I went on and translated all prompts in appsettings.json to German.
This resulted in the intent also be extracted as a german string and the vector search was done with that.
The relevance of the found snippet of the document went up (great!), but because the intent always gets formulated in another way that I formulate my initial question, other (wrong) parts of the document still always rank higher. And they take up more tokens than are left so that the actual relevant information isn't provided to the model when answering the question.

This is something that I already noticed with business documents and I also wrote an article about that problem:
https://labs.thinktecture.com/improved-rag-more-effective-semantic-search-with-content-transformations/

In essence, we need to transform either the content to better match the query, or we need to transform the query to better match the content, or transform both so that they are as similar as possible when doing semantic search.

So, my suggestion would be a way to add content transformations (i.e. generate embeddings not over the content of the document but over a rephrased "hypothetical" document, which matches the intent) to the indexing of documents, or otherwise much more control over how memories are extracted, to be able to really find the relevant parts.