MariaDB vector store implementation.

[rusher]

Description

This is the MariaDB vectorstore implementation. see https://mariadb.org/projects/mariadb-vector/

Key Features

• provide mariadb implementation to store and request embeddings
• provide a filter unifying proposal

Test

• test cases added

Filtering proposal

This PR contains a proposal to unify filtering data. I might have not understood stuctured query well, but goal of this framework is to permit unifying behavior and it doesn't feel this part has succeed in it.

I'll detail what i've submitted, i'll let you judge if accurate or not.
There is now a new package "@langchain/core/filter"

an example of database filter implementation:

class TestFilterExpressionConverter
  extends BaseFilterExpressionConverter
  implements FilterExpressionConverter
{
  constructor() {
    super();
  }

  convertExpression(expression: Expression): string {
    return super.convertExpression(expression);
  }

  convertExpressionToContext(expression: Expression, context: StringBuilder): void {
    super.convertOperandToContext(expression.left, context);
    super.convertSymbolToContext(expression, context);
    super.convertOperandToContext(expression.right!, context);
  }

  convertKeyToContext(key: Key, context: StringBuilder): void {
    context.append(`'$.${key.key}'`);
  }

  writeValueRangeStart(_listValue: Value, context: StringBuilder): void {
    context.append("[");
  }

  writeValueRangeEnd(_listValue: Value, context: StringBuilder): void {
    context.append("]");
  }

  writeGroupStart(_group: Group, context: StringBuilder): void {
    context.append("(");
  }

  writeGroupEnd(_group: Group, context: StringBuilder): void {
    context.append(")");
  }
}

A builder permit to create the filter, and vectorstore implementation will convert it to expected format :

const b = new FilterExpressionBuilder();
const impl = new TestFilterExpressionConverter();

let exp = b.eq("a", 2015); 
console.log(impl.convertExpression(exp));
// "'$.a' = 2015"


exp = b.in("a", [2015, 2018]); 
console.log(impl.convertExpression(exp));
// "'$.a' IN [2015,2018]"

// name = 'martin' AND (firstname = 'john' OR firstname = 'jack')
exp = b.and(
            b.eq("name", "martin"),
            b.group(b.or(b.eq("firstname", "john"), b.eq("firstname", "jack")))
        ); 
console.log(impl.convertExpression(exp));
// "'$.name' = 'martin' AND '$.firstname' = 'john'"

// the opposite
exp = b.not(
          b.and(
            b.eq("name", "martin"),
            b.group(b.or(b.eq("firstname", "john"), b.eq("firstname", "jack")))
          )
        ); 
console.log(impl.convertExpression(exp));
// "'$.name' != 'martin' OR ('$.firstname' != 'john' AND '$.firstname' != 'jack')"

(Ideally, a text parser using antlr to generate the expression from a standardized query would next the next step)

When requiring the corresponding SQL to a filter expression, each vector store needs to implement a filter implementation, extending BaseFilterExpressionConverter. This prevent SQL injection, unify behavior, and permit any kind of filtering, like IN, NOT, gouping ...

(a database filtering implementation must be probably maximum 20 significant lines of code, mainly because each database use his specific JSON functions)

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
langchainjs-docs	❌ Failed (Inspect)			Dec 13, 2024 2:45pm

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
langchainjs-api-refs	⬜️ Ignored (Inspect)			Dec 13, 2024 2:45pm

[rusher]

Can i still help to get this PR merged ? about the vercel failure, i cannot have access to it. so i leave it to you to indicate what's wrong