Looking for suggestions/ideas to make GLiNER better

[urchade]

Hi everyone,

We’re looking to make GLiNER better and need your help. Got ideas or suggestions? We want to hear them all – big or small.

[freinold]

Thanks for building this great model!

Could you provide more insight into the creation of your dataset used for the v2.1 models?
Your dataset didn't provide any info, but a LinkedIn-Post said you used Mistral for generating the training dataset.

I would like to do this myself in my field of research, so I'm highly interested in your approach.
Best regards!

[urchade]

Hi @freinold

Here is the prompt I used to generate the dataset

def create_prompt(text):
 out = """You are a sophisticated information extraction system designed to meticulously analyze text content. Your primary function is to identify and classify entities within a given text. Entity consist of span, type and subtype like the following example: ("Barack Obama", "Person", "Politician"). Follow this template:"""
 out += "\n[Start]\n<text>{Your input text goes here}</text>\n<extracted_entities>This section contain a of list entities found within the input text. Each entity is represented as a 3-uple: (entity span, entity type, entity subtype) based on the analysis of the content.</extracted_entities>[End]"
 out += f"\nExample:\n[Start]\n<text>{text}</text>\n<extracted_entities>"
 return out

[urchade]

Full code

import os
import pickle
import argparse
from vllm import LLM, SamplingParams
import ast
from tqdm import tqdm

def create_prompt(text):
    out = """You are a sophisticated information extraction system designed to meticulously analyze text content. Your primary function is to identify and classify entities within a given text. Entity consist of span, type and subtype like the following example: ("Barack Obama", "Person", "Politician"). Follow this template:"""
    out += "\n[Start]\n<text>{Your input text goes here}</text>\n<extracted_entities>This section contain a of list entities found within the input text. Each entity is represented as a 3-uple: (entity span, entity type, entity subtype) based on the analysis of the content.</extracted_entities>[End]"
    out += f"\nExample:\n[Start]\n<text>{text}</text>\n<extracted_entities>"
    return out

def process_and_extract_entities(outputs):
    """
    Process generated outputs and extract entities.

    Args:
        outputs (list): List of generated outputs.

    Returns:
        list: List of extracted entities.
    """
    def extract_(generated_text):
        try:
            # Correctly handle the extraction by trimming the unnecessary part
            extracted = ast.literal_eval(generated_text[:-len("</extracted_entities>")])
            return extracted
        except:
            return None

    all_ents = []
    for i, output in tqdm(enumerate(outputs), desc="Processing outputs"):
        prompt = output.prompt
        generated_text = output.outputs[0].text.strip()
        ents = extract_(generated_text)
        if ents is not None:
            all_ents.append((i, ents))

    return all_ents

def main(weight_path, pile_path, output_path):
    """
    Main function to perform entity extraction.

    Args:
        weight_path (str): Path to the model weights.
        pile_path (str): Path to the data pile.
        output_path (str): Path to save the output file.
    """
    # Set the TRANSFORMERS_OFFLINE environment variable to 1
    os.environ["TRANSFORMERS_OFFLINE"] = "1"

    # Load the model
    llm = LLM(model=weight_path)

    with open(pile_path, 'rb') as f:
        data = pickle.load(f)

    print("Number of texts:", len(data))

    prompts = [create_prompt(i) for i in data]

    # create chunks of 5000 prompts
    sampling_params = SamplingParams(top_k=1, max_tokens=1000, stop="[End]")
    outputs = llm.generate(prompts, sampling_params)

    all_ents = process_and_extract_entities(outputs)

    # print the first 10 results
    print(all_ents[:10])

    comb = {"data": data, "all_ents": all_ents}
    # save the results
    with open(output_path, 'wb') as f:
        pickle.dump(comb, f)

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description='Entity Extraction with Large Language Model')
    parser.add_argument('--weight_path', type=str, default="mistralai/Mistral-7B-Instruct-v0.2",
                        help='Path to the model weights (default: %(default)s)')
    parser.add_argument('--pile_path', type=str,
                        help='pickle file containing list of data')
    parser.add_argument('--output_path', type=str, default="output.pkl",
                        help='Path to save the output file (default: %(default)s)')

    args = parser.parse_args()

    main(args.weight_path, args.pile_path, args.output_path)

[freinold]

Thanks a lot for the quick reply and the code!

[aayush-5]

Hi Urchade ,

Really a great model and I truly appreciate for your contribution !!

Can you help me on this that how can I plot the validation loss and training loss after each epoch to see if the model is overfitting on the data?

[aayush-5]

Got it using WandB , we can do that !!

[daniel-scalioni]

Hi urchade,

I see there is a config parameter to enable wandb, but how can I use it?
because I install wandb library, do the login, but nothing happens on wandb page.
could you help me on how to make it work?

[urchade]

@Ingvarstep may respond to that

[daniel-scalioni]

@Ingvarstep may respond to that

ok, thanks.

[Ingvarstep]

@daniel-scalioni , you need to set report_to to be equal 'wandb'.

GLiNER/train.py

Line 85 in 99130ad

report_to="none",

[ChrisPatten]

Hi @urchade! Thanks so much for these models, I'm seeing great results pulling entities out of fantasy novels and its handling made up names and places really well.

One problem I'm trying to solve for is essentially entity mastering: if I have references to "Mr. H. Potter", "Harry Potter", and "Harry" I want to group them together as a single logical character (so I can include that as metadata for text chunks I'm throwing in a vector DB for a RAG application).

I'm currently using a simple cosine similarity function to try to group and it's okay, but I can't help imagining that there's contextual information that could inform more accurate groupings (e.g. the person was introduced as Harold Potter and is referred to as Harry in the next sentence). Is this something that can be done with the existing model? Any pointers on what I can go dig into or research if not?

Thanks!