Pipeline Evaluation Functionality Missing in Haystack 2.0

[leomaurodesenv]

Currently, Haystack 2.0 lacks the functionality to evaluate Extractive Question Answering pipelines directly, which is a crucial aspect for assessing the performance and effectiveness of various pipeline configurations.

I did not find any useful tutorial or documentation about it. Can you help me?

Currently

from haystack import Pipeline
from haystack.components.readers import ExtractiveReader

# Get the reader
reader = ExtractiveReader(model=DOC_READER)
# Build the pipeline
pipeline = Pipeline()
pipeline.add_component(instance=reader, name="reader")

# How can I evaluate?

Previous Behavior

from haystack import Pipeline
from haystack.nodes import FARMReader

# Get the reader
reader = FARMReader(DOC_READER, use_gpu=True)
# Build the pipeline
pipe = Pipeline()
pipe.add_node(component=reader, name='Reader', inputs=['Query'])

eval_result = pipe.eval(labels=eval_labels, documents=eval_docs, params={"Reader": {"top_k": NUM_K}})
metrics = eval_result.calculate_metrics()

[anakin87]

@julian-risch @silvanocerza can you help?

[mrm1001]

Hi Leonardo, thanks for reaching out! May I ask, are you interested in extractive QA specifically, or did you pick that as an example of evaluation? We are focusing our discussions on evaluation for RAG pipelines at the moment (to evaluate retriever and answer generation, as well as end-to-end).

[leomaurodesenv]

@julian-risch we have tested the ExactMatch and it is working fine!

[leomaurodesenv]

Doubt: @julian-risch @mrm1001 ,
Do you have anything planned for F1 Score? If no, I can open an Issue and propose a Pull Request. Does it make sense?

[julian-risch]

Hello @leomaurodesenv we are not working on an AnswerF1Evaluator component yet. A contribution would be very welcome, yes! Calculating the F1 score is a bit more complicated than the exact match because the F1 score is token based. So the evaluator first needs to tokenize the predicted answer and the ground truth answers. Then it needs to calculate precision and recall based on those tokens and then calculate the harmonic mean of those to get the final F1 score.
Looking forward to your PR! Creating an issue first would be great!

[leomaurodesenv]

Indeed, I agree that it is more complex. I will open the issue and link it here

[leomaurodesenv]

I just created the issue #7606

[mrm1001]

Hi @leomaurodesenv, how are you getting on? I am the product manager at Haystack and I'm trying to talk to user to understand how to improve the library and what to prioritise. Would you be open for a 15 min catch up? We would really appreciate it?

[leomaurodesenv]

Sure, let's schedule a catch-up 😉

[leomaurodesenv]

Also, if you would like to take a look in what I am playing with

• https://huggingface.co/spaces/leomaurodesenv/qasports-website