ICLTestbed is a framework specifically designed for researchers working on in-context learning. It encapsulates various commonly used models and datasets in a clear and concise process, aiming to free researchers from complex engineering code and allowing them to focus on creative research.
Major features
-
Modular design:
The in-context learning framework is decomposed into different components, enabling users to easily construct a customized in-context learning framework by combining different modules.
-
Low cost of getting started:
Most components are directly based on Hugging Face's library or PyTorch, allowing users to get started with very low learning costs.
-
Support for multiple models and tools out of the box:
The toolbox directly supports multiple models, datasets, and metrics.
I have divided the in-context learning process into four stages: data loading, model setup, model inference, and evaluation. These stages correspond to the three main modules of ICLTestbed: testbed.data, testbed.models, and testbed.evaluate. If you are interested in visual question answering task, you can see the complete usage process in this tutorial. For image captioning task, see this tutorial
If you want to customize new datasets, models, or metrics, you can follow the suggestions in the How-to guides, or directly raise an issue for me to implement it.
| Task | Dataset | Model | Metrics |
|---|---|---|---|
| Visual Question Answering | VQA v2 | Idefics Idefics2 |
vqa accuracy |
| OK-VQA | |||
| Image Captioning | COCO (Karpathy split) | CIDEr | |
| Visual Reasoning | Hateful memes | AUC ROC | |
| NLP | - | Mistral | - |
This project uses the MIT License.