Chat template not being used despite passing --apply_chat_template option

[juliafalcao]

I am having an issue when evaluating several instructed models using the LM Evaluation Harness and the --apply_chat_template option. I am passing this option and I ensured that the models have chat templates available in their tokenizer configurations, but when I check the results JSON file, the chat_template field is null, which, as I understand, means that the chat template was not used in the evaluation to obtain such results.

For example, I am evaluating the utter-project/EuroLLM-1.7B-Instruct model, which has a chat template defined in its tokenizer_config.json. I'm using a local clone of this model and I've confirmed the chat_template field is there:

$ cat tokenizer_config.json | grep chat_template
  "chat_template": "{% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}"

I ran evaluation on the catcola task with the following command:

accelerate launch -m lm_eval --model hf \
    --model_args pretrained=EuroLLM-1.7B-Instruct,trust_remote_code=True \
    --tasks catcola \
    --num_fewshot 5 \
    --batch_size 1 \
    --output_path chat_template_test_results.json \
    --log_samples \
    --trust_remote_code \
    --apply_chat_template \
    --fewshot_as_multiturn \
    --seed 1234

But in the results file, the chat template is null.

]$ cat chat_template_test_results.json | grep chat_template
  "chat_template": null,
  "chat_template_sha": null,

I'm attaching a zip file with the results file and the two log files from the Slurm job, with stdout in chat_template_test.out and stderr in chat_template_test.err. There doesn't seem to be anything about chat template issues in the logs.

Would appreciate any help to solve this issue!

chat_template_test.zip

[baberabb]

Hi! I'll have a look. Can you also confirm by having a look at the samples file? Should be obvious if the args are wrapped up in chat formatting