1890948 - LLM-based evaluator for the small models

Status: RESOLVED FIXED

(Core :: Machine Learning, enhancement)

Description

[Tarek Ziadé (:tarek)]

This will be a fully autonomous project on github. That will be used to evaluate the quality of small models in an automated way using an LLM.

Given a labeled dataset, the script will :

• call the small model and run its inference
• send the inputs and outpouts to an LLM and ask it to send back an assessment of the quality

The first version will run against an HTTP endpoint llammafile+llava+Mistral 7B.

Example of using LLMs to evaluate SLMs https://cookbook.openai.com/examples/evaluation/how_to_eval_abstractive_summarization

Comment 1

[Tarek Ziadé (:tarek)]

We should check existing projects, like https://github.com/confident-ai/deepeval or https://openpipe.ai/

Comment 2

[Tarek Ziadé (:tarek)]

done in https://github.com/mozilla/checkvite/blob/main/checkvite/accuracy.py and labels built for gpt in https://github.com/mozilla/distilvit/blob/main/distilvit/gpt4.py