Add comparisons with other open source models
Categories
(Firefox :: Translations, task)
Tracking
()
People
(Reporter: marco, Unassigned)
References
Details
In our evaluation code, we currently compare our student models against models from other vendors.
We should add open source models to the comparison.
For languages that we don't already have and for languages that we already have where our quality is lower, we could consider using their training configuration, or even using them as teacher models.
|
Reporter | |
Updated•2 years ago
|
|
|
Reporter | |
Comment 1•2 years ago
|
||
- libretranslate.com (https://github.com/argosopentech/argos-translate)
- Opus-MT (https://github.com/UKPLab/EasyNMT#Opus-MT)
- NNLB (https://github.com/thammegowda/nllb-serve)
https://github.com/alvations/lightyear/tree/main/lightyear/translators also has source to use Opus-MT and NNLB.
NNLB also offers already translated test sentences, so we wouldn't even need to run the models: https://github.com/facebookresearch/fairseq/blob/nllb/examples/nllb/evaluation/README.md.
We should start with the Helsinki ones, given mozilla/firefox-translations-training#117.
|
|
Reporter | |
Comment 2•2 years ago
|
||
We also have https://opus.nlpl.eu/dashboard.
|
||
Comment 3•2 years ago
|
||
+1, before investing in functionality to use OPUS models as teachers, we should run a full-scale evaluation and compare them with models that we trained
|
|
Reporter | |
Comment 4•1 years ago
|
||
Description
•