1821488 - Add the possibility of downloading the translation engine and a translation model when running tests

Status: RESOLVED DUPLICATE of bug 1924192

(Testing :: General, task)

Description

[Marco Castelluccio [:marco]]

We were thinking of doing the following:

1. Engines are downloaded in a fetch task (where we define URL and hash)
2. One model is downloaded in a fetch task (where we define URL and hash), e.g. https://firefox-settings-attachments.cdn.mozilla.net/main-workspace/translations-identification-models/fc2a4381-bc9e-47ad-8cff-a31163538e27.ftz
3. Reference output is stored in m-c
4. External CIs builds the engine as WASM and uploads it to remote settings as WIP
5. The training pipeline trains models and uploads them to remote settings as WIP
6. We push a patch to try, updating the engine and model URLs in the fetch tasks with the new WIPs from Remote Settings
7. If there are no changes/failures, we are good. If there are unexpected changes, go back. If there are only expected changes, update the references
8. Mark engine and models as released in Remote Settings
9. Land a patch in m-c that updates the engine and model in the fetch tasks to point to the new released model and updates the references

This bug tracks implementing the two fetch tasks and the addition of a dependency on such fetch tasks for test tasks that are running manifests under toolkit/components/translations/tests.

Comment 1

[Mike Hommey [:glandium]]

Can you give more context around this because it's not clear what the constraints are that make you want to go through these steps.

Comment 2

[Marco Castelluccio [:marco]]

We have the translations tests (which are mochitests) in m-c that would need to use the engine and models that are built externally (and uploaded to remote settings, served through a CDN). While we could potentially build the engine in m-c CI, like we do with other tools, the models need to be trained externally.

During tests, we can't access remote resources. So I was thinking of using fetch tasks so that we re-upload those files in our CI and we have no external dependencies during tests.

Comment 3

[Marco Castelluccio [:marco]]

Is the picture from comment 2 clear? Is this something you could help with?

Comment 4

[Mike Hommey [:glandium]]

Engines are downloaded in a fetch task (where we define URL and hash)
One model is downloaded in a fetch task (where we define URL and hash), e.g. https://firefox-settings-attachments.cdn.mozilla.net/main-workspace/translations-identification-models/fc2a4381-bc9e-47ad-8cff-a31163538e27.ftz

Add a file in taskcluster/ci/fetch/ and add a reference to it under jobs-from in taskcluster/ci/fetch/kind.yml.
In the file you created, add things like:

name-of-the-fetch-task:
    description: "what it is"
    fetch:
        type: static-url
        url: https://url
        sha256: sha256-of-the-download
        size: size-of-the-download

If the files are not zip or tar.something, it won't like it. There are other things you can put in there, see https://searchfox.org/mozilla-central/rev/ad732108b073742d7324f998c085f459674a6846/taskcluster/gecko_taskgraph/transforms/fetch.py#187-215

For each test task you want to get the fetched file from, add:

fetches:
    fetch:
        - name-of-the-fetch-task

Reference output is stored in m-c

Depending how big that is, you may want to use a fetch for that too.

Comment 5

[Mike Hommey [:glandium]]

If this is ever shipped in Firefox, fetches are going to be a problem, though.

Comment 6

[Erik Nordin [:nordzilla]]

I absent-mindedly created a new bug for this even though I knew that this bug existed.

Fixed by Bug 1924192.