1883590 - Implement the model hub client

Status: RESOLVED FIXED

(Core :: Machine Learning, enhancement)

Description

[Tarek Ziadé (:tarek)]

The model hub client will follow the HF layout and will run in the main thread to provide a class to fetch a models files given:

• a root URL
• a model organization, name and version
• a filename

let hub = new ModelHub(rootURL, cacheDirectory='.cache');
model_config = await hub.getFile('google', 't5-small', 'config.json');

// directly as an ArrayBuffer so it can be transferred to a worker
buffer = await hub.getFileAsArrayBuffer('google', 't5-small', 'config.json');

Fetched files will be stored on disk in a cache directory that follows the HF hug cache structure see https://huggingface.co/docs/huggingface_hub/en/guides/manage-cache

Comment 1

[Tarek Ziadé (:tarek)]

Attachment: Bug 1883590 - Implement the model hub client - r?gregtatum

Depends on D202563

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7b609d9f295f
Implement the model hub client - r=gregtatum

Comment 3

[Natalia Csoregi [:nataliaCs]]

Backed out for causing failures on browser_all_files_referenced.js

• backout: https://hg.mozilla.org/integration/autoland/rev/bd5a2ea9380391afc741daa08ac5f0dd3d376026
• push: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&revision=7b609d9f295fce7ab954f09492fea414b72843e6&selectedTaskRun=I8IGJsk9TV6RXHvBaY7n4A.0
• failure log: https://treeherder.mozilla.org/logviewer?job_id=451176151&repo=autoland&lineNumber=2040

[task 2024-03-18T20:43:17.632Z] 20:43:17     INFO - Buffered messages finished
[task 2024-03-18T20:43:17.632Z] 20:43:17     INFO - TEST-UNEXPECTED-FAIL | browser/base/content/test/static/browser_all_files_referenced.js | there should be no unreferenced files - Got 2, expected +0
[task 2024-03-18T20:43:17.632Z] 20:43:17     INFO - Stack trace:
[task 2024-03-18T20:43:17.632Z] 20:43:17     INFO - chrome://mochikit/content/browser-test.js:test_is:1620
[task 2024-03-18T20:43:17.633Z] 20:43:17     INFO - chrome://mochitests/content/browser/browser/base/content/test/static/browser_all_files_referenced.js:checkAllTheFiles:1039
[task 2024-03-18T20:43:17.633Z] 20:43:17     INFO - chrome://mochikit/content/browser-test.js:handleTask:1139
[task 2024-03-18T20:43:17.633Z] 20:43:17     INFO - chrome://mochikit/content/browser-test.js:_runTaskBasedTest:1211
[task 2024-03-18T20:43:17.634Z] 20:43:17     INFO - chrome://mochikit/content/browser-test.js:Tester_execTest:1353
[task 2024-03-18T20:43:17.635Z] 20:43:17     INFO - chrome://mochikit/content/browser-test.js:nextTest/<:1128
[task 2024-03-18T20:43:17.635Z] 20:43:17     INFO - chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:SimpleTest.waitForFocus/<:1058
[task 2024-03-18T20:43:17.635Z] 20:43:17     INFO - Not taking screenshot here: see the one that was previously logged
[task 2024-03-18T20:43:17.636Z] 20:43:17     INFO - TEST-UNEXPECTED-FAIL | browser/base/content/test/static/browser_all_files_referenced.js | unreferenced file: chrome://global/content/ml/ModelHub.sys.mjs - 
[task 2024-03-18T20:43:17.636Z] 20:43:17     INFO - Stack trace:
[task 2024-03-18T20:43:17.636Z] 20:43:17     INFO - chrome://mochikit/content/browser-test.js:test_ok:1592
[task 2024-03-18T20:43:17.636Z] 20:43:17     INFO - chrome://mochitests/content/browser/browser/base/content/test/static/browser_all_files_referenced.js:checkAllTheFiles:1043
[task 2024-03-18T20:43:17.636Z] 20:43:17     INFO - chrome://mochikit/content/browser-test.js:handleTask:1139
[task 2024-03-18T20:43:17.637Z] 20:43:17     INFO - chrome://mochikit/content/browser-test.js:_runTaskBasedTest:1211
[task 2024-03-18T20:43:17.637Z] 20:43:17     INFO - chrome://mochikit/content/browser-test.js:Tester_execTest:1353
[task 2024-03-18T20:43:17.637Z] 20:43:17     INFO - chrome://mochikit/content/browser-test.js:nextTest/<:1128
[task 2024-03-18T20:43:17.638Z] 20:43:17     INFO - chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:SimpleTest.waitForFocus/<:1058
[task 2024-03-18T20:43:17.639Z] 20:43:17     INFO - Not taking screenshot here: see the one that was previously logged
[task 2024-03-18T20:43:17.640Z] 20:43:17     INFO - TEST-UNEXPECTED-FAIL | browser/base/content/test/static/browser_all_files_referenced.js | unreferenced file: chrome://global/content/ml/tests/acme/bert/resolve/main/config.json - 

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/37ad73d4bc67
Implement the model hub client - r=gregtatum

Comment 5

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/37ad73d4bc67