1929932 - Perma [tier 2] toolkit/components/ml/tests/browser/browser_ml_engine_perf.js | Uncaught exception in test bound test_ml_generic_pipeline - at chrome://global/content/ml/ModelHub.sys.mjs:70 - Error: Error: Forbidden URL: http://localhost:37001/Xenova/all-M

Status: RESOLVED FIXED

(Core :: Machine Learning: General, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: sstanca [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=481639903&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/PGINLJZdSra0xHX3xhISWg/runs/0/artifacts/public/logs/live_backing.log

---

[task 2024-11-07T17:25:18.231Z] TEST-START | toolkit/components/ml/tests/browser/browser_ml_engine_perf.js
[task 2024-11-07T17:25:18.231Z] Entering test bound test_ml_generic_pipeline
[task 2024-11-07T17:25:18.231Z] Model Directory: /home/cltbld/tasks/task_173100025345993/fetches/onnx-models
[task 2024-11-07T17:25:18.231Z] ModelHubRootUrl: http://localhost:37001/
[task 2024-11-07T17:25:18.231Z] Get the engine process
[task 2024-11-07T17:25:18.231Z] Get Pipeline Options
[task 2024-11-07T17:25:18.231Z] Run the inference
[task 2024-11-07T17:25:18.231Z] GECKO(3134) | console.debug: ML:EngineParent: "Creating a new engine"
[task 2024-11-07T17:25:18.231Z] GECKO(3134) | console.debug: ML:EngineParent: "Ignored attempt to delete previous models when the engine is not fully initialized."
[task 2024-11-07T17:25:18.231Z] GECKO(3134) | console.debug: ML:EngineParent: "Using runtime ort-wasm-simd-threaded.jsep.wasm@2.0" ({id:"d0f44554-27b1-4025-a52f-7d6f5e901a5a", name:"ort-wasm-simd-threaded.jsep.wasm", version:"2.0", last_modified:1731000317872, schema:1731000317872})
[task 2024-11-07T17:25:18.231Z] GECKO(3134) | console.debug: ML:EngineChild: "Inference engine options:" ({engineId:"default-engine", featureId:null, taskName:"feature-extraction", timeoutMS:1000, modelHub:null, modelHubRootUrl:"http://localhost:37001/", modelHubUrlTemplate:"{model}/{revision}", modelId:"Xenova/all-MiniLM-L6-v2", modelRevision:"main", tokenizerId:null, tokenizerRevision:null, processorId:null, processorRevision:null, logLevel:"All", runtimeFilename:"ort-wasm-simd-threaded.jsep.wasm", device:null, dtype:null, numThreads:null, executionPriority:null})
[task 2024-11-07T17:25:18.231Z] GECKO(3134) | console.debug: ML:ONNXPipeline: "Initializing Pipeline for task feature-extraction"
[task 2024-11-07T17:25:18.231Z] GECKO(3134) | console.debug: ML:ONNXPipeline: "Unknown internal task feature-extraction"
[task 2024-11-07T17:25:18.231Z] GECKO(3134) | console.debug: ML:ONNXPipeline: "Nightly detected. Using transformers-dev.js"
[task 2024-11-07T17:25:18.231Z] GECKO(3134) | console.debug: ML:ONNXPipeline: "Transformers.js env" ({version:"3.0.0-alpha.15", backends:{onnx:{wasm:{wasmPaths:{}, proxy:false, numThreads:4, wasmBinary:{}}, webgl:{}, webgpu:{powerPreference:"high-performance"}, versions:{common:"1.20.0-dev.20240827-5d54dc1462", web:"1.20.0-dev.20240827-1d059b8702"}, get logLevel(){return dr}, set logLevel(e){if(e!==void 0){if(typeof e!="string"||["verbose","info","warning","error","fatal"].indexOf(e)===-1)throw new Error(`Unsupported logging level: ${e}`);dr=e}}, trace:true, debug:true}}, allowRemoteModels:true, remoteHost:"http://localhost:37001/", remotePathTemplate:"{model}/{revision}", allowLocalModels:false, localModelPath:"NO_LOCAL", useFS:false, useBrowserCache:false, useFSCache:false, cacheDir:null, useCustomCache:true, customCache:{}})
[task 2024-11-07T17:25:18.231Z] GECKO(3134) | console.debug: ML:ONNXPipeline: "Setting up pipeline for wasm using fp16 quantization."
[task 2024-11-07T17:25:18.231Z] GECKO(3134) | console.debug: ML:ONNXPipeline: "Using generic pipeline function"
[task 2024-11-07T17:25:18.231Z] GECKO(3134) | console.debug: ML:ONNXPipeline: "Pipeline initialized"
[task 2024-11-07T17:25:18.231Z] GECKO(3134) | console.debug: ML:EngineParent: "Creating model hub instance"
[task 2024-11-07T17:25:18.231Z] GECKO(3134) | console.debug: ML:EngineParent: "Creating model hub instance"
[task 2024-11-07T17:25:18.231Z] GECKO(3134) | console.debug: ML:EngineParent: "Creating model hub instance"
[task 2024-11-07T17:25:18.231Z] GECKO(3134) | console.debug: ML:ModelHub: "Getting model file from http://localhost:37001/Xenova/all-MiniLM-L6-v2/main/tokenizer.json?download=true"
[task 2024-11-07T17:25:18.231Z] GECKO(3134) | console.debug: ML:ModelHub: "Getting model file from http://localhost:37001/Xenova/all-MiniLM-L6-v2/main/tokenizer_config.json?download=true"
[task 2024-11-07T17:25:18.231Z] GECKO(3134) | console.debug: ML:ModelHub: "Getting model file from http://localhost:37001/Xenova/all-MiniLM-L6-v2/main/config.json?download=true"
[task 2024-11-07T17:25:18.231Z] GECKO(3134) | console.debug: ML:ONNXPipeline: "Initializing pipeline"
[task 2024-11-07T17:25:18.231Z] GECKO(3134) | console.debug: ML:ModelHub: "Creating index modelRevisionIndex on store headers"
[task 2024-11-07T17:25:18.231Z] GECKO(3134) | console.debug: ML:ModelHub: "Creating index modelRevisionIndex on store tasks"
[task 2024-11-07T17:25:18.231Z] GECKO(3134) | console.debug: ML:ModelHub: "Creating index modelRevisionFileIndex on store tasks"
[task 2024-11-07T17:25:18.231Z] GECKO(3134) | console.debug: ML:ModelHub: "Creating index taskModelRevisionIndex on store tasks"
[task 2024-11-07T17:25:18.231Z] GECKO(3134) | console.debug: ML:ModelHub: "Getting model file from http://localhost:37001/Xenova/all-MiniLM-L6-v2/main/tokenizer_config.json?download=true"
[task 2024-11-07T17:25:18.231Z] GECKO(3134) | JavaScript error: resource://gre/modules/workers/PromiseWorker.mjs, line 276: uncaught exception: Object
[task 2024-11-07T17:25:18.231Z] GECKO(3134) | console.error: ML:EngineChild: "Could not initalize the engine" (new Error("Error: Forbidden URL: http://localhost:37001/Xenova/all-MiniLM-L6-v2/main/tokenizer_config.json?download=true (DISALLOWED)", "chrome://global/content/ml/ModelHub.sys.mjs", 70))
[task 2024-11-07T17:25:18.231Z] GECKO(3134) | console.error: ML:EngineChild: "Failed to get the engine" (new Error("Error: Forbidden URL: http://localhost:37001/Xenova/all-MiniLM-L6-v2/main/tokenizer_config.json?download=true (DISALLOWED)", "chrome://global/content/ml/ModelHub.sys.mjs", 70))
[task 2024-11-07T17:25:18.231Z] GECKO(3134) | console.debug: ML:ModelHub: "Getting model file from http://localhost:37001/Xenova/all-MiniLM-L6-v2/main/config.json?download=true"
[task 2024-11-07T17:25:18.231Z] GECKO(3134) | console.error: ML:EngineChild: "Failed to get the engine" (new Error("Error: Forbidden URL: http://localhost:37001/Xenova/all-MiniLM-L6-v2/main/tokenizer_config.json?download=true (DISALLOWED)", "chrome://global/content/ml/ModelHub.sys.mjs", 70))
[task 2024-11-07T17:25:18.231Z] Console message: [JavaScript Error: "uncaught exception: Object" {file: "resource://gre/modules/workers/PromiseWorker.mjs" line: 276}]
[task 2024-11-07T17:25:18.231Z] TEST-UNEXPECTED-FAIL | toolkit/components/ml/tests/browser/browser_ml_engine_perf.js | Uncaught exception in test bound test_ml_generic_pipeline - at chrome://global/content/ml/ModelHub.sys.mjs:70 - Error: Error: Forbidden URL: http://localhost:37001/Xenova/all-MiniLM-L6-v2/main/tokenizer_config.json?download=true (DISALLOWED)
[task 2024-11-07T17:25:18.231Z] Stack trace:
[task 2024-11-07T17:25:18.231Z] postMessage@resource://gre/modules/PromiseWorker.sys.mjs:386:17
[task 2024-11-07T17:25:18.231Z] async*post@resource://gre/modules/PromiseWorker.sys.mjs:425:17
[task 2024-11-07T17:25:18.231Z] create@resource://gre/actors/MLEngineChild.sys.mjs:779:18
[task 2024-11-07T17:25:18.231Z] initializeInferenceEngine@resource://gre/actors/MLEngineChild.sys.mjs:350:28
[task 2024-11-07T17:25:18.231Z] async*EngineDispatcher@resource://gre/actors/MLEngineChild.sys.mjs:375:25
[task 2024-11-07T17:25:18.231Z] initialize@resource://gre/actors/MLEngineChild.sys.mjs:424:24
[task 2024-11-07T17:25:18.231Z] #onNewPortCreated@resource://gre/actors/MLEngineChild.sys.mjs:186:32
[task 2024-11-07T17:25:18.231Z] receiveMessage@resource://gre/actors/MLEngineChild.sys.mjs:114:37
[task 2024-11-07T17:25:18.231Z] JSActor query*setupPortCommunication@resource://gre/actors/MLEngineParent.sys.mjs:722:25
[task 2024-11-07T17:25:18.231Z] initialize@resource://gre/actors/MLEngineParent.sys.mjs:651:20
[task 2024-11-07T17:25:18.231Z] getEngine@resource://gre/actors/MLEngineParent.sys.mjs:172:37
[task 2024-11-07T17:25:18.231Z] runInference@chrome://mochitests/content/browser/toolkit/components/ml/tests/browser/head.js:196:47
[task 2024-11-07T17:25:18.231Z] async*test_ml_generic_pipeline@chrome://mochitests/content/browser/toolkit/components/ml/tests/browser/browser_ml_engine_perf.js:59:25
[task 2024-11-07T17:25:18.231Z] handleTask@chrome://mochikit/content/browser-test.js:1147:26
[task 2024-11-07T17:25:18.231Z] _runTaskBasedTest@chrome://mochikit/content/browser-test.js:1219:18
[task 2024-11-07T17:25:18.396Z] Tester_execTest@chrome://mochikit/content/browser-test.js:1360:14
[task 2024-11-07T17:25:18.396Z] nextTest/<@chrome://mochikit/content/browser-test.js:1136:14
[task 2024-11-07T17:25:18.396Z] SimpleTest.waitForFocus/<@chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:1058:13
[task 2024-11-07T17:25:18.396Z] Leaving test bound test_ml_generic_pipeline
[task 2024-11-07T17:25:18.396Z] GECKO(3134) | console.error: ML:EngineChild: "Failed to get the engine" (new Error("Error: Forbidden URL: http://localhost:37001/Xenova/all-MiniLM-L6-v2/main/tokenizer_config.json?download=true (DISALLOWED)", "chrome://global/content/ml/ModelHub.sys.mjs", 70))
[task 2024-11-07T17:25:18.396Z] GECKO(3134) | MEMORY STAT vsizeMaxContiguous not supported in this build configuration.
[task 2024-11-07T17:25:18.396Z] GECKO(3134) | MEMORY STAT | vsize 11370MB | residentFast 391MB | heapAllocated 181MB
[task 2024-11-07T17:25:18.396Z] TEST-OK | toolkit/components/ml/tests/browser/browser_ml_engine_perf.js | took 421ms
[task 2024-11-07T17:25:18.396Z] checking window state
[task 2024-11-07T17:25:18.396Z] GECKO(3134) | Completed ShutdownLeaks collections in process 3134
[task 2024-11-07T17:25:18.396Z] TEST-START | Shutdown
[task 2024-11-07T17:25:18.396Z] Browser Chrome Test Summary
[task 2024-11-07T17:25:18.396Z] Passed:  0
[task 2024-11-07T17:25:18.396Z] Failed:  1
[task 2024-11-07T17:25:18.396Z] Todo:    0
[task 2024-11-07T17:25:18.396Z] Mode:    e10s
[task 2024-11-07T17:25:18.396Z] *** End BrowserChrome Test Results ***
[task 2024-11-07T17:25:18.396Z] GECKO(3134) | Exiting due to channel error.
[task 2024-11-07T17:25:18.396Z] GECKO(3134) | Exiting due to channel error.
[task 2024-11-07T17:25:18.396Z] GECKO(3134) | Exiting due to channel error.
[task 2024-11-07T17:25:18.396Z] GECKO(3134) | Exiting due to channel error.
[task 2024-11-07T17:25:18.396Z] GECKO(3134) | Exiting due to channel error.
[task 2024-11-07T17:25:18.396Z] TEST-INFO | Main app process: exit 0
[task 2024-11-07T17:25:18.396Z] runtests.py | Application ran for: 0:00:02.926942
[task 2024-11-07T17:25:18.396Z] zombiecheck | Reading PID log: /tmp/tmpm8mf54inpidlog
[task 2024-11-07T17:25:18.396Z] ==> process 3134 launched child process 3210
[task 2024-11-07T17:25:18.396Z] zombiecheck | Checking for orphan process with PID: 3210
[task 2024-11-07T17:25:18.396Z] Stopping web server
[task 2024-11-07T17:25:18.396Z] Server shut down.
[task 2024-11-07T17:25:18.397Z] Web server killed.
[task 2024-11-07T17:25:18.397Z] Stopping web socket server
[task 2024-11-07T17:25:18.397Z] Stopping ssltunnel
[task 2024-11-07T17:25:18.397Z] leakcheck | refcount logging is off, so leaks can't be detected!
[task 2024-11-07T17:25:18.397Z] runtests.py | Running tests: end.
[task 2024-11-07T17:25:18.397Z] Buffered messages finished
[task 2024-11-07T17:25:18.397Z] TEST-INFO | checking window state
[task 2024-11-07T17:25:18.397Z] Browser Chrome Test Summary
[task 2024-11-07T17:25:18.397Z] 	Passed: 0
[task 2024-11-07T17:25:18.397Z] 	Failed: 1
[task 2024-11-07T17:25:18.397Z] 	Todo: 0
[task 2024-11-07T17:25:18.397Z] 	Mode: e10s
[task 2024-11-07T17:25:18.397Z] *** End BrowserChrome Test Results ***
[task 2024-11-07T17:25:18.397Z] Buffered messages finished
[task 2024-11-07T17:25:18.397Z] SUITE-END | took 3s
[task 2024-11-07T17:25:18.397Z] Traceback (most recent call last):
[task 2024-11-07T17:25:18.397Z]   File "python/mozperftest/mozperftest/runner.py", line 299, in <module>
[task 2024-11-07T17:25:18.397Z]     sys.exit(main())
[task 2024-11-07T17:25:18.397Z]   File "python/mozperftest/mozperftest/runner.py", line 295, in main
[task 2024-11-07T17:25:18.397Z]     run_tests(mach_cmd, args, user_args)
[task 2024-11-07T17:25:18.397Z]   File "python/mozperftest/mozperftest/runner.py", line 188, in run_tests
[task 2024-11-07T17:25:18.397Z]     e.run(metadata)
[task 2024-11-07T17:25:18.397Z]   File "/home/cltbld/tasks/task_173100025345993/fetches/python/mozperftest/mozperftest/environment.py", line 92, in run
[task 2024-11-07T17:25:18.397Z]     metadata = testlayer(syslayer(metadata))
[task 2024-11-07T17:25:18.397Z]   File "/home/cltbld/tasks/task_173100025345993/fetches/python/mozperftest/mozperftest/layers.py", line 157, in __call__
[task 2024-11-07T17:25:18.397Z]     metadata = layer(metadata)
[task 2024-11-07T17:25:18.397Z]   File "/home/cltbld/tasks/task_173100025345993/fetches/python/mozperftest/mozperftest/layers.py", line 69, in __call__
[task 2024-11-07T17:25:18.397Z]     metadata = self.run(metadata)
[task 2024-11-07T17:25:18.397Z]   File "/home/cltbld/tasks/task_173100025345993/fetches/python/mozperftest/mozperftest/test/mochitest.py", line 192, in run
[task 2024-11-07T17:25:18.397Z]     raise MochitestTestFailure("Test failed to run")
[task 2024-11-07T17:25:18.397Z] mozperftest.test.mochitest.MochitestTestFailure: Test failed to run
[taskcluster 2024-11-07T17:25:18.680Z]    Exit Code: 1
[taskcluster 2024-11-07T17:25:18.680Z]    User Time: 1m6.80807s
[taskcluster 2024-11-07T17:25:18.680Z]  Kernel Time: 10.989987s
[taskcluster 2024-11-07T17:25:18.680Z]    Wall Time: 1m3.364938754s
[taskcluster 2024-11-07T17:25:18.680Z]       Result: FAILED
[taskcluster 2024-11-07T17:25:18.680Z] === Task Finished ===
[taskcluster 2024-11-07T17:25:18.680Z] Task Duration: 1m3.368339553s
[taskcluster 2024-11-07T17:25:18.728Z] Uploading link artifact public/logs/live.log to artifact public/logs/live_backing.log with expiry 2025-11-07T17:07:17.429Z
[taskcluster:error] exit status 1

Comment 1

[Cosmin Sabou [:CosminS]]

I believe these are coming from changes in Bug 1929499 here.

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1926218

Comment 3

[Intermittent Failures Robot]

19 failures in 874 pushes (0.022 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 16
• mozilla-central: 3

Platform and build breakdown:

• linux1804-64-shippable: 11
  • opt: 11
• windows11-64-shippable-qr: 4
  • opt: 4
• macosx1015-64-shippable-qr: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1929932&startday=2024-11-07&endday=2024-11-07&tree=all

Comment 4

[Tarek Ziadé (:tarek)]

Attachment: Bug 1929932 - allow localhost with ports r?cgopal

Comment 5

[Tarek Ziadé (:tarek)]

Fixed with the patch, pending cgopal review https://treeherder.mozilla.org/jobs?repo=try&tier=1%2C2%2C3&revision=7bce08d6264782a8914513582950141d68698b69

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d6a66ab423a5
allow localhost with ports r=cgopal

Comment 7

[Intermittent Failures Robot]

18 failures in 764 pushes (0.024 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 9
• autoland: 9

Platform and build breakdown:

• windows11-64-shippable-qr: 6
  • opt: 6
• macosx1015-64-shippable-qr: 6
  • opt: 6
• linux1804-64-shippable: 6
  • opt: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1929932&startday=2024-11-08&endday=2024-11-08&tree=all

Comment 8

[agoloman]

https://hg.mozilla.org/mozilla-central/rev/d6a66ab423a5

Comment 9

[Intermittent Failures Robot]

37 failures in 4515 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 12
• autoland: 25

Platform and build breakdown:

• windows11-64-shippable-qr: 10
  • opt: 10
• macosx1015-64-shippable-qr: 10
  • opt: 10
• linux1804-64-shippable: 17
  • opt: 17

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1929932&startday=2024-11-04&endday=2024-11-10&tree=all