1933881 - Perma [tier 2] toolkit/components/ml/tests/browser/browser_ml_summarizer_perf.js | Uncaught exception in test bound test_ml_falconsai_small_article - at chrome://global/content/ml/EngineProcess.sys.mjs:86 - PipelineOptionsValidationError: Invalid value "o

Status: RESOLVED FIXED

(Core :: Machine Learning, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: nfay [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=484501420&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/LJ8xXDeSSCmlVQCIzZvCsg/runs/0/artifacts/public/logs/live_backing.log

---

[task 2024-11-28T03:26:41.343Z] TEST-PASS | toolkit/components/ml/tests/browser/browser_ml_summarizer_perf.js | true == true - 
[task 2024-11-28T03:26:41.343Z] GECKO(3155) | Results (ms)
[task 2024-11-28T03:26:41.343Z] GECKO(3155) | SUM-XENOVA-DISTILBART-CNN-12-6_BIG-pipeline-ready-latency:  3,800  3,828  3,733  3,811  3,824  3,768  3,821  3,799  3,854  3,817   median 3,814
[task 2024-11-28T03:26:41.343Z] GECKO(3155) | SUM-XENOVA-DISTILBART-CNN-12-6_BIG-initialization-latency:  3,819  3,848  3,753  3,831  3,843  3,788  3,841  3,819  3,873  3,836   median 3,834
[task 2024-11-28T03:26:41.343Z] GECKO(3155) | SUM-XENOVA-DISTILBART-CNN-12-6_BIG-model-run-latency:      18,560 18,520 19,000 18,900 19,030 18,820 18,580 18,860 18,840 18,400   median 18,830
[task 2024-11-28T03:26:41.343Z] GECKO(3155) | SUM-XENOVA-DISTILBART-CNN-12-6_BIG-pipeline-ready-memory:   1,116  1,115  1,215  1,116  1,214  1,115  1,114  1,115  1,115  1,115   median 1,115
[task 2024-11-28T03:26:41.343Z] GECKO(3155) | SUM-XENOVA-DISTILBART-CNN-12-6_BIG-initialization-memory:   1,116  1,115    864  1,116  863.7  1,115  1,114  1,115  1,115  1,115   median 1,115
[task 2024-11-28T03:26:41.343Z] GECKO(3155) | SUM-XENOVA-DISTILBART-CNN-12-6_BIG-model-run-memory:        220.3  211.4  452.5  149.2  453.2  141.4  221.2  149.8  142.6  204.4   median 207.9
[task 2024-11-28T03:26:41.343Z] GECKO(3155) | SUM-XENOVA-DISTILBART-CNN-12-6_BIG-total-memory-usage:        548    547    559    898    559    891    548    899    890    547   median 559
[task 2024-11-28T03:26:41.513Z] perfMetrics | {"SUM-XENOVA-DISTILBART-CNN-12-6_BIG-pipeline-ready-latency":3814,"SUM-XENOVA-DISTILBART-CNN-12-6_BIG-initialization-latency":3833.5,"SUM-XENOVA-DISTILBART-CNN-12-6_BIG-model-run-latency":18831,"SUM-XENOVA-DISTILBART-CNN-12-6_BIG-pipeline-ready-memory":1115.30078125,"SUM-XENOVA-DISTILBART-CNN-12-6_BIG-initialization-memory":1114.880859375,"SUM-XENOVA-DISTILBART-CNN-12-6_BIG-model-run-memory":207.900390625,"SUM-XENOVA-DISTILBART-CNN-12-6_BIG-total-memory-usage":559}
[task 2024-11-28T03:26:41.513Z] Leaving test bound test_ml_falconsai_tiny_article
[task 2024-11-28T03:26:41.513Z] Entering test bound test_ml_falconsai_small_article
[task 2024-11-28T03:26:41.513Z] TEST-UNEXPECTED-FAIL | toolkit/components/ml/tests/browser/browser_ml_summarizer_perf.js | Uncaught exception in test bound test_ml_falconsai_small_article - at chrome://global/content/ml/EngineProcess.sys.mjs:86 - PipelineOptionsValidationError: Invalid value "onnx-community-Qwen2.5-0.5B-Instruct_tiny" for field "engineId". Should contain only alphanumeric characters, underscores, or dashes.
[task 2024-11-28T03:26:41.513Z] Stack trace:
[task 2024-11-28T03:26:41.513Z] PipelineOptionsValidationError@chrome://global/content/ml/EngineProcess.sys.mjs:86:5
[task 2024-11-28T03:26:41.513Z] #validateTaskName@chrome://global/content/ml/EngineProcess.sys.mjs:386:13
[task 2024-11-28T03:26:41.513Z] updateOptions/<@chrome://global/content/ml/EngineProcess.sys.mjs:516:31
[task 2024-11-28T03:26:41.513Z] updateOptions@chrome://global/content/ml/EngineProcess.sys.mjs:501:17
[task 2024-11-28T03:26:41.513Z] PipelineOptions@chrome://global/content/ml/EngineProcess.sys.mjs:346:10
[task 2024-11-28T03:26:41.513Z] run_summarizer_with_perf@chrome://mochitests/content/browser/toolkit/components/ml/tests/browser/browser_ml_summarizer_perf.js:335:19
[task 2024-11-28T03:26:41.513Z] test_ml_falconsai_small_article@chrome://mochitests/content/browser/toolkit/components/ml/tests/browser/browser_ml_summarizer_perf.js:391:9
[task 2024-11-28T03:26:41.513Z] handleTask@chrome://mochikit/content/browser-test.js:1147:26
[task 2024-11-28T03:26:41.513Z] _runTaskBasedTest@chrome://mochikit/content/browser-test.js:1219:18
[task 2024-11-28T03:26:41.513Z] async*Tester_execTest@chrome://mochikit/content/browser-test.js:1360:14
[task 2024-11-28T03:26:41.513Z] nextTest/<@chrome://mochikit/content/browser-test.js:1136:14
[task 2024-11-28T03:26:41.513Z] SimpleTest.waitForFocus/<@chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:1058:13
[task 2024-11-28T03:26:41.513Z] Leaving test bound test_ml_falconsai_small_article
[task 2024-11-28T03:26:41.513Z] Entering test bound test_ml_falconsai_medium_article
[task 2024-11-28T03:26:41.513Z] TEST-UNEXPECTED-FAIL | toolkit/components/ml/tests/browser/browser_ml_summarizer_perf.js | Uncaught exception in test bound test_ml_falconsai_medium_article - at chrome://global/content/ml/EngineProcess.sys.mjs:86 - PipelineOptionsValidationError: Invalid value "onnx-community-Qwen2.5-0.5B-Instruct_small" for field "engineId". Should contain only alphanumeric characters, underscores, or dashes.
[task 2024-11-28T03:26:41.513Z] Stack trace:
[task 2024-11-28T03:26:41.513Z] PipelineOptionsValidationError@chrome://global/content/ml/EngineProcess.sys.mjs:86:5
[task 2024-11-28T03:26:41.513Z] #validateTaskName@chrome://global/content/ml/EngineProcess.sys.mjs:386:13
[task 2024-11-28T03:26:41.513Z] updateOptions/<@chrome://global/content/ml/EngineProcess.sys.mjs:516:31
[task 2024-11-28T03:26:41.513Z] updateOptions@chrome://global/content/ml/EngineProcess.sys.mjs:501:17
[task 2024-11-28T03:26:41.514Z] PipelineOptions@chrome://global/content/ml/EngineProcess.sys.mjs:346:10
[task 2024-11-28T03:26:41.514Z] run_summarizer_with_perf@chrome://mochitests/content/browser/toolkit/components/ml/tests/browser/browser_ml_summarizer_perf.js:335:19
[task 2024-11-28T03:26:41.514Z] test_ml_falconsai_medium_article@chrome://mochitests/content/browser/toolkit/components/ml/tests/browser/browser_ml_summarizer_perf.js:395:9
[task 2024-11-28T03:26:41.514Z] handleTask@chrome://mochikit/content/browser-test.js:1147:26
[task 2024-11-28T03:26:41.514Z] _runTaskBasedTest@chrome://mochikit/content/browser-test.js:1219:18
[task 2024-11-28T03:26:41.514Z] async*Tester_execTest@chrome://mochikit/content/browser-test.js:1360:14
[task 2024-11-28T03:26:41.514Z] nextTest/<@chrome://mochikit/content/browser-test.js:1136:14
[task 2024-11-28T03:26:41.514Z] SimpleTest.waitForFocus/<@chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:1058:13
[task 2024-11-28T03:26:41.514Z] Leaving test bound test_ml_falconsai_medium_article
[task 2024-11-28T03:26:41.514Z] Entering test bound test_ml_falconsai_medium_article
[task 2024-11-28T03:26:41.514Z] TEST-UNEXPECTED-FAIL | toolkit/components/ml/tests/browser/browser_ml_summarizer_perf.js | Uncaught exception in test bound test_ml_falconsai_medium_article - at chrome://global/content/ml/EngineProcess.sys.mjs:86 - PipelineOptionsValidationError: Invalid value "onnx-community-Qwen2.5-0.5B-Instruct_medium" for field "engineId". Should contain only alphanumeric characters, underscores, or dashes.
[task 2024-11-28T03:26:41.514Z] Stack trace:
[task 2024-11-28T03:26:41.514Z] PipelineOptionsValidationError@chrome://global/content/ml/EngineProcess.sys.mjs:86:5
[task 2024-11-28T03:26:41.514Z] #validateTaskName@chrome://global/content/ml/EngineProcess.sys.mjs:386:13
[task 2024-11-28T03:26:41.514Z] updateOptions/<@chrome://global/content/ml/EngineProcess.sys.mjs:516:31
[task 2024-11-28T03:26:41.514Z] updateOptions@chrome://global/content/ml/EngineProcess.sys.mjs:501:17
[task 2024-11-28T03:26:41.514Z] PipelineOptions@chrome://global/content/ml/EngineProcess.sys.mjs:346:10
[task 2024-11-28T03:26:41.514Z] run_summarizer_with_perf@chrome://mochitests/content/browser/toolkit/components/ml/tests/browser/browser_ml_summarizer_perf.js:335:19
[task 2024-11-28T03:26:41.514Z] test_ml_falconsai_medium_article@chrome://mochitests/content/browser/toolkit/components/ml/tests/browser/browser_ml_summarizer_perf.js:399:9
[task 2024-11-28T03:26:41.514Z] handleTask@chrome://mochikit/content/browser-test.js:1147:26
[task 2024-11-28T03:26:41.514Z] _runTaskBasedTest@chrome://mochikit/content/browser-test.js:1219:18
[task 2024-11-28T03:26:41.514Z] async*Tester_execTest@chrome://mochikit/content/browser-test.js:1360:14
[task 2024-11-28T03:26:41.514Z] nextTest/<@chrome://mochikit/content/browser-test.js:1136:14
[task 2024-11-28T03:26:41.514Z] SimpleTest.waitForFocus/<@chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:1058:13
[task 2024-11-28T03:26:41.514Z] Leaving test bound test_ml_falconsai_medium_article
[task 2024-11-28T03:26:41.514Z] Entering test bound test_ml_falconsai_medium_article
[task 2024-11-28T03:26:41.514Z] TEST-UNEXPECTED-FAIL | toolkit/components/ml/tests/browser/browser_ml_summarizer_perf.js | Uncaught exception in test bound test_ml_falconsai_medium_article - at chrome://global/content/ml/EngineProcess.sys.mjs:86 - PipelineOptionsValidationError: Invalid value "onnx-community-Qwen2.5-0.5B-Instruct_big" for field "engineId". Should contain only alphanumeric characters, underscores, or dashes.
[task 2024-11-28T03:26:41.514Z] Stack trace:
[task 2024-11-28T03:26:41.514Z] PipelineOptionsValidationError@chrome://global/content/ml/EngineProcess.sys.mjs:86:5
[task 2024-11-28T03:26:41.514Z] #validateTaskName@chrome://global/content/ml/EngineProcess.sys.mjs:386:13
[task 2024-11-28T03:26:41.514Z] updateOptions/<@chrome://global/content/ml/EngineProcess.sys.mjs:516:31
[task 2024-11-28T03:26:41.514Z] updateOptions@chrome://global/content/ml/EngineProcess.sys.mjs:501:17
[task 2024-11-28T03:26:41.514Z] PipelineOptions@chrome://global/content/ml/EngineProcess.sys.mjs:346:10
[task 2024-11-28T03:26:41.514Z] run_summarizer_with_perf@chrome://mochitests/content/browser/toolkit/components/ml/tests/browser/browser_ml_summarizer_perf.js:335:19
[task 2024-11-28T03:26:41.514Z] test_ml_falconsai_medium_article@chrome://mochitests/content/browser/toolkit/components/ml/tests/browser/browser_ml_summarizer_perf.js:403:9
[task 2024-11-28T03:26:41.514Z] handleTask@chrome://mochikit/content/browser-test.js:1147:26
[task 2024-11-28T03:26:41.514Z] _runTaskBasedTest@chrome://mochikit/content/browser-test.js:1219:18
[task 2024-11-28T03:26:41.514Z] async*Tester_execTest@chrome://mochikit/content/browser-test.js:1360:14
[task 2024-11-28T03:26:41.514Z] nextTest/<@chrome://mochikit/content/browser-test.js:1136:14
[task 2024-11-28T03:26:41.514Z] SimpleTest.waitForFocus/<@chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:1058:13
[task 2024-11-28T03:26:41.514Z] Leaving test bound test_ml_falconsai_medium_article
[task 2024-11-28T03:26:41.514Z] GECKO(3155) | MEMORY STAT vsizeMaxContiguous not supported in this build configuration.
[task 2024-11-28T03:26:41.514Z] GECKO(3155) | MEMORY STAT | vsize 3292MB | residentFast 325MB | heapAllocated 118MB
[task 2024-11-28T03:26:41.514Z] TEST-OK | toolkit/components/ml/tests/browser/browser_ml_summarizer_perf.js | took 857065ms
[task 2024-11-28T03:26:41.514Z] checking window state
[task 2024-11-28T03:26:41.514Z] GECKO(3155) | Completed ShutdownLeaks collections in process 3155
[task 2024-11-28T03:26:41.514Z] TEST-START | Shutdown
[task 2024-11-28T03:26:41.514Z] Browser Chrome Test Summary
[task 2024-11-28T03:26:41.514Z] Passed:  4
[task 2024-11-28T03:26:41.514Z] Failed:  4
[task 2024-11-28T03:26:41.514Z] Todo:    0
[task 2024-11-28T03:26:41.514Z] Mode:    e10s
[task 2024-11-28T03:26:41.514Z] *** End BrowserChrome Test Results ***
[task 2024-11-28T03:26:41.514Z] GECKO(3155) | Exiting due to channel error.
[task 2024-11-28T03:26:41.514Z] GECKO(3155) | Exiting due to channel error.
[task 2024-11-28T03:26:41.514Z] GECKO(3155) | Exiting due to channel error.
[task 2024-11-28T03:26:41.514Z] GECKO(3155) | Exiting due to channel error.
[task 2024-11-28T03:26:41.514Z] GECKO(3155) | Exiting due to channel error.
[task 2024-11-28T03:26:41.514Z] GECKO(3155) | Exiting due to channel error.
[task 2024-11-28T03:26:41.514Z] GECKO(3155) | Exiting due to channel error.
[task 2024-11-28T03:26:41.514Z] TEST-INFO | Main app process: exit 0
[task 2024-11-28T03:26:41.514Z] runtests.py | Application ran for: 0:14:20.463991
[task 2024-11-28T03:26:41.514Z] zombiecheck | Reading PID log: /tmp/tmpbc5nr7jppidlog
[task 2024-11-28T03:26:41.514Z] ==> process 3155 launched child process 3232
[task 2024-11-28T03:26:41.514Z] zombiecheck | Checking for orphan process with PID: 3232
[task 2024-11-28T03:26:41.514Z] Stopping web server
[task 2024-11-28T03:26:41.514Z] Server shut down.
[task 2024-11-28T03:26:41.514Z] Web server killed.
[task 2024-11-28T03:26:41.514Z] Stopping web socket server
[task 2024-11-28T03:26:41.514Z] Stopping ssltunnel
[task 2024-11-28T03:26:41.514Z] leakcheck | refcount logging is off, so leaks can't be detected!
[task 2024-11-28T03:26:41.514Z] runtests.py | Running tests: end.
[task 2024-11-28T03:26:41.514Z] Buffered messages finished
[task 2024-11-28T03:26:41.514Z] TEST-INFO | checking window state
[task 2024-11-28T03:26:41.514Z] Browser Chrome Test Summary
[task 2024-11-28T03:26:41.514Z] 	Passed: 4
[task 2024-11-28T03:26:41.514Z] 	Failed: 4
[task 2024-11-28T03:26:41.514Z] 	Todo: 0
[task 2024-11-28T03:26:41.514Z] 	Mode: e10s
[task 2024-11-28T03:26:41.514Z] *** End BrowserChrome Test Results ***
[task 2024-11-28T03:26:41.514Z] Buffered messages finished
[task 2024-11-28T03:26:41.514Z] SUITE-END | took 861s
[task 2024-11-28T03:26:41.514Z] Traceback (most recent call last):
[task 2024-11-28T03:26:41.514Z]   File "python/mozperftest/mozperftest/runner.py", line 299, in <module>
[task 2024-11-28T03:26:41.514Z]     sys.exit(main())
[task 2024-11-28T03:26:41.514Z]   File "python/mozperftest/mozperftest/runner.py", line 295, in main
[task 2024-11-28T03:26:41.514Z]     run_tests(mach_cmd, args, user_args)
[task 2024-11-28T03:26:41.514Z]   File "python/mozperftest/mozperftest/runner.py", line 188, in run_tests
[task 2024-11-28T03:26:41.514Z]     e.run(metadata)
[task 2024-11-28T03:26:41.514Z]   File "/home/cltbld/tasks/task_173275741192613/fetches/python/mozperftest/mozperftest/environment.py", line 92, in run
[task 2024-11-28T03:26:41.514Z]     metadata = testlayer(syslayer(metadata))
[task 2024-11-28T03:26:41.514Z]   File "/home/cltbld/tasks/task_173275741192613/fetches/python/mozperftest/mozperftest/layers.py", line 157, in __call__
[task 2024-11-28T03:26:41.514Z]     metadata = layer(metadata)
[task 2024-11-28T03:26:41.514Z]   File "/home/cltbld/tasks/task_173275741192613/fetches/python/mozperftest/mozperftest/layers.py", line 69, in __call__
[task 2024-11-28T03:26:41.514Z]     metadata = self.run(metadata)
[task 2024-11-28T03:26:41.514Z]   File "/home/cltbld/tasks/task_173275741192613/fetches/python/mozperftest/mozperftest/test/mochitest.py", line 192, in run
[task 2024-11-28T03:26:41.514Z]     raise MochitestTestFailure("Test failed to run")
[task 2024-11-28T03:26:41.514Z] mozperftest.test.mochitest.MochitestTestFailure: Test failed to run
[taskcluster 2024-11-28T03:26:41.787Z]    Exit Code: 1
[taskcluster 2024-11-28T03:26:41.787Z]    User Time: 3m50.704483s
[taskcluster 2024-11-28T03:26:41.787Z]  Kernel Time: 1m34.449199s
[taskcluster 2024-11-28T03:26:41.787Z]    Wall Time: 17m32.837820817s
[taskcluster 2024-11-28T03:26:41.787Z]       Result: FAILED
[taskcluster 2024-11-28T03:26:41.787Z] === Task Finished ===
[taskcluster 2024-11-28T03:26:41.787Z] Task Duration: 17m32.841265423s
[taskcluster 2024-11-28T03:26:41.833Z] Uploading link artifact public/logs/live.log to artifact public/logs/live_backing.log with expiry 2025-11-28T01:15:13.622Z
[taskcluster:error] exit status 1

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1931470

:atossou, since you are the author of the regressor, bug 1931470, could you take a look?

For more information, please visit BugBot documentation.

Comment 2

[Aristide Tossou]

Looking into it.

Comment 3

[Aristide Tossou]

Attachment: Bug 1933881 - Regression fix for summarizer perf test - r?tarek,cgopal

Comment 4

[Intermittent Failures Robot]

32 failures in 691 pushes (0.046 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 23
• mozilla-central: 9

Platform and build breakdown:

• windows11-64-shippable-qr: 10
  • opt: 10
• linux1804-64-shippable: 13
  • opt: 13
• macosx1015-64-shippable-qr: 9
  • opt: 9

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1933881&startday=2024-11-28&endday=2024-11-28&tree=all

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d70a1d55e1be
Regression fix for summarizer perf test  - r=tarek

Comment 6

[Serban Stanca [:SerbanS]]

https://hg.mozilla.org/mozilla-central/rev/d70a1d55e1be

Comment 7

[Intermittent Failures Robot]

15 failures in 540 pushes (0.028 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 6
• autoland: 9

Platform and build breakdown:

• windows11-64-shippable-qr: 5
  • opt: 5
• linux1804-64-shippable: 5
  • opt: 5
• macosx1015-64-shippable-qr: 5
  • opt: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1933881&startday=2024-11-29&endday=2024-11-29&tree=all

Comment 8

[Intermittent Failures Robot]

65 failures in 3953 pushes (0.016 failures/push) were associated with this bug in the last 7 days.

This is the #25 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 15
• autoland: 50

Platform and build breakdown:

• windows11-64-shippable-qr: 24
  • opt: 24
• linux1804-64-shippable: 27
  • opt: 27
• macosx1015-64-shippable-qr: 14
  • opt: 14

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1933881&startday=2024-11-25&endday=2024-12-01&tree=all