Permanent firefox/libxul.so differs. See the diff.html or diff.txt artifact
Categories
(Firefox Build System :: Toolchains, defect, P5)
Tracking
(firefox-esr78 unaffected, firefox-esr91 unaffected, firefox93 unaffected, firefox94 unaffected, firefox95 wontfix, firefox96 fixed)
Tracking | Status | |
---|---|---|
firefox-esr78 | --- | unaffected |
firefox-esr91 | --- | unaffected |
firefox93 | --- | unaffected |
firefox94 | --- | unaffected |
firefox95 | --- | wontfix |
firefox96 | --- | fixed |
People
(Reporter: intermittent-bug-filer, Assigned: glandium)
References
(Regression)
Details
(Keywords: intermittent-failure, regression, Whiteboard: [retriggered][stockwell needswork:owner])
Attachments
(1 file)
Filed by: csabou [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=355537888&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/TMVaHBVkRu6XNTSyk1m60A/runs/0/artifacts/public/logs/live_backing.log
[task 2021-10-21T05:06:58.718Z] + case "$ORIG_URL" in
[task 2021-10-21T05:06:58.718Z] + OMNIJAR=omni.ja
[task 2021-10-21T05:06:58.718Z] + POST=true
[task 2021-10-21T05:06:58.718Z] + for option in "$@"
[task 2021-10-21T05:06:58.718Z] + case "$option" in
[task 2021-10-21T05:06:58.718Z] + POST=fail
[task 2021-10-21T05:06:58.718Z] + '[' -n '' ']'
[task 2021-10-21T05:06:58.718Z] + diffoscope --html diff.html --text diff.txt --progress --output-empty --no-default-limits --max-page-size 100000000 --max-page-diff-block-lines 10000 --exclude-directory-metadata=yes --exclude-command .--line-numbers a b
[task 2021-10-21T05:06:58.821Z] | | 0% ETA: --:--:--
[task 2021-10-21T05:06:58.831Z] | | 0% a/firefox/application.ini ETA: 0:30:01
[task 2021-10-21T05:06:58.833Z] | | 0% a/firefox/browser ETA: 0:31:16
[task 2021-10-21T05:06:58.834Z] | | 0% …ns/default/default16.png ETA: 263 days, 20:34:42
[task 2021-10-21T05:06:58.835Z] | | 0% …ns/default/default32.png ETA: 262 days, 22:33:48
[task 2021-10-21T05:06:58.836Z] | | 0% …ns/default/default48.png ETA: 253 days, 20:58:14
[task 2021-10-21T05:06:58.836Z] | | 0% …ns/default/default64.png ETA: 235 days, 9:45:11
[task 2021-10-21T05:06:58.836Z] | | 0% ETA: 214 days, 3:40:31
[task 2021-10-21T05:06:58.837Z] | | 0% ETA: 214 days, 9:41:15
[task 2021-10-21T05:06:58.837Z] | | 0% ETA: 151 days, 15:13:50
[task 2021-10-21T05:06:58.839Z] | | 0% …ashreporter-override.ini ETA: 124 days, 1:08:45
[task 2021-10-21T05:06:58.841Z] | | 0% …firefox/browser/features ETA: 114 days, 10:52:40
[task 2021-10-21T05:06:58.845Z] | | 0% …autofill@mozilla.org.xpi ETA: 4694 days, 2:18:53
[task 2021-10-21T05:06:58.846Z] | | 0% …npicture@mozilla.org.xpi ETA: 706 days, 5:02:54
[task 2021-10-21T05:06:58.848Z] | | 0% …failover@mozilla.com.xpi ETA: 661 days, 19:12:13
[task 2021-10-21T05:06:58.852Z] | | 0% …eenshots@mozilla.org.xpi ETA: 610 days, 8:43:04
[task 2021-10-21T05:06:58.854Z] | | 0% …reporter@mozilla.org.xpi ETA: 252 days, 20:54:11
[task 2021-10-21T05:06:58.859Z] | | 0% …ebcompat@mozilla.org.xpi ETA: 247 days, 2:13:49
[task 2021-10-21T05:06:58.859Z] | | 0% ETA: 174 days, 8:20:54
[task 2021-10-21T05:06:58.889Z] | | 0% a/firefox/browser/omni.ja ETA: 174 days, 16:00:10
[task 2021-10-21T05:06:58.889Z] | | 0% ETA: 1:09:13
[task 2021-10-21T05:06:58.894Z] | | 0% a/firefox/crashreporter ETA: 1:09:19
[task 2021-10-21T05:06:58.898Z] | | 0% …irefox/crashreporter.ini ETA: 0:01:31
[task 2021-10-21T05:06:58.904Z] | | 0% a/firefox/defaults ETA: 0:01:33
[task 2021-10-21T05:06:58.904Z] | | 0% ETA: 0:01:36
[task 2021-10-21T05:06:58.904Z] | | 0% ETA: 0:01:36
[task 2021-10-21T05:06:58.909Z] | | 0% …refox/dependentlibs.list ETA: 0:01:36
[task 2021-10-21T05:06:58.911Z] | | 0% a/firefox/dmd.py ETA: 0:01:40
[task 2021-10-21T05:06:58.912Z] | | 0% a/firefox/firefox ETA: 0:01:31
[task 2021-10-21T05:06:58.920Z] | | 0% a/firefox/firefox-bin ETA: 0:01:27
[task 2021-10-21T05:06:58.922Z] | | 0% a/firefox/fix_stacks.py ETA: 0:00:25
[task 2021-10-21T05:06:58.931Z] | | 0% a/firefox/fonts ETA: 0:00:26
[task 2021-10-21T05:06:58.931Z] | | 0% ETA: 0:00:27
[task 2021-10-21T05:06:58.935Z] | | 0% a/firefox/gmp-clearkey ETA: 0:00:27
[task 2021-10-21T05:06:58.935Z] | | 0% …earkey/0.1/manifest.json ETA: 0:00:28
[task 2021-10-21T05:06:58.935Z] | | 0% ETA: 0:00:28
[task 2021-10-21T05:06:58.936Z] | | 0% ETA: 0:00:28
[task 2021-10-21T05:06:58.937Z] | | 0% a/firefox/icons ETA: 0:00:28
[task 2021-10-21T05:06:58.937Z] | | 0% ETA: 0:00:28
[task 2021-10-21T05:06:58.943Z] | | 0% …irefox/libfreeblpriv3.so ETA: 0:00:28
[task 2021-10-21T05:06:58.944Z] | | 0% a/firefox/liblgpllibs.so ETA: 0:00:18
[task 2021-10-21T05:06:58.948Z] | | 0% …firefox/libmozavcodec.so ETA: 0:00:18
[task 2021-10-21T05:06:58.951Z] | | 0% a/firefox/libmozavutil.so ETA: 0:00:16
[task 2021-10-21T05:06:58.952Z] | | 1% a/firefox/libmozgtk.so ETA: 0:00:15
[task 2021-10-21T05:06:58.955Z] | | 1% …firefox/libmozsandbox.so ETA: 0:00:15
[task 2021-10-21T05:06:58.963Z] | | 1% …firefox/libmozsqlite3.so ETA: 0:00:14
[task 2021-10-21T05:06:58.965Z] | | 1% …firefox/libmozwayland.so ETA: 0:00:10
[task 2021-10-21T05:06:58.968Z] | | 1% a/firefox/libnspr4.so ETA: 0:00:10
[task 2021-10-21T05:06:58.975Z] | | 1% a/firefox/libnss3.so ETA: 0:00:10
[task 2021-10-21T05:06:58.980Z] | | 2% a/firefox/libnssckbi.so ETA: 0:00:08
[task 2021-10-21T05:06:58.983Z] | | 2% a/firefox/libnssutil3.so ETA: 0:00:08
[task 2021-10-21T05:06:58.984Z] | | 2% a/firefox/libplc4.so ETA: 0:00:08
[task 2021-10-21T05:06:58.985Z] | | 2% a/firefox/libplds4.so ETA: 0:00:08
[task 2021-10-21T05:06:58.988Z] | | 2% a/firefox/libsmime3.so ETA: 0:00:08
[task 2021-10-21T05:06:58.992Z] | | 2% a/firefox/libsoftokn3.so ETA: 0:00:07
[task 2021-10-21T05:06:58.997Z] | | 2% a/firefox/libssl3.so ETA: 0:00:07
[task 2021-10-21T05:08:02.718Z] | | 2% a/firefox/libxul.so ETA: 0:00:07
[task 2021-10-21T05:08:03.051Z] |# | 6% …ox/libxul.so [.gnu.hash] ETA: 0:14:52
[task 2021-10-21T05:08:06.786Z] |## | 9% …efox/libxul.so [.dynstr] ETA: 0:10:04
[task 2021-10-21T05:08:07.680Z] |### | 12% …ul.so [.elfhack.data.v0] ETA: 0:07:54
[task 2021-10-21T05:08:07.967Z] |#### | 15% …ul.so [.elfhack.text.v0] ETA: 0:06:17
[task 2021-10-21T05:08:08.252Z] |#### | 18% …irefox/libxul.so [.init] ETA: 0:05:07
[task 2021-10-21T05:08:08.765Z] |##### | 21% …firefox/libxul.so [.plt] ETA: 0:04:16
[task 2021-10-21T05:08:09.058Z] |###### | 24% …fox/libxul.so [.plt.got] ETA: 0:03:39
[task 2021-10-21T05:16:04.336Z] |####### | 27% …irefox/libxul.so [.text] ETA: 0:03:08
[task 2021-10-21T05:16:04.706Z] |######## | 30% …fox/libxul.so [text_env] ETA: 0:21:09
[task 2021-10-21T05:16:04.981Z] |######## | 32% …irefox/libxul.so [.fini] ETA: 0:18:29
[task 2021-10-21T05:16:26.924Z] |######### | 35% …efox/libxul.so [.rodata] ETA: 0:16:15
[task 2021-10-21T05:16:29.172Z] |########## | 38% …ibxul.so [.eh_frame_hdr] ETA: 0:14:55
[task 2021-10-21T05:16:40.188Z] |########### | 41% …ox/libxul.so [.eh_frame] ETA: 0:13:15
[task 2021-10-21T05:16:40.221Z] |############ | 44% …l.so [.gcc_except_table] ETA: 0:12:00
[task 2021-10-21T05:16:40.241Z] |############ | 47% …refox/libxul.so [.tdata] ETA: 0:10:40
[task 2021-10-21T05:16:40.260Z] |############# | 50% …irefox/libxul.so [.tbss] ETA: 0:09:29
[task 2021-10-21T05:16:40.280Z] |############## | 53% …/libxul.so [.init_array] ETA: 0:08:26
[task 2021-10-21T05:16:40.299Z] |############### | 56% …/libxul.so [.fini_array] ETA: 0:07:30
[task 2021-10-21T05:16:42.681Z] |################ | 59% …libxul.so [.data.rel.ro] ETA: 0:06:39
[task 2021-10-21T05:16:42.708Z] |################ | 62% …firefox/libxul.so [.got] ETA: 0:05:54
[task 2021-10-21T05:16:42.741Z] |################# | 65% …fox/libxul.so [.got.plt] ETA: 0:05:12
[task 2021-10-21T05:16:42.906Z] |################## | 68% …irefox/libxul.so [.data] ETA: 0:04:34
[task 2021-10-21T05:16:42.927Z] |################### | 70% …firefox/libxul.so [.bss] ETA: 0:03:58
[task 2021-10-21T05:16:42.947Z] |################### | 73% …fox/libxul.so [.comment] ETA: 0:03:26
[task 2021-10-21T05:16:42.965Z] |#################### | 76% …bxul.so [.gnu_debuglink] ETA: 0:02:56
[task 2021-10-21T05:17:12.539Z] |##################### | 79% …efox/libxul.so [.strtab] ETA: 0:02:28
[task 2021-10-21T05:17:12.559Z] |###################### | 82% …ox/libxul.so [.shstrtab] ETA: 0:02:08
[task 2021-10-21T05:17:12.559Z] |####################### | 85% ETA: 0:01:43
[task 2021-10-21T05:17:12.570Z] |####################### | 85% …irefox/minidump-analyzer ETA: 0:01:43
[task 2021-10-21T05:17:12.594Z] |####################### | 86% a/firefox/omni.ja ETA: 0:01:36
[task 2021-10-21T05:17:12.602Z] |########################## | 99% a/firefox/pingsender ETA: 0:00:05
[task 2021-10-21T05:17:12.609Z] |########################## | 99% a/firefox/platform.ini ETA: 0:00:02
[task 2021-10-21T05:17:12.616Z] |########################## | 99% …firefox/plugin-container ETA: 0:00:02
[task 2021-10-21T05:17:12.623Z] |########################## | 99% a/firefox/precomplete ETA: 0:00:00
[task 2021-10-21T05:17:12.624Z] |########################## | 99% a/firefox/removed-files ETA: 0:00:00
[task 2021-10-21T05:17:12.630Z] |########################## | 99% …efox/update-settings.ini ETA: 0:00:00
[task 2021-10-21T05:17:12.633Z] |########################## | 99% a/firefox/updater ETA: 0:00:00
[task 2021-10-21T05:17:12.636Z] |########################## | 99% a/firefox/updater.ini ETA: 0:00:00
[task 2021-10-21T05:17:12.636Z] |###########################| 100% ETA: 0:00:00
[task 2021-10-21T05:17:12.636Z] |###########################| 100% ETA: 0:00:00
[task 2021-10-21T05:17:12.636Z] |###########################| 100% ETA: 0:00:00
[task 2021-10-21T05:18:21.494Z] |###########################| 100% Time: 0:10:13
[task 2021-10-21T05:18:21.494Z] ++ dirname /builds/worker/bin/get_and_diffoscope
[task 2021-10-21T05:18:21.494Z] + /builds/worker/bin/report_error diff
[task 2021-10-21T05:18:22.152Z] TEST-UNEXPECTED-FAIL | firefox/libxul.so differs. See the diff.html or diff.txt artifact
[task 2021-10-21T05:18:22.154Z] + fail
[task 2021-10-21T05:18:22.154Z] + exit 1
[taskcluster 2021-10-21 05:18:22.478Z] === Task Finished ===
[taskcluster 2021-10-21 05:18:27.696Z] Unsuccessful task run with exit code: 1 completed in 721.316 seconds
Comment 2•2 years ago
|
||
Set release status flags based on info from the regressing bug 1731582
Updated•2 years ago
|
Comment hidden (Intermittent Failures Robot) |
Updated•2 years ago
|
![]() |
||
Comment 4•2 years ago
|
||
Who can look into fixing these tasks? Mike, Andi, can you coordinate this?
Assignee | ||
Comment 5•2 years ago
|
||
I'm still bisecting this. There's a possible workaround for the linux32 one, but it doesn't work for both jobs.
Assignee | ||
Updated•2 years ago
|
Comment hidden (Intermittent Failures Robot) |
Comment 7•2 years ago
|
||
There are 44 total failures in the last 7 days on diff opt
Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=356582304&repo=mozilla-central&lineNumber=178
[task 2021-11-01T10:21:01.607Z] + diffoscope --html diff.html --text diff.txt --progress --output-empty --no-default-limits --max-page-size 100000000 --max-page-diff-block-lines 10000 --exclude-directory-metadata=yes --exclude-command .--line-numbers a b
[task 2021-11-01T10:21:01.723Z] | | 0% ETA: --:--:--
[task 2021-11-01T10:21:01.734Z] | | 0% a/firefox/application.ini ETA: 0:33:43
[task 2021-11-01T10:21:01.737Z] | | 0% a/firefox/browser ETA: 0:35:24
[task 2021-11-01T10:21:01.738Z] | | 0% …ns/default/default16.png ETA: 299 days, 12:14:19
[task 2021-11-01T10:21:01.739Z] | | 0% …ns/default/default32.png ETA: 298 days, 12:54:06
[task 2021-11-01T10:21:01.740Z] | | 0% …ns/default/default48.png ETA: 288 days, 4:10:01
[task 2021-11-01T10:21:01.741Z] | | 0% …ns/default/default64.png ETA: 267 days, 6:13:02
[task 2021-11-01T10:21:01.741Z] | | 0% ETA: 243 days, 2:09:35
[task 2021-11-01T10:21:01.741Z] | | 0% ETA: 243 days, 9:53:45
[task 2021-11-01T10:21:01.741Z] | | 0% ETA: 172 days, 3:38:14
[task 2021-11-01T10:21:01.744Z] | | 0% …ashreporter-override.ini ETA: 140 days, 20:43:36
[task 2021-11-01T10:21:01.746Z] | | 0% …firefox/browser/features ETA: 130 days, 8:17:17
[task 2021-11-01T10:21:01.750Z] | | 0% …autofill@mozilla.org.xpi ETA: 5393 days, 21:23:45
[task 2021-11-01T10:21:01.752Z] | | 0% …npicture@mozilla.org.xpi ETA: 810 days, 1:43:40
[task 2021-11-01T10:21:01.754Z] | | 0% …failover@mozilla.com.xpi ETA: 758 days, 10:30:44
[task 2021-11-01T10:21:01.759Z] | | 0% …eenshots@mozilla.org.xpi ETA: 699 days, 19:18:27
[task 2021-11-01T10:21:01.761Z] | | 0% …reporter@mozilla.org.xpi ETA: 289 days, 3:39:54
[task 2021-11-01T10:21:01.766Z] | | 0% …ebcompat@mozilla.org.xpi ETA: 282 days, 6:01:25
[task 2021-11-01T10:21:01.766Z] | | 0% ETA: 197 days, 8:10:37
[task 2021-11-01T10:21:01.799Z] | | 0% a/firefox/browser/omni.ja ETA: 197 days, 16:33:42
[task 2021-11-01T10:21:01.799Z] | | 0% ETA: 1:18:04
[task 2021-11-01T10:21:01.804Z] | | 0% a/firefox/crashreporter ETA: 1:18:10
[task 2021-11-01T10:21:01.808Z] | | 0% …irefox/crashreporter.ini ETA: 0:01:42
[task 2021-11-01T10:21:01.814Z] | | 0% a/firefox/defaults ETA: 0:01:43
[task 2021-11-01T10:21:01.814Z] | | 0% ETA: 0:01:47
[task 2021-11-01T10:21:01.815Z] | | 0% ETA: 0:01:47
[task 2021-11-01T10:21:01.820Z] | | 0% …refox/dependentlibs.list ETA: 0:01:47
[task 2021-11-01T10:21:01.823Z] | | 0% a/firefox/dmd.py ETA: 0:01:51
[task 2021-11-01T10:21:01.824Z] | | 0% a/firefox/firefox ETA: 0:01:41
[task 2021-11-01T10:21:01.833Z] | | 0% a/firefox/firefox-bin ETA: 0:01:37
[task 2021-11-01T10:21:01.836Z] | | 0% a/firefox/fix_stacks.py ETA: 0:00:28
[task 2021-11-01T10:21:01.846Z] | | 0% a/firefox/fonts ETA: 0:00:28
[task 2021-11-01T10:21:01.846Z] | | 0% ETA: 0:00:30
[task 2021-11-01T10:21:01.851Z] | | 0% a/firefox/gmp-clearkey ETA: 0:00:30
[task 2021-11-01T10:21:01.851Z] | | 0% …earkey/0.1/manifest.json ETA: 0:00:31
[task 2021-11-01T10:21:01.851Z] | | 0% ETA: 0:00:31
[task 2021-11-01T10:21:01.851Z] | | 0% ETA: 0:00:31
[task 2021-11-01T10:21:01.853Z] | | 0% a/firefox/icons ETA: 0:00:31
[task 2021-11-01T10:21:01.853Z] | | 0% ETA: 0:00:31
[task 2021-11-01T10:21:01.859Z] | | 0% …irefox/libfreeblpriv3.so ETA: 0:00:31
[task 2021-11-01T10:21:01.861Z] | | 0% a/firefox/liblgpllibs.so ETA: 0:00:20
[task 2021-11-01T10:21:01.864Z] | | 0% …firefox/libmozavcodec.so ETA: 0:00:20
[task 2021-11-01T10:21:01.868Z] | | 0% a/firefox/libmozavutil.so ETA: 0:00:18
[task 2021-11-01T10:21:01.870Z] | | 1% a/firefox/libmozgtk.so ETA: 0:00:17
[task 2021-11-01T10:21:01.873Z] | | 1% …firefox/libmozsandbox.so ETA: 0:00:17
[task 2021-11-01T10:21:01.881Z] | | 1% …firefox/libmozsqlite3.so ETA: 0:00:16
[task 2021-11-01T10:21:01.882Z] | | 1% …firefox/libmozwayland.so ETA: 0:00:11
[task 2021-11-01T10:21:01.886Z] | | 1% a/firefox/libnspr4.so ETA: 0:00:11
[task 2021-11-01T10:21:01.893Z] | | 1% a/firefox/libnss3.so ETA: 0:00:11
[task 2021-11-01T10:21:01.898Z] | | 2% a/firefox/libnssckbi.so ETA: 0:00:09
[task 2021-11-01T10:21:01.902Z] | | 2% a/firefox/libnssutil3.so ETA: 0:00:09
[task 2021-11-01T10:21:01.903Z] | | 2% a/firefox/libplc4.so ETA: 0:00:08
[task 2021-11-01T10:21:01.904Z] | | 2% a/firefox/libplds4.so ETA: 0:00:08
[task 2021-11-01T10:21:01.907Z] | | 2% a/firefox/libsmime3.so ETA: 0:00:08
[task 2021-11-01T10:21:01.912Z] | | 2% a/firefox/libsoftokn3.so ETA: 0:00:08
[task 2021-11-01T10:21:01.917Z] | | 2% a/firefox/libssl3.so ETA: 0:00:08
[task 2021-11-01T10:22:15.218Z] | | 2% a/firefox/libxul.so ETA: 0:00:08
[task 2021-11-01T10:22:15.586Z] |# | 6% …ox/libxul.so [.gnu.hash] ETA: 0:17:07
[task 2021-11-01T10:22:19.863Z] |## | 9% …efox/libxul.so [.dynstr] ETA: 0:11:35
[task 2021-11-01T10:22:21.048Z] |### | 12% …ul.so [.elfhack.data.v0] ETA: 0:09:06
[task 2021-11-01T10:22:21.349Z] |#### | 15% …ul.so [.elfhack.text.v0] ETA: 0:07:14
[task 2021-11-01T10:22:21.655Z] |#### | 18% …irefox/libxul.so [.init] ETA: 0:05:54
[task 2021-11-01T10:22:22.259Z] |##### | 21% …firefox/libxul.so [.plt] ETA: 0:04:55
[task 2021-11-01T10:22:22.571Z] |###### | 24% …fox/libxul.so [.plt.got] ETA: 0:04:12
[task 2021-11-01T10:31:42.676Z] |####### | 27% …irefox/libxul.so [.text] ETA: 0:03:37
[task 2021-11-01T10:31:43.074Z] |######## | 30% …fox/libxul.so [text_env] ETA: 0:24:52
[task 2021-11-01T10:31:43.363Z] |######## | 32% …irefox/libxul.so [.fini] ETA: 0:21:43
[task 2021-11-01T10:32:08.786Z] |######### | 35% …efox/libxul.so [.rodata] ETA: 0:19:06
[task 2021-11-01T10:32:11.396Z] |########## | 38% …ibxul.so [.eh_frame_hdr] ETA: 0:17:31
[task 2021-11-01T10:32:24.355Z] |########### | 41% …ox/libxul.so [.eh_frame] ETA: 0:15:34
[task 2021-11-01T10:32:24.392Z] |############ | 44% …l.so [.gcc_except_table] ETA: 0:14:06
[task 2021-11-01T10:32:24.413Z] |############ | 47% …refox/libxul.so [.tdata] ETA: 0:12:32
[task 2021-11-01T10:32:24.434Z] |############# | 50% …irefox/libxul.so [.tbss] ETA: 0:11:09
[task 2021-11-01T10:32:24.458Z] |############## | 53% …/libxul.so [.init_array] ETA: 0:09:55
[task 2021-11-01T10:32:24.478Z] |############### | 56% …/libxul.so [.fini_array] ETA: 0:08:49
[task 2021-11-01T10:32:27.251Z] |################ | 59% …libxul.so [.data.rel.ro] ETA: 0:07:49
[task 2021-11-01T10:32:27.280Z] |################ | 62% …firefox/libxul.so [.got] ETA: 0:06:56
[task 2021-11-01T10:32:27.314Z] |################# | 65% …fox/libxul.so [.got.plt] ETA: 0:06:07
[task 2021-11-01T10:32:27.505Z] |################## | 68% …irefox/libxul.so [.data] ETA: 0:05:22
[task 2021-11-01T10:32:27.530Z] |################### | 70% …firefox/libxul.so [.bss] ETA: 0:04:40
[task 2021-11-01T10:32:27.554Z] |################### | 73% …fox/libxul.so [.comment] ETA: 0:04:02
[task 2021-11-01T10:32:27.575Z] |#################### | 76% …bxul.so [.gnu_debuglink] ETA: 0:03:27
[task 2021-11-01T10:33:01.389Z] |##################### | 79% …efox/libxul.so [.strtab] ETA: 0:02:54
[task 2021-11-01T10:33:01.412Z] |###################### | 82% …ox/libxul.so [.shstrtab] ETA: 0:02:31
[task 2021-11-01T10:33:01.412Z] |####################### | 85% ETA: 0:02:01
[task 2021-11-01T10:33:01.424Z] |####################### | 85% …irefox/minidump-analyzer ETA: 0:02:01
[task 2021-11-01T10:33:01.453Z] |####################### | 86% a/firefox/omni.ja ETA: 0:01:53
[task 2021-11-01T10:33:01.463Z] |########################## | 99% a/firefox/pingsender ETA: 0:00:06
[task 2021-11-01T10:33:01.471Z] |########################## | 99% a/firefox/platform.ini ETA: 0:00:03
[task 2021-11-01T10:33:01.479Z] |########################## | 99% …firefox/plugin-container ETA: 0:00:03
[task 2021-11-01T10:33:01.486Z] |########################## | 99% a/firefox/precomplete ETA: 0:00:00
[task 2021-11-01T10:33:01.488Z] |########################## | 99% a/firefox/removed-files ETA: 0:00:00
[task 2021-11-01T10:33:01.494Z] |########################## | 99% …efox/update-settings.ini ETA: 0:00:00
[task 2021-11-01T10:33:01.498Z] |########################## | 99% a/firefox/updater ETA: 0:00:00
[task 2021-11-01T10:33:01.501Z] |########################## | 99% a/firefox/updater.ini ETA: 0:00:00
[task 2021-11-01T10:33:01.501Z] |###########################| 100% ETA: 0:00:00
[task 2021-11-01T10:33:01.501Z] |###########################| 100% ETA: 0:00:00
[task 2021-11-01T10:33:01.501Z] |###########################| 100% ETA: 0:00:00
[task 2021-11-01T10:34:21.592Z] |###########################| 100% Time: 0:11:59
[task 2021-11-01T10:34:21.592Z] ++ dirname /builds/worker/bin/get_and_diffoscope
[task 2021-11-01T10:34:21.592Z] + /builds/worker/bin/report_error diff
[task 2021-11-01T10:34:22.370Z] TEST-UNEXPECTED-FAIL | firefox/libxul.so differs. See the diff.html or diff.txt artifact
Comment hidden (Intermittent Failures Robot) |
Assignee | ||
Comment 9•2 years ago
|
||
Finally identified not one, but two sources of regression:
Now need to determine how to fix it.
Comment hidden (Intermittent Failures Robot) |
Assignee | ||
Comment 11•2 years ago
|
||
It turns out that both issues seem to have been fixed on trunk while I was looking for reproducers.
One by https://reviews.llvm.org/rG3c47c5ca13b8a502de3272e8105548715947b7a8 and I'm still bisecting the other.
Assignee | ||
Comment 12•2 years ago
|
||
So the other one is https://reviews.llvm.org/rGbadcd585897253e94b7b2d4e6f9f430a2020d642, but that only covers the two subcases I had pinned down to the regressors in comment 9, but it turns out there are still cases that aren't fixed on trunk.
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Assignee | ||
Comment 15•2 years ago
|
||
The revert doesn't have a lot of impact, as it's only for debug info
about enums with a 128-bits integer representation.
Assignee | ||
Comment 16•2 years ago
|
||
The patch is not enough. The LLVM regressions have made it into rust in the meantime, so we'll have to wait for rust 1.57 for the rest of the fix (thankfully, it releases next week), but there will likely be some remaining non-determinism left even after that.
Assignee | ||
Updated•2 years ago
|
Comment 17•2 years ago
|
||
Pushed by mh@glandium.org: https://hg.mozilla.org/integration/autoland/rev/2e36d2d47679 Apply 2 clang upstream fixes for reproducibility and revert one. r=andi
Updated•2 years ago
|
Comment 18•2 years ago
|
||
bugherder |
Comment hidden (Intermittent Failures Robot) |
Comment 20•2 years ago
|
||
There are 41 total failures in the last 7 days on diff opt
Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=359797037&repo=mozilla-central&lineNumber=174
[task 2021-12-01T06:45:11.948Z] + diffoscope --html diff.html --text diff.txt --progress --output-empty --no-default-limits --max-page-size 100000000 --max-page-diff-block-lines 10000 --exclude-directory-metadata=yes --exclude-command .--line-numbers a b
[task 2021-12-01T06:45:12.062Z] | | 0% ETA: --:--:--
[task 2021-12-01T06:45:12.073Z] | | 0% a/firefox/application.ini ETA: 0:34:50
[task 2021-12-01T06:45:12.075Z] | | 0% a/firefox/browser ETA: 0:36:28
[task 2021-12-01T06:45:12.076Z] | | 0% …ns/default/default16.png ETA: 309 days, 6:12:19
[task 2021-12-01T06:45:12.077Z] | | 0% …ns/default/default32.png ETA: 307 days, 18:56:48
[task 2021-12-01T06:45:12.078Z] | | 0% …ns/default/default48.png ETA: 296 days, 23:02:22
[task 2021-12-01T06:45:12.079Z] | | 0% …ns/default/default64.png ETA: 275 days, 0:36:38
[task 2021-12-01T06:45:12.079Z] | | 0% ETA: 249 days, 19:38:27
[task 2021-12-01T06:45:12.079Z] | | 0% ETA: 250 days, 3:10:45
[task 2021-12-01T06:45:12.079Z] | | 0% ETA: 176 days, 21:09:00
[task 2021-12-01T06:45:12.082Z] | | 0% …ashreporter-override.ini ETA: 144 days, 16:26:19
[task 2021-12-01T06:45:12.084Z] | | 0% …firefox/browser/features ETA: 134 days, 2:44:54
[task 2021-12-01T06:45:12.088Z] | | 0% …autofill@mozilla.org.xpi ETA: 5648 days, 12:43:20
[task 2021-12-01T06:45:12.091Z] | | 0% …npicture@mozilla.org.xpi ETA: 850 days, 20:23:11
[task 2021-12-01T06:45:12.092Z] | | 0% …failover@mozilla.com.xpi ETA: 799 days, 6:19:35
[task 2021-12-01T06:45:12.098Z] | | 0% …eenshots@mozilla.org.xpi ETA: 737 days, 17:22:44
[task 2021-12-01T06:45:12.100Z] | | 0% …reporter@mozilla.org.xpi ETA: 307 days, 17:33:09
[task 2021-12-01T06:45:12.106Z] | | 0% …ebcompat@mozilla.org.xpi ETA: 301 days, 3:11:31
[task 2021-12-01T06:45:12.106Z] | | 0% ETA: 208 days, 21:34:08
[task 2021-12-01T06:45:12.140Z] | | 0% a/firefox/browser/omni.ja ETA: 209 days, 7:55:59
[task 2021-12-01T06:45:12.140Z] | | 0% ETA: 1:22:11
[task 2021-12-01T06:45:12.146Z] | | 0% a/firefox/crashreporter ETA: 1:22:19
[task 2021-12-01T06:45:12.150Z] | | 0% …irefox/crashreporter.ini ETA: 0:01:45
[task 2021-12-01T06:45:12.157Z] | | 0% a/firefox/defaults ETA: 0:01:48
[task 2021-12-01T06:45:12.157Z] | | 0% ETA: 0:01:52
[task 2021-12-01T06:45:12.157Z] | | 0% ETA: 0:01:52
[task 2021-12-01T06:45:12.162Z] | | 0% …refox/dependentlibs.list ETA: 0:01:52
[task 2021-12-01T06:45:12.165Z] | | 0% a/firefox/dmd.py ETA: 0:01:56
[task 2021-12-01T06:45:12.166Z] | | 0% a/firefox/firefox ETA: 0:01:46
[task 2021-12-01T06:45:12.177Z] | | 0% a/firefox/firefox-bin ETA: 0:01:42
[task 2021-12-01T06:45:12.180Z] | | 0% a/firefox/fix_stacks.py ETA: 0:00:28
[task 2021-12-01T06:45:12.192Z] | | 0% a/firefox/fonts ETA: 0:00:29
[task 2021-12-01T06:45:12.193Z] | | 0% ETA: 0:00:31
[task 2021-12-01T06:45:12.197Z] | | 0% a/firefox/gmp-clearkey ETA: 0:00:31
[task 2021-12-01T06:45:12.198Z] | | 0% …earkey/0.1/manifest.json ETA: 0:00:31
[task 2021-12-01T06:45:12.198Z] | | 0% ETA: 0:00:32
[task 2021-12-01T06:45:12.198Z] | | 0% ETA: 0:00:32
[task 2021-12-01T06:45:12.200Z] | | 0% a/firefox/icons ETA: 0:00:32
[task 2021-12-01T06:45:12.200Z] | | 0% ETA: 0:00:32
[task 2021-12-01T06:45:12.209Z] | | 0% …irefox/libfreeblpriv3.so ETA: 0:00:32
[task 2021-12-01T06:45:12.210Z] | | 0% a/firefox/liblgpllibs.so ETA: 0:00:20
[task 2021-12-01T06:45:12.221Z] | | 0% …firefox/libmozavcodec.so ETA: 0:00:20
[task 2021-12-01T06:45:12.227Z] | | 1% a/firefox/libmozavutil.so ETA: 0:00:10
[task 2021-12-01T06:45:12.228Z] | | 2% a/firefox/libmozgtk.so ETA: 0:00:10
[task 2021-12-01T06:45:12.232Z] | | 2% …firefox/libmozsandbox.so ETA: 0:00:10
[task 2021-12-01T06:45:12.242Z] | | 2% …firefox/libmozsqlite3.so ETA: 0:00:09
[task 2021-12-01T06:45:12.243Z] | | 2% …firefox/libmozwayland.so ETA: 0:00:08
[task 2021-12-01T06:45:12.248Z] | | 2% a/firefox/libnspr4.so ETA: 0:00:08
[task 2021-12-01T06:45:12.257Z] | | 2% a/firefox/libnss3.so ETA: 0:00:08
[task 2021-12-01T06:45:12.263Z] | | 3% a/firefox/libnssckbi.so ETA: 0:00:07
[task 2021-12-01T06:45:12.267Z] | | 3% a/firefox/libnssutil3.so ETA: 0:00:07
[task 2021-12-01T06:45:12.269Z] | | 3% a/firefox/libplc4.so ETA: 0:00:07
[task 2021-12-01T06:45:12.269Z] | | 3% a/firefox/libplds4.so ETA: 0:00:07
[task 2021-12-01T06:45:12.273Z] | | 3% a/firefox/libsmime3.so ETA: 0:00:07
[task 2021-12-01T06:45:12.279Z] | | 3% a/firefox/libsoftokn3.so ETA: 0:00:07
[task 2021-12-01T06:45:12.285Z] | | 3% a/firefox/libssl3.so ETA: 0:00:06
[task 2021-12-01T06:46:21.260Z] |# | 3% a/firefox/libxul.so ETA: 0:00:06
[task 2021-12-01T06:46:21.588Z] |# | 6% …ox/libxul.so [.gnu.hash] ETA: 0:16:00
[task 2021-12-01T06:46:25.843Z] |## | 9% …efox/libxul.so [.dynstr] ETA: 0:10:50
[task 2021-12-01T06:46:26.980Z] |### | 12% …ul.so [.elfhack.data.v0] ETA: 0:08:31
[task 2021-12-01T06:46:27.324Z] |#### | 15% …ul.so [.elfhack.text.v0] ETA: 0:06:47
[task 2021-12-01T06:46:27.671Z] |#### | 18% …irefox/libxul.so [.init] ETA: 0:05:32
[task 2021-12-01T06:46:28.256Z] |##### | 21% …firefox/libxul.so [.plt] ETA: 0:04:37
[task 2021-12-01T06:46:28.634Z] |###### | 24% …fox/libxul.so [.plt.got] ETA: 0:03:56
[task 2021-12-01T06:53:23.401Z] |####### | 27% …irefox/libxul.so [.text] ETA: 0:03:24
[task 2021-12-01T06:53:23.832Z] |######## | 30% …fox/libxul.so [text_env] ETA: 0:18:53
[task 2021-12-01T06:53:24.138Z] |######## | 33% …irefox/libxul.so [.fini] ETA: 0:16:30
[task 2021-12-01T06:53:47.680Z] |######### | 36% …efox/libxul.so [.rodata] ETA: 0:14:30
[task 2021-12-01T06:53:50.566Z] |########## | 39% …ibxul.so [.eh_frame_hdr] ETA: 0:13:24
[task 2021-12-01T06:54:02.631Z] |########### | 41% …ox/libxul.so [.eh_frame] ETA: 0:11:56
[task 2021-12-01T06:54:02.679Z] |############ | 44% …l.so [.gcc_except_table] ETA: 0:10:50
[task 2021-12-01T06:54:02.709Z] |############ | 47% …refox/libxul.so [.tdata] ETA: 0:09:38
[task 2021-12-01T06:54:02.737Z] |############# | 50% …irefox/libxul.so [.tbss] ETA: 0:08:33
[task 2021-12-01T06:54:02.767Z] |############## | 53% …/libxul.so [.init_array] ETA: 0:07:36
[task 2021-12-01T06:54:02.795Z] |############### | 56% …/libxul.so [.fini_array] ETA: 0:06:45
[task 2021-12-01T06:54:07.286Z] |################ | 59% …libxul.so [.data.rel.ro] ETA: 0:05:59
[task 2021-12-01T06:54:07.408Z] |################ | 62% …firefox/libxul.so [.got] ETA: 0:05:20
[task 2021-12-01T06:54:07.462Z] |################# | 65% …fox/libxul.so [.got.plt] ETA: 0:04:42
[task 2021-12-01T06:54:07.744Z] |################## | 68% …irefox/libxul.so [.data] ETA: 0:04:06
[task 2021-12-01T06:54:07.775Z] |################### | 71% …firefox/libxul.so [.bss] ETA: 0:03:34
[task 2021-12-01T06:54:07.807Z] |#################### | 74% …fox/libxul.so [.comment] ETA: 0:03:05
[task 2021-12-01T06:54:07.839Z] |#################### | 77% …bxul.so [.gnu_debuglink] ETA: 0:02:37
[task 2021-12-01T06:54:39.228Z] |##################### | 80% …efox/libxul.so [.strtab] ETA: 0:02:12
[task 2021-12-01T06:54:39.263Z] |###################### | 83% …ox/libxul.so [.shstrtab] ETA: 0:01:55
[task 2021-12-01T06:54:39.263Z] |####################### | 86% ETA: 0:01:31
[task 2021-12-01T06:54:39.277Z] |####################### | 86% …irefox/minidump-analyzer ETA: 0:01:31
[task 2021-12-01T06:54:39.307Z] |####################### | 86% a/firefox/omni.ja ETA: 0:01:25
[task 2021-12-01T06:54:39.320Z] |########################## | 99% a/firefox/pingsender ETA: 0:00:05
[task 2021-12-01T06:54:39.328Z] |########################## | 99% a/firefox/platform.ini ETA: 0:00:02
[task 2021-12-01T06:54:39.340Z] |########################## | 99% …firefox/plugin-container ETA: 0:00:02
[task 2021-12-01T06:54:39.347Z] |########################## | 99% a/firefox/precomplete ETA: 0:00:00
[task 2021-12-01T06:54:39.349Z] |########################## | 99% a/firefox/removed-files ETA: 0:00:00
[task 2021-12-01T06:54:39.355Z] |########################## | 99% …efox/update-settings.ini ETA: 0:00:00
[task 2021-12-01T06:54:39.363Z] |########################## | 99% a/firefox/updater ETA: 0:00:00
[task 2021-12-01T06:54:39.366Z] |########################## | 99% a/firefox/updater.ini ETA: 0:00:00
[task 2021-12-01T06:54:39.366Z] |###########################| 100% ETA: 0:00:00
[task 2021-12-01T06:54:39.366Z] |###########################| 100% ETA: 0:00:00
[task 2021-12-01T06:54:39.366Z] |###########################| 100% ETA: 0:00:00
[task 2021-12-01T06:54:45.557Z] |###########################| 100% Time: 0:09:27
[task 2021-12-01T06:54:45.557Z] ++ dirname /builds/worker/bin/get_and_diffoscope
[task 2021-12-01T06:54:45.558Z] + /builds/worker/bin/report_error diff
[task 2021-12-01T06:54:45.834Z] TEST-UNEXPECTED-FAIL | firefox/libxul.so differs. See the diff.html or diff.txt artifact
[task 2021-12-01T06:54:45.836Z] + fail
[task 2021-12-01T06:54:45.836Z] + exit 1
[taskcluster 2021-12-01 06:54:46.181Z] === Task Finished ===
[taskcluster 2021-12-01 06:54:48.426Z] Unsuccessful task run with exit code: 1 completed in 601.376 seconds
Assignee | ||
Comment 21•2 years ago
|
||
(In reply to Mike Hommey [:glandium] from comment #16)
The patch is not enough. The LLVM regressions have made it into rust in the meantime, so we'll have to wait for rust 1.57 for the rest of the fix (thankfully, it releases next week), but there will likely be some remaining non-determinism left even after that.
Forgot to add this comment: it turns out the remaining non-determinismt goes away when PGO is enabled, so this will go away after bug 1742749 lands after rustc 1.57.0 is released officially later this week.
Assignee | ||
Comment 22•2 years ago
|
||
Confirmed fixed: https://treeherder.mozilla.org/jobs?repo=mozilla-central&searchStr=diff%2Clinux&revision=356c521826254fbae435b212923b20801d171950
Updated•2 years ago
|
Comment hidden (Intermittent Failures Robot) |
Updated•2 years ago
|
Description
•