Intermittent TEST-UNEXPECTED-FAIL | firefox/libxul.so differs. See the diff.html or diff.txt artifact
Categories
(Firefox Build System :: Toolchains, defect, P5)
Tracking
(firefox-esr68 unaffected, firefox-esr78 unaffected, firefox78 unaffected, firefox79 unaffected, firefox80 affected)
Tracking | Status | |
---|---|---|
firefox-esr68 | --- | unaffected |
firefox-esr78 | --- | unaffected |
firefox78 | --- | unaffected |
firefox79 | --- | unaffected |
firefox80 | --- | affected |
People
(Reporter: intermittent-bug-filer, Unassigned)
References
(Regression)
Details
(Keywords: intermittent-failure, regression)
Filed by: btara [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=309960181&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/e5envhNSS_OUZsMwRP7snw/runs/0/artifacts/public/logs/live_backing.log
[task 2020-07-16T05:24:14.321Z] + curl -L https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/OYvbUu0cRLWB6q-YtTJWug/artifacts/public/build/target.tar.bz2
[task 2020-07-16T05:24:14.321Z] + tar -C b -jxf -
[task 2020-07-16T05:24:14.326Z] % Total % Received % Xferd Average Speed Time Time Time Current
[task 2020-07-16T05:24:14.326Z] Dload Upload Total Spent Left Speed
[task 2020-07-16T05:24:14.326Z]
[task 2020-07-16T05:24:14.398Z] 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
[task 2020-07-16T05:24:14.398Z] 100 122 100 122 0 0 1718 0 --:--:-- --:--:-- --:--:-- 1718
[task 2020-07-16T05:24:14.599Z]
[task 2020-07-16T05:24:15.466Z] 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
[task 2020-07-16T05:24:16.477Z] 5 75.6M 5 4116k 0 0 3614k 0 0:00:21 0:00:01 0:00:20 4742k
[task 2020-07-16T05:24:17.460Z] 16 75.6M 16 12.3M 0 0 5862k 0 0:00:13 0:00:02 0:00:11 6711k
[task 2020-07-16T05:24:18.466Z] 28 75.6M 28 21.4M 0 0 7000k 0 0:00:11 0:00:03 0:00:08 7666k
[task 2020-07-16T05:24:19.439Z] 40 75.6M 40 30.5M 0 0 7558k 0 0:00:10 0:00:04 0:00:06 8087k
[task 2020-07-16T05:24:20.410Z] 52 75.6M 52 39.7M 0 0 7953k 0 0:00:09 0:00:05 0:00:04 8400k
[task 2020-07-16T05:24:21.475Z] 64 75.6M 64 48.7M 0 0 8200k 0 0:00:09 0:00:06 0:00:03 9256k
[task 2020-07-16T05:24:22.432Z] 74 75.6M 74 56.4M 0 0 8083k 0 0:00:09 0:00:07 0:00:02 9037k
[task 2020-07-16T05:24:23.445Z] 85 75.6M 85 64.8M 0 0 8187k 0 0:00:09 0:00:08 0:00:01 8933k
[task 2020-07-16T05:24:23.689Z] 96 75.6M 96 73.2M 0 0 8225k 0 0:00:09 0:00:09 --:--:-- 8779k
[task 2020-07-16T05:24:23.689Z] 100 75.6M 100 75.6M 0 0 8278k 0 0:00:09 0:00:09 --:--:-- 8671k
[task 2020-07-16T05:24:23.703Z] + case "$ORIG_URL" in
[task 2020-07-16T05:24:23.703Z] + OMNIJAR=omni.ja
[task 2020-07-16T05:24:23.703Z] + POST=true
[task 2020-07-16T05:24:23.703Z] + for option in "$@"
[task 2020-07-16T05:24:23.703Z] + case "$option" in
[task 2020-07-16T05:24:23.703Z] + POST=fail
[task 2020-07-16T05:24:23.703Z] + '[' -n '' ']'
[task 2020-07-16T05:24:23.704Z] + diffoscope --html diff.html --text diff.txt --progress --output-empty --no-default-limits --max-page-size 100000000 --max-page-diff-block-lines 10000 --exclude-directory-metadata --exclude-command .--line-numbers a b
[task 2020-07-16T05:24:23.773Z] | | 0% ETA: --:--:--
[task 2020-07-16T05:24:23.773Z] 2020-07-16 05:24:23 W: diffoscope.main: Fuzzy-matching is currently disabled as the "tlsh" module is unavailable.
[task 2020-07-16T05:24:23.820Z] | | 0% ETA: --:--:--
[task 2020-07-16T05:24:23.820Z] | | 0% application.ini ETA: 0:32:14
[task 2020-07-16T05:24:23.832Z] | | 0% browser ETA: 0:27:17
[task 2020-07-16T05:24:23.832Z] | | 0% default16.png ETA: 267 days, 10:50:56
[task 2020-07-16T05:24:23.832Z] | | 0% default32.png ETA: 262 days, 8:11:03
[task 2020-07-16T05:24:23.832Z] | | 0% default48.png ETA: 249 days, 8:12:13
[task 2020-07-16T05:24:23.832Z] | | 0% default64.png ETA: 227 days, 20:58:51
[task 2020-07-16T05:24:23.832Z] | | 0% ETA: 204 days, 3:44:16
[task 2020-07-16T05:24:23.832Z] | | 0% ETA: 204 days, 10:47:08
[task 2020-07-16T05:24:23.832Z] | | 0% ETA: 144 days, 14:46:24
[task 2020-07-16T05:24:23.833Z] | | 0% …ashreporter-override.ini ETA: 118 days, 7:44:07
[task 2020-07-16T05:24:23.835Z] | | 0% features ETA: 104 days, 11:47:34
[task 2020-07-16T05:24:23.837Z] | | 0% …autofill@mozilla.org.xpi ETA: 6175 days, 5:05:24
[task 2020-07-16T05:24:23.839Z] | | 0% …eenshots@mozilla.org.xpi ETA: 217 days, 17:00:23
[task 2020-07-16T05:24:23.839Z] | | 0% …reporter@mozilla.org.xpi ETA: 149 days, 4:07:24
[task 2020-07-16T05:24:23.841Z] | | 0% webcompat@mozilla.org.xpi ETA: 145 days, 12:50:09
[task 2020-07-16T05:24:23.841Z] | | 0% ETA: 134 days, 9:19:37
[task 2020-07-16T05:24:23.864Z] | | 0% omni.ja ETA: 134 days, 15:17:55
[task 2020-07-16T05:24:23.864Z] | | 0% ETA: 0:52:58
[task 2020-07-16T05:24:23.866Z] | | 0% crashreporter ETA: 0:53:02
[task 2020-07-16T05:24:23.866Z] | | 0% crashreporter.ini ETA: 0:01:08
[task 2020-07-16T05:24:23.874Z] | | 0% defaults ETA: 0:01:07
[task 2020-07-16T05:24:23.874Z] | | 0% ETA: 0:01:12
[task 2020-07-16T05:24:23.874Z] | | 0% ETA: 0:01:12
[task 2020-07-16T05:24:23.874Z] | | 0% dependentlibs.list ETA: 0:01:12
[task 2020-07-16T05:24:23.874Z] | | 0% dmd.py ETA: 0:01:12
[task 2020-07-16T05:24:23.874Z] | | 0% firefox ETA: 0:01:04
[task 2020-07-16T05:24:23.876Z] | | 0% firefox-bin ETA: 0:01:01
[task 2020-07-16T05:24:23.876Z] | | 0% fix_stacks.py ETA: 0:00:18
[task 2020-07-16T05:24:23.882Z] | | 0% fonts ETA: 0:00:18
[task 2020-07-16T05:24:23.882Z] | | 0% ETA: 0:00:18
[task 2020-07-16T05:24:23.889Z] | | 0% gmp-clearkey ETA: 0:00:19
[task 2020-07-16T05:24:23.889Z] | | 0% manifest.json ETA: 0:00:20
[task 2020-07-16T05:24:23.889Z] | | 0% ETA: 0:00:20
[task 2020-07-16T05:24:23.889Z] | | 0% ETA: 0:00:20
[task 2020-07-16T05:24:23.892Z] | | 0% gtk2 ETA: 0:00:20
[task 2020-07-16T05:24:23.892Z] | | 0% ETA: 0:00:20
[task 2020-07-16T05:24:23.894Z] | | 0% icons ETA: 0:00:20
[task 2020-07-16T05:24:23.894Z] | | 0% ETA: 0:00:20
[task 2020-07-16T05:24:23.896Z] | | 0% libfreeblpriv3.so ETA: 0:00:20
[task 2020-07-16T05:24:23.896Z] | | 0% liblgpllibs.so ETA: 0:00:14
[task 2020-07-16T05:24:23.897Z] | | 0% libmozavcodec.so ETA: 0:00:14
[task 2020-07-16T05:24:23.899Z] | | 0% libmozavutil.so ETA: 0:00:12
[task 2020-07-16T05:24:23.899Z] | | 1% libmozgtk.so ETA: 0:00:11
[task 2020-07-16T05:24:23.901Z] | | 1% libmozsandbox.so ETA: 0:00:11
[task 2020-07-16T05:24:23.902Z] | | 1% libmozsqlite3.so ETA: 0:00:10
[task 2020-07-16T05:24:23.903Z] | | 1% libmozwayland.so ETA: 0:00:07
[task 2020-07-16T05:24:23.904Z] | | 1% libnspr4.so ETA: 0:00:07
[task 2020-07-16T05:24:23.906Z] | | 1% libnss3.so ETA: 0:00:06
[task 2020-07-16T05:24:23.907Z] | | 2% libnssckbi.so ETA: 0:00:05
[task 2020-07-16T05:24:23.909Z] | | 2% libnssutil3.so ETA: 0:00:05
[task 2020-07-16T05:24:23.909Z] | | 2% libplc4.so ETA: 0:00:05
[task 2020-07-16T05:24:23.909Z] | | 2% libplds4.so ETA: 0:00:05
[task 2020-07-16T05:24:23.911Z] | | 2% libsmime3.so ETA: 0:00:05
[task 2020-07-16T05:24:23.912Z] | | 2% libsoftokn3.so ETA: 0:00:04
[task 2020-07-16T05:24:23.914Z] | | 2% libssl3.so ETA: 0:00:04
[task 2020-07-16T05:24:58.714Z] | | 3% libxul.so ETA: 0:00:04
[task 2020-07-16T05:24:58.735Z] |# | 6% …ox/libxul.so [.gnu.hash] ETA: 0:08:22
[task 2020-07-16T05:24:58.875Z] |## | 9% …efox/libxul.so [.dynstr] ETA: 0:05:40
[task 2020-07-16T05:24:59.526Z] |### | 12% …ul.so [.elfhack.data.v0] ETA: 0:04:14
[task 2020-07-16T05:24:59.762Z] |#### | 14% …ul.so [.elfhack.text.v0] ETA: 0:03:23
[task 2020-07-16T05:24:59.993Z] |#### | 17% …irefox/libxul.so [.init] ETA: 0:02:46
[task 2020-07-16T05:25:00.409Z] |##### | 20% …firefox/libxul.so [.plt] ETA: 0:02:19
[task 2020-07-16T05:25:00.647Z] |###### | 23% …fox/libxul.so [.plt.got] ETA: 0:02:00
[task 2020-07-16T05:32:15.561Z] |####### | 26% …irefox/libxul.so [.text] ETA: 0:01:43
[task 2020-07-16T05:32:15.863Z] |####### | 28% …fox/libxul.so [text_env] ETA: 0:19:15
[task 2020-07-16T05:32:16.096Z] |######## | 31% …irefox/libxul.so [.fini] ETA: 0:16:51
[task 2020-07-16T05:32:34.367Z] |######### | 34% …efox/libxul.so [.rodata] ETA: 0:14:51
[task 2020-07-16T05:32:36.138Z] |########## | 37% …ibxul.so [.eh_frame_hdr] ETA: 0:13:39
[task 2020-07-16T05:32:44.825Z] |########## | 40% …ox/libxul.so [.eh_frame] ETA: 0:12:11
[task 2020-07-16T05:32:44.848Z] |########### | 43% …l.so [.gcc_except_table] ETA: 0:11:02
[task 2020-07-16T05:32:44.870Z] |############ | 45% …refox/libxul.so [.tdata] ETA: 0:09:51
[task 2020-07-16T05:32:44.889Z] |############# | 48% …irefox/libxul.so [.tbss] ETA: 0:08:48
[task 2020-07-16T05:32:44.909Z] |############# | 51% …/libxul.so [.init_array] ETA: 0:07:52
[task 2020-07-16T05:32:44.928Z] |############## | 54% …/libxul.so [.fini_array] ETA: 0:07:01
[task 2020-07-16T05:32:46.888Z] |############### | 57% …libxul.so [.data.rel.ro] ETA: 0:06:16
[task 2020-07-16T05:32:46.913Z] |################ | 59% …firefox/libxul.so [.got] ETA: 0:05:36
[task 2020-07-16T05:32:46.944Z] |################ | 62% …fox/libxul.so [.got.plt] ETA: 0:04:58
[task 2020-07-16T05:32:47.067Z] |################# | 65% …irefox/libxul.so [.data] ETA: 0:04:24
[task 2020-07-16T05:32:47.088Z] |################## | 68% …firefox/libxul.so [.bss] ETA: 0:03:52
[task 2020-07-16T05:32:47.108Z] |################### | 71% …fox/libxul.so [.comment] ETA: 0:03:23
[task 2020-07-16T05:32:47.126Z] |################### | 73% ….__warn_memset_zero_len] ETA: 0:02:56
[task 2020-07-16T05:32:47.149Z] |#################### | 76% …bxul.so [.gnu_debuglink] ETA: 0:02:32
[task 2020-07-16T05:33:07.759Z] |##################### | 79% …efox/libxul.so [.strtab] ETA: 0:02:08
[task 2020-07-16T05:33:07.785Z] |###################### | 82% …ox/libxul.so [.shstrtab] ETA: 0:01:51
[task 2020-07-16T05:33:07.785Z] |####################### | 85% ETA: 0:01:30
[task 2020-07-16T05:33:07.791Z] |####################### | 85% minidump-analyzer ETA: 0:01:30
[task 2020-07-16T05:33:07.809Z] |####################### | 86% omni.ja ETA: 0:01:25
[task 2020-07-16T05:33:07.813Z] |########################## | 99% pingsender ETA: 0:00:04
[task 2020-07-16T05:33:07.814Z] |########################## | 99% platform.ini ETA: 0:00:02
[task 2020-07-16T05:33:07.818Z] |########################## | 99% plugin-container ETA: 0:00:02
[task 2020-07-16T05:33:07.818Z] |########################## | 99% precomplete ETA: 0:00:00
[task 2020-07-16T05:33:07.818Z] |########################## | 99% removed-files ETA: 0:00:00
[task 2020-07-16T05:33:07.819Z] |########################## | 99% update-settings.ini ETA: 0:00:00
[task 2020-07-16T05:33:07.823Z] |########################## | 99% updater ETA: 0:00:00
[task 2020-07-16T05:33:07.823Z] |########################## | 99% updater.ini ETA: 0:00:00
[task 2020-07-16T05:33:07.823Z] |###########################| 100% ETA: 0:00:00
[task 2020-07-16T05:33:07.823Z] |###########################| 100% ETA: 0:00:00
[task 2020-07-16T05:33:07.823Z] |###########################| 100% ETA: 0:00:00
[task 2020-07-16T05:34:01.768Z] |###########################| 100% Time: 0:08:44
[task 2020-07-16T05:34:01.768Z] + report_error diff
[task 2020-07-16T05:34:01.768Z] + python3
[task 2020-07-16T05:34:10.964Z] TEST-UNEXPECTED-FAIL | firefox/libxul.so differs. See the diff.html or diff.txt artifact
[task 2020-07-16T05:34:10.966Z] + fail
[task 2020-07-16T05:34:10.966Z] + exit 1
[taskcluster 2020-07-16 05:34:11.264Z] === Task Finished ===
[taskcluster 2020-07-16 05:36:29.548Z] Unsuccessful task run with exit code: 1 completed in 763.536 seconds
Comment 1•4 years ago
|
||
The difference, as per diffing the crashreporter symbols, starts with:
-FUNC 3923150 1016 0 std::sys_common::backtrace::__rust_begin_short_backtrace
+FUNC 3923150 100a 0 std::sys_common::backtrace::__rust_begin_short_backtrace
Comment 2•4 years ago
|
||
And the first significant difference in that function is:
│ │ │ - 3923789: 8b b5 5c ff ff ff mov -0xa4(%ebp),%esi
│ │ │ + 3923789: 8b b5 74 ff ff ff mov -0x8c(%ebp),%esi
Most differences in the function are related to %ebp-relative addresses being different, then different registers are picked, then things escalate.
Updated•4 years ago
|
Comment 4•4 years ago
•
|
||
(In reply to Mike Hommey [:glandium] from comment #3)
It looks like it started with bug 1650984.
That failure looks like it goes back well before I landed my patches. I guess I am not understanding how my patches could ever cause this, or why does it look like the failure predates my patches landing?
Comment 5•4 years ago
|
||
I was able to reproduce by only relinking (with xLTO) libxul. That is, with no changes to any .o/.a.
(In reply to Lee Salzman [:lsalzman] from comment #4)
(In reply to Mike Hommey [:glandium] from comment #3)
It looks like it started with bug 1650984.
That failure looks like it goes back well before I landed my patches. I guess I am not understanding how my patches could ever cause this, or why does it look like the failure predates my patches landing?
The ones before are different if you look at the actual diff.
Comment 6•4 years ago
|
||
Set release status flags based on info from the regressing bug 1650984
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment 26•3 years ago
|
||
This started to perma fail again. First push with failure: https://treeherder.mozilla.org/jobs?repo=mozilla-central&revision=e090b0e10f5de9c444d145caeff7b166fbc17736&selectedTaskRun=caJXSglcTcC1uP2rtS6Fcw.0
Comment 27•3 years ago
|
||
(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #26)
This started to perma fail again. First push with failure: https://treeherder.mozilla.org/jobs?repo=mozilla-central&revision=e090b0e10f5de9c444d145caeff7b166fbc17736&selectedTaskRun=caJXSglcTcC1uP2rtS6Fcw.0
The test is green on https://treeherder.mozilla.org/jobs?repo=mozilla-central&revision=1a164819adefcdb42bdba8a24976b62ee8d03c80 and none of the seven pushes from e090b0e10f5de9c444d145caeff7b166fbc17736 have been backed out, so the appearance of perma may have just been bad luck.
Updated•3 years ago
|
Comment 28•3 years ago
|
||
By the way, as a bit of good news for everyone who has had headaches from this bug, I've been seeing very high success rate on this task using clang 12 trunk, so the situation here should improve dramatically at the next release. Unfortunately that is still months away, but at least there is hope on the horizon.
Comment 29•3 years ago
|
||
(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #26)
This started to perma fail again. First push with failure: https://treeherder.mozilla.org/jobs?repo=mozilla-central&revision=e090b0e10f5de9c444d145caeff7b166fbc17736&selectedTaskRun=caJXSglcTcC1uP2rtS6Fcw.0
A retrigger will always perma-fail, since the job is just comparing the artifacts from two other builds that don't change. You'd need to retrigger the repr build + downstream.
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
Comment 37•3 years ago
|
||
https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
For more information, please visit auto_nag documentation.
Comment 38•3 years ago
|
||
Comment 39•3 years ago
|
||
Filed Bug 1736970 instead so closing this one.
Comment 40•3 years ago
|
||
More specifically, this one apparently went away with the upgrade to clang 12, but we now have a new one with clang 13.
Description
•