Closed Bug 1638607 Opened 2 years ago Closed 1 year ago

Intermittent Tier 2 Diffscope TEST-UNEXPECTED-FAIL | firefox/libxul.so differs. See the diff.html or diff.txt artifact

Categories

(Firefox Build System :: Toolchains, defect, P5)

defect

Tracking

(firefox87 fixed)

RESOLVED FIXED
87 Branch
Tracking Status
firefox87 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: away)

References

Details

(Keywords: intermittent-failure, Whiteboard: [stockwell unknown])

Attachments

(1 file)

Filed by: btara [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=302611386&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/FFtho2nIRaeTdr2cHJLnvg/runs/1/artifacts/public/logs/live_backing.log


...
[task 2020-05-16T23:47:36.369Z]  |                           |    3%                  libxul.so  ETA:  0:00:04 
[task 2020-05-16T23:47:36.393Z]  |#                          |    6%  …ox/libxul.so [.gnu.hash]  ETA:  0:06:41 
[task 2020-05-16T23:47:36.492Z]  |##                         |    9%  …efox/libxul.so [.dynstr]  ETA:  0:04:33 
[task 2020-05-16T23:47:37.344Z]  |###                        |   12%  …ul.so [.elfhack.data.v0]  ETA:  0:03:24 
[task 2020-05-16T23:47:37.574Z]  |####                       |   15%  …ul.so [.elfhack.text.v0]  ETA:  0:02:45 
[task 2020-05-16T23:47:37.811Z]  |#####                      |   18%  …irefox/libxul.so [.init]  ETA:  0:02:15 
[task 2020-05-16T23:47:38.233Z]  |#####                      |   21%  …firefox/libxul.so [.plt]  ETA:  0:01:53 
[task 2020-05-16T23:47:38.467Z]  |######                     |   24%  …fox/libxul.so [.plt.got]  ETA:  0:01:37 
[task 2020-05-16T23:54:04.509Z]  |#######                    |   27%  …irefox/libxul.so [.text]  ETA:  0:01:24 
[task 2020-05-16T23:54:04.822Z]  |########                   |   30%  …fox/libxul.so [text_env]  ETA:  0:15:59 
[task 2020-05-16T23:54:05.053Z]  |########                   |   33%  …irefox/libxul.so [.fini]  ETA:  0:13:58 
[task 2020-05-16T23:54:20.792Z]  |#########                  |   36%  …efox/libxul.so [.rodata]  ETA:  0:12:17 
[task 2020-05-16T23:54:22.173Z]  |##########                 |   39%  …ibxul.so [.eh_frame_hdr]  ETA:  0:11:14 
[task 2020-05-16T23:54:29.248Z]  |###########                |   42%  …ox/libxul.so [.eh_frame]  ETA:  0:09:59 
[task 2020-05-16T23:54:29.270Z]  |############               |   45%  …l.so [.gcc_except_table]  ETA:  0:09:00 
[task 2020-05-16T23:54:29.287Z]  |############               |   47%  …refox/libxul.so [.tdata]  ETA:  0:08:00 
[task 2020-05-16T23:54:29.302Z]  |#############              |   50%  …irefox/libxul.so [.tbss]  ETA:  0:07:07 
[task 2020-05-16T23:54:29.318Z]  |##############             |   53%  …/libxul.so [.init_array]  ETA:  0:06:20 
[task 2020-05-16T23:54:29.338Z]  |###############            |   56%  …/libxul.so [.fini_array]  ETA:  0:05:37 
[task 2020-05-16T23:54:31.246Z]  |################           |   59%  …libxul.so [.data.rel.ro]  ETA:  0:04:59 
[task 2020-05-16T23:54:31.266Z]  |################           |   62%  …firefox/libxul.so [.got]  ETA:  0:04:25 
[task 2020-05-16T23:54:31.294Z]  |#################          |   65%  …fox/libxul.so [.got.plt]  ETA:  0:03:53 
[task 2020-05-16T23:54:31.409Z]  |##################         |   68%  …irefox/libxul.so [.data]  ETA:  0:03:24 
[task 2020-05-16T23:54:31.426Z]  |###################        |   71%  …firefox/libxul.so [.bss]  ETA:  0:02:58 
[task 2020-05-16T23:54:31.441Z]  |####################       |   74%  …fox/libxul.so [.comment]  ETA:  0:02:33 
[task 2020-05-16T23:54:31.461Z]  |####################       |   77%  …bxul.so [.gnu_debuglink]  ETA:  0:02:10 
[task 2020-05-16T23:54:47.304Z]  |#####################      |   80%  …efox/libxul.so [.strtab]  ETA:  0:01:49 
[task 2020-05-16T23:54:47.326Z]  |######################     |   83%  …ox/libxul.so [.shstrtab]  ETA:  0:01:33 
[task 2020-05-16T23:54:47.326Z]  |#######################    |   86%                             ETA:  0:01:14 
[task 2020-05-16T23:54:47.331Z]  |#######################    |   86%          minidump-analyzer  ETA:  0:01:14 
[task 2020-05-16T23:54:47.346Z]  |#######################    |   86%                    omni.ja  ETA:  0:01:09 
[task 2020-05-16T23:54:47.350Z]  |########################## |   99%                 pingsender  ETA:  0:00:04 
[task 2020-05-16T23:54:47.350Z]  |########################## |   99%               platform.ini  ETA:  0:00:02 
[task 2020-05-16T23:54:47.354Z]  |########################## |   99%           plugin-container  ETA:  0:00:02 
[task 2020-05-16T23:54:47.354Z]  |########################## |   99%                precomplete  ETA:  0:00:00 
[task 2020-05-16T23:54:47.354Z]  |########################## |   99%              removed-files  ETA:  0:00:00 
[task 2020-05-16T23:54:47.354Z]  |########################## |   99%        update-settings.ini  ETA:  0:00:00 
[task 2020-05-16T23:54:47.358Z]  |########################## |   99%                    updater  ETA:  0:00:00 
[task 2020-05-16T23:54:47.358Z]  |########################## |   99%                updater.ini  ETA:  0:00:00 
[task 2020-05-16T23:54:47.358Z]  |###########################|  100%                             ETA:  0:00:00 
[task 2020-05-16T23:54:47.358Z]  |###########################|  100%                             ETA:  0:00:00 
[task 2020-05-16T23:54:47.358Z]  |###########################|  100%                             ETA:  0:00:00 
[task 2020-05-16T23:54:47.395Z]  |###########################|  100%                             Time: 0:07:40 
[task 2020-05-16T23:54:47.395Z] + report_error diff
[task 2020-05-16T23:54:47.395Z] + python3
[task 2020-05-16T23:54:47.404Z] TEST-UNEXPECTED-FAIL | firefox/libxul.so differs. See the diff.html or diff.txt artifact
[task 2020-05-16T23:54:47.406Z] + fail
[task 2020-05-16T23:54:47.406Z] + exit 1
[taskcluster 2020-05-16 23:54:47.741Z] === Task Finished ===
[taskcluster 2020-05-16 23:54:48.134Z] Unsuccessful task run with exit code: 1 completed in 494.441 seconds
Flags: needinfo?(mh+mozilla)

The relevant diff is:

  2c62113:	8b 74 11 1c          	mov    0x1c(%ecx,%edx,1),%esi
  2c62117:	89 44 11 1c          	mov    %eax,0x1c(%ecx,%edx,1)
- 2c6211b:	8b 85 c0 fe ff ff    	mov    -0x140(%ebp),%eax
- 2c62121:	89 44 11 28          	mov    %eax,0x28(%ecx,%edx,1)
- 2c62125:	f3 0f 7e 85 b8 fe ff 	movq   -0x148(%ebp),%xmm0
- 2c6212c:	ff 
- 2c6212d:	66 0f d6 44 11 20    	movq   %xmm0,0x20(%ecx,%edx,1)
+ 2c6211b:	f3 0f 7e 85 b8 fe ff 	movq   -0x148(%ebp),%xmm0
+ 2c62122:	ff 
+ 2c62123:	66 0f d6 44 11 20    	movq   %xmm0,0x20(%ecx,%edx,1)
+ 2c62129:	8b 85 c0 fe ff ff    	mov    -0x140(%ebp),%eax
+ 2c6212f:	89 44 11 28          	mov    %eax,0x28(%ecx,%edx,1)
  2c62133:	74 0e                	je     2c62143 <<fxa_client::http_client::Client as fxa_client::http_client::FxAClient>::scoped_key_data+0x1773>
  2c62135:	8b 45 bc             	mov    -0x44(%ebp),%eax

and it happens close to half the time, which is a lot.

Cross referencing with the crash symbols, this comes from:

copy_nonoverlapping(src, dst, count)

within core::intrinsics::copy_nonoverlapping. (intrinsics.rs line 1999, from commit 4fb7144ed159f94491249e86d5bbd033b5d60550 in the rust repo), all inlined in scoped_key_data from third_party/rust/fxa-client/src/http_client.rs. (https://searchfox.org/mozilla-central/rev/9193635dca8cfdcb68f114306194ffc860456044/third_party/rust/fxa-client/src/http_client.rs#362)

This only happens when both PGO and LTO (cross) are enabled.

It happens equally with clang 9 and clang 10.

Flags: needinfo?(mh+mozilla)

Are you able to get this into a link reproducer file? (Not sure if every linker supports this; lld does) And if so, do multiple links against the same reproducer give different results? That would make quite a shortcut to debugging this if it works.

It's the same error and not perma fail (next merge was green)

Flags: needinfo?(mh+mozilla)

Retriggering the Bpgo(repr) with force: true and downstream: true turned green, so it's still intermittent.

Flags: needinfo?(mh+mozilla)

Mike are you working on this?

The latest instances are bug 1654123.

Hi Mike, this issue got classified with this bug and bug 1654123 sometimes.

Can you please point out which is the correct one and how to make the difference?

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=314974128&repo=mozilla-central&lineNumber=173

[task 2020-09-06T23:49:53.888Z] |###################### | 82% minidump-analyzer ETA: 0:01:36
[task 2020-09-06T23:49:53.934Z] |###################### | 83% omni.ja ETA: 0:01:31
[task 2020-09-06T23:49:53.967Z] |########################## | 99% pingsender ETA: 0:00:03
[task 2020-09-06T23:49:53.967Z] |########################## | 99% platform.ini ETA: 0:00:01
[task 2020-09-06T23:49:54.001Z] |########################## | 99% plugin-container ETA: 0:00:01
[task 2020-09-06T23:49:54.001Z] |########################## | 99% precomplete ETA: 0:00:00
[task 2020-09-06T23:49:54.001Z] |########################## | 99% removed-files ETA: 0:00:00
[task 2020-09-06T23:49:54.001Z] |########################## | 99% update-settings.ini ETA: 0:00:00
[task 2020-09-06T23:49:54.035Z] |########################## | 99% updater ETA: 0:00:00
[task 2020-09-06T23:49:54.035Z] |########################## | 99% updater.ini ETA: 0:00:00
[task 2020-09-06T23:49:54.035Z] |###########################| 100% ETA: 0:00:00
[task 2020-09-06T23:49:54.035Z] |###########################| 100% ETA: 0:00:00
[task 2020-09-06T23:49:54.035Z] |###########################| 100% ETA: 0:00:00
[task 2020-09-06T23:49:54.119Z] |###########################| 100% Time: 0:07:35
[task 2020-09-06T23:49:54.119Z] + report_error diff
[task 2020-09-06T23:49:54.119Z] + python3
[task 2020-09-06T23:49:54.128Z] TEST-UNEXPECTED-FAIL | firefox/libxul.so differs. See the diff.html or diff.txt artifact
[task 2020-09-06T23:49:54.130Z] + fail
[task 2020-09-06T23:49:54.130Z] + exit 1
[taskcluster 2020-09-06 23:49:54.488Z] === Task Finished ===
[taskcluster 2020-09-06 23:49:55.177Z] Unsuccessful task run with exit code: 1 completed in 490.515 seconds

Flags: needinfo?(mh+mozilla)

There are 41 total failures in the last 30 days

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=325331889&repo=mozilla-central&lineNumber=180

[task 2020-12-23T22:58:03.809Z] |# | 6% …ox/libxul.so [.gnu.hash] ETA: 0:10:59
[task 2020-12-23T22:58:07.073Z] |## | 9% …efox/libxul.so [.dynstr] ETA: 0:07:27
[task 2020-12-23T22:58:07.941Z] |### | 12% …ul.so [.elfhack.data.v0] ETA: 0:05:54
[task 2020-12-23T22:58:08.258Z] |#### | 15% …ul.so [.elfhack.text.v0] ETA: 0:04:42
[task 2020-12-23T22:58:08.573Z] |#### | 18% …irefox/libxul.so [.init] ETA: 0:03:50
[task 2020-12-23T22:58:09.101Z] |##### | 21% …firefox/libxul.so [.plt] ETA: 0:03:13
[task 2020-12-23T22:58:09.423Z] |###### | 24% …fox/libxul.so [.plt.got] ETA: 0:02:45
[task 2020-12-23T23:05:42.978Z] |####### | 27% …irefox/libxul.so [.text] ETA: 0:02:22
[task 2020-12-23T23:05:43.371Z] |######## | 30% …fox/libxul.so [text_env] ETA: 0:19:29
[task 2020-12-23T23:05:43.678Z] |######## | 33% …irefox/libxul.so [.fini] ETA: 0:17:01
[task 2020-12-23T23:06:03.514Z] |######### | 36% …efox/libxul.so [.rodata] ETA: 0:14:57
[task 2020-12-23T23:06:05.588Z] |########## | 39% …ibxul.so [.eh_frame_hdr] ETA: 0:13:42
[task 2020-12-23T23:06:15.589Z] |########### | 42% …ox/libxul.so [.eh_frame] ETA: 0:12:11
[task 2020-12-23T23:06:15.624Z] |############ | 44% …l.so [.gcc_except_table] ETA: 0:11:00
[task 2020-12-23T23:06:15.644Z] |############ | 47% …refox/libxul.so [.tdata] ETA: 0:09:47
[task 2020-12-23T23:06:15.664Z] |############# | 50% …irefox/libxul.so [.tbss] ETA: 0:08:42
[task 2020-12-23T23:06:15.684Z] |############## | 53% …/libxul.so [.init_array] ETA: 0:07:44
[task 2020-12-23T23:06:15.703Z] |############### | 56% …/libxul.so [.fini_array] ETA: 0:06:52
[task 2020-12-23T23:06:17.925Z] |################ | 59% …libxul.so [.data.rel.ro] ETA: 0:06:05
[task 2020-12-23T23:06:17.952Z] |################ | 62% …firefox/libxul.so [.got] ETA: 0:05:23
[task 2020-12-23T23:06:17.982Z] |################# | 65% …fox/libxul.so [.got.plt] ETA: 0:04:45
[task 2020-12-23T23:06:18.108Z] |################## | 68% …irefox/libxul.so [.data] ETA: 0:04:09
[task 2020-12-23T23:06:18.128Z] |################### | 71% …firefox/libxul.so [.bss] ETA: 0:03:37
[task 2020-12-23T23:06:18.147Z] |#################### | 74% …fox/libxul.so [.comment] ETA: 0:03:07
[task 2020-12-23T23:06:18.166Z] |#################### | 77% …bxul.so [.gnu_debuglink] ETA: 0:02:39
[task 2020-12-23T23:06:43.525Z] |##################### | 80% …efox/libxul.so [.strtab] ETA: 0:02:13
[task 2020-12-23T23:06:43.546Z] |###################### | 83% …ox/libxul.so [.shstrtab] ETA: 0:01:54
[task 2020-12-23T23:06:43.546Z] |####################### | 86% ETA: 0:01:31
[task 2020-12-23T23:06:43.549Z] |####################### | 86% minidump-analyzer ETA: 0:01:31
[task 2020-12-23T23:06:43.565Z] |####################### | 86% omni.ja ETA: 0:01:26
[task 2020-12-23T23:06:43.568Z] |########################## | 99% pingsender ETA: 0:00:03
[task 2020-12-23T23:06:43.568Z] |########################## | 99% platform.ini ETA: 0:00:02
[task 2020-12-23T23:06:43.571Z] |########################## | 99% plugin-container ETA: 0:00:02
[task 2020-12-23T23:06:43.571Z] |########################## | 99% precomplete ETA: 0:00:00
[task 2020-12-23T23:06:43.571Z] |########################## | 99% removed-files ETA: 0:00:00
[task 2020-12-23T23:06:43.571Z] |########################## | 99% update-settings.ini ETA: 0:00:00
[task 2020-12-23T23:06:43.573Z] |########################## | 99% updater ETA: 0:00:00
[task 2020-12-23T23:06:43.573Z] |########################## | 99% updater.ini ETA: 0:00:00
[task 2020-12-23T23:06:43.573Z] |###########################| 100% ETA: 0:00:00
[task 2020-12-23T23:06:43.574Z] |###########################| 100% ETA: 0:00:00
[task 2020-12-23T23:06:43.574Z] |###########################| 100% ETA: 0:00:00
[task 2020-12-23T23:07:24.866Z] |###########################| 100% Time: 0:09:27
[task 2020-12-23T23:07:24.866Z] + report_error diff
[task 2020-12-23T23:07:24.866Z] + python3
[task 2020-12-23T23:07:25.332Z] TEST-UNEXPECTED-FAIL | firefox/libxul.so differs. See the diff.html or diff.txt artifact
[task 2020-12-23T23:07:25.333Z] + fail
[task 2020-12-23T23:07:25.333Z] + exit 1
[taskcluster 2020-12-23 23:07:25.662Z] === Task Finished ===
[taskcluster 2020-12-23 23:07:31.745Z] Unsuccessful task run with exit code: 1 completed in 651.939 seconds

Mike are there any updates on this?

Flags: needinfo?(mh+mozilla)
Flags: needinfo?(mh+mozilla)
Whiteboard: [stockwell needswork:owner]

There are 27 total failures in the last 7 days on diff opt

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=328024271&repo=mozilla-central&lineNumber=179

[task 2021-01-27T23:27:25.280Z] 2021-01-27 23:27:25 W: diffoscope.main: Fuzzy-matching is currently disabled as the "tlsh" module is unavailable.
[task 2021-01-27T23:27:25.321Z] | | 0% ETA: --:--:--
[task 2021-01-27T23:27:25.321Z] | | 0% application.ini ETA: 0:30:50
[task 2021-01-27T23:27:25.331Z] | | 0% browser ETA: 0:26:07
[task 2021-01-27T23:27:25.331Z] | | 0% default16.png ETA: 254 days, 1:32:36
[task 2021-01-27T23:27:25.331Z] | | 0% default32.png ETA: 249 days, 7:36:06
[task 2021-01-27T23:27:25.331Z] | | 0% default48.png ETA: 237 days, 0:31:16
[task 2021-01-27T23:27:25.331Z] | | 0% default64.png ETA: 216 days, 13:18:23
[task 2021-01-27T23:27:25.331Z] | | 0% ETA: 194 days, 3:18:55
[task 2021-01-27T23:27:25.331Z] | | 0% ETA: 194 days, 9:14:54
[task 2021-01-27T23:27:25.331Z] | | 0% ETA: 137 days, 12:27:27
[task 2021-01-27T23:27:25.331Z] | | 0% …ashreporter-override.ini ETA: 112 days, 12:36:38
[task 2021-01-27T23:27:25.334Z] | | 0% features ETA: 99 days, 8:45:37
[task 2021-01-27T23:27:25.336Z] | | 0% …autofill@mozilla.org.xpi ETA: 6268 days, 18:21:29
[task 2021-01-27T23:27:25.337Z] | | 0% …eenshots@mozilla.org.xpi ETA: 217 days, 19:06:59
[task 2021-01-27T23:27:25.337Z] | | 0% …reporter@mozilla.org.xpi ETA: 151 days, 12:22:44
[task 2021-01-27T23:27:25.339Z] | | 0% webcompat@mozilla.org.xpi ETA: 148 days, 4:16:40
[task 2021-01-27T23:27:25.339Z] | | 0% ETA: 129 days, 2:44:18
[task 2021-01-27T23:27:25.362Z] | | 0% omni.ja ETA: 129 days, 9:07:54
[task 2021-01-27T23:27:25.362Z] | | 0% ETA: 0:52:31
[task 2021-01-27T23:27:25.364Z] | | 0% crashreporter ETA: 0:52:36
[task 2021-01-27T23:27:25.364Z] | | 0% crashreporter.ini ETA: 0:01:06
[task 2021-01-27T23:27:25.368Z] | | 0% defaults ETA: 0:01:05
[task 2021-01-27T23:27:25.368Z] | | 0% ETA: 0:01:07
[task 2021-01-27T23:27:25.368Z] | | 0% ETA: 0:01:08
[task 2021-01-27T23:27:25.369Z] | | 0% dependentlibs.list ETA: 0:01:08
[task 2021-01-27T23:27:25.369Z] | | 0% dmd.py ETA: 0:01:08
[task 2021-01-27T23:27:25.369Z] | | 0% firefox ETA: 0:01:00
[task 2021-01-27T23:27:25.370Z] | | 0% firefox-bin ETA: 0:00:58
[task 2021-01-27T23:27:25.371Z] | | 0% fix_stacks.py ETA: 0:00:19
[task 2021-01-27T23:27:25.375Z] | | 0% fonts ETA: 0:00:19
[task 2021-01-27T23:27:25.375Z] | | 0% ETA: 0:00:19
[task 2021-01-27T23:27:25.381Z] | | 0% gmp-clearkey ETA: 0:00:19
[task 2021-01-27T23:27:25.381Z] | | 0% manifest.json ETA: 0:00:20
[task 2021-01-27T23:27:25.381Z] | | 0% ETA: 0:00:21
[task 2021-01-27T23:27:25.381Z] | | 0% ETA: 0:00:21
[task 2021-01-27T23:27:25.383Z] | | 0% gtk2 ETA: 0:00:21
[task 2021-01-27T23:27:25.383Z] | | 0% ETA: 0:00:21
[task 2021-01-27T23:27:25.386Z] | | 0% icons ETA: 0:00:21
[task 2021-01-27T23:27:25.386Z] | | 0% ETA: 0:00:21
[task 2021-01-27T23:27:25.387Z] | | 0% libfreeblpriv3.so ETA: 0:00:21
[task 2021-01-27T23:27:25.388Z] | | 0% liblgpllibs.so ETA: 0:00:12
[task 2021-01-27T23:27:25.390Z] | | 0% libmozavcodec.so ETA: 0:00:11
[task 2021-01-27T23:27:25.391Z] | | 1% libmozavutil.so ETA: 0:00:05
[task 2021-01-27T23:27:25.392Z] | | 2% libmozgtk.so ETA: 0:00:05
[task 2021-01-27T23:27:25.393Z] | | 2% libmozsandbox.so ETA: 0:00:05
[task 2021-01-27T23:27:25.395Z] | | 2% libmozsqlite3.so ETA: 0:00:05
[task 2021-01-27T23:27:25.395Z] | | 2% libmozwayland.so ETA: 0:00:04
[task 2021-01-27T23:27:25.396Z] | | 2% libnspr4.so ETA: 0:00:04
[task 2021-01-27T23:27:25.398Z] | | 2% libnss3.so ETA: 0:00:04
[task 2021-01-27T23:27:25.400Z] | | 3% libnssckbi.so ETA: 0:00:03
[task 2021-01-27T23:27:25.401Z] | | 3% libnssutil3.so ETA: 0:00:03
[task 2021-01-27T23:27:25.401Z] | | 3% libplc4.so ETA: 0:00:03
[task 2021-01-27T23:27:25.401Z] | | 3% libplds4.so ETA: 0:00:03
[task 2021-01-27T23:27:25.403Z] | | 3% libsmime3.so ETA: 0:00:03
[task 2021-01-27T23:27:25.404Z] | | 3% libsoftokn3.so ETA: 0:00:03
[task 2021-01-27T23:27:25.406Z] | | 3% libssl3.so ETA: 0:00:03
[task 2021-01-27T23:28:11.378Z] |# | 3% libxul.so ETA: 0:00:03
[task 2021-01-27T23:28:11.674Z] |# | 6% …ox/libxul.so [.gnu.hash] ETA: 0:10:34
[task 2021-01-27T23:28:14.677Z] |## | 9% …efox/libxul.so [.dynstr] ETA: 0:07:10
[task 2021-01-27T23:28:15.515Z] |### | 12% …ul.so [.elfhack.data.v0] ETA: 0:05:40
[task 2021-01-27T23:28:15.793Z] |#### | 15% …ul.so [.elfhack.text.v0] ETA: 0:04:31
[task 2021-01-27T23:28:16.072Z] |##### | 18% …irefox/libxul.so [.init] ETA: 0:03:41
[task 2021-01-27T23:28:16.534Z] |##### | 21% …firefox/libxul.so [.plt] ETA: 0:03:05
[task 2021-01-27T23:28:16.842Z] |###### | 24% …fox/libxul.so [.plt.got] ETA: 0:02:38
[task 2021-01-27T23:33:58.727Z] |####### | 27% …irefox/libxul.so [.text] ETA: 0:02:16
[task 2021-01-27T23:33:59.099Z] |######## | 30% …fox/libxul.so [text_env] ETA: 0:15:02
[task 2021-01-27T23:33:59.403Z] |######## | 33% …irefox/libxul.so [.fini] ETA: 0:13:08
[task 2021-01-27T23:34:18.361Z] |######### | 36% …efox/libxul.so [.rodata] ETA: 0:11:32
[task 2021-01-27T23:34:20.572Z] |########## | 39% …ibxul.so [.eh_frame_hdr] ETA: 0:10:40
[task 2021-01-27T23:34:30.589Z] |########### | 42% …ox/libxul.so [.eh_frame] ETA: 0:09:29
[task 2021-01-27T23:34:30.628Z] |############ | 45% …l.so [.gcc_except_table] ETA: 0:08:37
[task 2021-01-27T23:34:30.655Z] |############ | 48% …refox/libxul.so [.tdata] ETA: 0:07:39
[task 2021-01-27T23:34:30.683Z] |############# | 51% …irefox/libxul.so [.tbss] ETA: 0:06:48
[task 2021-01-27T23:34:30.711Z] |############## | 53% …/libxul.so [.init_array] ETA: 0:06:03
[task 2021-01-27T23:34:30.738Z] |############### | 56% …/libxul.so [.fini_array] ETA: 0:05:22
[task 2021-01-27T23:34:34.445Z] |################ | 59% …libxul.so [.data.rel.ro] ETA: 0:04:45
[task 2021-01-27T23:34:34.550Z] |################ | 62% …firefox/libxul.so [.got] ETA: 0:04:14
[task 2021-01-27T23:34:34.594Z] |################# | 65% …fox/libxul.so [.got.plt] ETA: 0:03:43
[task 2021-01-27T23:34:34.769Z] |################## | 68% …irefox/libxul.so [.data] ETA: 0:03:15
[task 2021-01-27T23:34:34.795Z] |################### | 71% …firefox/libxul.so [.bss] ETA: 0:02:49
[task 2021-01-27T23:34:34.822Z] |#################### | 74% …fox/libxul.so [.comment] ETA: 0:02:26
[task 2021-01-27T23:34:34.848Z] |#################### | 77% …bxul.so [.gnu_debuglink] ETA: 0:02:04
[task 2021-01-27T23:34:58.581Z] |##################### | 80% …efox/libxul.so [.strtab] ETA: 0:01:44
[task 2021-01-27T23:34:58.614Z] |###################### | 83% …ox/libxul.so [.shstrtab] ETA: 0:01:29
[task 2021-01-27T23:34:58.614Z] |####################### | 86% ETA: 0:01:11
[task 2021-01-27T23:34:58.621Z] |####################### | 86% minidump-analyzer ETA: 0:01:11
[task 2021-01-27T23:34:58.640Z] |####################### | 87% omni.ja ETA: 0:01:06
[task 2021-01-27T23:34:58.647Z] |########################## | 99% pingsender ETA: 0:00:03
[task 2021-01-27T23:34:58.647Z] |########################## | 99% platform.ini ETA: 0:00:01
[task 2021-01-27T23:34:58.654Z] |########################## | 99% plugin-container ETA: 0:00:01
[task 2021-01-27T23:34:58.654Z] |########################## | 99% precomplete ETA: 0:00:00
[task 2021-01-27T23:34:58.654Z] |########################## | 99% removed-files ETA: 0:00:00
[task 2021-01-27T23:34:58.654Z] |########################## | 99% update-settings.ini ETA: 0:00:00
[task 2021-01-27T23:34:58.660Z] |########################## | 99% updater ETA: 0:00:00
[task 2021-01-27T23:34:58.660Z] |########################## | 99% updater.ini ETA: 0:00:00
[task 2021-01-27T23:34:58.660Z] |###########################| 100% ETA: 0:00:00
[task 2021-01-27T23:34:58.660Z] |###########################| 100% ETA: 0:00:00
[task 2021-01-27T23:34:58.660Z] |###########################| 100% ETA: 0:00:00
[task 2021-01-27T23:35:31.464Z] |###########################| 100% Time: 0:07:33
[task 2021-01-27T23:35:31.464Z] + report_error diff
[task 2021-01-27T23:35:31.464Z] + python3
[task 2021-01-27T23:35:31.708Z] TEST-UNEXPECTED-FAIL | firefox/libxul.so differs. See the diff.html or diff.txt artifact
[task 2021-01-27T23:35:31.710Z] + fail
[task 2021-01-27T23:35:31.710Z] + exit 1
[taskcluster 2021-01-27 23:35:32.029Z] === Task Finished ===
[taskcluster 2021-01-27 23:35:34.669Z] Unsuccessful task run with exit code: 1 completed in 527.883 seconds.
Mike are you still working on this? Are there updates?

Flags: needinfo?(mh+mozilla)
Whiteboard: [stockwell unknown] → [stockwell needswork:owner]
Flags: needinfo?(mh+mozilla)

It seems the linux64 task is failing much more often than the linux32 one. There's a difference in codegen in nsCSSBorderRenderer::DrawBorders and a bunch of symbol differences. David, what do you think about backing out bug 1686510?

Flags: needinfo?(mh+mozilla) → needinfo?(dmajor)

Funny enough, in clang-12 it's the other way around, Linux64 has been rock solid while I stopped looking at Linux32 because there's a 50-50 random element in it (EDIT: in rust code, not c++).
How about we set them both to tier-3 until we upgrade to 12?

Flags: needinfo?(dmajor)

fair enough, I guess.

I expect that we'll be able to undo this (at least for Linux64) after clang-12.

Assignee: nobody → dmajor
Status: NEW → ASSIGNED
Pushed by dmajor@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/bab47ba5a65b
Make diffoscope tasks tier 3 in the short term r=glandium
Status: ASSIGNED → RESOLVED
Closed: 1 year ago
Resolution: --- → FIXED
Target Milestone: --- → 87 Branch
You need to log in before you can comment on or make changes to this bug.