1736970 - Permanent firefox/libxul.so differs. See the diff.html or diff.txt artifact

Status: RESOLVED FIXED

(Firefox Build System :: Toolchains, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: csabou [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=355537888&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/TMVaHBVkRu6XNTSyk1m60A/runs/0/artifacts/public/logs/live_backing.log

---

[task 2021-10-21T05:06:58.718Z] + case "$ORIG_URL" in
[task 2021-10-21T05:06:58.718Z] + OMNIJAR=omni.ja
[task 2021-10-21T05:06:58.718Z] + POST=true
[task 2021-10-21T05:06:58.718Z] + for option in "$@"
[task 2021-10-21T05:06:58.718Z] + case "$option" in
[task 2021-10-21T05:06:58.718Z] + POST=fail
[task 2021-10-21T05:06:58.718Z] + '[' -n '' ']'
[task 2021-10-21T05:06:58.718Z] + diffoscope --html diff.html --text diff.txt --progress --output-empty --no-default-limits --max-page-size 100000000 --max-page-diff-block-lines 10000 --exclude-directory-metadata=yes --exclude-command .--line-numbers a b
[task 2021-10-21T05:06:58.821Z]  |                          |    0%                             ETA:  --:--:-- 
[task 2021-10-21T05:06:58.831Z]  |                           |    0%  a/firefox/application.ini  ETA:  0:30:01 
[task 2021-10-21T05:06:58.833Z]  |                           |    0%          a/firefox/browser  ETA:  0:31:16 
[task 2021-10-21T05:06:58.834Z]  |                |    0%  …ns/default/default16.png  ETA:  263 days, 20:34:42 
[task 2021-10-21T05:06:58.835Z]  |                |    0%  …ns/default/default32.png  ETA:  262 days, 22:33:48 
[task 2021-10-21T05:06:58.836Z]  |                |    0%  …ns/default/default48.png  ETA:  253 days, 20:58:14 
[task 2021-10-21T05:06:58.836Z]  |                 |    0%  …ns/default/default64.png  ETA:  235 days, 9:45:11 
[task 2021-10-21T05:06:58.836Z]  |                 |    0%                             ETA:  214 days, 3:40:31 
[task 2021-10-21T05:06:58.837Z]  |                 |    0%                             ETA:  214 days, 9:41:15 
[task 2021-10-21T05:06:58.837Z]  |                |    0%                             ETA:  151 days, 15:13:50 
[task 2021-10-21T05:06:58.839Z]  |                 |    0%  …ashreporter-override.ini  ETA:  124 days, 1:08:45 
[task 2021-10-21T05:06:58.841Z]  |                |    0%  …firefox/browser/features  ETA:  114 days, 10:52:40 
[task 2021-10-21T05:06:58.845Z]  |                |    0%  …[e-mail]  ETA:  4694 days, 2:18:53 
[task 2021-10-21T05:06:58.846Z]  |                 |    0%  …[e-mail]  ETA:  706 days, 5:02:54 
[task 2021-10-21T05:06:58.848Z]  |                |    0%  …[e-mail]  ETA:  661 days, 19:12:13 
[task 2021-10-21T05:06:58.852Z]  |                 |    0%  …[e-mail]  ETA:  610 days, 8:43:04 
[task 2021-10-21T05:06:58.854Z]  |                |    0%  …[e-mail]  ETA:  252 days, 20:54:11 
[task 2021-10-21T05:06:58.859Z]  |                 |    0%  …[e-mail]  ETA:  247 days, 2:13:49 
[task 2021-10-21T05:06:58.859Z]  |                 |    0%                             ETA:  174 days, 8:20:54 
[task 2021-10-21T05:06:58.889Z]  |                |    0%  a/firefox/browser/omni.ja  ETA:  174 days, 16:00:10 
[task 2021-10-21T05:06:58.889Z]  |                           |    0%                             ETA:  1:09:13 
[task 2021-10-21T05:06:58.894Z]  |                           |    0%    a/firefox/crashreporter  ETA:  1:09:19 
[task 2021-10-21T05:06:58.898Z]  |                           |    0%  …irefox/crashreporter.ini  ETA:  0:01:31 
[task 2021-10-21T05:06:58.904Z]  |                           |    0%         a/firefox/defaults  ETA:  0:01:33 
[task 2021-10-21T05:06:58.904Z]  |                           |    0%                             ETA:  0:01:36 
[task 2021-10-21T05:06:58.904Z]  |                           |    0%                             ETA:  0:01:36 
[task 2021-10-21T05:06:58.909Z]  |                           |    0%  …refox/dependentlibs.list  ETA:  0:01:36 
[task 2021-10-21T05:06:58.911Z]  |                           |    0%           a/firefox/dmd.py  ETA:  0:01:40 
[task 2021-10-21T05:06:58.912Z]  |                           |    0%          a/firefox/firefox  ETA:  0:01:31 
[task 2021-10-21T05:06:58.920Z]  |                           |    0%      a/firefox/firefox-bin  ETA:  0:01:27 
[task 2021-10-21T05:06:58.922Z]  |                           |    0%    a/firefox/fix_stacks.py  ETA:  0:00:25 
[task 2021-10-21T05:06:58.931Z]  |                           |    0%            a/firefox/fonts  ETA:  0:00:26 
[task 2021-10-21T05:06:58.931Z]  |                           |    0%                             ETA:  0:00:27 
[task 2021-10-21T05:06:58.935Z]  |                           |    0%     a/firefox/gmp-clearkey  ETA:  0:00:27 
[task 2021-10-21T05:06:58.935Z]  |                           |    0%  …earkey/0.1/manifest.json  ETA:  0:00:28 
[task 2021-10-21T05:06:58.935Z]  |                           |    0%                             ETA:  0:00:28 
[task 2021-10-21T05:06:58.936Z]  |                           |    0%                             ETA:  0:00:28 
[task 2021-10-21T05:06:58.937Z]  |                           |    0%            a/firefox/icons  ETA:  0:00:28 
[task 2021-10-21T05:06:58.937Z]  |                           |    0%                             ETA:  0:00:28 
[task 2021-10-21T05:06:58.943Z]  |                           |    0%  …irefox/libfreeblpriv3.so  ETA:  0:00:28 
[task 2021-10-21T05:06:58.944Z]  |                           |    0%   a/firefox/liblgpllibs.so  ETA:  0:00:18 
[task 2021-10-21T05:06:58.948Z]  |                           |    0%  …firefox/libmozavcodec.so  ETA:  0:00:18 
[task 2021-10-21T05:06:58.951Z]  |                           |    0%  a/firefox/libmozavutil.so  ETA:  0:00:16 
[task 2021-10-21T05:06:58.952Z]  |                           |    1%     a/firefox/libmozgtk.so  ETA:  0:00:15 
[task 2021-10-21T05:06:58.955Z]  |                           |    1%  …firefox/libmozsandbox.so  ETA:  0:00:15 
[task 2021-10-21T05:06:58.963Z]  |                           |    1%  …firefox/libmozsqlite3.so  ETA:  0:00:14 
[task 2021-10-21T05:06:58.965Z]  |                           |    1%  …firefox/libmozwayland.so  ETA:  0:00:10 
[task 2021-10-21T05:06:58.968Z]  |                           |    1%      a/firefox/libnspr4.so  ETA:  0:00:10 
[task 2021-10-21T05:06:58.975Z]  |                           |    1%       a/firefox/libnss3.so  ETA:  0:00:10 
[task 2021-10-21T05:06:58.980Z]  |                           |    2%    a/firefox/libnssckbi.so  ETA:  0:00:08 
[task 2021-10-21T05:06:58.983Z]  |                           |    2%   a/firefox/libnssutil3.so  ETA:  0:00:08 
[task 2021-10-21T05:06:58.984Z]  |                           |    2%       a/firefox/libplc4.so  ETA:  0:00:08 
[task 2021-10-21T05:06:58.985Z]  |                           |    2%      a/firefox/libplds4.so  ETA:  0:00:08 
[task 2021-10-21T05:06:58.988Z]  |                           |    2%     a/firefox/libsmime3.so  ETA:  0:00:08 
[task 2021-10-21T05:06:58.992Z]  |                           |    2%   a/firefox/libsoftokn3.so  ETA:  0:00:07 
[task 2021-10-21T05:06:58.997Z]  |                           |    2%       a/firefox/libssl3.so  ETA:  0:00:07 
[task 2021-10-21T05:08:02.718Z]  |                           |    2%        a/firefox/libxul.so  ETA:  0:00:07 
[task 2021-10-21T05:08:03.051Z]  |#                          |    6%  …ox/libxul.so [.gnu.hash]  ETA:  0:14:52 
[task 2021-10-21T05:08:06.786Z]  |##                         |    9%  …efox/libxul.so [.dynstr]  ETA:  0:10:04 
[task 2021-10-21T05:08:07.680Z]  |###                        |   12%  …ul.so [.elfhack.data.v0]  ETA:  0:07:54 
[task 2021-10-21T05:08:07.967Z]  |####                       |   15%  …ul.so [.elfhack.text.v0]  ETA:  0:06:17 
[task 2021-10-21T05:08:08.252Z]  |####                       |   18%  …irefox/libxul.so [.init]  ETA:  0:05:07 
[task 2021-10-21T05:08:08.765Z]  |#####                      |   21%  …firefox/libxul.so [.plt]  ETA:  0:04:16 
[task 2021-10-21T05:08:09.058Z]  |######                     |   24%  …fox/libxul.so [.plt.got]  ETA:  0:03:39 
[task 2021-10-21T05:16:04.336Z]  |#######                    |   27%  …irefox/libxul.so [.text]  ETA:  0:03:08 
[task 2021-10-21T05:16:04.706Z]  |########                   |   30%  …fox/libxul.so [text_env]  ETA:  0:21:09 
[task 2021-10-21T05:16:04.981Z]  |########                   |   32%  …irefox/libxul.so [.fini]  ETA:  0:18:29 
[task 2021-10-21T05:16:26.924Z]  |#########                  |   35%  …efox/libxul.so [.rodata]  ETA:  0:16:15 
[task 2021-10-21T05:16:29.172Z]  |##########                 |   38%  …ibxul.so [.eh_frame_hdr]  ETA:  0:14:55 
[task 2021-10-21T05:16:40.188Z]  |###########                |   41%  …ox/libxul.so [.eh_frame]  ETA:  0:13:15 
[task 2021-10-21T05:16:40.221Z]  |############               |   44%  …l.so [.gcc_except_table]  ETA:  0:12:00 
[task 2021-10-21T05:16:40.241Z]  |############               |   47%  …refox/libxul.so [.tdata]  ETA:  0:10:40 
[task 2021-10-21T05:16:40.260Z]  |#############              |   50%  …irefox/libxul.so [.tbss]  ETA:  0:09:29 
[task 2021-10-21T05:16:40.280Z]  |##############             |   53%  …/libxul.so [.init_array]  ETA:  0:08:26 
[task 2021-10-21T05:16:40.299Z]  |###############            |   56%  …/libxul.so [.fini_array]  ETA:  0:07:30 
[task 2021-10-21T05:16:42.681Z]  |################           |   59%  …libxul.so [.data.rel.ro]  ETA:  0:06:39 
[task 2021-10-21T05:16:42.708Z]  |################           |   62%  …firefox/libxul.so [.got]  ETA:  0:05:54 
[task 2021-10-21T05:16:42.741Z]  |#################          |   65%  …fox/libxul.so [.got.plt]  ETA:  0:05:12 
[task 2021-10-21T05:16:42.906Z]  |##################         |   68%  …irefox/libxul.so [.data]  ETA:  0:04:34 
[task 2021-10-21T05:16:42.927Z]  |###################        |   70%  …firefox/libxul.so [.bss]  ETA:  0:03:58 
[task 2021-10-21T05:16:42.947Z]  |###################        |   73%  …fox/libxul.so [.comment]  ETA:  0:03:26 
[task 2021-10-21T05:16:42.965Z]  |####################       |   76%  …bxul.so [.gnu_debuglink]  ETA:  0:02:56 
[task 2021-10-21T05:17:12.539Z]  |#####################      |   79%  …efox/libxul.so [.strtab]  ETA:  0:02:28 
[task 2021-10-21T05:17:12.559Z]  |######################     |   82%  …ox/libxul.so [.shstrtab]  ETA:  0:02:08 
[task 2021-10-21T05:17:12.559Z]  |#######################    |   85%                             ETA:  0:01:43 
[task 2021-10-21T05:17:12.570Z]  |#######################    |   85%  …irefox/minidump-analyzer  ETA:  0:01:43 
[task 2021-10-21T05:17:12.594Z]  |#######################    |   86%          a/firefox/omni.ja  ETA:  0:01:36 
[task 2021-10-21T05:17:12.602Z]  |########################## |   99%       a/firefox/pingsender  ETA:  0:00:05 
[task 2021-10-21T05:17:12.609Z]  |########################## |   99%     a/firefox/platform.ini  ETA:  0:00:02 
[task 2021-10-21T05:17:12.616Z]  |########################## |   99%  …firefox/plugin-container  ETA:  0:00:02 
[task 2021-10-21T05:17:12.623Z]  |########################## |   99%      a/firefox/precomplete  ETA:  0:00:00 
[task 2021-10-21T05:17:12.624Z]  |########################## |   99%    a/firefox/removed-files  ETA:  0:00:00 
[task 2021-10-21T05:17:12.630Z]  |########################## |   99%  …efox/update-settings.ini  ETA:  0:00:00 
[task 2021-10-21T05:17:12.633Z]  |########################## |   99%          a/firefox/updater  ETA:  0:00:00 
[task 2021-10-21T05:17:12.636Z]  |########################## |   99%      a/firefox/updater.ini  ETA:  0:00:00 
[task 2021-10-21T05:17:12.636Z]  |###########################|  100%                             ETA:  0:00:00 
[task 2021-10-21T05:17:12.636Z]  |###########################|  100%                             ETA:  0:00:00 
[task 2021-10-21T05:17:12.636Z]  |###########################|  100%                             ETA:  0:00:00 
[task 2021-10-21T05:18:21.494Z]  |###########################|  100%                             Time: 0:10:13 
[task 2021-10-21T05:18:21.494Z] ++ dirname /builds/worker/bin/get_and_diffoscope
[task 2021-10-21T05:18:21.494Z] + /builds/worker/bin/report_error diff
[task 2021-10-21T05:18:22.152Z] TEST-UNEXPECTED-FAIL | firefox/libxul.so differs. See the diff.html or diff.txt artifact
[task 2021-10-21T05:18:22.154Z] + fail
[task 2021-10-21T05:18:22.154Z] + exit 1
[taskcluster 2021-10-21 05:18:22.478Z] === Task Finished ===
[taskcluster 2021-10-21 05:18:27.696Z] Unsuccessful task run with exit code: 1 completed in 721.316 seconds

Comment 2

[Release mgmt bot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1731582

Comment 3

[Intermittent Failures Robot]

34 failures in 3186 pushes (0.011 failures/push) were associated with this bug in the last 7 days.

This is the #24 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 34

Platform and build breakdown:

• diff: 34
  • opt: 34

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1736970&startday=2021-10-18&endday=2021-10-24&tree=all

Comment 4

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Who can look into fixing these tasks? Mike, Andi, can you coordinate this?

Comment 5

[Mike Hommey [:glandium]]

I'm still bisecting this. There's a possible workaround for the linux32 one, but it doesn't work for both jobs.

Comment 6

[Intermittent Failures Robot]

42 failures in 2845 pushes (0.015 failures/push) were associated with this bug in the last 7 days.

This is the #15 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 42

Platform and build breakdown:

• diff: 42
  • opt: 42

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1736970&startday=2021-10-25&endday=2021-10-31&tree=all

Comment 7

[Andreea Pavel [:apavel]]

There are 44 total failures in the last 7 days on diff opt
Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=356582304&repo=mozilla-central&lineNumber=178

[task 2021-11-01T10:21:01.607Z] + diffoscope --html diff.html --text diff.txt --progress --output-empty --no-default-limits --max-page-size 100000000 --max-page-diff-block-lines 10000 --exclude-directory-metadata=yes --exclude-command .--line-numbers a b
[task 2021-11-01T10:21:01.723Z]  |                          |    0%                             ETA:  --:--:-- 
[task 2021-11-01T10:21:01.734Z]  |                           |    0%  a/firefox/application.ini  ETA:  0:33:43 
[task 2021-11-01T10:21:01.737Z]  |                           |    0%          a/firefox/browser  ETA:  0:35:24 
[task 2021-11-01T10:21:01.738Z]  |                |    0%  …ns/default/default16.png  ETA:  299 days, 12:14:19 
[task 2021-11-01T10:21:01.739Z]  |                |    0%  …ns/default/default32.png  ETA:  298 days, 12:54:06 
[task 2021-11-01T10:21:01.740Z]  |                 |    0%  …ns/default/default48.png  ETA:  288 days, 4:10:01 
[task 2021-11-01T10:21:01.741Z]  |                 |    0%  …ns/default/default64.png  ETA:  267 days, 6:13:02 
[task 2021-11-01T10:21:01.741Z]  |                 |    0%                             ETA:  243 days, 2:09:35 
[task 2021-11-01T10:21:01.741Z]  |                 |    0%                             ETA:  243 days, 9:53:45 
[task 2021-11-01T10:21:01.741Z]  |                 |    0%                             ETA:  172 days, 3:38:14 
[task 2021-11-01T10:21:01.744Z]  |                |    0%  …ashreporter-override.ini  ETA:  140 days, 20:43:36 
[task 2021-11-01T10:21:01.746Z]  |                 |    0%  …firefox/browser/features  ETA:  130 days, 8:17:17 
[task 2021-11-01T10:21:01.750Z]  |               |    0%  …[e-mail]  ETA:  5393 days, 21:23:45 
[task 2021-11-01T10:21:01.752Z]  |                 |    0%  …[e-mail]  ETA:  810 days, 1:43:40 
[task 2021-11-01T10:21:01.754Z]  |                |    0%  …[e-mail]  ETA:  758 days, 10:30:44 
[task 2021-11-01T10:21:01.759Z]  |                |    0%  …[e-mail]  ETA:  699 days, 19:18:27 
[task 2021-11-01T10:21:01.761Z]  |                 |    0%  …[e-mail]  ETA:  289 days, 3:39:54 
[task 2021-11-01T10:21:01.766Z]  |                 |    0%  …[e-mail]  ETA:  282 days, 6:01:25 
[task 2021-11-01T10:21:01.766Z]  |                 |    0%                             ETA:  197 days, 8:10:37 
[task 2021-11-01T10:21:01.799Z]  |                |    0%  a/firefox/browser/omni.ja  ETA:  197 days, 16:33:42 
[task 2021-11-01T10:21:01.799Z]  |                           |    0%                             ETA:  1:18:04 
[task 2021-11-01T10:21:01.804Z]  |                           |    0%    a/firefox/crashreporter  ETA:  1:18:10 
[task 2021-11-01T10:21:01.808Z]  |                           |    0%  …irefox/crashreporter.ini  ETA:  0:01:42 
[task 2021-11-01T10:21:01.814Z]  |                           |    0%         a/firefox/defaults  ETA:  0:01:43 
[task 2021-11-01T10:21:01.814Z]  |                           |    0%                             ETA:  0:01:47 
[task 2021-11-01T10:21:01.815Z]  |                           |    0%                             ETA:  0:01:47 
[task 2021-11-01T10:21:01.820Z]  |                           |    0%  …refox/dependentlibs.list  ETA:  0:01:47 
[task 2021-11-01T10:21:01.823Z]  |                           |    0%           a/firefox/dmd.py  ETA:  0:01:51 
[task 2021-11-01T10:21:01.824Z]  |                           |    0%          a/firefox/firefox  ETA:  0:01:41 
[task 2021-11-01T10:21:01.833Z]  |                           |    0%      a/firefox/firefox-bin  ETA:  0:01:37 
[task 2021-11-01T10:21:01.836Z]  |                           |    0%    a/firefox/fix_stacks.py  ETA:  0:00:28 
[task 2021-11-01T10:21:01.846Z]  |                           |    0%            a/firefox/fonts  ETA:  0:00:28 
[task 2021-11-01T10:21:01.846Z]  |                           |    0%                             ETA:  0:00:30 
[task 2021-11-01T10:21:01.851Z]  |                           |    0%     a/firefox/gmp-clearkey  ETA:  0:00:30 
[task 2021-11-01T10:21:01.851Z]  |                           |    0%  …earkey/0.1/manifest.json  ETA:  0:00:31 
[task 2021-11-01T10:21:01.851Z]  |                           |    0%                             ETA:  0:00:31 
[task 2021-11-01T10:21:01.851Z]  |                           |    0%                             ETA:  0:00:31 
[task 2021-11-01T10:21:01.853Z]  |                           |    0%            a/firefox/icons  ETA:  0:00:31 
[task 2021-11-01T10:21:01.853Z]  |                           |    0%                             ETA:  0:00:31 
[task 2021-11-01T10:21:01.859Z]  |                           |    0%  …irefox/libfreeblpriv3.so  ETA:  0:00:31 
[task 2021-11-01T10:21:01.861Z]  |                           |    0%   a/firefox/liblgpllibs.so  ETA:  0:00:20 
[task 2021-11-01T10:21:01.864Z]  |                           |    0%  …firefox/libmozavcodec.so  ETA:  0:00:20 
[task 2021-11-01T10:21:01.868Z]  |                           |    0%  a/firefox/libmozavutil.so  ETA:  0:00:18 
[task 2021-11-01T10:21:01.870Z]  |                           |    1%     a/firefox/libmozgtk.so  ETA:  0:00:17 
[task 2021-11-01T10:21:01.873Z]  |                           |    1%  …firefox/libmozsandbox.so  ETA:  0:00:17 
[task 2021-11-01T10:21:01.881Z]  |                           |    1%  …firefox/libmozsqlite3.so  ETA:  0:00:16 
[task 2021-11-01T10:21:01.882Z]  |                           |    1%  …firefox/libmozwayland.so  ETA:  0:00:11 
[task 2021-11-01T10:21:01.886Z]  |                           |    1%      a/firefox/libnspr4.so  ETA:  0:00:11 
[task 2021-11-01T10:21:01.893Z]  |                           |    1%       a/firefox/libnss3.so  ETA:  0:00:11 
[task 2021-11-01T10:21:01.898Z]  |                           |    2%    a/firefox/libnssckbi.so  ETA:  0:00:09 
[task 2021-11-01T10:21:01.902Z]  |                           |    2%   a/firefox/libnssutil3.so  ETA:  0:00:09 
[task 2021-11-01T10:21:01.903Z]  |                           |    2%       a/firefox/libplc4.so  ETA:  0:00:08 
[task 2021-11-01T10:21:01.904Z]  |                           |    2%      a/firefox/libplds4.so  ETA:  0:00:08 
[task 2021-11-01T10:21:01.907Z]  |                           |    2%     a/firefox/libsmime3.so  ETA:  0:00:08 
[task 2021-11-01T10:21:01.912Z]  |                           |    2%   a/firefox/libsoftokn3.so  ETA:  0:00:08 
[task 2021-11-01T10:21:01.917Z]  |                           |    2%       a/firefox/libssl3.so  ETA:  0:00:08 
[task 2021-11-01T10:22:15.218Z]  |                           |    2%        a/firefox/libxul.so  ETA:  0:00:08 
[task 2021-11-01T10:22:15.586Z]  |#                          |    6%  …ox/libxul.so [.gnu.hash]  ETA:  0:17:07 
[task 2021-11-01T10:22:19.863Z]  |##                         |    9%  …efox/libxul.so [.dynstr]  ETA:  0:11:35 
[task 2021-11-01T10:22:21.048Z]  |###                        |   12%  …ul.so [.elfhack.data.v0]  ETA:  0:09:06 
[task 2021-11-01T10:22:21.349Z]  |####                       |   15%  …ul.so [.elfhack.text.v0]  ETA:  0:07:14 
[task 2021-11-01T10:22:21.655Z]  |####                       |   18%  …irefox/libxul.so [.init]  ETA:  0:05:54 
[task 2021-11-01T10:22:22.259Z]  |#####                      |   21%  …firefox/libxul.so [.plt]  ETA:  0:04:55 
[task 2021-11-01T10:22:22.571Z]  |######                     |   24%  …fox/libxul.so [.plt.got]  ETA:  0:04:12 
[task 2021-11-01T10:31:42.676Z]  |#######                    |   27%  …irefox/libxul.so [.text]  ETA:  0:03:37 
[task 2021-11-01T10:31:43.074Z]  |########                   |   30%  …fox/libxul.so [text_env]  ETA:  0:24:52 
[task 2021-11-01T10:31:43.363Z]  |########                   |   32%  …irefox/libxul.so [.fini]  ETA:  0:21:43 
[task 2021-11-01T10:32:08.786Z]  |#########                  |   35%  …efox/libxul.so [.rodata]  ETA:  0:19:06 
[task 2021-11-01T10:32:11.396Z]  |##########                 |   38%  …ibxul.so [.eh_frame_hdr]  ETA:  0:17:31 
[task 2021-11-01T10:32:24.355Z]  |###########                |   41%  …ox/libxul.so [.eh_frame]  ETA:  0:15:34 
[task 2021-11-01T10:32:24.392Z]  |############               |   44%  …l.so [.gcc_except_table]  ETA:  0:14:06 
[task 2021-11-01T10:32:24.413Z]  |############               |   47%  …refox/libxul.so [.tdata]  ETA:  0:12:32 
[task 2021-11-01T10:32:24.434Z]  |#############              |   50%  …irefox/libxul.so [.tbss]  ETA:  0:11:09 
[task 2021-11-01T10:32:24.458Z]  |##############             |   53%  …/libxul.so [.init_array]  ETA:  0:09:55 
[task 2021-11-01T10:32:24.478Z]  |###############            |   56%  …/libxul.so [.fini_array]  ETA:  0:08:49 
[task 2021-11-01T10:32:27.251Z]  |################           |   59%  …libxul.so [.data.rel.ro]  ETA:  0:07:49 
[task 2021-11-01T10:32:27.280Z]  |################           |   62%  …firefox/libxul.so [.got]  ETA:  0:06:56 
[task 2021-11-01T10:32:27.314Z]  |#################          |   65%  …fox/libxul.so [.got.plt]  ETA:  0:06:07 
[task 2021-11-01T10:32:27.505Z]  |##################         |   68%  …irefox/libxul.so [.data]  ETA:  0:05:22 
[task 2021-11-01T10:32:27.530Z]  |###################        |   70%  …firefox/libxul.so [.bss]  ETA:  0:04:40 
[task 2021-11-01T10:32:27.554Z]  |###################        |   73%  …fox/libxul.so [.comment]  ETA:  0:04:02 
[task 2021-11-01T10:32:27.575Z]  |####################       |   76%  …bxul.so [.gnu_debuglink]  ETA:  0:03:27 
[task 2021-11-01T10:33:01.389Z]  |#####################      |   79%  …efox/libxul.so [.strtab]  ETA:  0:02:54 
[task 2021-11-01T10:33:01.412Z]  |######################     |   82%  …ox/libxul.so [.shstrtab]  ETA:  0:02:31 
[task 2021-11-01T10:33:01.412Z]  |#######################    |   85%                             ETA:  0:02:01 
[task 2021-11-01T10:33:01.424Z]  |#######################    |   85%  …irefox/minidump-analyzer  ETA:  0:02:01 
[task 2021-11-01T10:33:01.453Z]  |#######################    |   86%          a/firefox/omni.ja  ETA:  0:01:53 
[task 2021-11-01T10:33:01.463Z]  |########################## |   99%       a/firefox/pingsender  ETA:  0:00:06 
[task 2021-11-01T10:33:01.471Z]  |########################## |   99%     a/firefox/platform.ini  ETA:  0:00:03 
[task 2021-11-01T10:33:01.479Z]  |########################## |   99%  …firefox/plugin-container  ETA:  0:00:03 
[task 2021-11-01T10:33:01.486Z]  |########################## |   99%      a/firefox/precomplete  ETA:  0:00:00 
[task 2021-11-01T10:33:01.488Z]  |########################## |   99%    a/firefox/removed-files  ETA:  0:00:00 
[task 2021-11-01T10:33:01.494Z]  |########################## |   99%  …efox/update-settings.ini  ETA:  0:00:00 
[task 2021-11-01T10:33:01.498Z]  |########################## |   99%          a/firefox/updater  ETA:  0:00:00 
[task 2021-11-01T10:33:01.501Z]  |########################## |   99%      a/firefox/updater.ini  ETA:  0:00:00 
[task 2021-11-01T10:33:01.501Z]  |###########################|  100%                             ETA:  0:00:00 
[task 2021-11-01T10:33:01.501Z]  |###########################|  100%                             ETA:  0:00:00 
[task 2021-11-01T10:33:01.501Z]  |###########################|  100%                             ETA:  0:00:00 
[task 2021-11-01T10:34:21.592Z]  |###########################|  100%                             Time: 0:11:59 
[task 2021-11-01T10:34:21.592Z] ++ dirname /builds/worker/bin/get_and_diffoscope
[task 2021-11-01T10:34:21.592Z] + /builds/worker/bin/report_error diff
[task 2021-11-01T10:34:22.370Z] TEST-UNEXPECTED-FAIL | firefox/libxul.so differs. See the diff.html or diff.txt artifact

Comment 8

[Intermittent Failures Robot]

18 failures in 495 pushes (0.036 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 18

Platform and build breakdown:

• diff: 18
  • opt: 18

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1736970&startday=2021-11-01&endday=2021-11-01&tree=all

Comment 9

[Mike Hommey [:glandium]]

Finally identified not one, but two sources of regression:

• https://reviews.llvm.org/D91722
• https://reviews.llvm.org/D106585

Now need to determine how to fix it.

Comment 10

[Intermittent Failures Robot]

55 failures in 3030 pushes (0.018 failures/push) were associated with this bug in the last 7 days.

This is the #19 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 55

Platform and build breakdown:

• diff: 55
  • opt: 55

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1736970&startday=2021-11-01&endday=2021-11-07&tree=all

Comment 11

[Mike Hommey [:glandium]]

It turns out that both issues seem to have been fixed on trunk while I was looking for reproducers.
One by https://reviews.llvm.org/rG3c47c5ca13b8a502de3272e8105548715947b7a8 and I'm still bisecting the other.

Comment 12

[Mike Hommey [:glandium]]

So the other one is https://reviews.llvm.org/rGbadcd585897253e94b7b2d4e6f9f430a2020d642, but that only covers the two subcases I had pinned down to the regressors in comment 9, but it turns out there are still cases that aren't fixed on trunk.

Comment 13

[Intermittent Failures Robot]

40 failures in 2723 pushes (0.015 failures/push) were associated with this bug in the last 7 days.

This is the #14 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 40

Platform and build breakdown:

• diff: 40
  • opt: 40

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1736970&startday=2021-11-08&endday=2021-11-14&tree=all

Comment 14

[Intermittent Failures Robot]

38 failures in 3259 pushes (0.012 failures/push) were associated with this bug in the last 7 days.

This is the #19 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 38

Platform and build breakdown:

• diff: 38
  • opt: 38

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1736970&startday=2021-11-15&endday=2021-11-21&tree=all

Comment 15

[Mike Hommey [:glandium]]

Attachment: Bug 1736970 - Apply 2 clang upstream fixes for reproducibility and revert one.

The revert doesn't have a lot of impact, as it's only for debug info
about enums with a 128-bits integer representation.

Comment 16

[Mike Hommey [:glandium]]

The patch is not enough. The LLVM regressions have made it into rust in the meantime, so we'll have to wait for rust 1.57 for the rest of the fix (thankfully, it releases next week), but there will likely be some remaining non-determinism left even after that.

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2e36d2d47679
Apply 2 clang upstream fixes for reproducibility and revert one. r=andi

Comment 18

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/2e36d2d47679

Comment 19

[Intermittent Failures Robot]

41 failures in 2819 pushes (0.015 failures/push) were associated with this bug in the last 7 days.

This is the #25 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 41

Platform and build breakdown:

• diff: 41
  • opt: 41

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1736970&startday=2021-11-22&endday=2021-11-28&tree=all

Comment 20

[Andreea Pavel [:apavel]]

There are 41 total failures in the last 7 days on diff opt
Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=359797037&repo=mozilla-central&lineNumber=174

[task 2021-12-01T06:45:11.948Z] + diffoscope --html diff.html --text diff.txt --progress --output-empty --no-default-limits --max-page-size 100000000 --max-page-diff-block-lines 10000 --exclude-directory-metadata=yes --exclude-command .--line-numbers a b
[task 2021-12-01T06:45:12.062Z]  |                          |    0%                             ETA:  --:--:-- 
[task 2021-12-01T06:45:12.073Z]  |                           |    0%  a/firefox/application.ini  ETA:  0:34:50 
[task 2021-12-01T06:45:12.075Z]  |                           |    0%          a/firefox/browser  ETA:  0:36:28 
[task 2021-12-01T06:45:12.076Z]  |                 |    0%  …ns/default/default16.png  ETA:  309 days, 6:12:19 
[task 2021-12-01T06:45:12.077Z]  |                |    0%  …ns/default/default32.png  ETA:  307 days, 18:56:48 
[task 2021-12-01T06:45:12.078Z]  |                |    0%  …ns/default/default48.png  ETA:  296 days, 23:02:22 
[task 2021-12-01T06:45:12.079Z]  |                 |    0%  …ns/default/default64.png  ETA:  275 days, 0:36:38 
[task 2021-12-01T06:45:12.079Z]  |                |    0%                             ETA:  249 days, 19:38:27 
[task 2021-12-01T06:45:12.079Z]  |                 |    0%                             ETA:  250 days, 3:10:45 
[task 2021-12-01T06:45:12.079Z]  |                |    0%                             ETA:  176 days, 21:09:00 
[task 2021-12-01T06:45:12.082Z]  |                |    0%  …ashreporter-override.ini  ETA:  144 days, 16:26:19 
[task 2021-12-01T06:45:12.084Z]  |                 |    0%  …firefox/browser/features  ETA:  134 days, 2:44:54 
[task 2021-12-01T06:45:12.088Z]  |               |    0%  …[e-mail]  ETA:  5648 days, 12:43:20 
[task 2021-12-01T06:45:12.091Z]  |                |    0%  …[e-mail]  ETA:  850 days, 20:23:11 
[task 2021-12-01T06:45:12.092Z]  |                 |    0%  …[e-mail]  ETA:  799 days, 6:19:35 
[task 2021-12-01T06:45:12.098Z]  |                |    0%  …[e-mail]  ETA:  737 days, 17:22:44 
[task 2021-12-01T06:45:12.100Z]  |                |    0%  …[e-mail]  ETA:  307 days, 17:33:09 
[task 2021-12-01T06:45:12.106Z]  |                 |    0%  …[e-mail]  ETA:  301 days, 3:11:31 
[task 2021-12-01T06:45:12.106Z]  |                |    0%                             ETA:  208 days, 21:34:08 
[task 2021-12-01T06:45:12.140Z]  |                 |    0%  a/firefox/browser/omni.ja  ETA:  209 days, 7:55:59 
[task 2021-12-01T06:45:12.140Z]  |                           |    0%                             ETA:  1:22:11 
[task 2021-12-01T06:45:12.146Z]  |                           |    0%    a/firefox/crashreporter  ETA:  1:22:19 
[task 2021-12-01T06:45:12.150Z]  |                           |    0%  …irefox/crashreporter.ini  ETA:  0:01:45 
[task 2021-12-01T06:45:12.157Z]  |                           |    0%         a/firefox/defaults  ETA:  0:01:48 
[task 2021-12-01T06:45:12.157Z]  |                           |    0%                             ETA:  0:01:52 
[task 2021-12-01T06:45:12.157Z]  |                           |    0%                             ETA:  0:01:52 
[task 2021-12-01T06:45:12.162Z]  |                           |    0%  …refox/dependentlibs.list  ETA:  0:01:52 
[task 2021-12-01T06:45:12.165Z]  |                           |    0%           a/firefox/dmd.py  ETA:  0:01:56 
[task 2021-12-01T06:45:12.166Z]  |                           |    0%          a/firefox/firefox  ETA:  0:01:46 
[task 2021-12-01T06:45:12.177Z]  |                           |    0%      a/firefox/firefox-bin  ETA:  0:01:42 
[task 2021-12-01T06:45:12.180Z]  |                           |    0%    a/firefox/fix_stacks.py  ETA:  0:00:28 
[task 2021-12-01T06:45:12.192Z]  |                           |    0%            a/firefox/fonts  ETA:  0:00:29 
[task 2021-12-01T06:45:12.193Z]  |                           |    0%                             ETA:  0:00:31 
[task 2021-12-01T06:45:12.197Z]  |                           |    0%     a/firefox/gmp-clearkey  ETA:  0:00:31 
[task 2021-12-01T06:45:12.198Z]  |                           |    0%  …earkey/0.1/manifest.json  ETA:  0:00:31 
[task 2021-12-01T06:45:12.198Z]  |                           |    0%                             ETA:  0:00:32 
[task 2021-12-01T06:45:12.198Z]  |                           |    0%                             ETA:  0:00:32 
[task 2021-12-01T06:45:12.200Z]  |                           |    0%            a/firefox/icons  ETA:  0:00:32 
[task 2021-12-01T06:45:12.200Z]  |                           |    0%                             ETA:  0:00:32 
[task 2021-12-01T06:45:12.209Z]  |                           |    0%  …irefox/libfreeblpriv3.so  ETA:  0:00:32 
[task 2021-12-01T06:45:12.210Z]  |                           |    0%   a/firefox/liblgpllibs.so  ETA:  0:00:20 
[task 2021-12-01T06:45:12.221Z]  |                           |    0%  …firefox/libmozavcodec.so  ETA:  0:00:20 
[task 2021-12-01T06:45:12.227Z]  |                           |    1%  a/firefox/libmozavutil.so  ETA:  0:00:10 
[task 2021-12-01T06:45:12.228Z]  |                           |    2%     a/firefox/libmozgtk.so  ETA:  0:00:10 
[task 2021-12-01T06:45:12.232Z]  |                           |    2%  …firefox/libmozsandbox.so  ETA:  0:00:10 
[task 2021-12-01T06:45:12.242Z]  |                           |    2%  …firefox/libmozsqlite3.so  ETA:  0:00:09 
[task 2021-12-01T06:45:12.243Z]  |                           |    2%  …firefox/libmozwayland.so  ETA:  0:00:08 
[task 2021-12-01T06:45:12.248Z]  |                           |    2%      a/firefox/libnspr4.so  ETA:  0:00:08 
[task 2021-12-01T06:45:12.257Z]  |                           |    2%       a/firefox/libnss3.so  ETA:  0:00:08 
[task 2021-12-01T06:45:12.263Z]  |                           |    3%    a/firefox/libnssckbi.so  ETA:  0:00:07 
[task 2021-12-01T06:45:12.267Z]  |                           |    3%   a/firefox/libnssutil3.so  ETA:  0:00:07 
[task 2021-12-01T06:45:12.269Z]  |                           |    3%       a/firefox/libplc4.so  ETA:  0:00:07 
[task 2021-12-01T06:45:12.269Z]  |                           |    3%      a/firefox/libplds4.so  ETA:  0:00:07 
[task 2021-12-01T06:45:12.273Z]  |                           |    3%     a/firefox/libsmime3.so  ETA:  0:00:07 
[task 2021-12-01T06:45:12.279Z]  |                           |    3%   a/firefox/libsoftokn3.so  ETA:  0:00:07 
[task 2021-12-01T06:45:12.285Z]  |                           |    3%       a/firefox/libssl3.so  ETA:  0:00:06 
[task 2021-12-01T06:46:21.260Z]  |#                          |    3%        a/firefox/libxul.so  ETA:  0:00:06 
[task 2021-12-01T06:46:21.588Z]  |#                          |    6%  …ox/libxul.so [.gnu.hash]  ETA:  0:16:00 
[task 2021-12-01T06:46:25.843Z]  |##                         |    9%  …efox/libxul.so [.dynstr]  ETA:  0:10:50 
[task 2021-12-01T06:46:26.980Z]  |###                        |   12%  …ul.so [.elfhack.data.v0]  ETA:  0:08:31 
[task 2021-12-01T06:46:27.324Z]  |####                       |   15%  …ul.so [.elfhack.text.v0]  ETA:  0:06:47 
[task 2021-12-01T06:46:27.671Z]  |####                       |   18%  …irefox/libxul.so [.init]  ETA:  0:05:32 
[task 2021-12-01T06:46:28.256Z]  |#####                      |   21%  …firefox/libxul.so [.plt]  ETA:  0:04:37 
[task 2021-12-01T06:46:28.634Z]  |######                     |   24%  …fox/libxul.so [.plt.got]  ETA:  0:03:56 
[task 2021-12-01T06:53:23.401Z]  |#######                    |   27%  …irefox/libxul.so [.text]  ETA:  0:03:24 
[task 2021-12-01T06:53:23.832Z]  |########                   |   30%  …fox/libxul.so [text_env]  ETA:  0:18:53 
[task 2021-12-01T06:53:24.138Z]  |########                   |   33%  …irefox/libxul.so [.fini]  ETA:  0:16:30 
[task 2021-12-01T06:53:47.680Z]  |#########                  |   36%  …efox/libxul.so [.rodata]  ETA:  0:14:30 
[task 2021-12-01T06:53:50.566Z]  |##########                 |   39%  …ibxul.so [.eh_frame_hdr]  ETA:  0:13:24 
[task 2021-12-01T06:54:02.631Z]  |###########                |   41%  …ox/libxul.so [.eh_frame]  ETA:  0:11:56 
[task 2021-12-01T06:54:02.679Z]  |############               |   44%  …l.so [.gcc_except_table]  ETA:  0:10:50 
[task 2021-12-01T06:54:02.709Z]  |############               |   47%  …refox/libxul.so [.tdata]  ETA:  0:09:38 
[task 2021-12-01T06:54:02.737Z]  |#############              |   50%  …irefox/libxul.so [.tbss]  ETA:  0:08:33 
[task 2021-12-01T06:54:02.767Z]  |##############             |   53%  …/libxul.so [.init_array]  ETA:  0:07:36 
[task 2021-12-01T06:54:02.795Z]  |###############            |   56%  …/libxul.so [.fini_array]  ETA:  0:06:45 
[task 2021-12-01T06:54:07.286Z]  |################           |   59%  …libxul.so [.data.rel.ro]  ETA:  0:05:59 
[task 2021-12-01T06:54:07.408Z]  |################           |   62%  …firefox/libxul.so [.got]  ETA:  0:05:20 
[task 2021-12-01T06:54:07.462Z]  |#################          |   65%  …fox/libxul.so [.got.plt]  ETA:  0:04:42 
[task 2021-12-01T06:54:07.744Z]  |##################         |   68%  …irefox/libxul.so [.data]  ETA:  0:04:06 
[task 2021-12-01T06:54:07.775Z]  |###################        |   71%  …firefox/libxul.so [.bss]  ETA:  0:03:34 
[task 2021-12-01T06:54:07.807Z]  |####################       |   74%  …fox/libxul.so [.comment]  ETA:  0:03:05 
[task 2021-12-01T06:54:07.839Z]  |####################       |   77%  …bxul.so [.gnu_debuglink]  ETA:  0:02:37 
[task 2021-12-01T06:54:39.228Z]  |#####################      |   80%  …efox/libxul.so [.strtab]  ETA:  0:02:12 
[task 2021-12-01T06:54:39.263Z]  |######################     |   83%  …ox/libxul.so [.shstrtab]  ETA:  0:01:55 
[task 2021-12-01T06:54:39.263Z]  |#######################    |   86%                             ETA:  0:01:31 
[task 2021-12-01T06:54:39.277Z]  |#######################    |   86%  …irefox/minidump-analyzer  ETA:  0:01:31 
[task 2021-12-01T06:54:39.307Z]  |#######################    |   86%          a/firefox/omni.ja  ETA:  0:01:25 
[task 2021-12-01T06:54:39.320Z]  |########################## |   99%       a/firefox/pingsender  ETA:  0:00:05 
[task 2021-12-01T06:54:39.328Z]  |########################## |   99%     a/firefox/platform.ini  ETA:  0:00:02 
[task 2021-12-01T06:54:39.340Z]  |########################## |   99%  …firefox/plugin-container  ETA:  0:00:02 
[task 2021-12-01T06:54:39.347Z]  |########################## |   99%      a/firefox/precomplete  ETA:  0:00:00 
[task 2021-12-01T06:54:39.349Z]  |########################## |   99%    a/firefox/removed-files  ETA:  0:00:00 
[task 2021-12-01T06:54:39.355Z]  |########################## |   99%  …efox/update-settings.ini  ETA:  0:00:00 
[task 2021-12-01T06:54:39.363Z]  |########################## |   99%          a/firefox/updater  ETA:  0:00:00 
[task 2021-12-01T06:54:39.366Z]  |########################## |   99%      a/firefox/updater.ini  ETA:  0:00:00 
[task 2021-12-01T06:54:39.366Z]  |###########################|  100%                             ETA:  0:00:00 
[task 2021-12-01T06:54:39.366Z]  |###########################|  100%                             ETA:  0:00:00 
[task 2021-12-01T06:54:39.366Z]  |###########################|  100%                             ETA:  0:00:00 
[task 2021-12-01T06:54:45.557Z]  |###########################|  100%                             Time: 0:09:27 
[task 2021-12-01T06:54:45.557Z] ++ dirname /builds/worker/bin/get_and_diffoscope
[task 2021-12-01T06:54:45.558Z] + /builds/worker/bin/report_error diff
[task 2021-12-01T06:54:45.834Z] TEST-UNEXPECTED-FAIL | firefox/libxul.so differs. See the diff.html or diff.txt artifact
[task 2021-12-01T06:54:45.836Z] + fail
[task 2021-12-01T06:54:45.836Z] + exit 1
[taskcluster 2021-12-01 06:54:46.181Z] === Task Finished ===
[taskcluster 2021-12-01 06:54:48.426Z] Unsuccessful task run with exit code: 1 completed in 601.376 seconds

Comment 21

[Mike Hommey [:glandium]]

(In reply to Mike Hommey [:glandium] from comment #16)

The patch is not enough. The LLVM regressions have made it into rust in the meantime, so we'll have to wait for rust 1.57 for the rest of the fix (thankfully, it releases next week), but there will likely be some remaining non-determinism left even after that.

Forgot to add this comment: it turns out the remaining non-determinismt goes away when PGO is enabled, so this will go away after bug 1742749 lands after rustc 1.57.0 is released officially later this week.

Comment 22

[Mike Hommey [:glandium]]

Confirmed fixed: https://treeherder.mozilla.org/jobs?repo=mozilla-central&searchStr=diff%2Clinux&revision=356c521826254fbae435b212923b20801d171950

Comment 23

[Intermittent Failures Robot]

24 failures in 3170 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

This is the #47 most frequent failure this week.

Repository breakdown:

• mozilla-central: 24

Platform and build breakdown:

• diff: 24
  • opt: 24

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1736970&startday=2021-11-29&endday=2021-12-05&tree=all