1851714 - Perma firefox/modules/backgroundtasks differs. See the diff.html or diff.txt artifact

Status: RESOLVED FIXED

(Toolkit :: Default Browser Agent, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: csabou [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=428067182&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/FdXVPwSaTIyphwSk_SZX5A/runs/0/artifacts/public/logs/live_backing.log

---

https://firefoxci.taskcluster-artifacts.net/FdXVPwSaTIyphwSk_SZX5A/0/public/diff.html

[task 2023-09-05T22:28:35.878Z]  |                          |    0%  …-remove-column-hover.gif  ETA:  --:--:-- 
[task 2023-09-05T22:28:35.880Z]  |                          |    0%  …/table-remove-column.gif  ETA:  --:--:-- 
[task 2023-09-05T22:28:35.881Z]  |                          |    0%  …le-remove-row-active.gif  ETA:  --:--:-- 
[task 2023-09-05T22:28:35.882Z]  |                          |    0%  …ble-remove-row-hover.gif  ETA:  --:--:-- 
[task 2023-09-05T22:28:35.884Z]  |                          |    0%  …res/table-remove-row.gif  ETA:  --:--:-- 
[task 2023-09-05T22:28:35.884Z]  |                          |    0%                             ETA:  --:--:-- 
[task 2023-09-05T22:28:35.887Z]  |                          |    0%     b/firefox/softokn3.dll  ETA:  --:--:-- 
[task 2023-09-05T22:28:35.887Z]  |                          |    0%        b/firefox/uninstall  ETA:  --:--:-- 
[task 2023-09-05T22:28:35.887Z]  |                          |    0%                             ETA:  --:--:-- 
[task 2023-09-05T22:28:35.889Z]  |                          |    0%  …efox/update-settings.ini  ETA:  --:--:-- 
[task 2023-09-05T22:28:35.891Z]  |                          |    0%    b/firefox/update.locale  ETA:  --:--:-- 
[task 2023-09-05T22:28:35.894Z]  |                          |    0%      b/firefox/updater.exe  ETA:  --:--:-- 
[task 2023-09-05T22:28:35.897Z]  |                          |    0%      b/firefox/updater.ini  ETA:  --:--:-- 
[task 2023-09-05T22:28:35.900Z]  |                          |    0%  …firefox/vcruntime140.dll  ETA:  --:--:-- 
[task 2023-09-05T22:28:35.902Z]  |                          |    0%  …refox/vcruntime140_1.dll  ETA:  --:--:-- 
[task 2023-09-05T22:28:35.988Z]  |                          |    0%          b/firefox/xul.dll  ETA:  --:--:-- 
[task 2023-09-05T22:28:35.988Z]  |                          |    0%                             ETA:  --:--:-- 
[task 2023-09-05T22:28:35.988Z]  |                          |    0%                             ETA:  --:--:-- 
[task 2023-09-05T22:28:35.988Z]  |                          |    0%                             ETA:  --:--:-- 
[task 2023-09-05T22:28:36.023Z]  |###########################|  100%                             Time: 0:00:15 
[task 2023-09-05T22:28:36.023Z] ++ dirname /builds/worker/bin/get_and_diffoscope
[task 2023-09-05T22:28:36.024Z] + /builds/worker/bin/report_error diff
[task 2023-09-05T22:28:36.034Z] TEST-UNEXPECTED-FAIL | firefox/modules/backgroundtasks differs. See the diff.html or diff.txt artifact
[task 2023-09-05T22:28:36.034Z] TEST-UNEXPECTED-FAIL | firefox/precomplete differs. See the diff.html or diff.txt artifact
[task 2023-09-05T22:28:36.036Z] + fail
[task 2023-09-05T22:28:36.037Z] + exit 1
[taskcluster 2023-09-05 22:28:37.512Z] === Task Finished ===
[taskcluster 2023-09-05 22:28:37.839Z] Unsuccessful task run with exit code: 1 completed in 60.691 seconds

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

:nrishel, since you are the author of the regressor, bug 1838754, could you take a look?

For more information, please visit BugBot documentation.

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1838754

Comment 3

[Intermittent Failures Robot]

15 failures in 724 pushes (0.021 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 12
• mozilla-central: 3

Platform and build breakdown:

• diff: 15
  • opt: 15

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1851714&startday=2023-09-06&endday=2023-09-06&tree=all

Comment 4

[Nick Rishel [:nrishel]]

:nalexander Not sure what I'm looking at here, new files added causing a test fail?

Comment 5

[Nick Alexander :nalexander [he/him] (PTO Feb 24-March 3)]

glandium: this rather looks like the artifacts fetched for the task are split around the landing or backout of Bug 1838754. Is it possible that the artifact packaging around EME artifacts grabbed an outdated build, leading to this error?

Can you investigate further?

Comment 6

[Mike Hommey [:glandium]]

Those builds are not using the "get lucky with artifacts download" code path, they actually give the explicit task that contains the artifacts via MOZ_ARTIFACT_TASK_WIN32_OPT, and that task is from the same push.

Comment 7

[Mike Hommey [:glandium]]

Essentially, this is a "artifact build doesn't produce the same thing as non-artifact build" problem.

Comment 8

[Nick Alexander :nalexander [he/him] (PTO Feb 24-March 3)]

(In reply to Mike Hommey [:glandium] from comment #7)

Essentially, this is a "artifact build doesn't produce the same thing as non-artifact build" problem.

oic. From https://firefoxci.taskcluster-artifacts.net/FdXVPwSaTIyphwSk_SZX5A/0/public/diff.html we're not shipping defaultagent.ini and BackgroundTask_defaultagent.sys.mjs in artifact builds.

And indeed, https://searchfox.org/mozilla-central/rev/146638366864f73ee8a697ea76480eb02c00eb3c/toolkit/moz.build#27-36 made this require a compile environment.

The action here is to weaken this to exclude some compilation pieces when !mingw rather than only allow clang, I think. That is, we still want the localized strings and the background task JS, even when Firefox is an artifact build.

nrishel: back to you to address this. (I can, of course, answer questions as needed.)

Comment 9

[Intermittent Failures Robot]

15 failures in 682 pushes (0.022 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 4
• autoland: 11

Platform and build breakdown:

• diff: 15
  • opt: 15

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1851714&startday=2023-09-08&endday=2023-09-08&tree=all

Comment 10

[Intermittent Failures Robot]

60 failures in 3267 pushes (0.018 failures/push) were associated with this bug in the last 7 days.

This is the #19 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 46
• mozilla-central: 14

Platform and build breakdown:

• diff: 60
  • opt: 60

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1851714&startday=2023-09-04&endday=2023-09-10&tree=all

Comment 11

[Intermittent Failures Robot]

18 failures in 559 pushes (0.032 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-central: 4
• autoland: 12
• try: 2

Platform and build breakdown:

• diff: 18
  • opt: 18

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1851714&startday=2023-09-12&endday=2023-09-12&tree=all

Comment 12

[Nick Rishel [:nrishel]]

I had referenced CONFIG['CC_TYPE'] != 'clang-cl' and previously found CONFIG["CC_TYPE"] != "gcc" had build errors, so what's the correct flag mixture here? Referencing moz.build files isn't fruitful due to inconsistency. I also see not CONFIG["CC_TYPE"] in ("gcc", "clang") in source?

Comment 13

[Mike Hommey [:glandium]]

There is no CC_TYPE when building artifact builds. You can detect artifact builds with MOZ_ARTIFACT_BUILDS.

Comment 14

[Intermittent Failures Robot]

16 failures in 697 pushes (0.023 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 13
• mozilla-central: 3

Platform and build breakdown:

• diff: 16
  • opt: 16

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1851714&startday=2023-09-15&endday=2023-09-15&tree=all

Comment 15

[Intermittent Failures Robot]

88 failures in 3337 pushes (0.026 failures/push) were associated with this bug in the last 7 days.

This is the #9 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 65
• mozilla-central: 21
• try: 2

Platform and build breakdown:

• diff: 88
  • opt: 88

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1851714&startday=2023-09-11&endday=2023-09-17&tree=all

Comment 16

[Intermittent Failures Robot]

17 failures in 822 pushes (0.021 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 14
• mozilla-central: 3

Platform and build breakdown:

• diff: 17
  • opt: 17

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1851714&startday=2023-09-19&endday=2023-09-19&tree=all

Comment 17

[Nick Rishel [:nrishel]]

Attachment: Bug 1851714 - Don't exclude Default Agent directory from artifact builds. r=glandium

This is necessary to bundle non-compiled files from the Default Agent, essentially .sys.mjs and .ini files.

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/efc3444da9bb
Don't exclude Default Agent directory from artifact builds. r=glandium

Comment 19

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/efc3444da9bb

Comment 20

[Intermittent Failures Robot]

41 failures in 3850 pushes (0.011 failures/push) were associated with this bug in the last 7 days.

This is the #23 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 32
• mozilla-central: 9

Platform and build breakdown:

• diff: 41
  • opt: 41

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1851714&startday=2023-09-18&endday=2023-09-24&tree=all

Comment 21

[Nick Rishel [:nrishel]]

Attachment: Backed out changeset efc3444da9bb (Bug 1851714)

Backed out due to changes breaking both Default Agent telemetry and launching Firefox in response to interaction with the Default Agent notifications.

Comment 22

[Donal Meehan [:dmeehan]]

Backed out of beta for Fx119
https://hg.mozilla.org/releases/mozilla-beta/rev/0bc85fc81651313ad0c87d04c23b813a947268d4

Setting Fx119 to disabled, the regressor Bug 1838754 was backed out of beta for Fx119