1760130 - Intermittent toolkit/components/nimbus/test/unit/test_ExperimentManager_enroll.js | single tracking bug

Status: RESOLVED FIXED

(Firefox :: Nimbus Desktop Client, defect, P3)

Comment 1

[Intermittent Failures Robot]

1 failures in 3341 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• linux1804-64-tsan-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1760130&startday=2022-03-21&endday=2022-03-27&tree=all

Comment 2

[Intermittent Failures Robot]

21 failures in 3317 pushes (0.006 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 5
• mozilla-central: 7
• try: 9

Platform and build breakdown:

• macosx1015-64-qr: 1
  • debug: 1
• windows10-64-2004-qr: 9
  • opt: 9
• windows10-32-2004-qr: 1
  • opt: 1
• windows10-64-2004-ccov-qr: 2
  • opt: 2
• windows10-64-2004-shippable-qr: 6
  • opt: 6
• windows10-64-2004-devedition-qr: 1
  • opt: 1
• windows7-32-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1760130&startday=2022-05-09&endday=2022-05-15&tree=all

Comment 3

[Intermittent Failures Robot]

27 failures in 667 pushes (0.04 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 24
• mozilla-central: 3

Platform and build breakdown:

• macosx1015-64-qr: 1
  • debug: 1
• windows10-64-2004-qr: 16
  • opt: 15
  • debug: 1
• windows10-32-2004-qr: 1
  • opt: 1
• windows10-64-2004-shippable-qr: 6
  • opt: 6
• windows7-32-shippable-qr: 2
  • opt: 2
• windows10-64-2004-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1760130&startday=2022-05-17&endday=2022-05-17&tree=all

Comment 4

[Joel Maher ( :jmaher ) (UTC -0800)]

./mach test-info failure-report --bugid 1760130
55 errors with:
15:26:03  WARNING -  TEST-UNEXPECTED-FAIL | toolkit/components/nimbus/test/unit/test_ExperimentManager_enroll.js | xpcshell return code: 0
15:26:03  WARNING -  TEST-UNEXPECTED-FAIL | toolkit/components/nimbus/test/unit/test_ExperimentManager_enroll.js | experiment_and_rollout_enroll_and_cleanup - [experiment_and_rollout_enroll_and_cleanup : 271] A promise chain failed to handle a rejection: Win error 32 during operation removeEmptyDir on file C:\\Users\\task_165288474324967\\AppData\\Local\\Temp\\testdir-690246 (The process cannot access the file because it is being used by another process.

  windows10-64-2004-shippable-qr/opt-xpcshell-spi-nw-fis: 15
  windows10-64-2004-qr/opt-xpcshell-spi-nw-fis: 16
  windows10-64-2004-qr/opt-xpcshell-fis: 10
  windows7-32-shippable-qr/opt-xpcshell-spi-nw-fis: 3
  windows10-64-2004-shippable-qr/opt-xpcshell-fis: 5
  windows7-32-qr/opt-xpcshell-fis: 1
  windows10-64-2004-ccov-qr/opt-xpcshell-fis: 1
  windows10-64-2004-ccov-qr/opt-xpcshell-spi-nw-fis: 2
  windows10-64-2004-qr/debug-test-verify-fis: 1
  windows10-64-2004-ccov-qr/opt-test-coverage-fis: 1
...

Comment 5

[Natalia Csoregi [:nataliaCs]]

Over 60 failures since yesterday: https://treeherder.mozilla.org/intermittent-failures/bugdetails?startday=2022-05-12&endday=2022-05-19&tree=trunk&bug=1760130 and it is permafailing: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&selectedTaskRun=d26iD5pVRVabln6j5Soaeg.0&revision=db80146ff3798a4df8658db09e49793737699b38&searchStr=windows%2C10%2Cx64%2C2004%2Cwebrender%2Copt%2Cxpcshell%2Ctests%2Cwith%2Cfission%2Cenabled%2Ctest-windows10-64-2004-qr%2Fopt-xpcshell-fis-e10s%2Cx2

Travis, can you take a look at this?
Thank you.

Comment 6

[Release mgmt bot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1754527

Comment 7

[Intermittent Failures Robot]

36 failures in 585 pushes (0.062 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 26
• mozilla-central: 8
• try: 2

Platform and build breakdown:

• windows7-32-qr: 1
  • opt: 1
• windows10-64-2004-qr: 18
  • opt: 18
• windows10-64-2004-ccov-qr: 1
  • opt: 1
• windows10-64-2004-shippable-qr: 14
  • opt: 14
• windows7-32-shippable-qr: 1
  • opt: 1
• windows10-64-2004-devedition-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1760130&startday=2022-05-18&endday=2022-05-18&tree=all

Comment 8

[Travis Long [:travis_]]

Oops, sorry. I thought I had this resolved (or so my try-pushes seemed to indicate). Yes, will look into this today, thanks for flagging me.

Comment 9

[Intermittent Failures Robot]

46 failures in 465 pushes (0.099 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 37
• mozilla-central: 6
• try: 3

Platform and build breakdown:

• windows10-64-2004-qr: 27
  • opt: 27
• windows10-64-2004-shippable-qr: 15
  • opt: 15
• windows10-64-2004-devedition-qr: 1
  • opt: 1
• windows7-32-shippable-qr: 2
  • opt: 2
• macosx1015-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1760130&startday=2022-05-19&endday=2022-05-19&tree=all

Comment 10

[Intermittent Failures Robot]

44 failures in 299 pushes (0.147 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 31
• mozilla-central: 7
• try: 6

Platform and build breakdown:

• windows10-64-2004-qr: 18
  • opt: 18
• windows7-32-shippable-qr: 4
  • opt: 4
• windows10-64-2004-shippable-qr: 16
  • opt: 16
• windows10-64-2004-ccov-qr: 1
  • opt: 1
• windows10-64-2004-devedition-qr: 2
  • opt: 2
• windows7-32-qr: 2
  • opt: 2
• windows10-32-2004-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1760130&startday=2022-05-20&endday=2022-05-20&tree=all

Comment 11

[Intermittent Failures Robot]

22 failures in 66 pushes (0.333 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 13
• mozilla-central: 6
• try: 3

Platform and build breakdown:

• windows10-64-2004-qr: 8
  • opt: 8
• windows10-64-2004-shippable-qr: 8
  • opt: 8
• windows7-32-shippable-qr: 3
  • opt: 3
• windows10-64-2004-devedition-qr: 1
  • opt: 1
• windows10-64-2004-ccov-qr: 1
  • opt: 1
• windows7-32-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1760130&startday=2022-05-21&endday=2022-05-21&tree=all

Comment 12

[Narcis Beleuzu [:NarcisB]]

Attachment: Bug 1760130 - Disable test_ExperimentManager_enroll.js on Win10. r?#intermittent-reviewers

Comment 13

[Narcis Beleuzu [:NarcisB]]

There are 170 failures in the last 7 days. Most of them (155) are on Win10.
Travis, I added a patch to disable the test on Win10 if it will take a little bit longer to fix it

Comment 14

[Intermittent Failures Robot]

195 failures in 2694 pushes (0.072 failures/push) were associated with this bug in the last 7 days.

This is the #4 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 141
• mozilla-central: 37
• try: 17

Platform and build breakdown:

• macosx1015-64-qr: 1
  • debug: 1
• windows10-64-2004-qr: 93
  • opt: 92
  • debug: 1
• windows10-32-2004-qr: 2
  • opt: 1
  • debug: 1
• windows10-64-2004-shippable-qr: 67
  • opt: 67
• windows7-32-shippable-qr: 14
  • opt: 14
• windows10-64-2004-ccov-qr: 6
  • opt: 6
• windows7-32-qr: 5
  • opt: 5
• windows10-64-2004-devedition-qr: 6
  • opt: 6
• macosx1015-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1760130&startday=2022-05-16&endday=2022-05-22&tree=all

Comment 15

[Intermittent Failures Robot]

34 failures in 348 pushes (0.098 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 21
• mozilla-central: 8
• try: 5

Platform and build breakdown:

• windows10-64-2004-qr: 14
  • opt: 14
• windows10-64-2004-shippable-qr: 14
  • opt: 14
• windows7-32-shippable-qr: 3
  • opt: 3
• windows10-64-2004-devedition-qr: 1
  • opt: 1
• windows10-64-2004-ccov-qr: 2
  • opt: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1760130&startday=2022-05-23&endday=2022-05-23&tree=all

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7c8213abea78
Disable test_ExperimentManager_enroll.js on windows 10 opt. r=MasterWayZ DONTBUILD

Comment 17

[Sandor Molnar]

https://hg.mozilla.org/mozilla-central/rev/7c8213abea78

Comment 18

[Intermittent Failures Robot]

61 failures in 2624 pushes (0.023 failures/push) were associated with this bug in the last 7 days.

This is the #9 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 33
• mozilla-central: 15
• try: 13

Platform and build breakdown:

• windows10-64-2004-qr: 16
  • opt: 16
• windows10-64-2004-shippable-qr: 21
  • opt: 21
• windows7-32-shippable-qr: 14
  • opt: 14
• windows10-64-2004-devedition-qr: 3
  • opt: 3
• windows10-64-2004-ccov-qr: 2
  • opt: 2
• windows7-32-qr: 5
  • opt: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1760130&startday=2022-05-23&endday=2022-05-29&tree=all

Comment 19

[Andreea Pavel [:apavel]]

This is now intermittently failing on windows7 32 bits, opt.

Comment 20

[Travis Long [:travis_]]

I'm at a bit of a loss at what is triggering this, :barret, do you have time to take a look at this and give me some advice on what might be happening here? My best guess is that the do_get_profile() is the source of this but I don't have much idea as to why.

Comment 21

[Barret Rennie [:barret] (they/them)]

This was caused by a race condition where JSONFile and TestFileUtils were trying to run their shutdown blocker simultaneously, which lead to attempting to blow away the directory that a JSONFile was saved in while it was saving.

Moving TestFileUtils to clear its file at OS.File.shutdown (instead of profileBeforeChange) fixes this issue.

Comment 22

[Barret Rennie [:barret] (they/them)]

Attachment: Bug 1760130 - Run FileTestUtils cleanup at the latest possible phase r?mak

Comment 23

[Release mgmt bot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1754527

Comment 24

[Barret Rennie [:barret] (they/them)]

Attachment: Bug 1760130 - Re-enable test_ExperimentManager_enroll.js on windows on Windows 10 r?emcminn

Depends on D148147

Comment 25

[Intermittent Failures Robot]

41 failures in 2917 pushes (0.014 failures/push) were associated with this bug in the last 7 days.

This is the #22 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 23
• mozilla-beta: 10
• mozilla-central: 5
• try: 2
• mozilla-esr102: 1

Platform and build breakdown:

• windows7-32-qr: 8
  • opt: 8
• windows7-32-shippable-qr: 30
  • opt: 30
• windows10-64-2004-qr: 1
  • debug: 1
• windows10-32-2004-qr: 1
  • debug: 1
• linux1804-64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1760130&startday=2022-05-30&endday=2022-06-05&tree=all

Comment 26

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5562cae92eb5
Run FileTestUtils cleanup at the latest possible phase r=mak
https://hg.mozilla.org/integration/autoland/rev/95e88f6bd6b1
Re-enable test_ExperimentManager_enroll.js on windows on Windows 10 r=emcminn

Comment 27

[Sandor Molnar]

https://hg.mozilla.org/mozilla-central/rev/5562cae92eb5
https://hg.mozilla.org/mozilla-central/rev/95e88f6bd6b1

Comment 28

[Release mgmt bot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:barret, is this bug important enough to require an uplift?
If not please set status_beta to wontfix.

For more information, please visit auto_nag documentation.

Comment 29

[Barret Rennie [:barret] (they/them)]

This is just a test, its not crucial to uplift to beta.

Comment 30

[Intermittent Failures Robot]

27 failures in 3336 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 6
• mozilla-central: 11
• try: 2
• mozilla-beta: 8

Platform and build breakdown:

• windows7-32-qr: 2
  • opt: 2
• windows7-32-shippable-qr: 24
  • opt: 24
• windows10-32-2004-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1760130&startday=2022-06-06&endday=2022-06-12&tree=all

Comment 31

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/releases/mozilla-beta/rev/3a5af2c71e02
https://hg.mozilla.org/releases/mozilla-beta/rev/e23487912d7c

Comment 32

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Uplifted to 102 because this version has to be supported for a year as ESR.