1638027 - Intermittent leakcheck | tab 248 bytes leaked (Mutex, PerformanceStorageWorker, WorkerRef)

Status: RESOLVED INVALID

(Core :: Performance, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: btara [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=302287693&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Ionqj6UqQTKwxckYg2srYA/runs/0/artifacts/public/logs/live_backing.log

---

...
[task 2020-05-14T15:55:11.008Z] 15:55:11     INFO - TEST-PASS | leakcheck | tab no leaks detected!
[task 2020-05-14T15:55:11.008Z] 15:55:11     INFO - leakcheck | Processing leak log file /tmp/tmp88Z3M0.mozrunner/runtests_leaks_tab_pid18064.log
[task 2020-05-14T15:55:11.009Z] 15:55:11     INFO - 
[task 2020-05-14T15:55:11.010Z] 15:55:11     INFO - == BloatView: ALL (cumulative) LEAK AND BLOAT STATISTICS, tab process 18064
[task 2020-05-14T15:55:11.010Z] 15:55:11     INFO - 
[task 2020-05-14T15:55:11.011Z] 15:55:11     INFO -      |<----------------Class--------------->|<-----Bytes------>|<----Objects---->|
[task 2020-05-14T15:55:11.011Z] 15:55:11     INFO -      |                                      | Per-Inst   Leaked|   Total      Rem|
[task 2020-05-14T15:55:11.012Z] 15:55:11     INFO -    0 |TOTAL                                 |       36      248|   55310        3|
[task 2020-05-14T15:55:11.013Z] 15:55:11     INFO -  256 |Mutex                                 |       72       72|     447        1|
[task 2020-05-14T15:55:11.013Z] 15:55:11     INFO -  388 |PerformanceStorageWorker              |       96       96|       2        1|
[task 2020-05-14T15:55:11.014Z] 15:55:11     INFO -  579 |WorkerRef                             |       80       80|      10        1|
[task 2020-05-14T15:55:11.016Z] 15:55:11     INFO - 
[task 2020-05-14T15:55:11.016Z] 15:55:11     INFO - nsTraceRefcnt::DumpStatistics: 943 entries
[task 2020-05-14T15:55:11.017Z] 15:55:11     INFO - TEST-INFO | leakcheck | tab leaked 1 Mutex
[task 2020-05-14T15:55:11.017Z] 15:55:11     INFO - TEST-INFO | leakcheck | tab leaked 1 PerformanceStorageWorker
[task 2020-05-14T15:55:11.018Z] 15:55:11     INFO - TEST-INFO | leakcheck | tab leaked 1 WorkerRef
[task 2020-05-14T15:55:11.018Z] 15:55:11     INFO - TEST-UNEXPECTED-FAIL | leakcheck | tab 248 bytes leaked (Mutex, PerformanceStorageWorker, WorkerRef)
[task 2020-05-14T15:55:11.018Z] 15:55:11     INFO - 
[task 2020-05-14T15:55:11.019Z] 15:55:11     INFO - leakcheck | Processing leak log file /tmp/tmp88Z3M0.mozrunner/runtests_leaks_tab_pid18037.log
...

Comment 2

[Intermittent Failures Robot]

16 failures in 1010 pushes (0.016 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 15
• mozilla-central: 1

Platform breakdown:

• linux1804-64: 16

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1638027&startday=2020-05-15&endday=2020-05-15&tree=all

Comment 3

[Bogdan Tara[:bogdan_tara | bogdant]]

This seems to have started with the changes on bug 1634259.

https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=linux%2C18.04%2Cx64%2Cdebug%2Cmochitests%2Cwith%2Cfission%2Cenabled%2Ctest-linux1804-64%2Fdebug-mochitest-browser-chrome-fis-e10s&tochange=f049c6635dc22894fc430664ea01e7dbb31b70d2&fromchange=c6808d59644cf49ca9e1f97815b616e18af33d71

asuth could you, please, take a look?

Comment 4

[Intermittent Failures Robot]

47 failures in 5488 pushes (0.009 failures/push) were associated with this bug in the last 7 days.

This is the #32 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 42
• mozilla-central: 5

Platform breakdown:

• linux1804-64: 47

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1638027&startday=2020-05-11&endday=2020-05-17&tree=all

Comment 5

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1634259

Comment 6

[Intermittent Failures Robot]

26 failures in 718 pushes (0.036 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 19
• mozilla-central: 4
• try: 3

Platform breakdown:

• linux1804-64: 26

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1638027&startday=2020-05-18&endday=2020-05-18&tree=all

Comment 7

[Intermittent Failures Robot]

27 failures in 1049 pushes (0.026 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 26
• try: 1

Platform breakdown:

• linux1804-64: 27

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1638027&startday=2020-05-19&endday=2020-05-19&tree=all

Comment 8

[Intermittent Failures Robot]

33 failures in 959 pushes (0.034 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 31
• mozilla-central: 2

Platform breakdown:

• linux1804-64: 33

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1638027&startday=2020-05-20&endday=2020-05-20&tree=all

Comment 9

[Intermittent Failures Robot]

22 failures in 759 pushes (0.029 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 21
• mozilla-central: 1

Platform breakdown:

• linux1804-64: 22

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1638027&startday=2020-05-21&endday=2020-05-21&tree=all

Comment 10

[Andreea Pavel [:apavel]]

Attachment: Bug 1638027 - Disabled browser_ext_windows_size.js, browser_readerMode_readingTime.js, browser_showMenu.js on linux18.04 debug r=jmaher DONTBUILD

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/37ab439d2e63
Disabled browser_ext_windows_size.js, browser_readerMode_readingTime.js, browser_showMenu.js on linux18.04 debug r=jmaher DONTBUILD

Comment 12

[Intermittent Failures Robot]

28 failures in 832 pushes (0.034 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 22
• mozilla-central: 3
• try: 3

Platform breakdown:

• linux1804-64: 28

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1638027&startday=2020-05-22&endday=2020-05-22&tree=all

Comment 13

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/37ab439d2e63

Comment 14

[Narcis Beleuzu [:NarcisB]]

Attachment: Bug 1638027 - Disable browser_readerMode_pocket.js on linux18.04 debug r?jmaher DONTBUILD

Comment 15

[Intermittent Failures Robot]

150 failures in 4727 pushes (0.032 failures/push) were associated with this bug in the last 7 days.

This is the #4 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 132
• mozilla-central: 11
• try: 7

Platform breakdown:

• linux1804-64: 149
• linux1804-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1638027&startday=2020-05-18&endday=2020-05-24&tree=all

Comment 16

[Intermittent Failures Robot]

33 failures in 710 pushes (0.046 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 30
• mozilla-central: 2
• try: 1

Platform breakdown:

• linux1804-64: 33

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1638027&startday=2020-05-25&endday=2020-05-25&tree=all

Comment 17

[Intermittent Failures Robot]

32 failures in 1025 pushes (0.031 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 25
• mozilla-central: 2
• try: 5

Platform breakdown:

• linux1804-64: 32

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1638027&startday=2020-05-26&endday=2020-05-26&tree=all

Comment 18

[Intermittent Failures Robot]

33 failures in 1215 pushes (0.027 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 28
• mozilla-central: 5

Platform breakdown:

• linux1804-64: 33

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1638027&startday=2020-05-27&endday=2020-05-27&tree=all

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b25757fea03b
Disable browser_readerMode_pocket.js on linux18.04 debug r=jmaher DONTBUILD

Comment 20

[Intermittent Failures Robot]

20 failures in 1048 pushes (0.019 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 13
• mozilla-central: 4
• try: 3

Platform breakdown:

• linux1804-64: 20

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1638027&startday=2020-05-28&endday=2020-05-28&tree=all

Comment 21

[Raul Gurzau (:RaulG)]

https://hg.mozilla.org/mozilla-central/rev/b25757fea03b

Comment 22

[Intermittent Failures Robot]

28 failures in 1051 pushes (0.027 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 26
• mozilla-central: 2

Platform breakdown:

• linux1804-64: 28

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1638027&startday=2020-05-29&endday=2020-05-29&tree=all

Comment 23

[Intermittent Failures Robot]

171 failures in 5458 pushes (0.031 failures/push) were associated with this bug in the last 7 days.

This is the #1 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 139
• mozilla-central: 19
• try: 13

Platform breakdown:

• linux1804-64: 171

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1638027&startday=2020-05-25&endday=2020-05-31&tree=all

Comment 24

[Intermittent Failures Robot]

27 failures in 722 pushes (0.037 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 24
• mozilla-central: 1
• try: 2

Platform breakdown:

• linux1804-64: 27

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1638027&startday=2020-06-01&endday=2020-06-01&tree=all

Comment 25

[Intermittent Failures Robot]

24 failures in 1092 pushes (0.022 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 21
• mozilla-central: 2
• try: 1

Platform breakdown:

• linux1804-64: 24

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1638027&startday=2020-06-02&endday=2020-06-02&tree=all

Comment 26

[Intermittent Failures Robot]

20 failures in 993 pushes (0.02 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 18
• mozilla-central: 2

Platform breakdown:

• linux1804-64: 20

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1638027&startday=2020-06-03&endday=2020-06-03&tree=all

Comment 27

[Intermittent Failures Robot]

17 failures in 950 pushes (0.018 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 15
• try: 1
• mozilla-central: 1

Platform breakdown:

• linux1804-64: 17

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1638027&startday=2020-06-04&endday=2020-06-04&tree=all

Comment 28

[Narcis Beleuzu [:NarcisB]]

There are 138 failures in the last 7 days.
The "new" ones are on "browser_readerMode_hidden_nodes.js" so I`ll add a disable patch for this one too.
Patricia with this one, there will be 5 tests disabled by this leakcheck.
Can you please take a look?

Comment 29

[Narcis Beleuzu [:NarcisB]]

Attachment: Bug 1638027 - Disable browser_readerMode_hidden_nodes.js on linux18.04 debug r=jmaher DONTBUILD

Comment 30

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b05abc0736ee
Disable browser_readerMode_hidden_nodes.js on linux18.04 debug r=jmaher DONTBUILD

Comment 31

[Narcis Beleuzu [:NarcisB]]

After the disable, the failure jumped on browser/components/extensions/test/browser/browser_ext_windows_remove.js

Comment 32

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Andrew, can you help stopping those leaks, please?

Comment 33

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/b05abc0736ee

Comment 34

[Andrew McCreight [:mccr8]]

These failures all have Fission enabled, so it would be possible to disable them only for Fission. Reader mode tests have caused issues with Fission before so it isn't too surprising we're seeing more. I think the issue is that they end up creating and destroying a lot of workers. I'll try to take a look soon, but this is probably going to be a worker issue. But maybe I can narrow it down a bit.

Comment 35

[Andrew McCreight [:mccr8]]

It looks like the closure for the weak worker reference for PerformanceStorageWorker has a strong reference to the worker. I assume there's some kind of shutdown race where we create the weak ref, but never notify the ref that the worker went away.

Comment 36

[Andrew McCreight [:mccr8]]

One approach might be to make the closure reference to the PerformanceStorageWorker weak. You'd clear the callback in the dtor, then. It would probably make some static analysis sad.

Comment 37

[Intermittent Failures Robot]

23 failures in 875 pushes (0.026 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 21
• mozilla-central: 2

Platform breakdown:

• linux1804-64: 23

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1638027&startday=2020-06-05&endday=2020-06-05&tree=all

Comment 38

[Intermittent Failures Robot]

121 failures in 4949 pushes (0.024 failures/push) were associated with this bug in the last 7 days.

This is the #7 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 108
• mozilla-central: 9
• try: 4

Platform breakdown:

• linux1804-64: 121

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1638027&startday=2020-06-01&endday=2020-06-07&tree=all

Comment 39

[Andrew Sutherland [:asuth] (he/him)]

I believe the problem here is a continuation of the saga related to starting workers late enough in shutdown that by the time their primary runnable gets a chance to run PBackground is already shutdown in xpcom-shutdown-threads. (But not so late that RuntimeService::Shutdown has set RuntimeService::mShuttingDown to true, thereby causing worker construction to fail.)

Bug 1638170 is the most recent work in this area where we introduced checks that assert strong worker refs aren't handed out prior to the worker reaching the Running state, but we allowed weak refs to be obtained, but WorkerPrivate::RunLoopNeverRan() didn't make the explicit notify transition that would cause the weak refs to be dropped or otherwise approximate it (because Notify* assumes an operating loop). And notably in bug 1615014 I explicitly had us move the creation of PerformanceStorageWorker prior to attempting to create the PBackground actor because the creation of the actor results in activity on the main thread that would otherwise result in a race.

The immediate action here is to make RunLoopNeverRan() cause the WeakWorkerRef instances to be dropped. I'm filing a bug for that in another tab that will block this bug (but that can be refactored).

Comment 40

[Andrew Sutherland [:asuth] (he/him)]

(In reply to Andrew Sutherland [:asuth] (he/him) from comment #39)

The immediate action here is to make RunLoopNeverRan() cause the WeakWorkerRef instances to be dropped. I'm filing a bug for that in another tab that will block this bug (but that can be refactored).

Meant to add: And the next fix after that is to help make sure we're not trying to start (chrome) workers during the shutdown process without a good reason. (And where "a good reason" means that a shutdown blocker is provided as proof-of-good-reason.)

Comment 41

[Intermittent Failures Robot]

2 failures in 5211 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 2

Platform breakdown:

• linux1804-64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1638027&startday=2020-06-08&endday=2020-06-14&tree=all

Comment 42

[Intermittent Failures Robot]

1 failures in 4516 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• linux1804-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1638027&startday=2020-06-29&endday=2020-07-05&tree=all

Comment 43

[Intermittent Failures Robot]

1 failures in 4289 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• linux1804-64-qr: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1638027&startday=2020-07-27&endday=2020-08-02&tree=all

Comment 44

[Bas Schouten (:bas.schouten)]

This is in the DOM performance API, which wasn't written or as far as I know is actively being worked on by the performance team. Baku might have an idea here.

Comment 45

[Andrea Marchesini [:baku]]

Andrew, it seems you have done the first round of debugging. Are you able to provide a fix for this leak?
Maybe we should not create a PerformanceStorageWorker when we are shutting down the browser.

Comment 46

[Intermittent Failures Robot]

15 failures in 3279 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 11
• mozilla-central: 4

Platform and build breakdown:

• linux1804-64-qr: 1
  • debug: 1
• linux1804-64: 14
  • debug: 14

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1638027&startday=2021-05-03&endday=2021-05-09&tree=all

Comment 47

[Intermittent Failures Robot]

12 failures in 3061 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 8
• mozilla-central: 4

Platform and build breakdown:

• linux1804-64: 12
  • debug: 12

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1638027&startday=2021-05-17&endday=2021-05-23&tree=all

Comment 48

[Intermittent Failures Robot]

14 failures in 2908 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 4
• autoland: 10

Platform and build breakdown:

• linux1804-64: 14
  • debug: 14

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1638027&startday=2021-05-24&endday=2021-05-30&tree=all

Comment 49

[Neha Kochar [:neha]]

These tests are not failing or disabled only for Fission so removing the fission dependency.

Comment 50

[Intermittent Failures Robot]

45 failures in 3065 pushes (0.015 failures/push) were associated with this bug in the last 7 days.

This is the #30 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 36
• mozilla-central: 8
• try: 1

Platform and build breakdown:

• linux1804-64: 13
  • debug: 13
• macosx1015-64-qr: 32
  • debug: 32

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1638027&startday=2021-05-31&endday=2021-06-06&tree=all

Comment 51

[Andrew McCreight [:mccr8]]

The failure rate has massively spiked up in the last week. Given the failures, I'm guessing that this is due to the enabling of browser chrome tests with Fission enabled. Every failure in the 3 weeks has been with Fission enabled.

Comment 52

[Intermittent Failures Robot]

23 failures in 681 pushes (0.034 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 19
• mozilla-central: 4

Platform and build breakdown:

• macosx1015-64-qr: 19
  • debug: 19
• linux1804-64: 4
  • debug: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1638027&startday=2021-06-07&endday=2021-06-07&tree=all

Comment 53

[Intermittent Failures Robot]

15 failures in 621 pushes (0.024 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 12
• mozilla-central: 3

Platform and build breakdown:

• linux1804-64: 4
  • debug: 4
• macosx1015-64-qr: 11
  • debug: 11

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1638027&startday=2021-06-10&endday=2021-06-10&tree=all

Comment 54

[Intermittent Failures Robot]

77 failures in 3502 pushes (0.022 failures/push) were associated with this bug in the last 7 days.

This is the #16 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 57
• mozilla-central: 20

Platform and build breakdown:

• macosx1015-64-qr: 64
  • debug: 64
• linux1804-64: 13
  • debug: 13

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1638027&startday=2021-06-07&endday=2021-06-13&tree=all

Comment 55

[Jens Stutte [:jstutte]]

Yaron will look into this.

Comment 56

[Intermittent Failures Robot]

17 failures in 774 pushes (0.022 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 9
• mozilla-central: 8

Platform and build breakdown:

• macosx1015-64-qr: 12
  • debug: 12
• linux1804-64: 5
  • debug: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1638027&startday=2021-06-17&endday=2021-06-17&tree=all

Comment 57

[Intermittent Failures Robot]

58 failures in 2888 pushes (0.02 failures/push) were associated with this bug in the last 7 days.

This is the #18 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 37
• mozilla-central: 20
• try: 1

Platform and build breakdown:

• macosx1015-64-qr: 46
  • debug: 46
• linux1804-64: 12
  • debug: 12

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1638027&startday=2021-06-14&endday=2021-06-20&tree=all

Comment 58

[Andreea Pavel [:apavel]]

There are 63 total failures in the last 7 days on

• macosx1015-64-qr debug
• linux1804-64 debug

Recent failure log: https://treeherder.mozilla.org/logviewer?job_id=343384323&repo=autoland&lineNumber=93005

[task 2021-06-21T15:15:52.854Z] 15:15:52 INFO - nsTraceRefcnt::DumpStatistics: 2081 entries
[task 2021-06-21T15:15:52.854Z] 15:15:52 INFO - TEST-PASS | leakcheck | default no leaks detected!
[task 2021-06-21T15:15:52.854Z] 15:15:52 INFO - leakcheck | Processing leak log file /var/folders/c2/gp9_mxbj7px3dsk7fl0bpv5r000014/T/tmpqlp51xqh.mozrunner/runtests_leaks_tab_pid3440.log
[task 2021-06-21T15:15:52.854Z] 15:15:52 INFO -
[task 2021-06-21T15:15:52.855Z] 15:15:52 INFO - == BloatView: ALL (cumulative) LEAK AND BLOAT STATISTICS, tab process 3440
[task 2021-06-21T15:15:52.855Z] 15:15:52 INFO -
[task 2021-06-21T15:15:52.855Z] 15:15:52 INFO - |<----------------Class--------------->|<-----Bytes------>|<----Objects---->|
[task 2021-06-21T15:15:52.855Z] 15:15:52 INFO - | | Per-Inst Leaked| Total Rem|
[task 2021-06-21T15:15:52.856Z] 15:15:52 INFO - 0 |TOTAL | 57 0| 88568 0|
[task 2021-06-21T15:15:52.857Z] 15:15:52 INFO -
[task 2021-06-21T15:15:52.857Z] 15:15:52 INFO - nsTraceRefcnt::DumpStatistics: 985 entries
[task 2021-06-21T15:15:52.857Z] 15:15:52 INFO - TEST-PASS | leakcheck | tab no leaks detected!
[task 2021-06-21T15:15:52.858Z] 15:15:52 INFO - leakcheck | Processing leak log file /var/folders/c2/gp9_mxbj7px3dsk7fl0bpv5r000014/T/tmpqlp51xqh.mozrunner/runtests_leaks_tab_pid3455.log
[task 2021-06-21T15:15:52.858Z] 15:15:52 INFO -
[task 2021-06-21T15:15:52.859Z] 15:15:52 INFO - == BloatView: ALL (cumulative) LEAK AND BLOAT STATISTICS, tab process 3455
[task 2021-06-21T15:15:52.859Z] 15:15:52 INFO -
[task 2021-06-21T15:15:52.859Z] 15:15:52 INFO - |<----------------Class--------------->|<-----Bytes------>|<----Objects---->|
[task 2021-06-21T15:15:52.860Z] 15:15:52 INFO - | | Per-Inst Leaked| Total Rem|
[task 2021-06-21T15:15:52.860Z] 15:15:52 INFO - 0 |TOTAL | 65 312| 59184 3|
[task 2021-06-21T15:15:52.860Z] 15:15:52 INFO - 253 |Mutex | 96 96| 668 1|
[task 2021-06-21T15:15:52.861Z] 15:15:52 INFO - 373 |PerformanceStorageWorker | 120 120| 2 1|
[task 2021-06-21T15:15:52.861Z] 15:15:52 INFO - 564 |WorkerRef | 96 96| 9 1|
[task 2021-06-21T15:15:52.861Z] 15:15:52 INFO -
[task 2021-06-21T15:15:52.862Z] 15:15:52 INFO - nsTraceRefcnt::DumpStatistics: 943 entries
[task 2021-06-21T15:15:52.862Z] 15:15:52 INFO - TEST-INFO | leakcheck | tab leaked 1 Mutex
[task 2021-06-21T15:15:52.862Z] 15:15:52 INFO - TEST-INFO | leakcheck | tab leaked 1 PerformanceStorageWorker
[task 2021-06-21T15:15:52.863Z] 15:15:52 INFO - TEST-INFO | leakcheck | tab leaked 1 WorkerRef
[task 2021-06-21T15:15:52.863Z] 15:15:52 INFO - TEST-UNEXPECTED-FAIL | leakcheck | tab 312 bytes leaked (Mutex, PerformanceStorageWorker, WorkerRef)
[task 2021-06-21T15:15:52.863Z] 15:15:52 INFO -
[task 2021-06-21T15:15:52.864Z] 15:15:52 INFO - leakcheck | Processing leak log file /var/folders/c2/gp9_mxbj7px3dsk7fl0bpv5r000014/T/tmpqlp51xqh.mozrunner/runtests_leaks_tab_pid3441.log
[task 2021-06-21T15:15:52.864Z] 15:15:52 INFO -
[task 2021-06-21T15:15:52.864Z] 15:15:52 INFO - == BloatView: ALL (cumulative) LEAK AND BLOAT STATISTICS, tab process 3441
[task 2021-06-21T15:15:52.865Z] 15:15:52 INFO -
[task 2021-06-21T15:15:52.865Z] 15:15:52 INFO - |<----------------Class--------------->|<-----Bytes------>|<----Objects---->|
[task 2021-06-21T15:15:52.865Z] 15:15:52 INFO - | | Per-Inst Leaked| Total Rem|
[task 2021-06-21T15:15:52.866Z] 15:15:52 INFO - 0 |TOTAL | 53 0| 21544 0|
[task 2021-06-21T15:15:52.866Z] 15:15:52 INFO -
[task 2021-06-21T15:15:52.866Z] 15:15:52 INFO - nsTraceRefcnt::DumpStatistics: 770 entries
[task 2021-06-21T15:15:52.867Z] 15:15:52 INFO - TEST-PASS | leakcheck | tab no leaks detected!

Comment 59

[Intermittent Failures Robot]

24 failures in 3115 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 18
• mozilla-central: 4
• try: 2

Platform and build breakdown:

• macosx1015-64-qr: 12
  • debug: 12
• linux1804-64: 12
  • debug: 12

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1638027&startday=2021-06-21&endday=2021-06-27&tree=all

Comment 60

[Intermittent Failures Robot]

12 failures in 723 pushes (0.017 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 8
• mozilla-central: 4

Platform and build breakdown:

• linux1804-64: 7
  • debug: 7
• macosx1015-64-qr: 5
  • debug: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1638027&startday=2021-06-28&endday=2021-07-04&tree=all

Comment 61

[Intermittent Failures Robot]

32 failures in 2387 pushes (0.013 failures/push) were associated with this bug in the last 7 days.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 19
• try: 4
• mozilla-central: 9

Platform and build breakdown:

• linux1804-64: 22
  • debug: 22
• macosx1015-64-qr: 10
  • debug: 10

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1638027&startday=2021-07-05&endday=2021-07-11&tree=all

Comment 62

[Intermittent Failures Robot]

17 failures in 2224 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 13
• mozilla-central: 4

Platform and build breakdown:

• linux1804-64: 12
  • debug: 12
• macosx1015-64-qr: 5
  • debug: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1638027&startday=2021-07-12&endday=2021-07-18&tree=all

Comment 63

[Intermittent Failures Robot]

35 failures in 2940 pushes (0.012 failures/push) were associated with this bug in the last 7 days.

This is the #28 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 26
• mozilla-central: 9

Platform and build breakdown:

• linux1804-64: 22
  • debug: 22
• macosx1015-64-qr: 12
  • debug: 12
• android-hw-p2-8-0-android-aarch64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1638027&startday=2021-07-19&endday=2021-07-25&tree=all

Comment 64

[Intermittent Failures Robot]

17 failures in 3131 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 9
• mozilla-central: 8

Platform and build breakdown:

• macosx1015-64-qr: 5
  • debug: 5
• linux1804-64: 3
  • debug: 3
• linux1804-64-qr: 9
  • debug: 9

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1638027&startday=2021-07-26&endday=2021-08-01&tree=all

Comment 65

[Intermittent Failures Robot]

26 failures in 2829 pushes (0.009 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1
• autoland: 17
• mozilla-central: 8

Platform and build breakdown:

• linux1804-64-qr: 16
  • debug: 16
• macosx1015-64-qr: 10
  • debug: 10

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1638027&startday=2021-08-02&endday=2021-08-08&tree=all

Comment 66

[Intermittent Failures Robot]

20 failures in 3025 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 13
• mozilla-central: 7

Platform and build breakdown:

• macosx1015-64-qr: 3
  • debug: 3
• linux1804-64-qr: 17
  • debug: 17

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1638027&startday=2021-08-09&endday=2021-08-15&tree=all

Comment 67

[Intermittent Failures Robot]

14 failures in 2899 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 9
• mozilla-central: 5

Platform and build breakdown:

• macosx1015-64-qr: 2
  • debug: 2
• linux1804-64-qr: 12
  • debug: 12

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1638027&startday=2021-08-16&endday=2021-08-22&tree=all

Comment 68

[Intermittent Failures Robot]

14 failures in 2580 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 8
• mozilla-central: 6

Platform and build breakdown:

• linux1804-64-qr: 6
  • debug: 6
• macosx1015-64-qr: 8
  • debug: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1638027&startday=2021-08-23&endday=2021-08-29&tree=all

Comment 69

[Intermittent Failures Robot]

23 failures in 3084 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 20
• mozilla-central: 3

Platform and build breakdown:

• macosx1015-64-qr: 9
  • debug: 9
• linux1804-64-qr: 14
  • debug: 14

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1638027&startday=2021-08-30&endday=2021-09-05&tree=all

Comment 70

[Intermittent Failures Robot]

14 failures in 3266 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 10
• mozilla-central: 4

Platform and build breakdown:

• linux1804-64-qr: 11
  • debug: 11
• macosx1015-64-qr: 3
  • debug: 3

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1638027&startday=2021-09-06&endday=2021-09-12&tree=all

Comment 71

[Intermittent Failures Robot]

5 failures in 3375 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3
• mozilla-central: 2

Platform and build breakdown:

• linux1804-64-qr: 3
  • debug: 3
• macosx1015-64-qr: 2
  • debug: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1638027&startday=2021-09-13&endday=2021-09-19&tree=all

Comment 72

[Jens Stutte [:jstutte]]

This is not happening since October.