1348547 - Intermittent devtools/client/aboutdebugging/test/browser_service_workers_status.js | Service worker is currently running - Got Stopped, expected Running

Status: RESOLVED FIXED

(DevTools :: about:debugging, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://treeherder.mozilla.org/logviewer.html#?job_id=84722558&repo=autoland

13:30:11     INFO - Entering test bound 
13:30:11     INFO - opening about:debugging
13:30:11     INFO - Adding a new tab with URL: about:debugging#workers
13:30:11     INFO - Buffered messages logged at 13:30:06
13:30:11     INFO - Tab added and finished loading
13:30:11     INFO - Adding a new tab with URL: http://example.com/browser/devtools/client/aboutdebugging/test/service-workers/delay-sw.html
13:30:11     INFO - Buffered messages logged at 13:30:07
13:30:11     INFO - Tab added and finished loading
13:30:11     INFO - Make the test page notify us when the service worker sends a message.
13:30:11     INFO - Buffered messages logged at 13:30:11
13:30:11     INFO - TEST-PASS | devtools/client/aboutdebugging/test/browser_service_workers_status.js | Found the service worker in the list - 
13:30:11     INFO - Buffered messages finished
13:30:11     INFO - TEST-UNEXPECTED-FAIL | devtools/client/aboutdebugging/test/browser_service_workers_status.js | Service worker is currently running - Got Stopped, expected Running
13:30:11     INFO - Stack trace:
13:30:11     INFO -     chrome://mochikit/content/browser-test.js:test_is:911
13:30:11     INFO -     chrome://mochitests/content/browser/devtools/client/aboutdebugging/test/browser_service_workers_status.js:null:44
13:30:11     INFO -     Tester_execTest@chrome://mochikit/content/browser-test.js:735:9
13:30:11     INFO -     Tester.prototype.nextTest</<@chrome://mochikit/content/browser-test.js:655:7
13:30:11     INFO -     SimpleTest.waitForFocus/waitForFocusInner/focusedOrLoaded/<@chrome://mochikit/content/tests/SimpleTest/SimpleTest.js:791:59

Comment 1

[Intermittent Failures Robot]

12 failures in 777 pushes (0.015 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* autoland: 9
* mozilla-central: 2
* mozilla-inbound: 1

Platform breakdown:
* linux64: 11
* windows7-32-vm: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1348547&startday=2017-03-13&endday=2017-03-19&tree=all

Comment 2

[Intermittent Failures Robot]

20 failures in 136 pushes (0.147 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 10
* mozilla-inbound: 6
* mozilla-central: 2
* try: 1
* graphics: 1

Platform breakdown:
* linux64: 19
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1348547&startday=2017-03-20&endday=2017-03-20&tree=all

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

another about:debugging failure from march 17th, possibly related to bug 1329836 ?

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

believe this might be fixed by bug 1329836

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

most likely related to https://bugzilla.mozilla.org/show_bug.cgi?id=1348112 which I narrowed down for bug 1329836.

Comment 6

[Intermittent Failures Robot]

17 failures in 174 pushes (0.098 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* try: 6
* autoland: 6
* mozilla-inbound: 4
* mozilla-central: 1

Platform breakdown:
* linux64: 17

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1348547&startday=2017-03-21&endday=2017-03-21&tree=all

Comment 7

[Intermittent Failures Robot]

21 failures in 174 pushes (0.121 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 10
* mozilla-inbound: 7
* mozilla-central: 2
* graphics: 2

Platform breakdown:
* linux64: 21

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1348547&startday=2017-03-23&endday=2017-03-23&tree=all

Comment 8

[Intermittent Failures Robot]

15 failures in 153 pushes (0.098 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 9
* mozilla-inbound: 6

Platform breakdown:
* linux64: 14
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1348547&startday=2017-03-24&endday=2017-03-24&tree=all

Comment 9

[Intermittent Failures Robot]

114 failures in 898 pushes (0.127 failures/push) were associated with this bug in the last 7 days. 

This is the #5 most frequent failure this week. 

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 51
* mozilla-inbound: 37
* try: 10
* mozilla-central: 9
* graphics: 6
* oak: 1

Platform breakdown:
* linux64: 111
* osx-10-10: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1348547&startday=2017-03-20&endday=2017-03-26&tree=all

Comment 10

[Intermittent Failures Robot]

17 failures in 141 pushes (0.121 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 11
* mozilla-inbound: 5
* graphics: 1

Platform breakdown:
* linux64: 14
* windows8-64: 1
* windows7-32-vm: 1
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1348547&startday=2017-03-27&endday=2017-03-27&tree=all

Comment 11

[Intermittent Failures Robot]

21 failures in 165 pushes (0.127 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-inbound: 8
* autoland: 8
* mozilla-central: 4
* graphics: 1

Platform breakdown:
* linux64: 19
* windows7-32-vm: 1
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1348547&startday=2017-03-28&endday=2017-03-28&tree=all

Comment 12

[Intermittent Failures Robot]

19 failures in 188 pushes (0.101 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 13
* graphics: 4
* try: 1
* mozilla-inbound: 1

Platform breakdown:
* linux64: 13
* osx-10-10: 5
* windows7-32-vm: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1348547&startday=2017-03-29&endday=2017-03-29&tree=all

Comment 13

[Geoff Brown [:gbrown]]

(In reply to Joel Maher ( :jmaher) from comment #5)
> most likely related to https://bugzilla.mozilla.org/show_bug.cgi?id=1348112
> which I narrowed down for bug 1329836.

I am not so sure, since bug 1329836 started much earlier (though 1329836 did increase in frequency just before 03-18).

Comment 14

[Geoff Brown [:gbrown]]

This test was updated around 03-16 - bug 1345932. But failures did not start with those changes.

Comment 15

[Geoff Brown [:gbrown]]

https://treeherder.mozilla.org/#/jobs?repo=autoland&filter-searchStr=dt3%20asan&tochange=1374720a22135135edbad4b41d23ae9eb69158ec&fromchange=576abf29e19cefe2a68f053245d81a65df46c99b&selectedJob=85399870

Comment 16

[Geoff Brown [:gbrown]]

Retriggers on comment 15 now clearly indicate that this started with https://hg.mozilla.org/integration/autoland/rev/5e664d9cc14d3331264be607ae27c650352066b6 / bug 1336398, which simply increased the process count.

:mrbkap -- mostly just making you aware, but wouldn't mind if you had a look at the failure here! (It is a frequent failure, so we need someone to fix or disable the test soon).

Comment 17

[Blake Kaplan (:mrbkap) (inactive)]

That is very surprising. Service worker debugging doesn't work in e10s-multi and these tests attempt to force a single content process [1][2]. So bug 1336398 really should have little to no effect on these tests. That suggests a bug in our process selector.

[1] http://searchfox.org/mozilla-central/rev/72fe012899a1b27d34838ab463ad1ae5b116d76b/devtools/client/aboutdebugging/test/browser_service_workers_status.js#15
[2] http://searchfox.org/mozilla-central/rev/72fe012899a1b27d34838ab463ad1ae5b116d76b/devtools/client/aboutdebugging/test/head.js#425,433

Comment 18

[Intermittent Failures Robot]

17 failures in 123 pushes (0.138 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-inbound: 7
* autoland: 6
* try: 2
* mozilla-central: 2

Platform breakdown:
* linux64: 15
* windows8-64: 1
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1348547&startday=2017-03-30&endday=2017-03-30&tree=all

Comment 19

[Gabor Krizsanits (INACTIVE)]

(In reply to Blake Kaplan (:mrbkap) from comment #17)
> That is very surprising. Service worker debugging doesn't work in e10s-multi
> and these tests attempt to force a single content process [1][2]. So bug
> 1336398 really should have little to no effect on these tests. That suggests
> a bug in our process selector.
> 
> [1]
> http://searchfox.org/mozilla-central/rev/
> 72fe012899a1b27d34838ab463ad1ae5b116d76b/devtools/client/aboutdebugging/test/
> browser_service_workers_status.js#15
> [2]
> http://searchfox.org/mozilla-central/rev/
> 72fe012899a1b27d34838ab463ad1ae5b116d76b/devtools/client/aboutdebugging/test/
> head.js#425,433

Maybe it's about the tables processes we keep alive... Have you tried calling Services.ppmm.releaseCachedProcesses(); if it fixes the problem?

Comment 20

[Geoff Brown [:gbrown]]

(In reply to Gabor Krizsanits [:krizsa :gabor] from comment #19)
> Maybe it's about the tables processes we keep alive... Have you tried
> calling Services.ppmm.releaseCachedProcesses(); if it fixes the problem?

Thanks! I'm trying https://treeherder.mozilla.org/#/jobs?repo=try&revision=ebde66316c44e062561ef39beaf438bb6484299a ... mixture of positive and negative signs so far. I'll check on that again on Monday...

Comment 21

[Intermittent Failures Robot]

23 failures in 146 pushes (0.158 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* mozilla-inbound: 11
* autoland: 7
* try: 3
* mozilla-central: 2

Platform breakdown:
* linux64: 17
* osx-10-10: 4
* windows7-32-vm: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1348547&startday=2017-03-31&endday=2017-03-31&tree=all

Comment 22

[Intermittent Failures Robot]

133 failures in 845 pushes (0.157 failures/push) were associated with this bug in the last 7 days. 

This is the #8 most frequent failure this week. 

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **  

Repository breakdown:
* autoland: 60
* mozilla-inbound: 39
* mozilla-central: 19
* try: 8
* graphics: 6
* oak: 1

Platform breakdown:
* linux64: 109
* osx-10-10: 16
* windows7-32-vm: 5
* windows8-64: 3

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1348547&startday=2017-03-27&endday=2017-04-02&tree=all

Comment 23

[Geoff Brown [:gbrown]]

Attachment: call releaseCachedProcesses()

Following up on your comment 19, I found that calling releaseCachedProcesses() does indeed appear to avoid this failure, but I saw new failures in non-e10s mode, particularly leaks. See https://treeherder.mozilla.org/#/jobs?repo=try&revision=ebde66316c44e062561ef39beaf438bb6484299a.

If I call releaseCachedProcesses() only if gMultiProcessBrowser, all is well:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=cc19e2dd2dddfe0b913313fe58473b572b8eb7c3

This appears to also avoid the frequent intermittent leaks in aboutdebugging mochitests reported in bug 1329836! :)

Comment 24

[Blake Kaplan (:mrbkap) (inactive)]

(In reply to Geoff Brown [:gbrown] from comment #23)
> Following up on your comment 19, I found that calling
> releaseCachedProcesses() does indeed appear to avoid this failure, but I saw
> new failures in non-e10s mode, particularly leaks. 

I wonder if ContentParent::ReleaseCachedProcesses should avoid creating sBrowserContentParents if it doesn't already exist.

Comment 25

[Intermittent Failures Robot]

15 failures in 147 pushes (0.102 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 9
* mozilla-inbound: 4
* try: 1
* mozilla-central: 1

Platform breakdown:
* linux64: 10
* osx-10-10: 3
* windows7-32-vm: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1348547&startday=2017-04-03&endday=2017-04-03&tree=all

Comment 26

[Gabor Krizsanits (INACTIVE)]

(In reply to Blake Kaplan (:mrbkap) from comment #24)
> (In reply to Geoff Brown [:gbrown] from comment #23)
> > Following up on your comment 19, I found that calling
> > releaseCachedProcesses() does indeed appear to avoid this failure, but I saw
> > new failures in non-e10s mode, particularly leaks. 
> 
> I wonder if ContentParent::ReleaseCachedProcesses should avoid creating
> sBrowserContentParents if it doesn't already exist.

Yeah... I have a patch for that for the in preallocated process manager work just haven't landed it yet :(

Comment 27

[Gabor Krizsanits (INACTIVE)]

Comment on attachment 8854152 [details] [diff] [review]
call releaseCachedProcesses()

Review of attachment 8854152 [details] [diff] [review]:
-----------------------------------------------------------------

Could you please try instead of using gMultiProcessBrowser in the test file adding this changeset to ContentParent.cpp and calling Services.ppmm.releaseCachedProcesses() unconditionally in the test?

 /*static*/ void
 ContentParent::ReleaseCachedProcesses()
 {
+  if (!GetPoolSize(NS_LITERAL_STRING(DEFAULT_REMOTE_TYPE))) {
+    return;
+  }
+

If that works please land it that way (it should), in case it does not then land your version and I'll look into it in a followup.

Comment 28

[Geoff Brown [:gbrown]]

That does work - thanks!

https://treeherder.mozilla.org/#/jobs?repo=try&revision=9a6f6fd4ad02458c2a3287d0d676baabb3caab40

Comment 29

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/8d8379d9a545
ReleaseCachedProcesses() between aboutdebugging tests; r=gabor

Comment 30

[Intermittent Failures Robot]

15 failures in 134 pushes (0.112 failures/push) were associated with this bug yesterday.   

Repository breakdown:
* autoland: 9
* mozilla-inbound: 4
* mozilla-central: 1
* mozilla-aurora: 1

Platform breakdown:
* linux64: 12
* windows8-64: 1
* windows7-32-vm: 1
* osx-10-10: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1348547&startday=2017-04-04&endday=2017-04-04&tree=all

Comment 31

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/8d8379d9a545

Comment 32

[Ryan VanderMeulen [:RyanVM]]

Can we get a rebased patch for Aurora and an approval request please?

Comment 33

[Geoff Brown [:gbrown]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #32)
> Can we get a rebased patch for Aurora and an approval request please?

We'll need bug 1345932 on aurora first. There are very few of these failures on aurora...worth the effort?

Comment 34

[Ryan VanderMeulen [:RyanVM]]

Only because they've been getting incorrectly starred as bug 1352586.

Comment 35

[Intermittent Failures Robot]

48 failures in 867 pushes (0.055 failures/push) were associated with this bug in the last 7 days. 

This is the #38 most frequent failure this week.  

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. ** 

Repository breakdown:
* autoland: 23
* mozilla-inbound: 11
* try: 5
* mozilla-central: 4
* mozilla-aurora: 3
* oak: 1
* graphics: 1

Platform breakdown:
* linux64: 39
* osx-10-10: 5
* windows7-32-vm: 3
* windows8-64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1348547&startday=2017-04-03&endday=2017-04-09&tree=all

Comment 37

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/fe788c5c3a4f

Comment 38

[Intermittent Failures Robot]

1 failures in 720 pushes (0.001 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* mozilla-esr52: 1

Platform breakdown:
* windows8-64: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1348547&startday=2017-07-10&endday=2017-07-16&tree=all