1386412 - Perma failure when 56 merges to beta in browser_service_workers_multi_content_process.js | Test timed out -

Status: RESOLVED FIXED

(DevTools :: about:debugging, defect, P3)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Merge is tomorrow (August 2nd)

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=119734413&repo=try

TEST-UNEXPECTED-FAIL | devtools/client/aboutdebugging/test/browser_service_workers_multi_content_process.js | Test timed out -

TEST-UNEXPECTED-FAIL | devtools/client/aboutdebugging/test/browser_service_workers_push.js | The service-workers url appears in the list: http://example.com/browser/devtools/client/aboutdebugging/test/service-workers/empty-sw.js - Got false, expected true

Comment 1

[Julian Descottes [:jdescottes]]

Are you sure about the perma failure? 

From what I see in the central as beta push https://treeherder.mozilla.org/#/jobs?repo=try&revision=b47de49ad58a9f315a1a6ddcbe3c8fbd5db1a38d&selectedJob=119734009, there are successful jobs where this test doesn't fail. Given that this test is already intermittent, are we sure it's a blocker?

Comment 2

[Julian Descottes [:jdescottes]]

Attachment: Bug 1386412 - fix race conditions in browser_service_workers_multi_content_process.js;

Review commit: https://reviewboard.mozilla.org/r/163830/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/163830/

Comment 3

[Julian Descottes [:jdescottes]]

This patch should normally address intermittents from Bug 1340199 and Bug 1349121

Comment 4

[Alexandre Poirot [:ochameau]]

Comment on attachment 8892832 [details]
Bug 1386412 - fix race conditions in browser_service_workers_multi_content_process.js;

https://reviewboard.mozilla.org/r/163830/#review169184

Thanks for chasing this intermittent.

It looks like there migh be other suspicious usages of waitForMutation:
http://searchfox.org/mozilla-central/source/devtools/client/aboutdebugging/test/browser_page_not_found.js#24
http://searchfox.org/mozilla-central/source/devtools/client/aboutdebugging/test/browser_service_workers_push_service.js#97
http://searchfox.org/mozilla-central/source/devtools/client/aboutdebugging/test/browser_service_workers_status.js#46
http://searchfox.org/mozilla-central/source/devtools/client/aboutdebugging/test/browser_service_workers_timeout.js#66

That makes me wonder if we should warn from eslint usages of "yield waitForXXXX"...
but unfortunately, some usages are valid.

::: devtools/client/aboutdebugging/test/browser_service_workers_multi_content_process.js:30
(Diff revision 1)
> +  let onMutation = waitForMutation(serviceWorkersElement, { childList: true });
> +
> +  let swTab = yield addTab(TAB_URL, { background: true });
>  
> -  yield waitForMutation(serviceWorkersElement, { childList: true });
> +  info("Wait for service worker to appear in the list");
> +  yield onMutation;

If that's what caused this intermittent, it looks like there is other potential intermittents:
http://searchfox.org/mozilla-central/source/devtools/client/aboutdebugging/test/browser_service_workers.js#20
http://searchfox.org/mozilla-central/source/devtools/client/aboutdebugging/test/browser_service_workers_fetch_flag.js#18
http://searchfox.org/mozilla-central/source/devtools/client/aboutdebugging/test/browser_service_workers_timeout.js#23

Comment 5

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

(In reply to Julian Descottes [:jdescottes] from comment #1)
> Are you sure about the perma failure? 
The test failed on Linux x64 and Windows 7, both on opt and pgo (which is an opt). These are at least 5 failures for this in pne test run: https://treeherder.mozilla.org/#/jobs?repo=try&revision=b47de49ad58a9f315a1a6ddcbe3c8fbd5db1a38d&filter-searchStr=devtools%20e10s&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=running&filter-resultStatus=pending&filter-resultStatus=runnable&selectedJob=119781861

Comment 6

[Julian Descottes [:jdescottes]]

Thanks for the review! I forgot to mention but I did a few try pushes to confirm if this was useful.

First the central as beta simulation without my patch: https://treeherder.mozilla.org/#/jobs?repo=try&revision=221c12eb40bfd6e9d04936c5b2aea085fa39aba1
we can see frequent dt9 failures, for the timeout issue mentioned.

Then same simulation with a first version of my patch: https://treeherder.mozilla.org/#/jobs?repo=try&revision=ffea93d1a821c5bd2ab567dc548c033941514a52 
still a dt9 failure (but less frequent) and failing later in the test

Finally a last test with a patch also trying to fix the issue found in the second try push:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=482f6a8b794cc4558e9bb6274a8b9fbb6b28d9f6
(I did this last one based off central, instead of the beta simulation, so the chunk was dt8 instead of dt9)

(In reply to Alexandre Poirot [:ochameau] from comment #4)
> Comment on attachment 8892832 [details]
> Bug 1386412 - fix race conditions in
> browser_service_workers_multi_content_process.js;
> 
> https://reviewboard.mozilla.org/r/163830/#review169184
> 
> Thanks for chasing this intermittent.
> 
> It looks like there migh be other suspicious usages of waitForMutation:
> http://searchfox.org/mozilla-central/source/devtools/client/aboutdebugging/
> test/browser_page_not_found.js#24
> http://searchfox.org/mozilla-central/source/devtools/client/aboutdebugging/
> test/browser_service_workers_push_service.js#97
> http://searchfox.org/mozilla-central/source/devtools/client/aboutdebugging/
> test/browser_service_workers_status.js#46
> http://searchfox.org/mozilla-central/source/devtools/client/aboutdebugging/
> test/browser_service_workers_timeout.js#66
> 
> That makes me wonder if we should warn from eslint usages of "yield
> waitForXXXX"...
> but unfortunately, some usages are valid.

Do you have any example in mind?

> 
> :::
> devtools/client/aboutdebugging/test/
> browser_service_workers_multi_content_process.js:30
> (Diff revision 1)
> > +  let onMutation = waitForMutation(serviceWorkersElement, { childList: true });
> > +
> > +  let swTab = yield addTab(TAB_URL, { background: true });
> >  
> > -  yield waitForMutation(serviceWorkersElement, { childList: true });
> > +  info("Wait for service worker to appear in the list");
> > +  yield onMutation;
> 
> If that's what caused this intermittent, it looks like there is other
> potential intermittents:
> http://searchfox.org/mozilla-central/source/devtools/client/aboutdebugging/
> test/browser_service_workers.js#20
> http://searchfox.org/mozilla-central/source/devtools/client/aboutdebugging/
> test/browser_service_workers_fetch_flag.js#18
> http://searchfox.org/mozilla-central/source/devtools/client/aboutdebugging/
> test/browser_service_workers_timeout.js#23

I'll file a follow up to cleanup other about:debugging tests. The pattern is racy anyway, I guess depending on the test they are more or less likely to fail, but fixing it shouldn't hurt.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0bf2574ad9ff
fix race conditions in browser_service_workers_multi_content_process.js;r=ochameau

Comment 8

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/0bf2574ad9ff

Comment 9

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/00ead85af43b

Comment 10

[Intermittent Failures Robot]

5 failures in 888 pushes (0.006 failures/push) were associated with this bug in the last 7 days.   

Repository breakdown:
* try: 5

Platform breakdown:
* linux64: 3
* windows7-32: 2

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1386412&startday=2017-07-31&endday=2017-08-06&tree=all