1268177 - Intermittent test_session_zombification.html | application timed out after 330 seconds with no output

Status: RESOLVED FIXED

(Firefox for Android Graveyard :: Testing, defect)

Description

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://treeherder.mozilla.org/logviewer.html#?job_id=26648365&repo=mozilla-inbound

Comment 1

[Jan Henning [:JanH]]

That log indicates that the test is hanging here (https://hg.mozilla.org/mozilla-central/file/fc15477ce628599519cb0055f52cc195d640dc94/mobile/android/tests/browser/chrome/test_session_zombification.html#l77), which is somewhat mystifying. Looking at the log, we do receive a DOMContentLoaded event, so promiseBrowserEvent should resolve and return control back to the test, but in this case that never happens.
However at precisely that point in time, we've also received a memory-pressure event (https://treeherder.mozilla.org/logviewer.html#?job_id=26648365&repo=mozilla-inbound#L7772).
I can't test my theory right now because I've manged to bork my development VM while trying to upgrade Ubuntu, but I strongly suspect that the mochitest main page - which was in a background tab at that time - was zombified into oblivion and with it the test script as well.
If so, a possible solution could be to just generally disallow automatic background tab zombifications during mochitests.

Comment 2

[Jan Henning [:JanH]]

Attachment: MozReview Request: Bug 1268177 - Disable background tab zombifications during tests. r=margaret

During tests that open additional tabs, the main test tab controlling the test is at risk of being zombified if a memory pressure event arrives, which breaks the test and leads to the test timing out. Therefore, we now disable background tab zombifications during tests.

This means that there's a slightly increased risk of being OOM killed instead, however
- the tabs opened by the tests themselves are normally relatively short-lived anyway
- we're no worse off than if the tab containing the test harness code had been zombified.

Review commit: https://reviewboard.mozilla.org/r/49797/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/49797/

Comment 3

[Jan Henning [:JanH]]

Comment on attachment 8747280 [details]
MozReview Request: Bug 1268177 - Disable background tab zombifications during tests. r=margaret

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/49797/diff/1-2/

Comment 4

[:Margaret Leibovic]

Comment on attachment 8747280 [details]
MozReview Request: Bug 1268177 - Disable background tab zombifications during tests. r=margaret

https://reviewboard.mozilla.org/r/49797/#review47535

Sounds like a reasonable approach to me. Have you run this on try to verify this reduces the occurence of this failure?

Comment 5

[Jan Henning [:JanH]]

No, but I can reproduce it locally by forcing a memory pressure event (adb shell am broadcast -a org.mozilla.gecko.FORCE_MEMORY_PRESSURE) while the test harness tab is in background. With this patch enabled, background tabs are no longer zombified and the test just keeps going, so I'm fairly confident it'll work on try, too.

I don't know if it's just a coincidence or not, but while looking through some logcats from try a common theme seems to be that often the backup service is active on the emulator around the time the memory pressure event is triggered - I'm wondering how easy it would be to disable the backup service on the emulator images used for try.

Comment 6

[Geoff Brown [:gbrown]]

(In reply to Jan Henning [:JanH] from comment #5)

> I'm wondering how easy it would be to disable the backup
> service on the emulator images used for try.

That sounds like a good idea. If you can tell me how to disable the backup service, I can update the emulator images.

Comment 7

[Jan Henning [:JanH]]

Good question - maybe Sebastian knows better?

Comment 8

[Jan Henning [:JanH]]

Looking at an emulator, it might be enough to turn backups off under "Settings -> Backup & reset", but I'm not sure whether something might be hidden elsewhere - my normal phone is somewhat older, so I'm not really familiar with it

Comment 9

[Jan Henning [:JanH]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=de8e9b59fb25&selectedJob=20436098

Comment 10

[Pulsebot]

https://hg.mozilla.org/integration/fx-team/rev/509fe4d4251e

Comment 11

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/509fe4d4251e

Comment 13

[Sebastian Kaspari (:sebastian; :pocmo)]

(In reply to Jan Henning [:JanH] from comment #7)
> Good question - maybe Sebastian knows better?

No idea. Backup for the app is disabled[1] and I assume we do not have a Google account assigned with the emulator images? So what is creating the backup and uploading to where? Is it actually doing anything? Maybe worth investigating in a follow-up?

[1] https://dxr.mozilla.org/mozilla-central/source/mobile/android/base/AndroidManifest.xml.in#30

Comment 14

[Geoff Brown [:gbrown]]

(In reply to Sebastian Kaspari (:sebastian) from comment #13)
> (In reply to Jan Henning [:JanH] from comment #7)
> > Good question - maybe Sebastian knows better?
> 
> No idea. Backup for the app is disabled[1] and I assume we do not have a
> Google account assigned with the emulator images? 

That's right - no account.

> So what is creating the
> backup and uploading to where? Is it actually doing anything? Maybe worth
> investigating in a follow-up?

I don't know what it is doing. Maybe I'll just look into disabling backup for future images, but leave this one as-is for now.

Comment 15

[Jan Henning [:JanH]]

(In reply to Sebastian Kaspari (:sebastian) from comment #13)
> (In reply to Jan Henning [:JanH] from comment #7)
> > Good question - maybe Sebastian knows better?
> 
> So what is creating the backup and uploading to where? Is it actually doing anything?
> Maybe worth investigating in a follow-up?

If you look at a random logcat (e.g. http://mozilla-releng-blobs.s3.amazonaws.com/blobs/mozilla-central/sha512/f8e92a1fc1bb7ea61779be9a2c8c1d9b477711787962bbbdf8a32e63991a56392ce0924854ee030e09dca1bed97ab0ba6e73d7059d48f88218a1db328b43deee) and search for "backup" you'll find a number of entries from the BackupManagerService.