1754572 - Intermittent ipc/glue/test/browser/browser_utility_memoryReport.js | application terminated with exit code 11

Status: RESOLVED FIXED

(Core :: IPC, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: alissy [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=367295822&repo=try
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/emp4g_ILT-G1pobDsHEUWA/runs/0/artifacts/public/logs/live_backing.log

---

I think this might be because I directly pass `Maybe<int32_t> utilityPid = utilityProc->ProcessPid();` to the `promise->MaybeResolve` without verifying that we have a `utilityPid` value?

Comment 1

[:gerard-majax]

Locally forcing Maybe<int32_t> utilityPid = Nothing(); gets me some similar stack, so I think this is the reason why. According to the retries in https://treeherder.mozilla.org/jobs?repo=try&revision=0d55c5b72ff4a9e5be4d6939b7b21466b0e7edce it seems to be not a frequent failure.

Comment 2

[:gerard-majax]

try with no fix: https://treeherder.mozilla.org/jobs?repo=try&revision=0511d2555d6d5e072ac86f69a14abf675ad7189d
try with a fix?: https://treeherder.mozilla.org/jobs?repo=try&revision=5c45f30f6db2b800a6fbd25769a8a60907537b22

Comment 3

[:gerard-majax]

Attachment: Bug 1754572 - Correct error handling in UtilityProcessTest r?nika!

Comment 4

[:gerard-majax]

(In reply to Alexandre LISSY :gerard-majax from comment #2)

try with no fix: https://treeherder.mozilla.org/jobs?repo=try&revision=0511d2555d6d5e072ac86f69a14abf675ad7189d
try with a fix?: https://treeherder.mozilla.org/jobs?repo=try&revision=5c45f30f6db2b800a6fbd25769a8a60907537b22

yet https://treeherder.mozilla.org/jobs?repo=try&revision=e87a2ae1363152b8a966e354b912ad98bd866890 shows green with 2079 (to date) retries, while the best one of those two shows at elast two failures over 1486 runs

This would advocate browser_utility_hard_kill.js is the culprit here. My guess for now is that the way we kill the process and quickly end the test leaves some dangling bits in UtilityProcessManager and when browser_utility_memoryReport.js test starts querying for a running process, then it gets into an incorrect state where it behaves likes there is still a process alive (maybe there is one?), but when querying the pid, it's dead.

Comment 5

[:gerard-majax]

green: https://treeherder.mozilla.org/logviewer?job_id=367361729&repo=try&lineNumber=2425 mProcessParent = nullptr
fail: https://treeherder.mozilla.org/logviewer?job_id=367361730&repo=try&lineNumber=2425-2426 mProcessParent != nullptr

Comment 6

[:gerard-majax]

original test:
WIP - hard_kill without the fix https://treeherder.mozilla.org/jobs?repo=try&revision=804e59681853930b7ae9856aa4affa22bff52f9c 25 failures, 2478 runs
Fix hard_kill test by waiting on process death - https://treeherder.mozilla.org/jobs?repo=try&revision=7950daf9937d70ebdb5665b965ca271c7ebc84b5 no failure

adding new delayed kill test:
WIP - hard_kill_delayed to repro: https://treeherder.mozilla.org/jobs?repo=try&revision=10e6beae03181bdc572bd52365f84dfec25c728b many failures (all)
WIP - fix hard_kill_delayed with process waiting https://treeherder.mozilla.org/jobs?repo=try&revision=d96df0044fa92270d41a9647a4659a934b304382 no failure

Comment 7

[:gerard-majax]

Attachment: Bug 1754572 - Fix hard_kill test by waiting on process death r?nika!

Depends on D138356

Comment 8

[:gerard-majax]

Attachment: Bug 1754572 - Add a delayed kill test r?nika!

Depends on D138482

Comment 9

[:gerard-majax]

Attachment: Bug 1754572 - Ensure we stop utility process after running tests r?nika!

Depends on D138483

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b4ed010bc971
Correct error handling in UtilityProcessTest r=nika
https://hg.mozilla.org/integration/autoland/rev/752d39a0201b
Fix hard_kill test by waiting on process death r=nika
https://hg.mozilla.org/integration/autoland/rev/5cea2c885fe7
Add a delayed kill test r=nika
https://hg.mozilla.org/integration/autoland/rev/364a06e45540
Ensure we stop utility process after running tests r=nika

Comment 11

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/b4ed010bc971
https://hg.mozilla.org/mozilla-central/rev/752d39a0201b
https://hg.mozilla.org/mozilla-central/rev/5cea2c885fe7
https://hg.mozilla.org/mozilla-central/rev/364a06e45540

Comment 12

[Intermittent Failures Robot]

13 failures in 3711 pushes (0.004 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 12
• autoland: 1

Platform and build breakdown:

• linux1804-64-qr: 10
  • debug: 4
  • opt: 6
• linux1804-64-asan-qr: 2
  • opt: 2
• linux1804-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1754572&startday=2022-02-07&endday=2022-02-13&tree=all