Failing to produce a leak log does not cause a test failure
Categories
(Testing :: Mochitest, defect, P2)
Tracking
(firefox-esr60 unaffected, firefox64 unaffected, firefox65 wontfix, firefox66 wontfix, firefox67 wontfix, firefox68 wontfix, firefox69 fix-optional, firefox70 fix-optional)
Tracking | Status | |
---|---|---|
firefox-esr60 | --- | unaffected |
firefox64 | --- | unaffected |
firefox65 | --- | wontfix |
firefox66 | --- | wontfix |
firefox67 | --- | wontfix |
firefox68 | --- | wontfix |
firefox69 | --- | fix-optional |
firefox70 | --- | fix-optional |
People
(Reporter: mccr8, Unassigned)
References
(Depends on 1 open bug, Blocks 1 open bug)
Details
(Keywords: regression, Whiteboard: [MemShrink:P2])
Attachments
(4 files, 1 obsolete file)
I did a try push with a patch that makes it so that it never prints out a leak log, and this does not cause an orange on TreeHerder:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=222757782&repo=try&lineNumber=5844
Note that you do get a FAIL if you run the tests locally through mach.
Bug 1518115 is about how we print out the "missing output line" message even if the failure is expected, while this is how the test job does not turn orange if we actually hit that failure.
This is probably not hiding any leaks because the most common cause of this error is a crash, but we should make sure of that.
Reporter | ||
Updated•6 years ago
|
Reporter | ||
Comment 1•6 years ago
|
||
I'm not exactly sure when this code is run, but it only handles the leaking failure case from mozleak_total, and not the missing log case, so maybe that's the reason for not failing? https://hg.mozilla.org/mozilla-central/rev/d40ebdfc91a0
Reporter | ||
Comment 2•6 years ago
|
||
Reporter | ||
Comment 3•6 years ago
|
||
James, can you take a look at this? Thanks.
Reporter | ||
Comment 4•6 years ago
|
||
I think I have a handle on how to fix this. I think the logic for mozleak_total in statushandler.py needs to be extended to handle this case.
Reporter | ||
Updated•6 years ago
|
Reporter | ||
Comment 5•6 years ago
|
||
In order to test the test harness's handling of a process failing to
produce a leak log, add a special function that disables the bloat log
output.
Reporter | ||
Comment 6•6 years ago
|
||
Depends on D17534
Reporter | ||
Comment 7•6 years ago
|
||
This is needed to test the test harness's handling of a negative leak
being reported by the XPCOM leak checker.
Depends on D17535
Reporter | ||
Comment 8•6 years ago
|
||
Depends on D17536
Reporter | ||
Updated•6 years ago
|
Comment 10•6 years ago
|
||
Backed out 4 changesets (bug 1521191) for causing leaks on multiple tests
Log:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=224150195&repo=autoland&lineNumber=53008
Backout:
https://hg.mozilla.org/integration/autoland/rev/044a86a4032cf1ab76e4cd43f5d7929fcff0c9a0
Reporter | ||
Comment 11•6 years ago
|
||
There's a leak in the webrtc/ directory, which is hitting the "NSS is still alive" fatal assertion, so we crash and don't produce a leak log. That assertion should probably be changed because "crash on leak" is not helpful. Anyways, I'm not sure if this leak is new or I just failed to do a try push with these patches.
Reporter | ||
Updated•6 years ago
|
Reporter | ||
Comment 12•6 years ago
|
||
There's a number of issues here:
- I think this WebRTC leak is a regression, so that should get tracked down.
- The "NSS_Shutdown failed" crash should be turned into a leak, not a crash.
- There needs to be a way to specify in a dir.ini file that there is no leak for a process type, and that needs to be updated on import. I don't see anything like that right now, but maybe I'm just missing it. I meant to look at this before I landed this, but I forgot about it.
Reporter | ||
Comment 13•6 years ago
|
||
I found my try push for these patches:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=42f5f6be042d6a915f37bc3943cf3d11ee9411ba
It does contain the NSS_Shutdown crash, but it is still somehow green. I think this implies that:
- The WebRTC leak isn't a regression.
- My patch somehow doesn't actually detect a missing leak log for the WebRTC leak.
- There's some other crash that has started happening?
The logs don't seem to contain the pids of things that crash which makes it hard to figure out what is going on.
In fact, with my try push, there are a bunch of other crashes in WebRTC that show up in the "failure summary" but didn't cause the test to go orange, which is peculiar:
PROCESS-CRASH | /webrtc/RTCPeerConnection-removeTrack.https.html | application crashed [@ libxul.so + 0xd1a42f]
Reporter | ||
Comment 14•6 years ago
|
||
Oh, I see, bug 1519862 recently landed and made the leak checker ignore missing logs for content processes. The orange in my attempted landed was from a main process crash. That at least explains that.
Updated•6 years ago
|
Updated•6 years ago
|
Updated•6 years ago
|
Updated•6 years ago
|
Updated•6 years ago
|
Reporter | ||
Comment 16•6 years ago
|
||
I'm not going to be able to get to this any time soon.
Updated•5 years ago
|
Comment 17•5 years ago
|
||
Marking fix-optional across all versions to remove this from weekly regression triage.
Updated•2 years ago
|
Description
•