1282862 - [meta] Intermittent failure, PR_ASSERT during xpcshell shutdown on Mac

Status: NEW

(Testing :: XPCShell Harness, defect, P3)

Description

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

This seems to happen in basically every test.

Comment 1

[Ryan VanderMeulen [:RyanVM]]

Is anybody actually working on this or should we just continue to file dupes indefinitely?

Comment 2

[Bevis Tseng[:bevis][:btseng](Exited)]

All these crashes are the same and are only happened in xpcshell child test in OS X debug build but we lost the assertion info in the log for further analysis.

ni :gps for bug 1282522 to help to identify the reason of the assertion.

Comment 3

[Gregory Szorc [:gps]]

I'm not actively working on bug 1282522. I'll send an email to auto-tools to see if we can find an owner.

Comment 4

[Chris Manchester (limited bugmail, email directly)]

I have a candidate patch for bug 1282522 that I ran through try with some re-triggers. The results are here: https://treeherder.mozilla.org/#/jobs?repo=try&revision=8e592ef51b64d348f8e5271721190257bcc3a09b

4 of the jobs contain failures that appear to match this pattern (2 are in indexedDB tests). All 4 point to the assertion at http://searchfox.org/mozilla-central/rev/bcc9ea6947878ca6378e5b7d6f08c1c090ed9eb7/nsprpub/pr/src/pthreads/ptthread.c#870 in _pt_thread_death.

Comment 5

[Bevis Tseng[:bevis][:btseng](Exited)]

(In reply to Chris Manchester (:chmanchester) from comment #4)
> 4 of the jobs contain failures that appear to match this pattern (2 are in
> indexedDB tests). All 4 point to the assertion at
> http://searchfox.org/mozilla-central/rev/
> bcc9ea6947878ca6378e5b7d6f08c1c090ed9eb7/nsprpub/pr/src/pthreads/ptthread.
> c#870 in _pt_thread_death.

thanks for pointing out this!

Comment 6

[Bevis Tseng[:bevis][:btseng](Exited)]

I am now working on bug 1300454 or bug 1286210 more specifically in priority which happens more frequent according to the orange robot's summary.

I'll revisit this later.

Comment 7

[Gregory Szorc [:gps]]

ICYMI, bug 1282522 landed, which means test output should contain stderr if we crash on shutdown. Have fun finding crashes!

Comment 8

[Bevis Tseng[:bevis][:btseng](Exited)]

After more observation in treeherder:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=3fec30eb0e24&selectedJob=28481069

This is more likely a common issue in xpcshell test instead of an IDB issue while doing run_test_in_child() in OS X debug build.
We can see all those failures caused by assertion are happened at nsprpub/pr/src/pthreads/ptthread.c#870 in _pt_thread_death and all these failed tests are run with run_test_in_child():
http://searchfox.org/mozilla-central/search?q=run_test_in_child&case=false&regexp=false&path=

Comment 9

[Gregory Szorc [:gps]]

Ted: You know the xpcshell child process foo better than anyone (I think). Care to weigh in regarding comment #8?

Comment 10

[(not currently active) Ted Mielczarek]

The logs from that try run apparently expired. :-( Bevis: can you find me an instance of this with a log that still exists?

Comment 11

[Bevis Tseng[:bevis][:btseng](Exited)]

I've re-triggered 200 trials again and the failed ones(1 in idb 6 in netwerk) with the same symptom are filtered as unclassified failures for your reference:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=3fec30eb0e24&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=pending&filter-resultStatus=running&filter-classifiedState=unclassified

These assertions all happened after test is completed.

BTW, for IDB, these test scripts run in both xpcshell test and mochitest but only died in xpcshell-child process.
Not pretty sure if there is something we have to take more care about when test is shutting down.

If it's still more related to each implementation instead of the test harness, please let me know, thanks!

Comment 12

[(not currently active) Ted Mielczarek]

Someone CCed me on bug 1295139, which has a suspiciously similar set of crashes, and in bug 1295139 comment 17 jld proposed a pretty sensible-sounding analysis. I bet it's the same root cause.

Comment 13

[Ryan VanderMeulen [:RyanVM]]

This is basically permafailing now on OSX debug.

Comment 14

[Bevis Tseng[:bevis][:btseng](Exited)]

The crash happens in various XPCShell tests, and the permafailing one recently in central and inbound is browser/components/sessionstore/test/unit/test_startup_invalid_session.js.
We need test harness or sessionstore owners' help to analyze this instead.

Comment 15

[Sylvestre Ledru [:Sylvestre]]

Moving to p3 because no activity for at least 24 weeks.