983848 - QuotaManager: Assertion failure: mPendingEventCount (Mismatched calls to observer methods!), at /home/work/B2G-emulator-arm/b2g-inbound/xpcom/threads/LazyIdleThread.cpp:533

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Runtime, defect)

Description

[Dave Hylands [:dhylands]]

Attachment: Emulator log file

When running the updateManagerXLM.js xpcshell test I see the following:

INFO | Running tests sequentially.
TEST-INFO | /home/work/B2G-emulator-arm/objdir-gecko-debug-b-i/dist/test-package-stage/xpcshell/tests/toolkit/mozapps/update/tests/unit_aus_update/updateManagerXML.js | running test ...
TEST-PASS | /home/work/B2G-emulator-arm/objdir-gecko-debug-b-i/dist/test-package-stage/xpcshell/tests/toolkit/mozapps/update/tests/unit_aus_update/updateManagerXML.js | test passed (time: 15652.522ms)
PROCESS-CRASH | /home/work/B2G-emulator-arm/objdir-gecko-debug-b-i/dist/test-package-stage/xpcshell/tests/toolkit/mozapps/update/tests/unit_aus_update/updateManagerXML.js | application crashed [Unknown top frame]

From the attached emulator log file, on line 1424 is the assertion which triggers the PROCESS-CRASH.

03-14 21:05:39.356   730   747 F MOZ_Assert: Assertion failure: mPendingEventCount (Mismatched calls to observer methods!), at /home/work/B2G-emulator-arm/b2g-inbound/xpcom/threads/LazyIdleThread.cpp:533

Line 1157 is where the LazyIdle thread was constructed. 
Process ID 730 is the xpcshell process.

There are a bunch of extra prints in the emulator log file.

Line 1414 shows the GonkMemoryPressure: Observed XPCOM shutdown
Line 1415 shows the next call (that I added prints to) into LazyIdleThread

These failures are currently not showing up on TBPL due to bug 959327

This particular bug is 100% reproducible.

Comment 1

[Dave Hylands [:dhylands]]

Attachment: Backtrace from QuoteManager::Init (which creates the LazyIdleThread)

Comment 2

[Dave Hylands [:dhylands]]

The test which triggers this is:

http://dxr.mozilla.org/mozilla-central/source/toolkit/mozapps/update/tests/unit_aus_update/updateManagerXML.js

Comment 3

[Ben Turner (not reading bugmail, use the needinfo flag!)]

This is just a testing bug, xpcshell tests that use indexeddb need to have a valid profile.

Comment 4

[Ben Turner (not reading bugmail, use the needinfo flag!)]

Hm, not sure about the right component for this...

Comment 5

[Dave Hylands [:dhylands]]

Attachment: Backtrace after adding do_get_profile() call

bent suggested that I put a do_get_profile() call in, so I added the following to setupTestCommon:

  if (IS_TOOLKIT_GONK) {
    do_get_profile();
  }

and now when I run the test, xpcshell dies with the attached backtrace.

Comment 6

[Dave Hylands [:dhylands]]

Adding ted (jgriffin suggested this).

Talking with bent on IRC, it seems that xpchsell is shutting down while there are still unprocessed events in the event queue.

Ted - do you know how the shutdown gets dealt with in xpcshell?

The tests that fail are updater tests (which aren't currently running on TBPL - see bug 959327), which use SettingsService, which on b2g uses the SettingsDB, which uses IndexedDB, which uses QuotaManager.

Well the updater tests actually pass, but xpcshell is triggering a PROCESS-CRASH.

Comment 7

[Dave Hylands [:dhylands]]

I discovered that if I #if 0 out the 3 places in UpdatePrompt.js that it uses the settings service, then the tests pass.

Upon further investigation, the settings are being set to convey information to other apps in b2g and aren't actively being used by the updater.

When running under xpcshell, there is no settingsdb, which winds up triggering all of the above issues.

It there is a way to detect that I'm running a test under xpcshell, then I can just key off of that and not call the settings service.

If there isn't an officual way of doing this, then I'll just set a preference and use that.

Comment 8

[Ben Turner (not reading bugmail, use the needinfo flag!)]

It's a bit hacky but I did this in another patch:

https://mxr.mozilla.org/mozilla-central/source/dom/ipc/ContentParent.cpp#271

Comment 9

[Dave Hylands [:dhylands]]

Attachment: bug-983848.patch

Fix B2G's UpdatePrompt to not use settings db when running xpcshell tests.

Comment 10

[Dave Hylands [:dhylands]]

Comment on attachment 8391966 [details] [diff] [review]
bug-983848.patch

Review of attachment 8391966 [details] [diff] [review]:
-----------------------------------------------------------------

Putting bent down to review since he r+'d at least one of the bugs which caused this :)

Comment 11

[Dave Hylands [:dhylands]]

Clearing needinfo on ted, since I don't think we need that info any more.

Comment 12

[(not currently active) Ted Mielczarek]

I feel like it would be nicer to handle this in the test harness than in ugly hacks inside application code, but whatever floats your boat.

Comment 13

[Ben Turner (not reading bugmail, use the needinfo flag!)]

Comment on attachment 8391966 [details] [diff] [review]
bug-983848.patch

Review of attachment 8391966 [details] [diff] [review]:
-----------------------------------------------------------------

I think this is the easiest fix for getting these tests enabled so r+. However, I agree with ted that this sort of thing should be fixed properly in the test harness somehow. Can you file a followup for that?

Comment 14

[Dave Hylands [:dhylands]]

https://hg.mozilla.org/integration/b2g-inbound/rev/d2e113d3f6f9

Comment 15

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/d2e113d3f6f9

Comment 16

[Dave Hylands [:dhylands]]

Comment on attachment 8391966 [details] [diff] [review]
bug-983848.patch

[Approval Request Comment]
Bug caused by (feature/regressing bug #): not sure - happened many months ago
User impact if declined: none (although something could happen since updater tests have been disabled)
Testing completed (on m-c, etc.): updater tests are now running in master.
Risk to taking this patch (and alternatives if risky): Seems low
String or IDL/UUID changes made by this patch: None

Now that bug 994926 has landed, updater tests have been disabled on aurora (since mozharness is shared between master and aurora)

Comment 17

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/8ca89a9ca5be