921721 - NS_ERROR_UNEXPECTED in airplane_mode.js when restarting B2G

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia, defect)

Description

[Dave Hunt [:davehunt] [he/him] ⌚BST]

The Gaia UI test, b2gperf, and b2gpopulate often stop/start the B2G process, and I've recently noticed the following exception a few times during the startup. At this point we're running asynchronous JavaScript and waiting for the FTU/Homescreen app to fire the appload event, so we catch any JavaScript exceptions thrown in that time:

Traceback (most recent call last):
  File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.master.perf.fps/.env/bin/b2gpopulate", line 9, in <module>
    load_entry_point('b2gpopulate==0.9', 'console_scripts', 'b2gpopulate')()
  File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.master.perf.fps/.env/local/lib/python2.7/site-packages/b2gpopulate/b2gpopulate.py", line 297, in cli
    options.video_count)
  File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.master.perf.fps/.env/local/lib/python2.7/site-packages/b2gpopulate/b2gpopulate.py", line 69, in populate
    self.populate_messages(message_count)
  File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.master.perf.fps/.env/local/lib/python2.7/site-packages/b2gpopulate/b2gpopulate.py", line 164, in populate_messages
    self.device.start_b2g()
  File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.master.perf.fps/.env/local/lib/python2.7/site-packages/gaiatest/gaia_test.py", line 391, in start_b2g
    });""", script_timeout=60000)
  File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.master.perf.fps/.env/local/lib/python2.7/site-packages/marionette/marionette.py", line 624, in execute_async_script
    scriptTimeout=script_timeout)
  File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.master.perf.fps/.env/local/lib/python2.7/site-packages/marionette/marionette.py", line 350, in _send_message
    self._handle_error(response)
  File "/var/jenkins/workspace/b2g.hamachi.mozilla-central.master.perf.fps/.env/local/lib/python2.7/site-packages/marionette/marionette.py", line 385, in _handle_error
    raise JavascriptException(message=message, status=status, stacktrace=stacktrace)
marionette.errors.JavascriptException: NS_ERROR_UNEXPECTED: 
stacktrace:
	@app://system.gaiamobile.org/js/airplane_mode.js, line 65

When this happens we lose a lot of test coverage.

Comment 1

[Zac C (:zac)]

Still affecting gaia UI test automation too.

Device Hamachi
Gecko  http://hg.mozilla.org/mozilla-central/rev/febfe3c7732b
Gaia  17e871ae1f82699793e3cd28acda805ba724a8b6
BuildID 20131015040202
Version 27.0a1

Comment 2

[Jason Smith [:jsmith]]

Tim - Could someone from your team help diagnose and fix this bug? It's impacting testing reliability of running Gaia UI Tests, b2gperf and b2gpopulate.

Comment 3

[Tim Guan-tin Chien [:timdream] (please needinfo)]

https://github.com/mozilla-b2g/gaia/blob/master/apps/system/js/airplane_mode.js#L65

I wonder if mozSettings is really the cause of this bug, there is not enough information here.

Comment 4

[Jason Smith [:jsmith]]

Gregor - We're hitting this intermittent failure on perf & ui automation. As a result, it's causing various tests to unexpectedly fail. Is there someone on the System FE team that could help out on this bug?

Comment 5

[Gregor Wagner [:gwagner]]

Yeah the current approach with timeOut doesn't work and is error prone.
Lets see if this fixes it: https://github.com/mozilla-b2g/gaia/pull/13154

Comment 6

[Jason Smith [:jsmith]]

(In reply to Gregor Wagner [:gwagner] from comment #5)
> Yeah the current approach with timeOut doesn't work and is error prone.
> Lets see if this fixes it: https://github.com/mozilla-b2g/gaia/pull/13154

Should this patch be put out for review formally? Or do you want us to try testing this patch in Gaia UI Tests to see if it resolves the issue?

Comment 7

[Gregor Wagner [:gwagner]]

Attachment: PR

Comment 8

[Gregor Wagner [:gwagner]]

If it's easy to test, please check if this patch fixes the problem. Otherwise we can wait until it gets merged.

Comment 9

[Dave Hunt [:davehunt] [he/him] ⌚BST]

I've never seen this issue locally, only in CI, so it will be difficult to confirm if this patch fixes the issue. I would say if this passes review from a peer and merges, we should know if the issue is fixed by looking through the CI console logs after a few days.

Comment 10

[Gregor Wagner [:gwagner]]

https://github.com/mozilla-b2g/gaia/commit/0423c447978a591e42d9db489ca7420039ead150

Comment 11

[Zac C (:zac)]

Yes we get this about 1-2 in 100 tests but fairly reliably. I should know in a day or two whether this patch has done the trick.

Comment 12

[Zac C (:zac)]

This patch has caused a new error and would seem to be related:

JavascriptException: NS_ERROR_UNEXPECTED: 
	stacktrace:
	@app://system.gaiamobile.org/js/storage.js, line 36

which would seem to be related up the chain to the change to shared getSettingsLock

Comment 13

[Zac C (:zac)]

It actually doesn't fix the original failure but as the new trace occurs after the original one it's the one that is picked up by Marionette.

This affects *every* test because it stops the framework from starting up the device cleanly.

This is 100% replicable.




E/GeckoConsole( 1626): [JavaScript Error: "Permission denied to access property 'toString'"]
E/GeckoConsole( 1626): [JavaScript Error: "Permission denied to access property 'message'"]
E/GeckoConsole( 1626): [JavaScript Error: "uncaught exception: unknown (can't convert to string)"]
E/GeckoConsole( 1626): [JavaScript Error: "NS_ERROR_UNEXPECTED: " {file: "app://system.gaiamobile.org/js/storage.js" line: 36}]
I/GeckoDump( 1626): XXX FIXME : Got a mozContentEvent: accessibility-screenreader
E/GeckoConsole( 1626): [JavaScript Error: "Permission denied to access property 'toString'"]
E/GeckoConsole( 1626): [JavaScript Error: "Permission denied to access property 'message'"]
E/GeckoConsole( 1626): [JavaScript Error: "uncaught exception: unknown (can't convert to string)"]
E/GeckoConsole( 1626): [JavaScript Error: "NS_ERROR_UNEXPECTED: " {file: "app://system.gaiamobile.org/js/airplane_mode.js" line: 65}]

Comment 14

[Gregor Wagner [:gwagner]]

Where does this happen? Travis and tbpl look green to me. How can I reproduce this?

Comment 15

[Zac C (:zac)]

desktopb2g, travis, tbpl etc won't pick this up. You need to test it on a device. 

You can see the exception just by setting up "adb logcat | grep Gecko" , starting up the device and looking out for the exception.

I am using a Hamachi with a Gaia revision that includes your patch.

Comment 16

[Gregor Wagner [:gwagner]]

Hm it worked on my nexus4 :(
Please back out

Comment 17

[Zac C (:zac)]

Well Firefox OS won't fail, it will boot fine and seem functional. but the error is in the logcat and may affect some thing else deeper in the OS?

Comment 18

[Jason Smith [:jsmith]]

(In reply to Gregor Wagner [:gwagner] from comment #16)
> Hm it worked on my nexus4 :(
> Please back out

John - Can you back this patch out?

Comment 19

[John Ford [:jhford] CET/CEST Berlin Time]

[master b2659e1] bug 921721 - Revert "Merge pull request #13154 from gregorwagner/settings3"

Comment 20

[Gregor Wagner [:gwagner]]

I couldn't reproduce this on the phone with debug build and b2g-destkop with debug and optimize build.
Todays build should have it in. Jason do you see this on any phone?

Comment 21

[Jason Smith [:jsmith]]

(In reply to Gregor Wagner [:gwagner] from comment #20)
> I couldn't reproduce this on the phone with debug build and b2g-destkop with
> debug and optimize build.
> Todays build should have it in. Jason do you see this on any phone?

I'm probably not the right person to verify this - Zac & Dave were able to reproduce this originally. Moving needinfo to Zac.

Comment 22

[Gregor Wagner [:gwagner]]

Ok I can see this error now. It seems to be a local problem with storage.js
debugging now.

Comment 23

[Gregor Wagner [:gwagner]]

Attachment: bug2.diff

Comment 24

[Gregor Wagner [:gwagner]]

remote:   https://hg.mozilla.org/integration/b2g-inbound/rev/13fcbae69a9e

Comment 25

[Gregor Wagner [:gwagner]]

I landed the gecko part on b2g-inbound and it seems like the gaia part never got backed out. So I don't have to land it again :)

Zac, let me know how our testing framework is doing once it picks up the gecko patch

Comment 26

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/13fcbae69a9e

Comment 27

[Zac C (:zac)]

Gregor, yes now it is working fine. 

We have a workaround we can use which is why we were not affected by the non-back out, but now I'll revert the workaround :)

Comment 28

[Dave Hunt [:davehunt] [he/him] ⌚BST]

The performance tests (b2gperf and eideticker) have also recovered from this now. Thanks.