Closed
Bug 1062255
Opened 11 years ago
Closed 11 years ago
Crash while opening IDB Database prevents any further tests from running
Categories
(Core :: DOM: Device Interfaces, defect)
Tracking
()
RESOLVED
WORKSFORME
| Tracking | Status | |
|---|---|---|
| b2g-v2.1 | --- | unaffected |
| b2g-v2.2 | --- | affected |
People
(Reporter: Bebe, Unassigned)
References
Details
(Keywords: qablocker, regression)
Attachments
(5 files, 1 obsolete file)
Description:
When running tests on the grid marionette can's start a session on the grid
Environment:
application_buildid: 20140903010251
application_changeset: e5d7e65f7087
application_display_name: B2G
application_name: B2G
application_repository: https://hg.mozilla.org/integration/b2g-inbound
application_version: 35.0a1
build_changeset: 74465af039d2809454afdfef285285bb63146e1b
device_firmware_date: 1409732525
device_firmware_version_incremental: eng.cltbld.20140903.042155
device_firmware_version_release: 4.3
device_id: flame
gaia_changeset: b5c782b267e9ef4587c2d76a4d96d5eeda952e97
gaia_date: 1409730158
platform_buildid: 20140903010251
platform_changeset: e5d7e65f7087
platform_repository: https://hg.mozilla.org/integration/b2g-inbound
Traceback (most recent call last):
File "/var/jenkins/2/workspace/flame.b2g-inbound.ui.functional.smoke/.env/local/lib/python2.7/site-packages/marionette_client-0.8.3-py2.7.egg/marionette/marionette_test.py", line 153, in run
self.setUp()
File "/var/jenkins/2/workspace/flame.b2g-inbound.ui.functional.smoke/tests/python/gaia-ui-tests/gaiatest/tests/functional/dialer/test_dialer_receive_call_with_locked_screen.py", line 19, in setUp
GaiaTestCase.setUp(self)
File "/var/jenkins/2/workspace/flame.b2g-inbound.ui.functional.smoke/tests/python/gaia-ui-tests/gaiatest/gaia_test.py", line 744, in setUp
self.device.start_b2g()
File "/var/jenkins/2/workspace/flame.b2g-inbound.ui.functional.smoke/tests/python/gaia-ui-tests/gaiatest/gaia_test.py", line 562, in start_b2g
self.marionette.start_session()
File "/var/jenkins/2/workspace/flame.b2g-inbound.ui.functional.smoke/.env/local/lib/python2.7/site-packages/marionette_client-0.8.3-py2.7.egg/marionette/marionette.py", line 815, in start_session
self.session = self._send_message('newSession', 'value')
File "/var/jenkins/2/workspace/flame.b2g-inbound.ui.functional.smoke/.env/local/lib/python2.7/site-packages/marionette_client-0.8.3-py2.7.egg/marionette/decorators.py", line 35, in _
return func(*args, **kwargs)
File "/var/jenkins/2/workspace/flame.b2g-inbound.ui.functional.smoke/.env/local/lib/python2.7/site-packages/marionette_client-0.8.3-py2.7.egg/marionette/marionette.py", line 621, in _send_message
"Connection timed out", status=errors.ErrorCodes.TIMEOUT)
Reproducible manually: NO
STR:
I can't reproduce the issue locally but we can see it constantly on the grid
http://jenkins1.qa.scl3.mozilla.com/job/flame.b2g-inbound.ui.functional.smoke/1252/consoleFull
Comment 1•11 years ago
|
||
I have seen it locally but very rarely.
B2G just completely fails to start up, the screen is black and the device needs an `adb reboot` to get it going again.
I did b2g-ps and saw that b2g process was running but the device is unusable.
Comment 2•11 years ago
|
||
This is causing us lots of issues in automation at the moment. We started doing full flashes, which coincides with this, so the regression may not be in gaia or gecko. Our flame.b2g-inbound.ui.functional.smoke shows the following regression range:
last good:
application_buildid: 20140902072512
application_changeset: 2f65aa582f06
application_version: 34.0a1
build_changeset: 74465af039d2809454afdfef285285bb63146e1b
device_firmware_date: 1409239766
device_firmware_version_incremental: eng.cltbld.20140828.112916
device_firmware_version_release: 4.3
gaia_changeset: a081afab97fe2d1a25b08f870aa5007ff929dfbb
gaia_date: 1409666241
first bad:
application_buildid: 20140902113703
application_changeset: 1d3b0ec6e32d
application_version: 35.0a1
build_changeset: 74465af039d2809454afdfef285285bb63146e1b
device_firmware_date: 1409685070
device_firmware_version_incremental: eng.cltbld.20140902.151100
device_firmware_version_release: 4.3
gaia_changeset: 7e469783859785a8bd4bf02a802f32561c84be7b
gaia_date: 1409669127
I also noticed the following in the logcat, which looks suspicious to me:
E/GeckoConsole(12048): [JavaScript Error: "IndexedDB UnknownErr: OpenDatabaseHelper.cpp:1901"]
E/GeckoConsole(12048): [JavaScript Error: "NS_ERROR_FAILURE: Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIInputStream.available]" {file: "resource://gre/modules/SettingsDB.jsm" line: 81}]
E/GeckoConsole(12052): Failed to load native module at path '/system/b2g/components/libxpcomsample.so': (80004005) dlopen failed: library "/system/b2g/components/libxpcomsample.so" not found
I/Gecko (12048): -*- SettingsRequestManager: SETTINGS DATABASE ERROR: Cannot make DB connection!
Comment 3•11 years ago
|
||
Attachment #8483444 -
Attachment is obsolete: true
Comment 4•11 years ago
|
||
I've been running tests against the following build without any crashes so far. It's not conclusive but I can pick this up again tomorrow if needed.
application_buildid: 20140902072512
application_changeset: 2f65aa582f06
application_version: 34.0a1
device_firmware_date: 1409671774
device_firmware_version_incremental: eng.cltbld.20140902.112925
device_firmware_version_release: 4.3
gaia_changeset: a081afab97fe2d1a25b08f870aa5007ff929dfbb
gaia_date: 1409666241
Jason: Could you keep an eye out for anything similar, and maybe have a look over the logcat to see if something jumps out at you? Thanks!
Flags: needinfo?(jsmith)
Summary: Marionette can't start a session on the device → Crash prevents any further tests from running
Comment 5•11 years ago
|
||
Comment 6•11 years ago
|
||
(In reply to Dave Hunt (:davehunt) from comment #2)
> This is causing us lots of issues in automation at the moment. We started
> doing full flashes, which coincides with this, so the regression may not be
> in gaia or gecko. Our flame.b2g-inbound.ui.functional.smoke shows the
> following regression range:
>
> last good:
> application_buildid: 20140902072512
> application_changeset: 2f65aa582f06
> application_version: 34.0a1
> build_changeset: 74465af039d2809454afdfef285285bb63146e1b
> device_firmware_date: 1409239766
> device_firmware_version_incremental: eng.cltbld.20140828.112916
> device_firmware_version_release: 4.3
> gaia_changeset: a081afab97fe2d1a25b08f870aa5007ff929dfbb
> gaia_date: 1409666241
>
> first bad:
> application_buildid: 20140902113703
> application_changeset: 1d3b0ec6e32d
> application_version: 35.0a1
> build_changeset: 74465af039d2809454afdfef285285bb63146e1b
> device_firmware_date: 1409685070
> device_firmware_version_incremental: eng.cltbld.20140902.151100
> device_firmware_version_release: 4.3
> gaia_changeset: 7e469783859785a8bd4bf02a802f32561c84be7b
> gaia_date: 1409669127
>
> I also noticed the following in the logcat, which looks suspicious to me:
>
> E/GeckoConsole(12048): [JavaScript Error: "IndexedDB UnknownErr:
> OpenDatabaseHelper.cpp:1901"]
> E/GeckoConsole(12048): [JavaScript Error: "NS_ERROR_FAILURE: Component
> returned failure code: 0x80004005 (NS_ERROR_FAILURE)
> [nsIInputStream.available]" {file: "resource://gre/modules/SettingsDB.jsm"
> line: 81}]
> E/GeckoConsole(12052): Failed to load native module at path
> '/system/b2g/components/libxpcomsample.so': (80004005) dlopen failed:
> library "/system/b2g/components/libxpcomsample.so" not found
> I/Gecko (12048): -*- SettingsRequestManager: SETTINGS DATABASE ERROR:
> Cannot make DB connection!
Kyle - Any ideas on what's failing here?
Flags: needinfo?(jsmith) → needinfo?(kyle)
Comment 7•11 years ago
|
||
Wow, ok, not being able to open the database is REALLY catastrophic. That's like "ran out of disk space or memory" level issues.
I think in this case settings might be our canary for this particular coal mine, as it may be the first thing on the system to try and open/create a file. It's rigged to yell like crazy if anything goes wrong at that point because, well, catastrophic. So I wouldn't blame the API for this, but something is definitely wrong.
Flags: needinfo?(kyle)
Comment 8•11 years ago
|
||
Bebe - How often are we seeing this failure?
Flags: needinfo?(florin.strugariu)
Comment 9•11 years ago
|
||
(In reply to Jason Smith [:jsmith] from comment #8)
> Bebe - How often are we seeing this failure?
According to No-Jun this is consistently happening.
Can we please pull one of these devices out of the pool so that a dev can take a look?
Comment 11•11 years ago
|
||
Dunno if we need a full device. We should just be able to pull the image, since they're just flames. Just takes being on the VPN and I haven't had time to set that up yet.
Comment 12•11 years ago
|
||
(In reply to Jason Smith [:jsmith] from comment #8)
> Bebe - How often are we seeing this failure?
It's affecting most (if not all) of our test runs.
(In reply to ben turner [:bent] (use the needinfo? flag!) from comment #10)
> Can we please pull one of these devices out of the pool so that a dev can
> take a look?
We've just seen this on b2g-23.2, which I've taken temporarily offline so we can investigate. Let me know what you need me to pull from this device (and how) and I will do what I can.
Flags: needinfo?(bent.mozilla)
Comment 13•11 years ago
|
||
The incident mentioned in comment 12 was after performing a full flash from the 31st August and then a shallow flash on top. This implies that the issue was either present in an earlier full flash build than the previous range suggests, or that it is not related to it being a full flash.
Flags: needinfo?(florin.strugariu)
Comment 14•11 years ago
|
||
[Blocking Requested - why for this release]:
Critical QA blocking regression causing all on device automation to be down.
blocking-b2g: --- → 2.1?
Component: Gaia::UI Tests → DOM: Device Interfaces
Keywords: qablocker,
regression
Product: Firefox OS → Core
Updated•11 years ago
|
blocking-b2g: 2.1? → 2.2?
Updated•11 years ago
|
status-b2g-v2.1:
--- → unaffected
status-b2g-v2.2:
--- → affected
Comment 15•11 years ago
|
||
(In reply to Dave Hunt (:davehunt) from comment #13)
> The incident mentioned in comment 12 was after performing a full flash from
> the 31st August and then a shallow flash on top. This implies that the issue
> was either present in an earlier full flash build than the previous range
> suggests, or that it is not related to it being a full flash.
Hey guys, so what can we do here? seems the flashing part is not the culprit. Kyle, any luck on getting your VPN setup yet? do you still need a physical device for debugging this?
Comment 16•11 years ago
|
||
This is proving difficult to get a regression range on. We've been able to reproduce the crash by running a single test 20 times, and if it's going to happen it tends to do so in that time. That said, we've had at least one 'good' build that on a rerun turned out to be not so good. Our best effort at a current regression range is below.
Last good:
application_buildid: 20140902132849
application_changeset: aad03f3096f4
build_changeset: 74465af039d2809454afdfef285285bb63146e1b
device_firmware_date: 1409517083
device_firmware_version_incremental: eng.cltbld.20140831.163113
device_firmware_version_release: 4.3
gaia_changeset: 9f09cbb3d5aaafdf2252442f82f2fc84392fb137
gaia_date: 1409684437
First bad:
application_buildid: 20140903075351
application_changeset: 142047b2389e
build_changeset: 74465af039d2809454afdfef285285bb63146e1b
device_firmware_date: 1409517083
device_firmware_version_incremental: eng.cltbld.20140831.163113
device_firmware_version_release: 4.3
gaia_changeset: d2a1710b3336050a427aca197b44d5821eb68266
gaia_date: 1409753617
Pushlog:
http://hg.mozilla.org/integration/b2g-inbound/pushloghtml?fromchange=aad03f3096f4&tochange=142047b2389e
As we get closer we may find this range shifts a little, but I hope this is at least helpful. Please let me know if there's anything more valuable I can provide.
The most helpful thing somebody could do is step through http://mxr.mozilla.org/mozilla-central/source/dom/indexedDB/OpenDatabaseHelper.cpp#1969 in gdb and see which early return we are taking.
Comment 18•11 years ago
|
||
I have my personal python marionette test environment running daily cronjob on flame device, and also saw this timeout issue over the weekend.
Following is what i got from the zipped archive of the test result:
Last good build for me:
20140901160203 in pvtbuild mozilla central
First bad build for me:
20140902160202, in pvtbuild mozilla central
Comment 19•11 years ago
|
||
(In reply to Tony Chung [:tchung] from comment #15)
> (In reply to Dave Hunt (:davehunt) from comment #13)
> > The incident mentioned in comment 12 was after performing a full flash from
> > the 31st August and then a shallow flash on top. This implies that the issue
> > was either present in an earlier full flash build than the previous range
> > suggests, or that it is not related to it being a full flash.
>
> Hey guys, so what can we do here? seems the flashing part is not the
> culprit. Kyle, any luck on getting your VPN setup yet? do you still need
> a physical device for debugging this?
Based on No-Jun's comment above, Kyle could use this build below to analyze this bug:
https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/mozilla-central-flame/2014/09/2014-09-02-16-02-02/
Comment 20•11 years ago
|
||
(In reply to No-Jun Park [:njpark] from comment #18)
> I have my personal python marionette test environment running daily cronjob
> on flame device, and also saw this timeout issue over the weekend.
>
> Following is what i got from the zipped archive of the test result:
>
> Last good build for me:
> 20140901160203 in pvtbuild mozilla central
>
> First bad build for me:
> 20140902160202, in pvtbuild mozilla central
just to note that this is m-c nightly build in pvtbuilds.
Comment 21•11 years ago
|
||
Here's the latest range according to our automation results:
Last good:
application_buildid: 20140902211838
application_changeset: 01962a67b5d8
build_changeset: 74465af039d2809454afdfef285285bb63146e1b
device_firmware_date: 1409517083
device_firmware_version_incremental: eng.cltbld.20140831.163113
device_firmware_version_release: 4.3
gaia_changeset: e1f6d88db39084c98a7d4e361ba30447a8b4264e
gaia_date: 1409716490
First bad:
application_buildid: 20140903010251
application_changeset: e5d7e65f7087
build_changeset: 74465af039d2809454afdfef285285bb63146e1b
device_firmware_date: 1409517083
device_firmware_version_incremental: eng.cltbld.20140831.163113
device_firmware_version_release: 4.3
gaia_changeset: b5c782b267e9ef4587c2d76a4d96d5eeda952e97
gaia_date: 1409730158
Pushlog: http://hg.mozilla.org/integration/b2g-inbound/pushloghtml?fromchange=01962a67b5d8&tochange=e5d7e65f7087
Again, we can't be 100% confident on the 'good' because of the intermittent behaviour of the crash, but we're not considering a build to be 'bad' unless we see a crash detected by Marionette. There's just one build left to test within this range, however if the pushlog looks unlikely to have caused this regression then we can continue to test some of the builds marked as good, to see if we can improve the range.
Comment 22•11 years ago
|
||
(In reply to Jason Smith [:jsmith] from comment #19)
> (In reply to Tony Chung [:tchung] from comment #15)
> > (In reply to Dave Hunt (:davehunt) from comment #13)
> > > The incident mentioned in comment 12 was after performing a full flash from
> > > the 31st August and then a shallow flash on top. This implies that the issue
> > > was either present in an earlier full flash build than the previous range
> > > suggests, or that it is not related to it being a full flash.
> >
> > Hey guys, so what can we do here? seems the flashing part is not the
> > culprit. Kyle, any luck on getting your VPN setup yet? do you still need
> > a physical device for debugging this?
>
> Based on No-Jun's comment above, Kyle could use this build below to analyze
> this bug:
>
> https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/mozilla-
> central-flame/2014/09/2014-09-02-16-02-02/
Do we have a minidump anywhere when the crash happens that can be attached to this bug?
Comment 23•11 years ago
|
||
(In reply to Jason Smith [:jsmith] from comment #19)
> https://pvtbuilds.mozilla.org/pvt/mozilla.org/b2gotoro/nightly/mozilla-
> central-flame/2014/09/2014-09-02-16-02-02/
Just flashed this to the phone. Boots fine here.
Comment 24•11 years ago
|
||
And hit enter too early on that last comment. Now running automation tests to see if I can get a repro.
Comment 25•11 years ago
|
||
So do we know if this has something to do with running tests on the phone, or is it just something that happens when you reboot a whole bunch? Maybe someone should just try running "adb reboot" in a loop for a while?
Comment 26•11 years ago
|
||
Ok, just hit this myself, but my logcat looks different. We aren't even getting to the point of settings complaining it can't open its database now. Lots of errors from indexeddb.
Comment 27•11 years ago
|
||
Reboot is NOT required to fix this. Simply bringing the b2g process down and back up using stop/start fixes it and B2G comes up fine.
Comment 28•11 years ago
|
||
Ran into the same problem while running the smoketest on my local flame, with v123 base + fullflash of mozilla-eng from today. Attaching stack trace in case it can provide any more clues.
Comment 29•11 years ago
|
||
(In reply to Robert Wood [:rwood] from comment #28)
> Created attachment 8485918 [details]
> timeout_console.txt
>
> Ran into the same problem while running the smoketest on my local flame,
> with v123 base + fullflash of mozilla-eng from today. Attaching stack trace
> in case it can provide any more clues.
did restarting b2g process fix things for you?
Comment 30•11 years ago
|
||
I'm running a build right now with instrumentation and debug messages that :ehsan put in. That said, I have some thoughts on what might be causing this. Note that this is a pure guess until I actually get some proof. :)
Between the automation tests, we just run something equivilent to adb stop b2g/adb start b2g. We don't actually reboot the phone. This just kills the b2g process. This means that we're basically simulating a crash in b2g, then bringing the process back up. While we've done that for a long time, it could be that our changes to how things work with settings and IDB mean we're accessing the IDB files before OS level exclusivity locks have cleared. Hence OpenDatabaseHelper gets stuck trying to open the DB.
Comment 31•11 years ago
|
||
What's happening here is |transaction.Commit();| here <http://mxr.mozilla.org/mozilla-central/source/dom/indexedDB/OpenDatabaseHelper.cpp#2160> fails with 0x80630002 which is NS_ERROR_STORAGE_IOERR. That is raised from <http://mxr.mozilla.org/mozilla-central/source/storage/src/mozStoragePrivateHelpers.cpp#57> which is a result of SQLITE_IOERR. I don't know exactly what that means, but google suggests that it has something to do with the db being locked. Comment 30 might be on to something.
Comment 32•11 years ago
|
||
Content of idb directory at the time of freeze
Comment 33•11 years ago
|
||
https://pastebin.mozilla.org/6390073 contains the dmesg output of the device.
Comment 34•11 years ago
|
||
I'm still following up on Dave's comment #21 but we keep getting the range changed due to false positives in early regression range finding. I'll update later today.
Comment 35•11 years ago
|
||
(In reply to Jason Smith [:jsmith] from comment #6)
> > I also noticed the following in the logcat, which looks suspicious to me:
> >
> > E/GeckoConsole(12048): [JavaScript Error: "IndexedDB UnknownErr:
> > OpenDatabaseHelper.cpp:1901"]
> > E/GeckoConsole(12048): [JavaScript Error: "NS_ERROR_FAILURE: Component
> > returned failure code: 0x80004005 (NS_ERROR_FAILURE)
> > [nsIInputStream.available]" {file: "resource://gre/modules/SettingsDB.jsm"
> > line: 81}]
> > E/GeckoConsole(12052): Failed to load native module at path
> > '/system/b2g/components/libxpcomsample.so': (80004005) dlopen failed:
> > library "/system/b2g/components/libxpcomsample.so" not found
> > I/Gecko (12048): -*- SettingsRequestManager: SETTINGS DATABASE ERROR:
> > Cannot make DB connection!
>
> Kyle - Any ideas on what's failing here?
This is also mentioned in bug 1064718.
Comment 36•11 years ago
|
||
If the harness (or a component of it or the way we run it) changed, could the range of interest be in that code and not in gecko?
Comment 37•11 years ago
|
||
My regression hunting today on b2g-i range narrowed it down to a m-c -> b2g-i merge commit.
The m-c commits that it included are:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=504e18c45b86&tochange=1d3b0ec6e32d
(I know this range is older than those cited but we found that going back in builds this became more difficult to replicate and thus led to several false 'last good builds' in the range).
Comment 38•11 years ago
|
||
I've narrowed it further to:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=c24ec895b156&tochange=7753c52d5976
Comment 39•11 years ago
|
||
(In reply to Zac C (:zac) from comment #38)
> I've narrowed it further to:
>
> http://hg.mozilla.org/mozilla-central/
> pushloghtml?fromchange=c24ec895b156&tochange=7753c52d5976
I think the following are suspicious (but I'm just shooting in the dark):
http://hg.mozilla.org/mozilla-central/rev/0a76964b8d8d
http://hg.mozilla.org/mozilla-central/rev/9678f9a596d8
http://hg.mozilla.org/mozilla-central/rev/99934eed086c
http://hg.mozilla.org/mozilla-central/rev/4d991522bbe3
http://hg.mozilla.org/mozilla-central/rev/6baeffddba47
http://hg.mozilla.org/mozilla-central/rev/bab6f837fc64
Comment 40•11 years ago
|
||
Shouldn't be http://hg.mozilla.org/mozilla-central/rev/9678f9a596d8 because that got backed out later and has yet to reland
Comment 41•11 years ago
|
||
Absolutely no clue if this is related, but today I've started to receive complaints of phones slowing down after about an hour of usage. See bug 1064624, bug 1064718.
mac- on IRC ran cleopatra profiles against their phone, seeing things dying in places (the system browser history module) there, mostly on IDB calls.
http://people.mozilla.org/~bgirard/cleopatra/#report=9444aa143c5f1b100a76f4602edd2339176309fa
http://people.mozilla.org/~bgirard/cleopatra/#report=5ec6ddc4d4087d8c82d96a806cbf951780d137e2
Attaching logcat to this bug, but I'm not seeing much in the way of IDB errors in it.
Comment 42•11 years ago
|
||
Making title more descriptive.
Summary: Crash prevents any further tests from running → Crash while opening IDB Database prevents any further tests from running
Comment 43•11 years ago
|
||
Comment 44•11 years ago
|
||
Zac, have you bisected this on the gaia side? Given the fact that many other tests examine indexed db and this is the only thing that is broken, it's extremely likely that the regression is on the gaia side (either in gaia code, or the test harness, etc.)
Flags: needinfo?(zcampbell)
Comment 45•11 years ago
|
||
Fabrice mentioned that backing out http://hg.mozilla.org/mozilla-central/rev/0a76964b8d8d cleared up error messages for him on Tarako v2.2 testing. This might be worth doing. I'd do it now except it's midnight. Zac, if you could try a build with this commit backed out, that'd be great.
Comment 46•11 years ago
|
||
(In reply to :Ehsan Akhgari (not reading bugmail, needinfo? me!) from comment #44)
> Zac, have you bisected this on the gaia side? Given the fact that many
> other tests examine indexed db and this is the only thing that is broken,
> it's extremely likely that the regression is on the gaia side (either in
> gaia code, or the test harness, etc.)
While testing the b2g-i range we just used the releng/tbpl built Gaia profile and the range also initially included a lot of Bumperbot commits, so yes we have, but for the tests we've always just run from the tip. I don't feel the latter has affected my ability to replicate the failure. I agree though, it smells like a Gaia problem and I'll be upset if I find my regression range wrong again!
Flags: needinfo?(zcampbell)
Comment 47•11 years ago
|
||
(In reply to Kyle Machulis [:kmachulis] [:qdot] (USE NEEDINFO?) from comment #45)
> Fabrice mentioned that backing out
> http://hg.mozilla.org/mozilla-central/rev/0a76964b8d8d cleared up error
> messages for him on Tarako v2.2 testing. This might be worth doing. I'd do
> it now except it's midnight. Zac, if you could try a build with this commit
> backed out, that'd be great.
This is in my m-i target so I'll run with the tbpl builds before and after (easier for me to just use the pre-built ones).
Comment 48•11 years ago
|
||
Running locally with gaiatest/test framework and packages on a static commit (to eliminate comment #45) I've narrowed down the last good and first bad mozilla-inbound builds.
I only used TBPL builds owing to time/capacity which has given us a range of 3 commits:
First failing
https://tbpl.mozilla.org/?showall=1&tree=Mozilla-Inbound&rev=e567693fa55f
Crashed after 3 tests run on two successive attempts.
Last successful
https://tbpl.mozilla.org/?showall=1&tree=Mozilla-Inbound&rev=9678f9a596d8
On the successful build there were 100 tests run without crash whereas in previous regression hunting a crash can be found in <60.
The range includes bug 1048011 that Kyle referred to in comment #45 and which Fabrice also validated.
http://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=9678f9a596d8&tochange=e567693fa55f
I'll ask for a backout now.
Depends on: 1048011
Comment 49•11 years ago
|
||
Once we confirm post backout that this no longer reproduces, let's get this bug closed out.
Comment 50•11 years ago
|
||
(In reply to Jason Smith [:jsmith] from comment #49)
> Once we confirm post backout that this no longer reproduces, let's get this
> bug closed out.
Zac, thank you for the deep investigation!
Kyle, Fabrice, can you help with the backout per comment 48?
Flags: needinfo?(kyle)
Flags: needinfo?(fabrice)
Comment 51•11 years ago
|
||
(In reply to Tony Chung [:tchung] from comment #50)
> (In reply to Jason Smith [:jsmith] from comment #49)
> > Once we confirm post backout that this no longer reproduces, let's get this
> > bug closed out.
>
> Zac, thank you for the deep investigation!
>
> Kyle, Fabrice, can you help with the backout per comment 48?
sorry, ignore me. We'll have QA confirm the backout fixes as soon as we can.
Flags: needinfo?(kyle)
Flags: needinfo?(florin.strugariu)
Flags: needinfo?(fabrice)
Comment 52•11 years ago
|
||
Once we've verified that the backout fixes this, if you need assistance with the backout from mozilla-central, ping me on IRC please.
Comment 53•11 years ago
|
||
(In reply to :Ehsan Akhgari (not reading bugmail, needinfo? me!) from comment #52)
> Once we've verified that the backout fixes this, if you need assistance with
> the backout from mozilla-central, ping me on IRC please.
Nevermind, it has already been backed out!
Comment 54•11 years ago
|
||
The bug that caused this was backed out in https://bugzilla.mozilla.org/show_bug.cgi?id=1048011
Device tests are back up and running again!
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → WORKSFORME
| Reporter | ||
Comment 55•11 years ago
|
||
As Zac said looks like the builds are coming back to normal.
Flags: needinfo?(florin.strugariu)
Updated•11 years ago
|
Flags: needinfo?(bent.mozilla)
Comment 56•11 years ago
|
||
After opened a few webpages I've got again same as in Comment 41, the new profile:
http://people.mozilla.org/~bgirard/cleopatra/#report=30441400a456e1d2a6fb6aff7349a0fcda7b0338
taken on:
Alcatel One Touch Fire production (got from T-mobile Poland)
B2G version: 2.2.0.0-prerelease master
Platform version: 35.0a1
Build Identifier: 20140914040220
Git commit info: 2014-09-13 09:23:34 e5da0e46
Comment 57•11 years ago
|
||
(In reply to mac from comment #56)
> After opened a few webpages I've got again same as in Comment 41, the new
> profile:
>
> http://people.mozilla.org/~bgirard/cleopatra/
> #report=30441400a456e1d2a6fb6aff7349a0fcda7b0338
>
> taken on:
>
> Alcatel One Touch Fire production (got from T-mobile Poland)
> B2G version: 2.2.0.0-prerelease master
> Platform version: 35.0a1
> Build Identifier: 20140914040220
> Git commit info: 2014-09-13 09:23:34 e5da0e46
Please file a new bug with more information if this is still reproducing for you. Developers request a crash log or mini dump if possible. This original bug is now marked resolved after backing out the culprit landing from comment 45.
You need to log in
before you can comment on or make changes to this bug.
Description
•