878878 - Intermittent B2G/ Gaia crash during UI automation - [@ FreeArenaList]

Status: RESOLVED INVALID

(Firefox OS Graveyard :: General, defect)

Description

[Zac C (:zac)]

We've suffered this crash during test automation and need some more help debugging it.

It caused the device to crash/reboot and the test run cannot recover. 

Gaia: b2g18/v1-train
Build ID 20130530230210

https://crash-stats.mozilla.com/report/index/4d6d0451-9279-4b90-a8e9-5d1652130601

The console trace from the test failure is here:
http://pastebin.mozilla.org/2472425
but I don't suspect that command directly because usually that snippet of code in this test is pretty bulletproof; we don't suffer failures here often.

Comment 1

[Jason Smith [:jsmith]]

Looks very SSL-esque. bsmith - Any ideas?

Comment 2

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

Is this happening a lot? Is it happening mostly/only during shutdown?

We have seen this crash on other platforms and I am working on a fix. 

Please see bug 874049 and bug 874147. This crashing during test runs indicates that the test infrastructure is not set up to disable the features that make network requests outside the local network. You should investigate this aspect independently of my fix for the bug, because those network requests tend to destabilize tests (increase intermittent orange).

Comment 3

[Zac C (:zac)]

Brian, it happens once every couple of hundred tests but when it does it knocks out the remainder of the test suite so the impact is large, we might lose a few dozen test results. These tests don't report to TBPL and they use outside resources quite often (indeed, need outside resources due to the nature of the tests).

It doesn't happen during shutdown; I've only seen it during a test case 'normal' operation as controlled by Marionette.

I'll try a bit more to replicate it today as I have a spare device on hand. That way I can submit the crash report before the build environment overwrites it.

Comment 4

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Zac C (:zac) from comment #3)
> Brian, it happens once every couple of hundred tests but when it does it
> knocks out the remainder of the test suite so the impact is large, we might
> lose a few dozen test results. These tests don't report to TBPL and they use
> outside resources quite often (indeed, need outside resources due to the
> nature of the tests).

Just for clarity "couple of hundred tests" is two test-suite runs, as we have 120 or so tests that we run each time, per-suite; thx.

Comment 5

[Stephen Donner [:stephend] Not actively reading bugmail]

Zac, is this, or bug 879451 (or both?) caused by the delete_bookmark() call you reference in the now-removed https://github.com/mozilla/gaia-ui-tests/commit/2a9ae790aed793c05c1769efea081e0e77b6e5f4?

Comment 6

[Zac C (:zac)]

Stephen, for the same "steps to replicate" I caused 3 different crash reports: this one, bug 879451 and bug 879303.

I can't tell if they're all related but they are definitely caused by that snippet of code, but also not only that snippet of code. I've seen crashes again today since I removed that piece of code so it's deeper than just that.

There is even a 4th one I caused today for which I haven't filed a bug but seems related to this bug:
https://crash-stats.mozilla.com/report/index/647e0207-8bea-45d0-a18b-cd58d2130606

I caused that one using a different test case but not as easily replicated.

Comment 7

[Stephen Donner [:stephend] Not actively reading bugmail]

Talked to :qdot today, and while he's getting Valgrind set up to again work with B2G, there's a chance he could look at this and the other crashes we're seeing.

Comment 8

[Please do not use this account; type :nhirata]

Zac, is it possible to either put the list of bookmarks here, or send me a pointer to where I can find that list of bookmarks please?

Comment 9

[Zac C (:zac)]

nhirata, the bookmark added was
http://mozqa.com/data/firefox/layout/mozilla.html
with a name of something like "gaia 

After I made the changed commented upon in c#5 I saw this crash during this test again so I think now that I was deceived by randomness and further so after the comment in #879451.

Is this popping up in manual testing?

I've never seen this on b2g24, incidentally.

Comment 10

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Is the bookmark list prepopulated even more than that?  I find it odd that it would crash on that particular webpage.  It's pretty plain unless I've overlooked something.
I was thinking maybe it has to do with loading an icon from a https page that is off screen or something...

Are you using a VPN tunnel?  I kinda doubt that would cause an issue, but you never know...

Comment 11

[Zac C (:zac)]

All the test does is open the browser app, load that page, tap the star, add to homescreen, type in the name and then check it appears. It's tedious to try and replicate it manually, I tried briefly but no luck.

There's no VPN tunnel, just a connection to whatever WiFi is in the area.

Comment 12

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

I think I was a little vague before; I am concerned about the data that the Top Sites, Bookmarks, and History contains.  I am asking for the profile/database for the places db.  Would it be possible to provide that please?  I think just zipping the profile and then having a look might be easier.

Comment 13

[Kyle Machulis [:qdot] [:kmachulis] (INACTIVE)]

I wouldn't block this waiting for valgrind, we're having problems getting it running stable on phones at the moment due to memory issues. Once we /do/ get that going, I'm pretty sure a test suite like this may take literally a full day to run.

Comment 14

[Legacy TBPL/Treeherder Robot]

jesup
https://tbpl.mozilla.org/php/getParsedLog.php?id=25265202&tree=Mozilla-Beta
Android Tegra 250 mozilla-beta opt test plain-reftest-4 on 2013-07-13 10:46:02
revision: ea93b56e969c
slave: tegra-208

PROCESS-CRASH | Shutdown | application crashed [@ FreeArenaList]

Comment 15

[Zac C (:zac)]

We can now recover from and thus pinpoint individual crashes