990837 - [MTBF] Phone Hang/Crashed with no adb/fastboot

Status: RESOLVED WORKSFORME

(Firefox OS Graveyard :: Gaia::System, defect, P3)

Description

[Walter Chen[:ypwalter][:wachen]]

Got 3/3 files.
Pulled files into about-memory-153.

To view this report, open Firefox on this machine and load the
following URL:

  about:memory?file=/home/wachen/FirefoxOS/MTBF-Driver/about-memory-153/memory-reports

Pulling GC/CC logs...
Got 6/6 files.
Pulled files into about-memory-153.
Compressing logs...
starting httpd
running webserver on http://10.247.24.65:40837/
test_keyboard_basic (test_mtbf_keyboard.TestKeyboard) ... ok

----------------------------------------------------------------------
Ran 1 test in 56.519s

OK
test_number_keyboard (test_mtbf_number_keyboard.TestNumberKeyboard) ... ok

----------------------------------------------------------------------
Ran 1 test in 29.131s

OK
test_keyboard_predictive_key (test_mtbf_keyboard_predictive_key.TestKeyboardPredictiveKey) ... ok

----------------------------------------------------------------------
Ran 1 test in 38.276s

OK
test_basic_email_keyboard (test_mtbf_email_keyboard.TestEmailKeyboard) ... ERROR

======================================================================
ERROR: None
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/marionette_client-0.7.1-py2.7.egg/marionette/marionette_test.py", line 127, in run
    self.setUp()
  File "/home/wachen/FirefoxOS/gaia_for_uitest/tests/python/gaia-ui-tests/gaiatest/tests/mtbf/keyboard/test_mtbf_email_keyboard.py", line 14, in setUp
    GaiaMtbfTestCase.setUp(self)
  File "/home/wachen/FirefoxOS/MTBF-Driver/MtbfTestCase.py", line 74, in setUp
    self.cleanup_gaia(full_reset=True)
  File "/home/wachen/FirefoxOS/MTBF-Driver/MtbfTestCase.py", line 106, in cleanup_gaia
    for filename in self.data_layer.media_files:
  File "/home/wachen/FirefoxOS/gaia_for_uitest/tests/python/gaia-ui-tests/gaiatest/gaia_test.py", line 372, in media_files
    result.extend(self.music_files)
  File "/home/wachen/FirefoxOS/gaia_for_uitest/tests/python/gaia-ui-tests/gaiatest/gaia_test.py", line 392, in music_files
    'return GaiaDataLayer.getAllMusic();')
  File "/usr/local/lib/python2.7/dist-packages/marionette_client-0.7.1-py2.7.egg/marionette/marionette.py", line 1073, in execute_async_script
    filename=os.path.basename(frame[0]))
  File "/usr/local/lib/python2.7/dist-packages/marionette_client-0.7.1-py2.7.egg/marionette/marionette.py", line 567, in _send_message
    raise TimeoutException(message='socket.timeout', status=ErrorCodes.TIMEOUT, stacktrace=None)
TEST-UNEXPECTED-FAIL | test_mtbf_email_keyboard.py test_mtbf_email_keyboard.TestEmailKeyboard.test_basic_email_keyboard | TimeoutException: socket.timeout
----------------------------------------------------------------------
Ran 1 test in 363.185s

FAILED (errors=1)
test_url_keyboard (test_mtbf_url_keyboard.TestUrlKeyboard) ... ERROR

======================================================================
ERROR: None
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/marionette_client-0.7.1-py2.7.egg/marionette/marionette_test.py", line 127, in run
    self.setUp()
  File "/home/wachen/FirefoxOS/gaia_for_uitest/tests/python/gaia-ui-tests/gaiatest/tests/mtbf/keyboard/test_mtbf_url_keyboard.py", line 13, in setUp
    GaiaMtbfTestCase.setUp(self)
  File "/home/wachen/FirefoxOS/MTBF-Driver/MtbfTestCase.py", line 29, in setUp
    MarionetteTestCase.setUp(self)
  File "/usr/local/lib/python2.7/dist-packages/marionette_client-0.7.1-py2.7.egg/marionette/marionette_test.py", line 312, in setUp
    CommonTestCase.setUp(self)
  File "/usr/local/lib/python2.7/dist-packages/marionette_client-0.7.1-py2.7.egg/marionette/marionette_test.py", line 247, in setUp
    self.marionette.start_session()
  File "/usr/local/lib/python2.7/dist-packages/marionette_client-0.7.1-py2.7.egg/marionette/marionette.py", line 681, in start_session
    self.session = self._send_message('newSession', 'value')
  File "/usr/local/lib/python2.7/dist-packages/marionette_client-0.7.1-py2.7.egg/marionette/marionette.py", line 567, in _send_message
    raise TimeoutException(message='socket.timeout', status=ErrorCodes.TIMEOUT, stacktrace=None)
TEST-UNEXPECTED-FAIL | test_mtbf_url_keyboard.py test_mtbf_url_keyboard.TestUrlKeyboard.test_url_keyboard | TimeoutException: socket.timeout
----------------------------------------------------------------------
Ran 1 test in 360.078s

FAILED (errors=1)

SUMMARY
-------
passed: 3
failed: 2
todo: 0

FAILED TESTS
-------
test_mtbf_email_keyboard.py test_mtbf_email_keyboard.TestEmailKeyboard.test_basic_email_keyboard
test_mtbf_url_keyboard.py test_mtbf_url_keyboard.TestUrlKeyboard.test_url_keyboard

Comment 1

[Walter Chen[:ypwalter][:wachen]]

Attachment: Phone after ran the MTBF testing

no adb connection, no fastboot connection, lsusb works:
Bus 003 Device 067: ID 05c6:9025 Qualcomm, Inc.

I reproduced this in v1.2, v1.3, and v1.4. Simply use MTBF testing with keyboard test suite.

Comment 2

[Walter Chen[:ypwalter][:wachen]]

Memory Report Dropbox location:
https://drive.google.com/file/d/0BwDTW9d_sBJMYTRpRjd2QXZ1ZEE/edit?usp=sharing

about-memory-1:
40.62 MB (100.0%) -- explicit
92 (100.0%) -- event-counts
about-memory-70:
67.33 MB (100.0%) -- explicit
6,925 (100.0%) -- event-counts
about-memory-153:
92.76 MB (100.0%) -- explicit
14,395 (100.0%) -- event-counts

This is increasing rapidly 
13.90 MB (14.98%) -- compartment([System Principal], inProcessTabChildGlobal?ownedBy=chrome://browser/content/shell.html)

Comment 3

[Walter Chen[:ypwalter][:wachen]]

Hi, Robert,
Can you help us once again? (If you don't have free time, please ignore this request)
If possible, try to run keyboard testing for 12+ hours.

Comment 4

[Walter Chen[:ypwalter][:wachen]]

Also, there is no crash report found after reboot.

Comment 5

[Walter Chen[:ypwalter][:wachen]]

Besides https://drive.google.com/file/d/0BwDTW9d_sBJMYTRpRjd2QXZ1ZEE/edit?usp=sharing
Same memory report is available in https://www.dropbox.com/s/gla8kj0tu17lpn5/about-memory-bug990837.tgz

Comment 7

[Walter Chen[:ypwalter][:wachen]]

Hi, Brian and Kevin,
This can be duplicated in multiple branches, and it needs to be concerned. Please do help us on this on.

Comment 8

[Brian Huang [:brianhuang][:brhuang]]

Ivan, Could you have someone to take a look on this? We use keyboard test scenarios to do test.

Comment 9

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

I don't see the thing from comment 2 in the files in comment 5.  I do see the "UI Test" process increasing a lot, and I see

9.47 MB (100.0%) -- explicit
├──3.92 MB (41.39%) ── freetype
├──3.87 MB (40.85%) -- workers
│  ├──4.00 MB (42.23%) -- workers(gaiamobile.org)/worker(js/imes/latin/worker.js, 0xNNN)

In the parent.  I don't really know anything about IMEs, unfortunately.

Comment 10

[Kai-Chih Hu [:khu]]

Do we know what operations were done right before the crash? Thanks.

Comment 11

[Walter Chen[:ypwalter][:wachen]]

I'd say, we kind of know it's running basic keyboard testing. From the log, we can tell it is ok doing OK
test_keyboard_predictive_key. That is to say, we were entering some string and see if predictive of words are correct. And, then it crashed in general cleaning up in setUp()

Comment 12

[Walter Chen[:ypwalter][:wachen]]

Hi, Kyle,

1. Result in comment 2
It's in memory-reports file. I was just pointing out the explicit memory of Main process going up from 40.62 MB -> 67.33 MB -> 92.76 MB. Also, event-counts from other measurement is going up from 92 to 6925 to 14395. 

2. "UI Tests" process is not marionette or gaia-ui-tests, it's an app you can find in engineer build of any kind. You can open it up and get an overview of almost all UI. It's like Phone, Gallery, Camera, or any other kind of app. I saw it varies from 8MB to 15MB from time to time.

Comment 13

[Ivan Tsay (:ITsay)]

Hi Gary,

Mind if you can help QA for the trouble shooting on this case. So far not sure why this is happening. Need help to narrow down cause of the issue.

Comment 14

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Oh.  about-memory-0 from that folder seems to be from relatively late in the run ... which is why I was seeing small diffs.  Compare about-memory-0 and about-memory-1 ...

about-memory-134 also appears to have a complete GC/CC log pair for the b2g process.  I'm investigating the issue now.

Comment 15

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

So perusing the heap logs I notice that we have several hundred settings observers floating around.  These come from apps/system/js/quick_settings.js.  They're entraining requests which is what causes the inflated event-targets and event-counts numbers.  Somehow it looks like quick_settings.js is being initialized multiple times.

I wonder if http://mxr.mozilla.org/gaia/source/apps/system/js/quick_settings.js#341 should remove itself as an event listener.  Is it possible to fire the "localized" event multiple times?  Regardless, this is a gaia issue and needs to be owned by someone from that team.

Comment 16

[Walter Chen[:ypwalter][:wachen]]

Hi, Alive, can you help us to look into this issue?

Comment 17

[Robert Wood [:rwood]]

See comment I added in bug 992715

Comment 18

[(inactive after 6/18) Alive Kuo [:alive]]

It's not obvious what should be done here...I could make a patch to fix comment 15 anyway.

Comment 19

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

(In reply to Alive Kuo [:alive][NEEDINFO!][God bless Taiwan.] from comment #18)
> It's not obvious what should be done here...I could make a patch to fix
> comment 15 anyway.

Figuring out how this is actually happening would be a good start ...

Comment 20

[Kai-Chih Hu [:khu]]

(In reply to (Away 4/19-5/7) from comment #19)
> (In reply to Alive Kuo [:alive][NEEDINFO!][God bless Taiwan.] from comment
> #18)
> > It's not obvious what should be done here...I could make a patch to fix
> > comment 15 anyway.
> 
> Figuring out how this is actually happening would be a good start ...

+1 to this. Alive, if you need more information about the environment, can you check with Walter? 
And, do you think you're the good fit to own this bug? 

Thank you.

Comment 21

[(inactive after 6/18) Alive Kuo [:alive]]

Blocked by blocker enough, flag Tim to find someone.

Comment 22

[Tim Guan-tin Chien [:timdream] (please needinfo)]

Gregor, could you assign someone on your team for this script in the System app?

Comment 23

[Gregor Wagner [:gwagner]]

You mean fixing quick_settings.js?
I dont think anyone from my team has touched this file before.
We can put it in our backlog but I am not sure when we get to it if it's not a blocker or a 2.0 feature.

Comment 24

[Michael Henretty [:mikehenrty][:mhenretty]]

Walter are you still seeing this issue with all the latest settings changes?

Comment 25

[Walter Chen[:ypwalter][:wachen]]

We are currently in a totally different branch, and I am not seeing this on v2.1 currently because of the blocking of 1064800. I will need 1064800 to be solved first, and then I can look into it.

Comment 26

[Walter Chen[:ypwalter][:wachen]]

Haven't seem this for awhile