1140225 - [MTBF][Memory Report] "Cannot allocate memory" exception for memory tool

Status: RESOLVED FIXED

(Firefox OS Graveyard :: MTBF, defect)

Description

[Walter Chen[:ypwalter][:wachen]]

http://mtbf-1:8080/job/flamekk.v2.2.moztwlab01.319.mtbf_op/label=moztwlab-01/62/console

This one ends due to failures of 5 tests (after memory report pulls)

For 56 & 58-61, they all have one similar exception:
21:17:45 Pulling GC/CC logs...
21:17:45 Traceback (most recent call last):
21:17:45   File "mtbf_job_runner.py", line 368, in <module>
21:17:45     mjr.run()
21:17:45   File "mtbf_job_runner.py", line 361, in run
21:17:45     self.execute()
21:17:45   File "mtbf_job_runner.py", line 333, in execute
21:17:45     mtbf.main(marionette=self.marionette, testvars=self.options.testvars, **kwargs)
21:17:45   File "/var/jenkins/workspace/flamekk.v2.2.moztwlab01.319.mtbf_op@5/label/moztwlab-01/mtbf-env/local/lib/python2.7/site-packages/mtbf_driver-0.2.0-py2.7.egg/mtbf_driver/mtbf.py", line 285, in main
21:17:45     mtbf.start_gaiatest()
21:17:45   File "/var/jenkins/workspace/flamekk.v2.2.moztwlab01.319.mtbf_op@5/label/moztwlab-01/mtbf-env/local/lib/python2.7/site-packages/mtbf_driver-0.2.0-py2.7.egg/mtbf_driver/mtbf.py", line 135, in start_gaiatest
21:17:45     self.collect_metrics(current_round)
21:17:45   File "/var/jenkins/workspace/flamekk.v2.2.moztwlab01.319.mtbf_op@5/label/moztwlab-01/mtbf-env/local/lib/python2.7/site-packages/mtbf_driver-0.2.0-py2.7.egg/mtbf_driver/mtbf.py", line 235, in collect_metrics
21:17:45     tools.get_about_memory.get_and_show_info(memory_report_args(output_directory=mem_dir))
21:17:45   File "/var/jenkins/workspace/flamekk.v2.2.moztwlab01.319.mtbf_op@5/label/moztwlab-01/tools/get_about_memory.py", line 288, in get_and_show_info
21:17:46     get_gc_cc_log.get_logs(args, out_dir=out_dir, get_procrank_etc=False)
21:17:47   File "/var/jenkins/workspace/flamekk.v2.2.moztwlab01.319.mtbf_op@5/label/moztwlab-01/tools/get_gc_cc_log.py", line 89, in get_logs
21:17:47     utils.run_and_delete_dir_on_exception(do_work, out_dir)
21:17:47   File "/var/jenkins/workspace/flamekk.v2.2.moztwlab01.319.mtbf_op@5/label/moztwlab-01/tools/include/device_utils.py", line 177, in run_and_delete_dir_on_exception
21:17:47     return fun()
21:17:47   File "/var/jenkins/workspace/flamekk.v2.2.moztwlab01.319.mtbf_op@5/label/moztwlab-01/tools/get_gc_cc_log.py", line 81, in do_work
21:17:47     out_dir=out_dir)
21:17:47   File "/var/jenkins/workspace/flamekk.v2.2.moztwlab01.319.mtbf_op@5/label/moztwlab-01/tools/include/device_utils.py", line 265, in notify_and_pull_files
21:17:47     new_files = _list_remote_temp_files(outfiles_prefixes) - old_files
21:17:47   File "/var/jenkins/workspace/flamekk.v2.2.moztwlab01.319.mtbf_op@5/label/moztwlab-01/tools/include/device_utils.py", line 352, in _list_remote_temp_files
21:17:47     os.path.basename(d) in remote_ls(os.path.dirname(d))]
21:17:47   File "/var/jenkins/workspace/flamekk.v2.2.moztwlab01.319.mtbf_op@5/label/moztwlab-01/tools/include/device_utils.py", line 65, in remote_ls
21:17:47     return {f.strip() for f in remote_toolbox_cmd('ls', dir, verbose).split('\n')}
21:17:47   File "/var/jenkins/workspace/flamekk.v2.2.moztwlab01.319.mtbf_op@5/label/moztwlab-01/tools/include/device_utils.py", line 60, in remote_toolbox_cmd
21:17:47     return remote_shell('/system/bin/toolbox "%s" %s' % (cmd, args), verbose)
21:17:47   File "/var/jenkins/workspace/flamekk.v2.2.moztwlab01.319.mtbf_op@5/label/moztwlab-01/tools/include/device_utils.py", line 26, in remote_shell
21:17:47     out = shell(r"""adb shell '%s; echo -n "|$?"'""" % cmd)
21:17:47   File "/var/jenkins/workspace/flamekk.v2.2.moztwlab01.319.mtbf_op@5/label/moztwlab-01/tools/include/device_utils.py", line 76, in shell
21:17:47     stdout=subprocess.PIPE, stderr=subprocess.PIPE)
21:17:47   File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
21:17:48     errread, errwrite)
21:17:48   File "/usr/lib/python2.7/subprocess.py", line 1223, in _execute_child
21:17:48     self.pid = os.fork()
21:17:48 OSError: [Errno 12] Cannot allocate memory

It might have something to do with memory report tools.

Comment 1

[Walter Chen[:ypwalter][:wachen]]

Hi, Eric,

We started to use memory report again. We used to meet exception of running out of spaces. Now, we encountered a memory issue for memory report. When we are trying to run it, we got this exception of "Cannot allocate memory". 

Can you help us with investigating this?

Comment 2

[Eric Rahm [:erahm]]

That seems to imply your host machine is out of memory. I'll see if I can reproduce this locally.

I assume this is the code you guys are using? https://github.com/Mozilla-TWQA/MTBF-Driver

Comment 3

[Eric Rahm [:erahm]]

We use |multiprocess.Pool| to speed up compression of logs [1], but the once |pool| goes out of scope the subporcesses are not closed.

[1] https://github.com/mozilla-b2g/B2G/blob/1da86b8fab50d6e2c01ff5955fa9be52698deace/tools/get_gc_cc_log.py#L60

Comment 4

[Eric Rahm [:erahm]]

Attachment: Link to Github pull-request: https://github.com/mozilla-b2g/B2G/pull/420

Comment 5

[Andrew McCreight [:mccr8]]

Comment on attachment 8574112 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/B2G/pull/420

Oops.

Comment 6

[Eric Rahm [:erahm]]

https://github.com/mozilla-b2g/B2G/commit/a871bc6b88fb16de02d55a707affb1b44b28fc55

Comment 7

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

Oh wow that's amazing.