[MTBF][Memory Report] "Cannot allocate memory" exception for memory tool

RESOLVED FIXED

Status

Firefox OS
MTBF
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: ypwalter, Assigned: erahm)

Tracking

(Blocks: 2 bugs)

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Reporter)

Description

3 years ago
http://mtbf-1:8080/job/flamekk.v2.2.moztwlab01.319.mtbf_op/label=moztwlab-01/62/console

This one ends due to failures of 5 tests (after memory report pulls)

For 56 & 58-61, they all have one similar exception:
21:17:45 Pulling GC/CC logs...
21:17:45 Traceback (most recent call last):
21:17:45   File "mtbf_job_runner.py", line 368, in <module>
21:17:45     mjr.run()
21:17:45   File "mtbf_job_runner.py", line 361, in run
21:17:45     self.execute()
21:17:45   File "mtbf_job_runner.py", line 333, in execute
21:17:45     mtbf.main(marionette=self.marionette, testvars=self.options.testvars, **kwargs)
21:17:45   File "/var/jenkins/workspace/flamekk.v2.2.moztwlab01.319.mtbf_op@5/label/moztwlab-01/mtbf-env/local/lib/python2.7/site-packages/mtbf_driver-0.2.0-py2.7.egg/mtbf_driver/mtbf.py", line 285, in main
21:17:45     mtbf.start_gaiatest()
21:17:45   File "/var/jenkins/workspace/flamekk.v2.2.moztwlab01.319.mtbf_op@5/label/moztwlab-01/mtbf-env/local/lib/python2.7/site-packages/mtbf_driver-0.2.0-py2.7.egg/mtbf_driver/mtbf.py", line 135, in start_gaiatest
21:17:45     self.collect_metrics(current_round)
21:17:45   File "/var/jenkins/workspace/flamekk.v2.2.moztwlab01.319.mtbf_op@5/label/moztwlab-01/mtbf-env/local/lib/python2.7/site-packages/mtbf_driver-0.2.0-py2.7.egg/mtbf_driver/mtbf.py", line 235, in collect_metrics
21:17:45     tools.get_about_memory.get_and_show_info(memory_report_args(output_directory=mem_dir))
21:17:45   File "/var/jenkins/workspace/flamekk.v2.2.moztwlab01.319.mtbf_op@5/label/moztwlab-01/tools/get_about_memory.py", line 288, in get_and_show_info
21:17:46     get_gc_cc_log.get_logs(args, out_dir=out_dir, get_procrank_etc=False)
21:17:47   File "/var/jenkins/workspace/flamekk.v2.2.moztwlab01.319.mtbf_op@5/label/moztwlab-01/tools/get_gc_cc_log.py", line 89, in get_logs
21:17:47     utils.run_and_delete_dir_on_exception(do_work, out_dir)
21:17:47   File "/var/jenkins/workspace/flamekk.v2.2.moztwlab01.319.mtbf_op@5/label/moztwlab-01/tools/include/device_utils.py", line 177, in run_and_delete_dir_on_exception
21:17:47     return fun()
21:17:47   File "/var/jenkins/workspace/flamekk.v2.2.moztwlab01.319.mtbf_op@5/label/moztwlab-01/tools/get_gc_cc_log.py", line 81, in do_work
21:17:47     out_dir=out_dir)
21:17:47   File "/var/jenkins/workspace/flamekk.v2.2.moztwlab01.319.mtbf_op@5/label/moztwlab-01/tools/include/device_utils.py", line 265, in notify_and_pull_files
21:17:47     new_files = _list_remote_temp_files(outfiles_prefixes) - old_files
21:17:47   File "/var/jenkins/workspace/flamekk.v2.2.moztwlab01.319.mtbf_op@5/label/moztwlab-01/tools/include/device_utils.py", line 352, in _list_remote_temp_files
21:17:47     os.path.basename(d) in remote_ls(os.path.dirname(d))]
21:17:47   File "/var/jenkins/workspace/flamekk.v2.2.moztwlab01.319.mtbf_op@5/label/moztwlab-01/tools/include/device_utils.py", line 65, in remote_ls
21:17:47     return {f.strip() for f in remote_toolbox_cmd('ls', dir, verbose).split('\n')}
21:17:47   File "/var/jenkins/workspace/flamekk.v2.2.moztwlab01.319.mtbf_op@5/label/moztwlab-01/tools/include/device_utils.py", line 60, in remote_toolbox_cmd
21:17:47     return remote_shell('/system/bin/toolbox "%s" %s' % (cmd, args), verbose)
21:17:47   File "/var/jenkins/workspace/flamekk.v2.2.moztwlab01.319.mtbf_op@5/label/moztwlab-01/tools/include/device_utils.py", line 26, in remote_shell
21:17:47     out = shell(r"""adb shell '%s; echo -n "|$?"'""" % cmd)
21:17:47   File "/var/jenkins/workspace/flamekk.v2.2.moztwlab01.319.mtbf_op@5/label/moztwlab-01/tools/include/device_utils.py", line 76, in shell
21:17:47     stdout=subprocess.PIPE, stderr=subprocess.PIPE)
21:17:47   File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
21:17:48     errread, errwrite)
21:17:48   File "/usr/lib/python2.7/subprocess.py", line 1223, in _execute_child
21:17:48     self.pid = os.fork()
21:17:48 OSError: [Errno 12] Cannot allocate memory

It might have something to do with memory report tools.
(Reporter)

Updated

3 years ago
Depends on: 990888
(Reporter)

Updated

3 years ago
Blocks: 990888, 1121812
No longer depends on: 990888
(Reporter)

Comment 1

3 years ago
Hi, Eric,

We started to use memory report again. We used to meet exception of running out of spaces. Now, we encountered a memory issue for memory report. When we are trying to run it, we got this exception of "Cannot allocate memory". 

Can you help us with investigating this?
Flags: needinfo?(erahm)
That seems to imply your host machine is out of memory. I'll see if I can reproduce this locally.

I assume this is the code you guys are using? https://github.com/Mozilla-TWQA/MTBF-Driver
Assignee: nobody → erahm
Flags: needinfo?(erahm)
We use |multiprocess.Pool| to speed up compression of logs [1], but the once |pool| goes out of scope the subporcesses are not closed.

[1] https://github.com/mozilla-b2g/B2G/blob/1da86b8fab50d6e2c01ff5955fa9be52698deace/tools/get_gc_cc_log.py#L60
Created attachment 8574112 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/B2G/pull/420
Attachment #8574112 - Flags: review?(continuation)
Comment on attachment 8574112 [details] [review]
Link to Github pull-request: https://github.com/mozilla-b2g/B2G/pull/420

Oops.
Attachment #8574112 - Flags: review?(continuation) → review+
https://github.com/mozilla-b2g/B2G/commit/a871bc6b88fb16de02d55a707affb1b44b28fc55
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.