983563 - [Memory Report] get_about_memory.py failed

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: General, defect)

Description

[Walter Chen[:ypwalter][:wachen]]

15:02:25 Started by upstream project "mtbf.memory.multi" build number 18
15:02:25 originally caused by:
15:02:25  Started by user Walter Chen
15:02:25 [EnvInject] - Loading node environment variables.
15:02:25 Building on master in workspace /var/lib/jenkins/workspace/mtbf.memory.multi/label/master
15:02:25 [copy-to-slave] Copying 'resource/B2G-flash-tool/B2G/tools/*, resource/B2G-flash-tool/B2G/tools/**/*, resource/B2G-flash-tool/B2G/tools/**/**/*', excluding nothing, from 'file:/var/lib/jenkins/userContent/' on the master to 'file:/var/lib/jenkins/workspace/mtbf.memory.multi/label/master/' on 'Jenkins'.
15:02:25 [master] $ /bin/sh -xe /tmp/hudson4227254278020434294.sh
15:02:25 + cd resource/B2G-flash-tool/B2G/tools
15:02:25 + chmod 775 get_about_memory.py
15:02:25 + true
15:02:25 + MOZ_IGNORE_NUWA_PROCESS=true ./get_about_memory.py
15:02:26 
Got 0/2 files.
Got 0/2 files.
Got 0/2 files.
Got 1/2 files.
Got 1/2 files.
Got 2/2 files.
15:02:33 Pulled files into about-memory-0.
15:02:33 
15:02:33 I just tried to open the memory report in Firefox.  If that didn't
15:02:33 work for some reason, or if you want to open this report at a later
15:02:33 time, open the following URL in a Firefox nightly build:
15:02:33 
15:02:33   about:memory?file=/var/lib/jenkins/workspace/mtbf.memory.multi/label/master/resource/B2G-flash-tool/B2G/tools/about-memory-0/memory-reports
15:02:33 
15:02:33 Pulling GC/CC logs...

15:02:33 Got 4/4 files.
15:03:33 Pulled files into about-memory-0.
15:03:33 Compressing logs...
15:03:33 + adb shell rm /data/local/tmp/memory-reports/*
15:03:33 rm failed for /data/local/tmp/memory-reports/*, No such file or directory
15:03:33 + sleep 600

15:13:33 + true
15:13:33 + MOZ_IGNORE_NUWA_PROCESS=true ./get_about_memory.py

15:13:34 Got 3/3 files.
15:13:43 Pulled files into about-memory-1.
15:13:43 
15:13:43 I just tried to open the memory report in Firefox.  If that didn't
15:13:43 work for some reason, or if you want to open this report at a later
15:13:43 time, open the following URL in a Firefox nightly build:
15:13:43 
15:13:43   about:memory?file=/var/lib/jenkins/workspace/mtbf.memory.multi/label/master/resource/B2G-flash-tool/B2G/tools/about-memory-1/memory-reports
15:13:43 
15:13:43 Pulling GC/CC logs...

15:13:43 Got 4/4 files.
15:14:53 Pulled files into about-memory-1.
15:14:53 Compressing logs...
15:14:53 + adb shell rm /data/local/tmp/memory-reports/*
15:14:53 rm failed for /data/local/tmp/memory-reports/*, No such file or directory
15:14:53 + sleep 600

15:24:53 + true
15:24:53 + MOZ_IGNORE_NUWA_PROCESS=true ./get_about_memory.py

15:24:54 Got 2/3 files.
Got 2/3 files.We've waited 120s but the only relevant files we see are
15:29:13   /data/local/tmp/memory-reports/memory-report-1394781887-28673.json.gz
15:29:13   /data/local/tmp/memory-reports/memory-report-1394781887-138.json.gz
15:29:13 We expected 3 but see only 2 files.  Giving up...
15:29:13 Traceback (most recent call last):
15:29:13   File "./get_about_memory.py", line 289, in <module>
15:29:13     get_and_show_info(args)
15:29:13   File "./get_about_memory.py", line 182, in get_and_show_info
15:29:13     (out_dir, merged_reports_path, dmd_files) = get_dumps(args)
15:29:13   File "./get_about_memory.py", line 179, in get_dumps
15:29:13     return utils.run_and_delete_dir_on_exception(do_work, out_dir)
15:29:13   File "/var/lib/jenkins/workspace/mtbf.memory.multi/label/master/resource/B2G-flash-tool/B2G/tools/include/device_utils.py", line 147, in run_and_delete_dir_on_exception
15:29:13     return fun()
15:29:13   File "./get_about_memory.py", line 164, in do_work
15:29:13     optional_outfiles_prefixes=['dmd-'])
15:29:13   File "/var/lib/jenkins/workspace/mtbf.memory.multi/label/master/resource/B2G-flash-tool/B2G/tools/include/device_utils.py", line 222, in notify_and_pull_files
15:29:13     _wait_for_remote_files(outfiles_prefixes, num_expected_files, old_files)
15:29:13   File "/var/lib/jenkins/workspace/mtbf.memory.multi/label/master/resource/B2G-flash-tool/B2G/tools/include/device_utils.py", line 304, in _wait_for_remote_files
15:29:13     raise Exception("Unable to pull some files.")
15:29:13 Exception: Unable to pull some files.
15:29:13 Build step 'Execute shell' marked build as failure
15:29:13 Finished: FAILURE

Comment 1

[Jason Smith [:jsmith]]

Is this happening outside of the MTBF framework?

Comment 2

[Walter Chen[:ypwalter][:wachen]]

I basically try to get memory report when doing MTBF or gaia-ui-tests. like 70% of time when running MTBF, it will fail.

However, it is outside of MTBF framework. I used jenkins to launch up few jobs including MTBF, adb logcat, and memory report (runs every 10 minutes). They are all seperated jobs.

Comment 3

[Walter Chen[:ypwalter][:wachen]]

PS. I was doing the same way before NUWA in. It works fine, but now not.

Comment 4

[James Ho]

Cervantes, could you comment on comment 2?

Comment 5

[Cervantes Yu [:cyu] [:cervantes]]

I can take a look.

Comment 6

[Cervantes Yu [:cyu] [:cervantes]]

I cannot reproduce this problem locally or on Paul Yang's device. From the log MOZ_IGNORE_NUWA_PROCESS is also set. I can't think of any reason this will block and timeout. Walter, can I see the problem reproduced on your device?

Comment 7

[Walter Chen[:ypwalter][:wachen]]

After talked with Cervantes, I will investigate further and get back to Cervantes.

Comment 8

[Walter Chen[:ypwalter][:wachen]]

Kyle, is it possible that you can look into this issue? Anything you need, please ask me for it :P

Comment 9

[Walter Chen[:ypwalter][:wachen]]

Additional information:
Memory report is started by a shell script in Jenkins CI(mtbf-1.corp.tpe1.mozilla.com:8080, intranet only). It will be blocked if previous memory report fetching didn't finish, it will not continue to do next get_about_memory.py

Previously:
1. 1 hour / pull
2. 10 minutes / pull
both failed

Thanks.

Comment 10

[Eric Rahm [:erahm]]

Walter is this an issue with |get_about_memory.py| or just the shell script you are using in Jenkins to call it?

Comment 11

[Walter Chen[:ypwalter][:wachen]]

It's the python mtbf-driver in Jenkins. We include libraries from get_about_memory.py and fetch data from it. We stopped using it after this happened so there is no updates on this bug.

Comment 12

[BMO Automation]

Firefox OS is not being worked on