1170056 - command timed out: 1200 seconds without output running ['python', 'scripts/buildfarm/utils/run_jetpack.py'

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Erik Vold [:erikvold] (please needinfo? me)]

http://ftp.mozilla.org/pub/mozilla.org/jetpack/tinderbox-builds/addon-sdk-snowleopard/jetpack-fx-team-snowleopard-opt-bm107-tests1-macosx-build1.txt.gz

https://treeherder.mozilla.org/logviewer.html#?job_id=7403&repo=addon-sdk

x addon-sdk-75965aef38a611498fd7ea64a180af6337a44453/test/windows/test-fennec-windows.js
x addon-sdk-75965aef38a611498fd7ea64a180af6337a44453/test/windows/test-firefox-windows.js
x addon-sdk-75965aef38a611498fd7ea64a180af6337a44453/test/zip/utils.js
Checksumming Driver Descriptor Map (DDM : 0)…
Driver Descriptor Map (DDM : 0): verified CRC32 $3805ED61
Checksumming Apple (Apple_partition_map : 1)…
Apple (Apple_partition_map : 1): verified CRC32 $B039068E
Checksumming DiscRecording 6.0d1 (Apple_HFS : 2)…
DiscRecording 6.0d1 (Apple_HFS : 2): verified CRC32 $8D6C6910
verified CRC32 $55577E11
/dev/disk1 Apple_partition_scheme
/dev/disk1s1 Apple_partition_map
/dev/disk1s2 Apple_HFS /builds/slave/test/addonsdk-poller/mnt
"disk1" unmounted.
"disk1" ejected.
command timed out: 1200 seconds without output running ['python', 'scripts/buildfarm/utils/run_jetpack.py', '-p', 'snowleopard', '-t', u'https://hg.mozilla.org/projects/addon-sdk/archive/75965aef38a611498fd7ea64a180af6337a44453.tar.bz2', '-b', 'fx-team', '-f', 'ftp://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/%(branch)s-%(platform)s', '-e', '(mac|mac64).dmg'], attempting to kill
process killed by signal 9
program finished with exit code -1
elapsedTime=1246.698075
========= Finished 'python scripts/buildfarm/utils/run_jetpack.py ...' failed (results: 2, elapsed: 20 mins, 46 secs) (at 2015-05-31 18:36:07.164301) =========

Comment 1

[Erik Vold [:erikvold] (please needinfo? me)]

From #releng:

10:07 AM <erikvold> jlund I was wondering if you could guess what might be going on with bug 1170056, or if you think bug 1170191 is the best way to debug this
10:08 AM — jlund looks
10:08 AM <erikvold> I'm requesting a loaner in bug 1170191
10:08 AM ⇐ JoeS1 quit ([e-mail]) Ping timeout: 121 seconds
10:09 AM <erikvold> jlund https://treeherder.mozilla.org/#/jobs?repo=addon-sdk might be the best place to look
10:10 AM <jlund> erikvold: this looks like we are not using mozharness. So this is all happening within buildbot steps (automation). Might be hard to reproduce with a loaner
10:10 AM <erikvold> jlund ok glad I asked then!
10:10 AM <jlund> erikvold: is the output timeout intermittent? how recent have we been hitting this? Which platforms are affected
10:11 AM bobowen → bobowen|afk
10:11 AM <erikvold> jlund it seems to be every platform, and it's prob due to something that happened after Tue May 19, 8:30:08
10:11 AM <erikvold> jlund also I don't think it's intermittent
10:12 AM AutomatedTester|AFK → AutomatedTester
10:13 AM <erikvold> jlund there were no commits to the sdk between may 19 and may 27
10:13 AM <erikvold> and the may 27 commit is all red and purple
10:14 AM → satdav joined ([e-mail])
10:14 AM <jlund> hm, so I also see "Error: missing Firefox executable" in other jobs
10:14 AM <erikvold> jlund oh hmm maybe it is intermittent
10:15 AM <jlund> http://ftp.mozilla.org/pub/mozilla.org/jetpack/tinderbox-builds/addon-sdk-ubuntu32_vm/jetpack-fx-team-ubuntu32_vm-opt-bm04-tests1-linux32-build0.txt.gz
10:15 AM <•selenamarie> catlee: got it! what if i make a kill list trello? :D
10:15 AM → JoeS1 joined ([e-mail])
10:16 AM <jlund> erikvold: looks like something may have changed to the archive structure as we are hitting: http://mxr.mozilla.org/build/source/tools/buildfarm/utils/run_jetpack.py#190

Comment 2

[Erik Vold [:erikvold] (please needinfo? me)]

I guess I need to know if we should revert the possible archive structure change, or work on updating run_jetpack.py#190 somehow?

Comment 3

[Erik Vold [:erikvold] (please needinfo? me)]

Attachment: Link to Github pull-request: https://github.com/mozilla/build-tools/pull/12

Comment 4

[Erik Vold [:erikvold] (please needinfo? me)]

Comment on attachment 8613611 [details] [review]
Link to Github pull-request: https://github.com/mozilla/build-tools/pull/12

This patch is only adding two extra print statements to the run_jetpack.py file to show the ftp and ftp directory being used.

Comment 5

[Erik Vold [:erikvold] (please needinfo? me)]

I suspect bug 1167768 would fix this issue, by using s3 instead of the ftp.

Comment 6

[Jordan Lund (:jlund)]

Comment on attachment 8613611 [details] [review]
Link to Github pull-request: https://github.com/mozilla/build-tools/pull/12

the github repo is read only so we will need to land this on http://hg.mozilla.org/build/tools/ but the patch diff itself looks good to land :)

Comment 7

[Erik Vold [:erikvold] (please needinfo? me)]

Landing patches from bug 1167053 to the addon-sdk seemed to resolve this.

Comment 8

[Erik Vold [:erikvold] (please needinfo? me)]

(In reply to Jordan Lund (:jlund) from comment #6)
> Comment on attachment 8613611 [details] [review]
> Link to Github pull-request: https://github.com/mozilla/build-tools/pull/12
> 
> the github repo is read only so we will need to land this on
> http://hg.mozilla.org/build/tools/ but the patch diff itself looks good to
> land :)

ok cool, I think I'll try landing this patch even tho the problem seems resolved now, because this might help us in the future.

Comment 9

[Erik Vold [:erikvold] (please needinfo? me)]

https://hg.mozilla.org/build/tools/rev/544f9d9b8ddf