command timed out: 1200 seconds without output running ['python', 'scripts/buildfarm/utils/run_jetpack.py'

RESOLVED FIXED

Status

Release Engineering
General
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: erikvold, Assigned: erikvold)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

http://ftp.mozilla.org/pub/mozilla.org/jetpack/tinderbox-builds/addon-sdk-snowleopard/jetpack-fx-team-snowleopard-opt-bm107-tests1-macosx-build1.txt.gz

https://treeherder.mozilla.org/logviewer.html#?job_id=7403&repo=addon-sdk

x addon-sdk-75965aef38a611498fd7ea64a180af6337a44453/test/windows/test-fennec-windows.js
x addon-sdk-75965aef38a611498fd7ea64a180af6337a44453/test/windows/test-firefox-windows.js
x addon-sdk-75965aef38a611498fd7ea64a180af6337a44453/test/zip/utils.js
Checksumming Driver Descriptor Map (DDM : 0)…
Driver Descriptor Map (DDM : 0): verified CRC32 $3805ED61
Checksumming Apple (Apple_partition_map : 1)…
Apple (Apple_partition_map : 1): verified CRC32 $B039068E
Checksumming DiscRecording 6.0d1 (Apple_HFS : 2)…
DiscRecording 6.0d1 (Apple_HFS : 2): verified CRC32 $8D6C6910
verified CRC32 $55577E11
/dev/disk1 Apple_partition_scheme
/dev/disk1s1 Apple_partition_map
/dev/disk1s2 Apple_HFS /builds/slave/test/addonsdk-poller/mnt
"disk1" unmounted.
"disk1" ejected.
command timed out: 1200 seconds without output running ['python', 'scripts/buildfarm/utils/run_jetpack.py', '-p', 'snowleopard', '-t', u'https://hg.mozilla.org/projects/addon-sdk/archive/75965aef38a611498fd7ea64a180af6337a44453.tar.bz2', '-b', 'fx-team', '-f', 'ftp://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/%(branch)s-%(platform)s', '-e', '(mac|mac64).dmg'], attempting to kill
process killed by signal 9
program finished with exit code -1
elapsedTime=1246.698075
========= Finished 'python scripts/buildfarm/utils/run_jetpack.py ...' failed (results: 2, elapsed: 20 mins, 46 secs) (at 2015-05-31 18:36:07.164301) =========
From #releng:

10:07 AM <erikvold> jlund I was wondering if you could guess what might be going on with bug 1170056, or if you think bug 1170191 is the best way to debug this
10:08 AM — jlund looks
10:08 AM <erikvold> I'm requesting a loaner in bug 1170191
10:08 AM ⇐ JoeS1 quit (Thunderbird@moz-ql06qu.east.verizon.net) Ping timeout: 121 seconds
10:09 AM <erikvold> jlund https://treeherder.mozilla.org/#/jobs?repo=addon-sdk might be the best place to look
10:10 AM <jlund> erikvold: this looks like we are not using mozharness. So this is all happening within buildbot steps (automation). Might be hard to reproduce with a loaner
10:10 AM <erikvold> jlund ok glad I asked then!
10:10 AM <jlund> erikvold: is the output timeout intermittent? how recent have we been hitting this? Which platforms are affected
10:11 AM bobowen → bobowen|afk
10:11 AM <erikvold> jlund it seems to be every platform, and it's prob due to something that happened after Tue May 19, 8:30:08
10:11 AM <erikvold> jlund also I don't think it's intermittent
10:12 AM AutomatedTester|AFK → AutomatedTester
10:13 AM <erikvold> jlund there were no commits to the sdk between may 19 and may 27
10:13 AM <erikvold> and the may 27 commit is all red and purple
10:14 AM → satdav joined (uid15780@moz-avs1e8.highgate.irccloud.com)
10:14 AM <jlund> hm, so I also see "Error: missing Firefox executable" in other jobs
10:14 AM <erikvold> jlund oh hmm maybe it is intermittent
10:15 AM <jlund> http://ftp.mozilla.org/pub/mozilla.org/jetpack/tinderbox-builds/addon-sdk-ubuntu32_vm/jetpack-fx-team-ubuntu32_vm-opt-bm04-tests1-linux32-build0.txt.gz
10:15 AM <•selenamarie> catlee: got it! what if i make a kill list trello? :D
10:15 AM → JoeS1 joined (Thunderbird@moz-ql06qu.east.verizon.net)
10:16 AM <jlund> erikvold: looks like something may have changed to the archive structure as we are hitting: http://mxr.mozilla.org/build/source/tools/buildfarm/utils/run_jetpack.py#190
I guess I need to know if we should revert the possible archive structure change, or work on updating run_jetpack.py#190 somehow?
Created attachment 8613611 [details] [review]
Link to Github pull-request: https://github.com/mozilla/build-tools/pull/12
Attachment #8613611 - Flags: review?(jlund)
Comment on attachment 8613611 [details] [review]
Link to Github pull-request: https://github.com/mozilla/build-tools/pull/12

This patch is only adding two extra print statements to the run_jetpack.py file to show the ftp and ftp directory being used.
I suspect bug 1167768 would fix this issue, by using s3 instead of the ftp.

Comment 6

3 years ago
Comment on attachment 8613611 [details] [review]
Link to Github pull-request: https://github.com/mozilla/build-tools/pull/12

the github repo is read only so we will need to land this on http://hg.mozilla.org/build/tools/ but the patch diff itself looks good to land :)
Attachment #8613611 - Flags: review?(jlund) → review+
Landing patches from bug 1167053 to the addon-sdk seemed to resolve this.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Depends on: 1167053
Resolution: --- → FIXED
(In reply to Jordan Lund (:jlund) from comment #6)
> Comment on attachment 8613611 [details] [review]
> Link to Github pull-request: https://github.com/mozilla/build-tools/pull/12
> 
> the github repo is read only so we will need to land this on
> http://hg.mozilla.org/build/tools/ but the patch diff itself looks good to
> land :)

ok cool, I think I'll try landing this patch even tho the problem seems resolved now, because this might help us in the future.
Assignee: nobody → evold
https://hg.mozilla.org/build/tools/rev/544f9d9b8ddf
You need to log in before you can comment on or make changes to this bug.