http://ftp.mozilla.org/pub/mozilla.org/jetpack/tinderbox-builds/addon-sdk-snowleopard/jetpack-fx-team-snowleopard-opt-bm107-tests1-macosx-build1.txt.gz https://treeherder.mozilla.org/logviewer.html#?job_id=7403&repo=addon-sdk x addon-sdk-75965aef38a611498fd7ea64a180af6337a44453/test/windows/test-fennec-windows.js x addon-sdk-75965aef38a611498fd7ea64a180af6337a44453/test/windows/test-firefox-windows.js x addon-sdk-75965aef38a611498fd7ea64a180af6337a44453/test/zip/utils.js Checksumming Driver Descriptor Map (DDM : 0)… Driver Descriptor Map (DDM : 0): verified CRC32 $3805ED61 Checksumming Apple (Apple_partition_map : 1)… Apple (Apple_partition_map : 1): verified CRC32 $B039068E Checksumming DiscRecording 6.0d1 (Apple_HFS : 2)… DiscRecording 6.0d1 (Apple_HFS : 2): verified CRC32 $8D6C6910 verified CRC32 $55577E11 /dev/disk1 Apple_partition_scheme /dev/disk1s1 Apple_partition_map /dev/disk1s2 Apple_HFS /builds/slave/test/addonsdk-poller/mnt "disk1" unmounted. "disk1" ejected. command timed out: 1200 seconds without output running ['python', 'scripts/buildfarm/utils/run_jetpack.py', '-p', 'snowleopard', '-t', u'https://hg.mozilla.org/projects/addon-sdk/archive/75965aef38a611498fd7ea64a180af6337a44453.tar.bz2', '-b', 'fx-team', '-f', 'ftp://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/%(branch)s-%(platform)s', '-e', '(mac|mac64).dmg'], attempting to kill process killed by signal 9 program finished with exit code -1 elapsedTime=1246.698075 ========= Finished 'python scripts/buildfarm/utils/run_jetpack.py ...' failed (results: 2, elapsed: 20 mins, 46 secs) (at 2015-05-31 18:36:07.164301) =========
From #releng: 10:07 AM <erikvold> jlund I was wondering if you could guess what might be going on with bug 1170056, or if you think bug 1170191 is the best way to debug this 10:08 AM — jlund looks 10:08 AM <erikvold> I'm requesting a loaner in bug 1170191 10:08 AM ⇐ JoeS1 quit (Thunderbird@moz-ql06qu.east.verizon.net) Ping timeout: 121 seconds 10:09 AM <erikvold> jlund https://treeherder.mozilla.org/#/jobs?repo=addon-sdk might be the best place to look 10:10 AM <jlund> erikvold: this looks like we are not using mozharness. So this is all happening within buildbot steps (automation). Might be hard to reproduce with a loaner 10:10 AM <erikvold> jlund ok glad I asked then! 10:10 AM <jlund> erikvold: is the output timeout intermittent? how recent have we been hitting this? Which platforms are affected 10:11 AM bobowen → bobowen|afk 10:11 AM <erikvold> jlund it seems to be every platform, and it's prob due to something that happened after Tue May 19, 8:30:08 10:11 AM <erikvold> jlund also I don't think it's intermittent 10:12 AM AutomatedTester|AFK → AutomatedTester 10:13 AM <erikvold> jlund there were no commits to the sdk between may 19 and may 27 10:13 AM <erikvold> and the may 27 commit is all red and purple 10:14 AM → satdav joined (firstname.lastname@example.org) 10:14 AM <jlund> hm, so I also see "Error: missing Firefox executable" in other jobs 10:14 AM <erikvold> jlund oh hmm maybe it is intermittent 10:15 AM <jlund> http://ftp.mozilla.org/pub/mozilla.org/jetpack/tinderbox-builds/addon-sdk-ubuntu32_vm/jetpack-fx-team-ubuntu32_vm-opt-bm04-tests1-linux32-build0.txt.gz 10:15 AM <•selenamarie> catlee: got it! what if i make a kill list trello? :D 10:15 AM → JoeS1 joined (Thunderbird@moz-ql06qu.east.verizon.net) 10:16 AM <jlund> erikvold: looks like something may have changed to the archive structure as we are hitting: http://mxr.mozilla.org/build/source/tools/buildfarm/utils/run_jetpack.py#190
I guess I need to know if we should revert the possible archive structure change, or work on updating run_jetpack.py#190 somehow?
Created attachment 8613611 [details] [review] Link to Github pull-request: https://github.com/mozilla/build-tools/pull/12
Comment on attachment 8613611 [details] [review] Link to Github pull-request: https://github.com/mozilla/build-tools/pull/12 This patch is only adding two extra print statements to the run_jetpack.py file to show the ftp and ftp directory being used.
I suspect bug 1167768 would fix this issue, by using s3 instead of the ftp.
Comment on attachment 8613611 [details] [review] Link to Github pull-request: https://github.com/mozilla/build-tools/pull/12 the github repo is read only so we will need to land this on http://hg.mozilla.org/build/tools/ but the patch diff itself looks good to land :)
Landing patches from bug 1167053 to the addon-sdk seemed to resolve this.
(In reply to Jordan Lund (:jlund) from comment #6) > Comment on attachment 8613611 [details] [review] > Link to Github pull-request: https://github.com/mozilla/build-tools/pull/12 > > the github repo is read only so we will need to land this on > http://hg.mozilla.org/build/tools/ but the patch diff itself looks good to > land :) ok cool, I think I'll try landing this patch even tho the problem seems resolved now, because this might help us in the future.