Closed Bug 1233577 Opened 4 years ago Closed 4 years ago
Thunderbird builds failing with upload error (mach not found)
Thunderbird nighties missing for Dec 16 and 17. https://archive.mozilla.org/pub/thunderbird/nightly/latest-comm-central/ http://archive.mozilla.org/pub/thunderbird/tinderbox-builds/comm-central-win32/1450387959/comm-central-win32-bm94-build1-build3.txt.gz .... Uploading c:\builds\moz2_slave\tb-c-cen-w32-00000000000000000\build\objdir-tb\dist\thunderbird-46.0a1.en-US.win32.checksums.asc Running post-upload command: post_upload.py --tinderbox-builds-dir comm-central-win32 -p thunderbird -i c:/mozilla-build/python27/python: can't open file 'c:/builds/moz2_slave/tb-c-cen-w32-00000000000000000/build/mach': [Errno 2] No such file or directory --revision a4999a28f9f0 --release-to-tinderbox-dated-builds --release-to-latest-tinderbox-builds c:/builds/moz2_slave/tb-c-cen-w32-00000000000000000/build/mozilla/toolkit/mozapps/installer/packager.mk:215: recipe for target 'upload' failed mozmake.exe: Leaving directory 'c:/builds/moz2_slave/tb-c-cen-w32-00000000000000000/build/objdir-tb/mail/installer' c:/builds/moz2_slave/tb-c-cen-w32-00000000000000000/build/mozilla/../mail/build.mk:25: recipe for target 'upload' failed Traceback (most recent call last): File "c:/builds/moz2_slave/tb-c-cen-w32-00000000000000000/build/mozilla/build/upload.py", line 328, in <module> package=options.package, verbose=True) File "c:/builds/moz2_slave/tb-c-cen-w32-00000000000000000/build/mozilla/build/upload.py", line 232, in UploadFiles output = DoSSHCommand('%s "%s" %s' % (post_upload_command, path, file_list), user, host, port=port, ssh_key=ssh_key) File "c:/builds/moz2_slave/tb-c-cen-w32-00000000000000000/build/mozilla/build/upload.py", line 96, in DoSSHCommand output = f(cmdline, stderr=STDOUT).strip() File "c:\builds\moz2_slave\tb-c-cen-w32-00000000000000000\build\mozilla\python\redo\redo\__init__.py", line 185, in _retriable_wrapper **retry_kwargs) File "c:\builds\moz2_slave\tb-c-cen-w32-00000000000000000\build\mozilla\python\redo\redo\__init__.py", line 140, in retry return action(*args, **kwargs) File "c:\mozilla-build\python27\Lib\subprocess.py", line 575, in check_output raise CalledProcessError(retcode, cmd, output=output)
Component: Build Config → Release Automation
Product: Thunderbird → Release Engineering
QA Contact: bhearsum
This looks very wrong: Running post-upload command: post_upload.py --tinderbox-builds-dir comm-central-win32 -p thunderbird -i c:/mozilla-build/python27/python: can't open file 'c:/builds/moz2_slave/tb-c-cen-w32-00000000000000000/build/mach': [Errno 2] No such file or directory --revision a4999a28f9f0 --release-to-tinderbox-dated-builds --release-to-latest-tinderbox-builds The -i should be followed by the buildID. Seems to be all platforms, likely this buildbot change http://hg.mozilla.org/build/buildbotcustom/rev/73e830d00659 from bug 1232466.
Component: Release Automation → General Automation
QA Contact: bhearsum → catlee
Seems to be nightly & dep, across all branches. Probably affects release builds too.
Summary: Thunderbird nightly builds missing from comm-central. upload error → Thunderbird builds failing with upload error
We'll need a fix very quickly to have any hope of doing two releases over the next two weeks.
Severity: normal → major
rail, can you do a quick fix for this? This is blocking building 38.5.0 - sylvestre says we should fix it before building. TIA
Unfortunately I can't look at that today. :( Just to clarify, actually http://hg.mozilla.org/build/buildbotcustom/rev/b99e35634c32 needs to be adjsuted to coop with the TB build system. We can use the same approach Fallen used in http://hg.mozilla.org/build/buildbotcustom/rev/7bef77bbaff8
I haven't made this critical or blocker so that people don't get paged - but it is that important, so that we get our builds done in time for SHA2, etc
Whiteboard: [blocking all TB builds and releases]
Possibly naive guess at a fix. Maybe there's a better way?
Attachment #8700352 - Flags: review?(rail)
Comment on attachment 8700352 [details] [diff] [review] Fix wrong path to mach in printconfig calls for comm builds I don't know what's the status of mach in TB, but the patch looks straight forward.
Attachment #8700352 - Flags: review?(rail) → review+
https://hg.mozilla.org/build/buildbotcustom/rev/30f63e792b59cf9916ac20b36e807ddd1ca73702 Bug 1233577 - Fix wrong path to mach in printconfig calls for comm builds. r=rail
https://hg.mozilla.org/build/buildbotcustom/rev/6d5e55dbf91af207cad3324d5a8225ca33cb2c3b Bug 1233577 - Fix typo in 30f63e792b59. rs=bustage-fix
callek wsmwsk: land that patch [from bug 1233577] to default and if the travis jobs pass (they'll report to #releng) merge them to the "production" (or "production-0.8") branch Callek_cloud9 wsmwk: I'd say we should wait a bit and see if dep builds on beta are good first before kicking the release [38.5.0] Callek_cloud9 just to be extra sure that `mach` works for TB correctly (when called at the correct location) [... then we could consider doing Bug 1233829] (In reply to aleth [:aleth] from comment #12) > https://hg.mozilla.org/build/buildbotcustom/rev/ > 6d5e55dbf91af207cad3324d5a8225ca33cb2c3b > Bug 1233577 - Fix typo in 30f63e792b59. rs=bustage-fix travis-ci mozilla/build-buildbotcustom#592 (master - adafe8e : aleth): The build was broken. travis-ci Change view : https://github.com/mozilla/build-buildbotcustom/compare/3880b2cd5283...adafe8eae5d6 travis-ci Build details : https://travis-ci.org/mozilla/build-buildbotcustom/builds/97979178 travis-ci Commit message : Bug 1233577 - Fix wrong path to mach in printconfig calls for comm builds. r=rail travis-ci mozilla/build-buildbotcustom#593 (master - a654e10 : aleth): The build was fixed. travis-ci Change view : https://github.com/mozilla/build-buildbotcustom/compare/adafe8eae5d6...a654e10e470d travis-ci Build details : https://travis-ci.org/mozilla/build-buildbotcustom/builds/97979681 travis-ci Commit message : Bug 1233577 - Fix typo in 30f63e792b59. rs=bustage-fix travis-ci mozilla/build-buildbotcustom#594 (production-0.8 - fc293c0 : aleth): The build passed. travis-ci Change view : https://github.com/mozilla/build-buildbotcustom/compare/06f6ec86bad9...fc293c0a927e travis-ci Build details : https://travis-ci.org/mozilla/build-buildbotcustom/builds/97980300 travis-ci Commit message : Merging from default. nice! thanks aleth
aceman posted a try build https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&selectedJob=13826 TEST-UNEXPECTED-FAIL | check-sync-dirs.py | build file copies are not in sync 1148954 TEST-UNEXPECTED-FAIL | check-sync-dirs.py | build file copies are not in sync: differing file mozconfig.common 1137406 TEST-UNEXPECTED-FAIL | check-sync-dirs.py | build file copies are not in sync: differing file: ./mozconfig.automation mozmake.exe: *** [check-sync-dirs] Error 1 pymake\..\..\mozmake.exe: *** [check] Error 2
callek triggered a beta build, which I assume is https://archive.mozilla.org/pub/thunderbird/tinderbox-builds/comm-beta-linux/1450216170/comm-beta-linux-bm72-build1-build4.txt.gz The only error I see is TEST-UNEXPECTED-FAIL | check-sync-dirs.py | build file copies are not in sync TEST-INFO | check-sync-dirs.py | file(s) found in: /builds/slave/tb-c-beta-lx-00000000000000000/build/mail/config/tooltool-manifests TEST-INFO | check-sync-dirs.py | differ from their originals in: /builds/slave/tb-c-beta-lx-00000000000000000/build/mozilla/browser/config/tooltool-manifests TEST-INFO | check-sync-dirs.py | differing file: ./linux64/releng.manifest TEST-INFO | check-sync-dirs.py | differing file: ./linux64/clang.manifest TEST-INFO | check-sync-dirs.py | differing file: ./linux32/releng.manifest In general, the files in '/builds/slave/tb-c-beta-... ditto https://archive.mozilla.org/pub/thunderbird/tinderbox-builds/comm-beta-win32/1450216170/comm-beta-win32-bm94-build1-build0.txt.gz So this bug is fixed? And the update just needs to be uplifted?
Windows upload seems to work. Linux still report problems during upload step and there still seems to be problem with BuildID (see comment 1). From https://treeherder.mozilla.org/logviewer.html#?job_id=29751&repo=comm-central > Running post-upload command: post_upload.py --tinderbox-builds-dir comm-central-linux-debug -p thunderbird -i Error loading mozconfig: /builds/slave/tb-c-cen-lx-d-0000000000000000/build/.mozconfig > /builds/slave/tb-c-cen-lx-d-0000000000000000/build/mozilla/build/unix/mozconfig.gtk: line 29: > /builds/slave/tb-c-cen-lx-d-0000000000000000/build/mozilla/gtk3/setup.sh: > No such file or directory > buildid: Error loading mozconfig: /builds/slave/tb-c-cen-lx-d-0000000000000000/build/.mozconfig
(In reply to Stefan Sitter from comment #16) > /builds/slave/tb-c-cen-lx-d-0000000000000000/build/mozilla/gtk3/setup.sh: No such file or directory I found the same error reported in Bug 1221502 a month ago, therefore it might be unrelated to the current failure tracked in this bug here.
Mac and Windows look the same - check-sync-dirs failed, 1 warning so that's good? https://archive.mozilla.org/pub/thunderbird/tinderbox-builds/comm-beta-macosx64/1450216170/comm-beta-macosx64-bm82-build1-build8.txt.gz https://archive.mozilla.org/pub/thunderbird/tinderbox-builds/comm-beta-win32/1450216170/comm-beta-win32-bm94-build1-build0.txt.gz nightly however is failing https://archive.mozilla.org/pub/thunderbird/tinderbox-builds/comm-central-win32/1450695726/comm-central-win32-nightly-bm77-build1-build0.txt.gz https://archive.mozilla.org/pub/thunderbird/tinderbox-builds/comm-central-macosx64/1450695726/comm-central-macosx64-nightly-bm84-build1-build7.txt.gz
check-sync-dirs failures are unrelated (and should now be resolved).
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Summary: Thunderbird builds failing with upload error → Thunderbird builds failing with upload error (mach not found)
You need to log in before you can comment on or make changes to this bug.