1233577 - Thunderbird builds failing with upload error (mach not found)

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Wayne Mery (:wsmwk)]

Thunderbird nighties missing for Dec 16 and 17.
https://archive.mozilla.org/pub/thunderbird/nightly/latest-comm-central/


http://archive.mozilla.org/pub/thunderbird/tinderbox-builds/comm-central-win32/1450387959/comm-central-win32-bm94-build1-build3.txt.gz

....
Uploading c:\builds\moz2_slave\tb-c-cen-w32-00000000000000000\build\objdir-tb\dist\thunderbird-46.0a1.en-US.win32.checksums.asc
Running post-upload command: post_upload.py --tinderbox-builds-dir comm-central-win32 -p thunderbird -i c:/mozilla-build/python27/python: can't open file 'c:/builds/moz2_slave/tb-c-cen-w32-00000000000000000/build/mach': [Errno 2] No such file or directory --revision a4999a28f9f0 --release-to-tinderbox-dated-builds --release-to-latest-tinderbox-builds
c:/builds/moz2_slave/tb-c-cen-w32-00000000000000000/build/mozilla/toolkit/mozapps/installer/packager.mk:215: recipe for target 'upload' failed
mozmake.exe[1]: Leaving directory 'c:/builds/moz2_slave/tb-c-cen-w32-00000000000000000/build/objdir-tb/mail/installer'
c:/builds/moz2_slave/tb-c-cen-w32-00000000000000000/build/mozilla/../mail/build.mk:25: recipe for target 'upload' failed
Traceback (most recent call last):
  File "c:/builds/moz2_slave/tb-c-cen-w32-00000000000000000/build/mozilla/build/upload.py", line 328, in <module>
    package=options.package, verbose=True)
  File "c:/builds/moz2_slave/tb-c-cen-w32-00000000000000000/build/mozilla/build/upload.py", line 232, in UploadFiles
    output = DoSSHCommand('%s "%s" %s' % (post_upload_command, path, file_list), user, host, port=port, ssh_key=ssh_key)
  File "c:/builds/moz2_slave/tb-c-cen-w32-00000000000000000/build/mozilla/build/upload.py", line 96, in DoSSHCommand
    output = f(cmdline, stderr=STDOUT).strip()
  File "c:\builds\moz2_slave\tb-c-cen-w32-00000000000000000\build\mozilla\python\redo\redo\__init__.py", line 185, in _retriable_wrapper
    **retry_kwargs)
  File "c:\builds\moz2_slave\tb-c-cen-w32-00000000000000000\build\mozilla\python\redo\redo\__init__.py", line 140, in retry
    return action(*args, **kwargs)
  File "c:\mozilla-build\python27\Lib\subprocess.py", line 575, in check_output
    raise CalledProcessError(retcode, cmd, output=output)

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

This looks very wrong:

Running post-upload command: post_upload.py --tinderbox-builds-dir comm-central-win32 -p thunderbird -i c:/mozilla-build/python27/python: can't open file 'c:/builds/moz2_slave/tb-c-cen-w32-00000000000000000/build/mach': [Errno 2] No such file or directory --revision a4999a28f9f0 --release-to-tinderbox-dated-builds --release-to-latest-tinderbox-builds

The -i should be followed by the buildID.  Seems to be all platforms, likely this buildbot change 
  http://hg.mozilla.org/build/buildbotcustom/rev/73e830d00659
from bug 1232466.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Seems to be nightly & dep, across all branches. Probably affects release builds too.

Comment 3

[Wayne Mery (:wsmwk)]

We'll need a fix very quickly to have any hope of doing two releases over the next two weeks.

Comment 4

[Wayne Mery (:wsmwk)]

rail, can you do a quick fix for this?  
This is blocking building 38.5.0 - sylvestre says we should fix it before building.
TIA

Comment 5

[Rail Aliiev [:rail]]

Unfortunately I can't look at that today. :(

Just to clarify, actually http://hg.mozilla.org/build/buildbotcustom/rev/b99e35634c32 needs to be adjsuted to coop with the TB build system. We can use the same approach Fallen used in http://hg.mozilla.org/build/buildbotcustom/rev/7bef77bbaff8

Comment 7

[Wayne Mery (:wsmwk)]

I haven't made this critical or blocker so that people don't get paged - but it is that important, so that we get our builds done in time for SHA2, etc

Comment 8

[aleth [:aleth]]

Attachment: Fix wrong path to mach in printconfig calls for comm builds

Possibly naive guess at a fix. Maybe there's a better way?

Comment 9

[aleth [:aleth]]

Attachment: Fix wrong path to mach in printconfig calls for comm builds

Corrected patch.

Comment 10

[Rail Aliiev [:rail]]

Comment on attachment 8700352 [details] [diff] [review]
Fix wrong path to mach in printconfig calls for comm builds

I don't know what's the status of mach in TB, but the patch looks straight forward.

Comment 11

[aleth [:aleth]]

https://hg.mozilla.org/build/buildbotcustom/rev/30f63e792b59cf9916ac20b36e807ddd1ca73702
Bug 1233577 - Fix wrong path to mach in printconfig calls for comm builds. r=rail

Comment 12

[aleth [:aleth]]

https://hg.mozilla.org/build/buildbotcustom/rev/6d5e55dbf91af207cad3324d5a8225ca33cb2c3b
Bug 1233577 - Fix typo in 30f63e792b59. rs=bustage-fix

Comment 13

[Wayne Mery (:wsmwk)]

callek  wsmwsk: land that patch [from bug 1233577] to default and if the travis jobs pass (they'll report to #releng) merge them to the "production" (or "production-0.8") branch 

Callek_cloud9	wsmwk: I'd say we should wait a bit and see if dep builds on beta are good first before kicking the release [38.5.0]
Callek_cloud9	just to be extra sure that `mach` works for TB correctly (when called at the correct location) [... then we could consider doing Bug 1233829]


(In reply to aleth [:aleth] from comment #12)
> https://hg.mozilla.org/build/buildbotcustom/rev/
> 6d5e55dbf91af207cad3324d5a8225ca33cb2c3b
> Bug 1233577 - Fix typo in 30f63e792b59. rs=bustage-fix

travis-ci mozilla/build-buildbotcustom#592 (master - adafe8e : aleth): The build was broken.
travis-ci	Change view : https://github.com/mozilla/build-buildbotcustom/compare/3880b2cd5283...adafe8eae5d6
travis-ci	Build details : https://travis-ci.org/mozilla/build-buildbotcustom/builds/97979178
travis-ci	Commit message : Bug 1233577 - Fix wrong path to mach in printconfig calls for comm builds. r=rail
travis-ci mozilla/build-buildbotcustom#593 (master - a654e10 : aleth): The build was fixed.
travis-ci	Change view : https://github.com/mozilla/build-buildbotcustom/compare/adafe8eae5d6...a654e10e470d
travis-ci	Build details : https://travis-ci.org/mozilla/build-buildbotcustom/builds/97979681
travis-ci	Commit message : Bug 1233577 - Fix typo in 30f63e792b59. rs=bustage-fix
travis-ci mozilla/build-buildbotcustom#594 (production-0.8 - fc293c0 : aleth): The build passed.
travis-ci	Change view : https://github.com/mozilla/build-buildbotcustom/compare/06f6ec86bad9...fc293c0a927e
travis-ci	Build details : https://travis-ci.org/mozilla/build-buildbotcustom/builds/97980300
travis-ci	Commit message : Merging from default.

nice!  thanks aleth

Comment 14

[Wayne Mery (:wsmwk)]

aceman posted a try build
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&selectedJob=13826

 TEST-UNEXPECTED-FAIL | check-sync-dirs.py | build file copies are not in sync
1148954 TEST-UNEXPECTED-FAIL | check-sync-dirs.py | build file copies are not in sync: differing file mozconfig.common
1137406 TEST-UNEXPECTED-FAIL | check-sync-dirs.py | build file copies are not in sync: differing file: ./mozconfig.automation

mozmake.exe[1]: *** [check-sync-dirs] Error 1
pymake\..\..\mozmake.exe: *** [check] Error 2

Comment 15

[Wayne Mery (:wsmwk)]

callek triggered a beta build, which I assume is https://archive.mozilla.org/pub/thunderbird/tinderbox-builds/comm-beta-linux/1450216170/comm-beta-linux-bm72-build1-build4.txt.gz

The only error I see is 
TEST-UNEXPECTED-FAIL | check-sync-dirs.py | build file copies are not in sync
TEST-INFO | check-sync-dirs.py | file(s) found in:               /builds/slave/tb-c-beta-lx-00000000000000000/build/mail/config/tooltool-manifests
TEST-INFO | check-sync-dirs.py | differ from their originals in: /builds/slave/tb-c-beta-lx-00000000000000000/build/mozilla/browser/config/tooltool-manifests
TEST-INFO | check-sync-dirs.py | differing file:                 ./linux64/releng.manifest
TEST-INFO | check-sync-dirs.py | differing file:                 ./linux64/clang.manifest
TEST-INFO | check-sync-dirs.py | differing file:                 ./linux32/releng.manifest
In general, the files in '/builds/slave/tb-c-beta-...

ditto https://archive.mozilla.org/pub/thunderbird/tinderbox-builds/comm-beta-win32/1450216170/comm-beta-win32-bm94-build1-build0.txt.gz


So this bug is fixed?
And the update just needs to be uplifted?

Comment 16

[Stefan Sitter [:ssitter]]

Windows upload seems to work. Linux still report problems during upload step and there still seems to be problem with BuildID (see comment 1). From https://treeherder.mozilla.org/logviewer.html#?job_id=29751&repo=comm-central

> Running post-upload command: post_upload.py --tinderbox-builds-dir comm-central-linux-debug -p thunderbird -i Error loading mozconfig: /builds/slave/tb-c-cen-lx-d-0000000000000000/build/.mozconfig

> /builds/slave/tb-c-cen-lx-d-0000000000000000/build/mozilla/build/unix/mozconfig.gtk: line 29:
> /builds/slave/tb-c-cen-lx-d-0000000000000000/build/mozilla/gtk3/setup.sh: 
> No such file or directory

> buildid: Error loading mozconfig: /builds/slave/tb-c-cen-lx-d-0000000000000000/build/.mozconfig

Comment 17

[Stefan Sitter [:ssitter]]

(In reply to Stefan Sitter from comment #16)
> /builds/slave/tb-c-cen-lx-d-0000000000000000/build/mozilla/gtk3/setup.sh: No such file or directory
I found the same error reported in Bug 1221502 a month ago, therefore it might be unrelated to the current failure tracked in this bug here.

Comment 18

[Wayne Mery (:wsmwk)]

Mac and Windows look the same - check-sync-dirs failed, 1 warning so that's good?

https://archive.mozilla.org/pub/thunderbird/tinderbox-builds/comm-beta-macosx64/1450216170/comm-beta-macosx64-bm82-build1-build8.txt.gz
https://archive.mozilla.org/pub/thunderbird/tinderbox-builds/comm-beta-win32/1450216170/comm-beta-win32-bm94-build1-build0.txt.gz

nightly however is failing
https://archive.mozilla.org/pub/thunderbird/tinderbox-builds/comm-central-win32/1450695726/comm-central-win32-nightly-bm77-build1-build0.txt.gz
https://archive.mozilla.org/pub/thunderbird/tinderbox-builds/comm-central-macosx64/1450695726/comm-central-macosx64-nightly-bm84-build1-build7.txt.gz

Comment 19

[aleth [:aleth]]

check-sync-dirs failures are unrelated (and should now be resolved).