Thunderbird builds failing with upload error (mach not found)

RESOLVED FIXED

Status

Release Engineering
General Automation
--
major
RESOLVED FIXED
2 years ago
2 years ago

People

(Reporter: wsmwk, Assigned: aleth)

Tracking

(Blocks: 1 bug)

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [blocking all TB builds and releases])

Attachments

(1 attachment, 1 obsolete attachment)

(Reporter)

Description

2 years ago
Thunderbird nighties missing for Dec 16 and 17.
https://archive.mozilla.org/pub/thunderbird/nightly/latest-comm-central/


http://archive.mozilla.org/pub/thunderbird/tinderbox-builds/comm-central-win32/1450387959/comm-central-win32-bm94-build1-build3.txt.gz

....
Uploading c:\builds\moz2_slave\tb-c-cen-w32-00000000000000000\build\objdir-tb\dist\thunderbird-46.0a1.en-US.win32.checksums.asc
Running post-upload command: post_upload.py --tinderbox-builds-dir comm-central-win32 -p thunderbird -i c:/mozilla-build/python27/python: can't open file 'c:/builds/moz2_slave/tb-c-cen-w32-00000000000000000/build/mach': [Errno 2] No such file or directory --revision a4999a28f9f0 --release-to-tinderbox-dated-builds --release-to-latest-tinderbox-builds
c:/builds/moz2_slave/tb-c-cen-w32-00000000000000000/build/mozilla/toolkit/mozapps/installer/packager.mk:215: recipe for target 'upload' failed
mozmake.exe[1]: Leaving directory 'c:/builds/moz2_slave/tb-c-cen-w32-00000000000000000/build/objdir-tb/mail/installer'
c:/builds/moz2_slave/tb-c-cen-w32-00000000000000000/build/mozilla/../mail/build.mk:25: recipe for target 'upload' failed
Traceback (most recent call last):
  File "c:/builds/moz2_slave/tb-c-cen-w32-00000000000000000/build/mozilla/build/upload.py", line 328, in <module>
    package=options.package, verbose=True)
  File "c:/builds/moz2_slave/tb-c-cen-w32-00000000000000000/build/mozilla/build/upload.py", line 232, in UploadFiles
    output = DoSSHCommand('%s "%s" %s' % (post_upload_command, path, file_list), user, host, port=port, ssh_key=ssh_key)
  File "c:/builds/moz2_slave/tb-c-cen-w32-00000000000000000/build/mozilla/build/upload.py", line 96, in DoSSHCommand
    output = f(cmdline, stderr=STDOUT).strip()
  File "c:\builds\moz2_slave\tb-c-cen-w32-00000000000000000\build\mozilla\python\redo\redo\__init__.py", line 185, in _retriable_wrapper
    **retry_kwargs)
  File "c:\builds\moz2_slave\tb-c-cen-w32-00000000000000000\build\mozilla\python\redo\redo\__init__.py", line 140, in retry
    return action(*args, **kwargs)
  File "c:\mozilla-build\python27\Lib\subprocess.py", line 575, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
(Reporter)

Updated

2 years ago
Component: Build Config → Release Automation
Product: Thunderbird → Release Engineering
QA Contact: bhearsum
This looks very wrong:

Running post-upload command: post_upload.py --tinderbox-builds-dir comm-central-win32 -p thunderbird -i c:/mozilla-build/python27/python: can't open file 'c:/builds/moz2_slave/tb-c-cen-w32-00000000000000000/build/mach': [Errno 2] No such file or directory --revision a4999a28f9f0 --release-to-tinderbox-dated-builds --release-to-latest-tinderbox-builds

The -i should be followed by the buildID.  Seems to be all platforms, likely this buildbot change 
  http://hg.mozilla.org/build/buildbotcustom/rev/73e830d00659
from bug 1232466.
Blocks: 1232466
Component: Release Automation → General Automation
QA Contact: bhearsum → catlee
Seems to be nightly & dep, across all branches. Probably affects release builds too.
Summary: Thunderbird nightly builds missing from comm-central. upload error → Thunderbird builds failing with upload error
(Reporter)

Comment 3

2 years ago
We'll need a fix very quickly to have any hope of doing two releases over the next two weeks.
Severity: normal → major
(Reporter)

Updated

2 years ago
Blocks: 1233667
(Reporter)

Comment 4

2 years ago
rail, can you do a quick fix for this?  
This is blocking building 38.5.0 - sylvestre says we should fix it before building.
TIA
Flags: needinfo?(rail)
Unfortunately I can't look at that today. :(

Just to clarify, actually http://hg.mozilla.org/build/buildbotcustom/rev/b99e35634c32 needs to be adjsuted to coop with the TB build system. We can use the same approach Fallen used in http://hg.mozilla.org/build/buildbotcustom/rev/7bef77bbaff8
Flags: needinfo?(rail)

Updated

2 years ago
Duplicate of this bug: 1233972
(Reporter)

Comment 7

2 years ago
I haven't made this critical or blocker so that people don't get paged - but it is that important, so that we get our builds done in time for SHA2, etc
Whiteboard: [blocking all TB builds and releases]
(Assignee)

Comment 8

2 years ago
Created attachment 8700351 [details] [diff] [review]
Fix wrong path to mach in printconfig calls for comm builds

Possibly naive guess at a fix. Maybe there's a better way?
Attachment #8700351 - Flags: review?(rail)
(Assignee)

Updated

2 years ago
Assignee: nobody → aleth
Status: NEW → ASSIGNED
(Assignee)

Updated

2 years ago
Status: ASSIGNED → NEW
(Assignee)

Comment 9

2 years ago
Created attachment 8700352 [details] [diff] [review]
Fix wrong path to mach in printconfig calls for comm builds

Corrected patch.
Attachment #8700352 - Flags: review?(rail)
(Assignee)

Updated

2 years ago
Attachment #8700351 - Attachment is obsolete: true
Attachment #8700351 - Flags: review?(rail)
Comment on attachment 8700352 [details] [diff] [review]
Fix wrong path to mach in printconfig calls for comm builds

I don't know what's the status of mach in TB, but the patch looks straight forward.
Attachment #8700352 - Flags: review?(rail) → review+
(Assignee)

Comment 11

2 years ago
https://hg.mozilla.org/build/buildbotcustom/rev/30f63e792b59cf9916ac20b36e807ddd1ca73702
Bug 1233577 - Fix wrong path to mach in printconfig calls for comm builds. r=rail
(Reporter)

Comment 13

2 years ago
callek  wsmwsk: land that patch [from bug 1233577] to default and if the travis jobs pass (they'll report to #releng) merge them to the "production" (or "production-0.8") branch 

Callek_cloud9	wsmwk: I'd say we should wait a bit and see if dep builds on beta are good first before kicking the release [38.5.0]
Callek_cloud9	just to be extra sure that `mach` works for TB correctly (when called at the correct location) [... then we could consider doing Bug 1233829]


(In reply to aleth [:aleth] from comment #12)
> https://hg.mozilla.org/build/buildbotcustom/rev/
> 6d5e55dbf91af207cad3324d5a8225ca33cb2c3b
> Bug 1233577 - Fix typo in 30f63e792b59. rs=bustage-fix

travis-ci mozilla/build-buildbotcustom#592 (master - adafe8e : aleth): The build was broken.
travis-ci	Change view : https://github.com/mozilla/build-buildbotcustom/compare/3880b2cd5283...adafe8eae5d6
travis-ci	Build details : https://travis-ci.org/mozilla/build-buildbotcustom/builds/97979178
travis-ci	Commit message : Bug 1233577 - Fix wrong path to mach in printconfig calls for comm builds. r=rail
travis-ci mozilla/build-buildbotcustom#593 (master - a654e10 : aleth): The build was fixed.
travis-ci	Change view : https://github.com/mozilla/build-buildbotcustom/compare/adafe8eae5d6...a654e10e470d
travis-ci	Build details : https://travis-ci.org/mozilla/build-buildbotcustom/builds/97979681
travis-ci	Commit message : Bug 1233577 - Fix typo in 30f63e792b59. rs=bustage-fix
travis-ci mozilla/build-buildbotcustom#594 (production-0.8 - fc293c0 : aleth): The build passed.
travis-ci	Change view : https://github.com/mozilla/build-buildbotcustom/compare/06f6ec86bad9...fc293c0a927e
travis-ci	Build details : https://travis-ci.org/mozilla/build-buildbotcustom/builds/97980300
travis-ci	Commit message : Merging from default.

nice!  thanks aleth
(Reporter)

Comment 14

2 years ago
aceman posted a try build
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&selectedJob=13826

 TEST-UNEXPECTED-FAIL | check-sync-dirs.py | build file copies are not in sync
1148954 TEST-UNEXPECTED-FAIL | check-sync-dirs.py | build file copies are not in sync: differing file mozconfig.common
1137406 TEST-UNEXPECTED-FAIL | check-sync-dirs.py | build file copies are not in sync: differing file: ./mozconfig.automation

mozmake.exe[1]: *** [check-sync-dirs] Error 1
pymake\..\..\mozmake.exe: *** [check] Error 2
(Reporter)

Comment 15

2 years ago
callek triggered a beta build, which I assume is https://archive.mozilla.org/pub/thunderbird/tinderbox-builds/comm-beta-linux/1450216170/comm-beta-linux-bm72-build1-build4.txt.gz

The only error I see is 
TEST-UNEXPECTED-FAIL | check-sync-dirs.py | build file copies are not in sync
TEST-INFO | check-sync-dirs.py | file(s) found in:               /builds/slave/tb-c-beta-lx-00000000000000000/build/mail/config/tooltool-manifests
TEST-INFO | check-sync-dirs.py | differ from their originals in: /builds/slave/tb-c-beta-lx-00000000000000000/build/mozilla/browser/config/tooltool-manifests
TEST-INFO | check-sync-dirs.py | differing file:                 ./linux64/releng.manifest
TEST-INFO | check-sync-dirs.py | differing file:                 ./linux64/clang.manifest
TEST-INFO | check-sync-dirs.py | differing file:                 ./linux32/releng.manifest
In general, the files in '/builds/slave/tb-c-beta-...

ditto https://archive.mozilla.org/pub/thunderbird/tinderbox-builds/comm-beta-win32/1450216170/comm-beta-win32-bm94-build1-build0.txt.gz


So this bug is fixed?
And the update just needs to be uplifted?
Flags: needinfo?(sledru)

Comment 16

2 years ago
Windows upload seems to work. Linux still report problems during upload step and there still seems to be problem with BuildID (see comment 1). From https://treeherder.mozilla.org/logviewer.html#?job_id=29751&repo=comm-central

> Running post-upload command: post_upload.py --tinderbox-builds-dir comm-central-linux-debug -p thunderbird -i Error loading mozconfig: /builds/slave/tb-c-cen-lx-d-0000000000000000/build/.mozconfig

> /builds/slave/tb-c-cen-lx-d-0000000000000000/build/mozilla/build/unix/mozconfig.gtk: line 29:
> /builds/slave/tb-c-cen-lx-d-0000000000000000/build/mozilla/gtk3/setup.sh: 
> No such file or directory

> buildid: Error loading mozconfig: /builds/slave/tb-c-cen-lx-d-0000000000000000/build/.mozconfig

Comment 17

2 years ago
(In reply to Stefan Sitter from comment #16)
> /builds/slave/tb-c-cen-lx-d-0000000000000000/build/mozilla/gtk3/setup.sh: No such file or directory
I found the same error reported in Bug 1221502 a month ago, therefore it might be unrelated to the current failure tracked in this bug here.
(Assignee)

Comment 19

2 years ago
check-sync-dirs failures are unrelated (and should now be resolved).
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → FIXED
(Assignee)

Updated

2 years ago
Summary: Thunderbird builds failing with upload error → Thunderbird builds failing with upload error (mach not found)

Updated

2 years ago
Duplicate of this bug: 1233829
You need to log in before you can comment on or make changes to this bug.