1381774 - Build log files for Tinderbox builds on OS X ending up in tinderbox-builds/mozilla-beta-macosx64/

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Since recently (regressed between beta 9 and beta 10) the build log files are no longer saved into the specific build folder but ending up in the root folder of tinderbox builds which is:

https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-beta-macosx64/

This actually breaks various automation systems including mozdownload, which now cannot find appropriate files anymore. As result ondemand firefox-ui update tests can no longer be run.

Please also see this difference in reporting to Treeherder:

Pass: https://treeherder.mozilla.org//api/project/mozilla-beta/job-log-url/?job_id=114097664

> https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-beta-macosx64/1499976498/mozilla-beta-macosx64-bm82-build1-build28.txt.gz

Fail: https://treeherder.mozilla.org//api/project/mozilla-beta/job-log-url/?job_id=114841189

> https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-beta-macosx64/mozilla-beta-macosx64-bm82-build1-build30.txt.gz

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

55.0b10 uses rev e26b1f5d635e/ of mozilla-beta. The products of the opt compile job made it to 
> https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-beta-macosx64/1500298880/
but the log did not. It's at
> https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-beta-macosx64/mozilla-beta-macosx64-bm82-build1-build30.txt.gz

So something went wrong with the log upload, and treeherder passed that one. This appears to be an intermittent fault, based on the contents of mozilla-beta-macosx64/:

File	mozilla-beta-macosx64-bm82-build1-build0.txt.gz	361K	13-Jun-2017 02:30
File	mozilla-beta-macosx64-bm82-build1-build24.txt.gz	349K	06-Jul-2017 22:58
File	mozilla-beta-macosx64-bm82-build1-build30.txt.gz	350K	17-Jul-2017 14:33
File	mozilla-beta-macosx64-bm82-build1-build4.txt.gz	352K	20-Jun-2017 19:09
File	mozilla-beta-macosx64-bm84-build1-build14.txt.gz	360K	30-Jun-2017 15:44
File	mozilla-beta-macosx64-bm84-build1-build26.txt.gz	353K	07-Jul-2017 15:40
File	mozilla-beta-macosx64-bm84-build1-build27.txt.gz	349K	07-Jul-2017 15:26
File	mozilla-beta-macosx64-bm84-build1-build9.txt.gz	353K	20-Jun-2017 11:52
File	mozilla-beta-macosx64-bm86-build1-build17.txt.gz	352K	28-Jun-2017 11:39
File	mozilla-beta-macosx64-bm86-build1-build21.txt.gz	350K	11-Jul-2017 19:01

ie 10 events since 20th of June, on 3 of the 4 masters doing mac compile jobs (coincidentally the three that also run buildbot-bridge ??). Nothing like this any of the win32 or win64 builds, or the mac addon-devel or deved. We don't retain logging of our uploads for more than a few minutes so it's going to be hard to debug this.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

https://dxr.mozilla.org/build-central/source/buildbotcustom/bin/log_uploader.py#93 is the code figuring out the buildID, which it passes to post_upload on the upload server. It turns out buildID is not a property set on a build with the broken upload.

The properties come from reading files in /builds/slave/m-beta-m64-0000000000000000000/properties/. Early in the build we set

06:41:20     INFO - Run as scripts/scripts/fx_desktop_build.py --config builds/releng_base_mac_64_builds.py --config balrog/production.py --branch mozilla-beta --build-pool production
06:41:20     INFO - Creating buildid through current time
06:41:20     INFO - Setting buildbot property buildid to 20170717064120
06:41:20     INFO - Writing buildbot properties ['buildid'] to /builds/slave/m-beta-m64-0000000000000000000/properties/buildid
06:41:20     INFO - Writing to file /builds/slave/m-beta-m64-0000000000000000000/properties/buildid
06:41:20     INFO - Contents:
06:41:20     INFO -  buildid:20170717064120

similar in working and busted logs. That's disappeared by the time the 2nd last step runs and sucks everything in properties/ back into buildbot.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

[[e-mail] properties]$ ls -lt
total 112
-rw-rw-r--  1 cltbld  staff    16 Jul 17 07:24 appName
-rw-rw-r--  1 cltbld  staff    16 Jul 17 07:24 appVersion
-rw-rw-r--  1 cltbld  staff    53 Jul 17 07:24 sourcestamp
-rw-rw-r--  1 cltbld  staff    56 Jul 17 07:24 completeMarFilename
-rw-rw-r--  1 cltbld  staff   145 Jul 17 07:24 completeMarHash
-rw-rw-r--  1 cltbld  staff    25 Jul 17 07:24 completeMarSize
-rw-rw-r--  1 cltbld  staff   117 Jul 17 07:24 jsshellUrl
-rw-rw-r--  1 cltbld  staff    43 Jul 17 07:24 packageFilename
-rw-rw-r--  1 cltbld  staff   128 Jul 17 07:24 packageUrl
-rw-rw-r--  1 cltbld  staff   150 Jul 17 07:24 symbolsUrl
-rw-rw-r--  1 cltbld  staff   148 Jul 17 07:24 testPackagesUrl
-rw-rw-r--  1 cltbld  staff   137 Jul 17 07:24 testsUrl
-rw-rw-r--  1 cltbld  staff  2798 Jul 17 07:24 uploadFiles
-rw-rw-r--  1 cltbld  staff    54 Jul 17 06:50 got_revision

So yeah, no sign of the buildid file there. Nothing i can find in the log to explain this.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

The mac opt jobs are going to convert to run as cross-compile on taskcluster when 56 merges to beta on August 2nd, so I'm going to WONTFIX this.