Closed Bug 1299702 Opened 8 years ago Closed 8 years ago

Add Linux32, Win32, Win64 and OSX artifact builds

Categories

(Testing :: General, defect, P1)

Product:
Testing
Component:
General
Type:
defect

Tracking

(firefox52 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla52
Tracking Flags:
Tracking Status
firefox52 --- fixed

People

(Reporter: impossibus, Assigned: armenzg)

References

(Blocks 1 open bug)

Details

(Whiteboard: [artifact-builds-on-automation])

Attachments

(4 files, 1 obsolete file)

Bug 1299702 - Add linux32 artifact configs
58 bytes, text/x-review-board-request
chmanchester
: review+
Details
Bug 1299702 - Add win64 artifact configs
58 bytes, text/x-review-board-request
chmanchester
: review+
Details
Bug 1299702 - Add mac64 artifact configs
58 bytes, text/x-review-board-request
chmanchester
: review+
Details
Bug 1299702 - Add win32 artifact configs
58 bytes, text/x-review-board-request
chmanchester
: review+
Details
For artifact builds for Windows and OS X desktop builds especially.
Filing this as a separate bug from Bug 1278698, because it doesn't actually affect the implementation of the --artifact flag in try syntax. Chris, are you available to help out with this? I'm happy to take it on, but I would need a lot of guidance to avoid wild guesswork.
Flags: needinfo?(cmanchester)
I'm not sure when I'd be able to do this but I'm glad to field any questions. It shouldn't be an enormous amount of work. We can use what we have for the linux taskcluster artifact builds as a rough guide. The mozharness config for them is here: http://searchfox.org/mozilla-central/source/testing/mozharness/configs/builds/releng_sub_linux_configs/64_artifact.py The in-tree mozconfig is here: http://searchfox.org/mozilla-central/source/browser/config/mozconfigs/linux64/artifact
Flags: needinfo?(cmanchester)
Yes, I had already looked at those. It's not obvious to me how the configs should vary per platform, so I'll just try some stuff out (wild guesswork) and we'll go from there.
The strategy should be the same for all the platforms: the mozharness config should essentially be the same as for a regular build, but pointing to a different in-tree mozconfig. The job of the in-tree mozconfig will be the same as for a regular build, except with "ac_add_options --enable-artifact-builds". Enabling artifact builds changes the set of valid configure options, so some additional tweaks may be necessary to get from there to a working build on each platform.
Summary: Define mozconfigs and mozharness configs for artifact builds on platforms other and Linux64 → Define mozconfigs and mozharness configs for artifact builds on platforms other than Linux64
From the Mac logs: > 10:17:26 INFO - Ambiguous object directory detected. We detected that both /builds/slave/try-m64-0000000000000000000000/build/src/obj-firefox/x86_64 and /builds/slave/try-m64-0000000000000000000000/build/src/obj-firefox could be object directories. This is typically caused by having a mozconfig pointing to a different object directory from the current working directory. To solve this problem, ensure you do not have a default mozconfig in searched paths. From the Windows logs: > 09:24:03 INFO - mozbuild.configure.options.InvalidOptionError: Unknown option: --enable-jemalloc
Have you had a chance to reproduce these issues locally? chmanchester: should the artifact mozconfigs have less flags in there? https://hg.mozilla.org/try/rev/6d5035c5c10849c54613309d5ccfca566a560549 https://hg.mozilla.org/try/rev/c3e0449d1d10cf7daa83f1923628d93d1b6e0a20
For the mac build, I'm not sure exactly what's causing that error but it's something to do with our mechanism to do universal builds. Doing a universal artifact doesn't make a lot of sense (the binaries we download will either be fat or not, we can't control that), so let's try inheriting from the non-universal mozconfig at browser/config/mozconfigs/macosx64/nightly instead. For Windows our issue is trying to set --enable-jemalloc in an artifact build. We need to keep from passing that option in the artifact case, either by not including the common-opt mozconfig that contains it (and moving anything else we need from there to the artifact mozconfig) or setting a variable we can check before passing it.
Depends on: 1303089
Depends on: 1303090
Thanks Chris. I will be grabbing this. I will start on Monday.
Assignee: nobody → armenzg
For some reason, I'm having the hardest time in the world to make this build for Mac checkout the right code on a loaner. It's only when I patch this file that it works: https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/buildbot.py#57 > > + self.buildbot_properties = buildbot_properties > self.info("Using buildbot properties:") I don't understand why we would say "Using buildbot properties" yet not storing it. aki: is it intentional that we don't set self.buildbot_properties? The build is now running and I will have a look at it tomorrow. Below are the steps I followed [1] [1] On the Mac loaner: > mkdir -p /builds/slave/try-m64-0000000000000000000000 > cd /builds/slave/try-m64-0000000000000000000000 > export PROPERTIES_FILE=/builds/slave/try-m64-0000000000000000000000/buildprops.json > # Created handcrafted buildprops.json [2] > wget -Oarchiver_client.py --no-check-certificate --tries=10 --waitretry=3 https://hg.mozilla.org/build/tools/raw-file/default/buildfarm/utils/archiver_client.py > python archiver_client.py mozharness --repo try --rev 010de1d900f9815e543050418e51409d6da2ea3f --destination scripts --debug > /tools/buildbot/bin/python scripts/scripts/fx_desktop_build.py --config builds/releng_base_mac_64_builds.py --config balrog/production.py --branch try --build-pool production [2] { "sourcestamp": { "changes": [{ "comments": "try: -b do -p linux,linux64,win64,mac64 -u none -t none --artifact", "revision": "010de1d900f9815e543050418e51409d6da2ea3f", "branch": "try" }] }, "properties": { "upload_to_task_id": "aOB9gQeAQgCGix2u6qQiwg", "project": "", "product": "firefox", "who": "mjzffr@gmail.com", "script_repo_revision": "production", "taskId": "LbwPwO1pQce8XiSrAAKmSg", "repository": "", "buildername": "OS X 10.7 try build", "basedir": "/builds/slave/try-m64-0000000000000000000000", "buildnumber": 21983, "platform": "macosx64", "master": "http://buildbot-master87.bb.releng.scl3.mozilla.com:8101/", "branch": "try", "revision": "010de1d900f9815e543050418e51409d6da2ea3f", "slavename": "bld-lion-r5-036", "repo_path": "try" } }
(In reply to Armen Zambrano [:armenzg] (EDT/UTC-4) from comment #11) > It's only when I patch this file that it works: > https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/ > mozilla/buildbot.py#57 > > > > + self.buildbot_properties = buildbot_properties > > self.info("Using buildbot properties:") > > I don't understand why we would say "Using buildbot properties" yet not > storing it. > aki: is it intentional that we don't set self.buildbot_properties? We set self.buildbot_config here: https://hg.mozilla.org/mozilla-central/annotate/2c0d64cea673/testing/mozharness/mozharness/mozilla/buildbot.py#l64 buildbot_properties is solely for logging purposes, so treeherder's log parsing doesn't mark buildbot config values as errors: https://hg.mozilla.org/mozilla-central/rev/2c0d64cea673 Most likely you want to point at self.buildbot_config['properties'] instead of buildbot_properties.
Maja: I can reproduce the issue on the loaner. I will see what is needed to make it work. Thanks Aki! I will look into it.
Note from yesterdat: The buildprops.json sourcestamp needed to include the revision.
chmanchester: this is what I'm getting when I use the universal mozconfig This is where I see --enable-rust: https://dxr.mozilla.org/mozilla-central/search?q=enable-rust&redirect=false 587 11:21:29 INFO - Adding configure options from /builds/slave/try-m64-0000000000000000000000/build/src/.mozconfig 588 11:21:29 INFO - --enable-crashreporter 589 11:21:29 INFO - --enable-release 590 11:21:29 INFO - --enable-js-shell 591 11:21:29 INFO - --enable-verify-mar 592 11:21:29 INFO - --with-branding=browser/branding/nightly 593 11:21:29 INFO - --enable-rust 594 11:21:29 INFO - --with-compiler-wrapper=python2.7 /builds/slave/try-m64-0000000000000000000000/build/src/sccache/sccache.py 595 11:21:29 INFO - --disable-install-strip 596 11:21:29 INFO - --enable-instruments 597 11:21:29 INFO - --enable-dtrace 598 11:21:29 INFO - --enable-artifact-builds 599 11:21:29 INFO - CC=/builds/slave/try-m64-0000000000000000000000/build/src/clang/bin/clang 600 11:21:29 INFO - DSYMUTIL=/builds/slave/try-m64-0000000000000000000000/build/src/clang/bin/llvm-dsymutil 601 11:21:29 INFO - MOZ_TELEMETRY_REPORTING=1 602 11:21:29 INFO - CXX=/builds/slave/try-m64-0000000000000000000000/build/src/clang/bin/clang++ 603 11:21:29 INFO - MOZ_PACKAGE_JSSHELL=1 604 11:21:29 INFO - SOCORRO_SYMBOL_UPLOAD_TOKEN_FILE=/builds/crash-stats-api.token 605 11:21:29 INFO - MOZILLA_OFFICIAL=1 606 11:21:29 INFO - LDFLAGS=-B/builds/slave/try-m64-0000000000000000000000/build/src/cctools/bin -Wl,-no_data_in_code_info 607 11:21:29 INFO - LLVMCONFIG=/builds/slave/try-m64-0000000000000000000000/build/src/clang/bin/llvm-config 608 11:21:29 INFO - CARGO=/builds/slave/try-m64-0000000000000000000000/build/src/cargo/bin/cargo 609 11:21:29 INFO - MOZ_ADDON_SIGNING=1 610 11:21:29 INFO - MOZ_REQUIRE_SIGNING=0 611 11:21:29 INFO - RUSTC=/builds/slave/try-m64-0000000000000000000000/build/src/rustc/bin/rustc 612 11:21:30 INFO - checking for a shell... /usr/bin/sh 613 11:21:30 INFO - checking for host system type... x86_64-apple-darwin11.2.0 614 11:21:30 INFO - checking for target system type... x86_64-apple-darwin11.2.0 615 11:21:30 INFO - Traceback (most recent call last): 616 11:21:30 INFO - File "/builds/slave/try-m64-0000000000000000000000/build/src/configure.py", line 107, in <module> 617 11:21:30 INFO - sys.exit(main(sys.argv)) 618 11:21:30 INFO - File "/builds/slave/try-m64-0000000000000000000000/build/src/configure.py", line 26, in main 619 11:21:30 INFO - sandbox.run(os.path.join(os.path.dirname(__file__), 'moz.configure')) 620 11:21:30 INFO - File "/builds/slave/try-m64-0000000000000000000000/build/src/python/mozbuild/mozbuild/configure/__init__.py", line 256, in run 621 11:21:30 INFO - raise InvalidOptionError('Unknown option: %s' % without_value) 622 11:21:30 INFO - mozbuild.configure.options.InvalidOptionError: Unknown option: --enable-rust 623 11:21:30 INFO - *** Fix above errors and then restart with "/usr/bin/make -f client.mk build" bash-3.2$ hg diff diff --git a/browser/config/mozconfigs/macosx-universal/artifact b/browser/config/mozconfigs/macosx-universal/artifact --- a/browser/config/mozconfigs/macosx-universal/artifact +++ b/browser/config/mozconfigs/macosx-universal/artifact @@ -1,7 +1,7 @@ MOZ_AUTOMATION_BUILD_SYMBOLS=0 MOZ_AUTOMATION_L10N_CHECK=0 -. "$topsrcdir/browser/config/mozconfigs/macosx-universal/common-opt" +. "$topsrcdir/browser/config/mozconfigs/macosx64/nightly" ac_add_options --disable-install-strip ac_add_options --enable-instruments ################################################################################ For Windows: > wget -Oarchiver_client.py --no-check-certificate --tries=10 --waitretry=3 https://hg.mozilla.org/build/tools/raw-file/default/buildfarm/utils/archiver_client.py > python archiver_client.py mozharness --repo try --rev 010de1d900f9815e543050418e51409d6da2ea3f --destination scripts --debug > # Download and install a recent Mercurial > wget http://puppetagain.pub.build.mozilla.org/data/repos/EXEs/Mercurial-3.9.1-x64.exe > /c/mozilla-build/python27/python -u scripts/scripts/fx_desktop_build.py --config builds/releng_base_windows_64_builds.py --config balrog/production.py --branch try --build-pool production
We went through this in bug 1282889 for the Linux artifact builds. A similar fix should work here, by including the other mozconfigs included by browser/config/mozconfigs/macosx64/nightly in our artifact mozconfig but not including mozconfig.rust.
Attached file wip - mac mozconfig changes (obsolete) —
Attachment #8793958 - Flags: feedback?(cmanchester)
Comment on attachment 8793958 [details] wip - mac mozconfig changes This is headed in the right direction. Is this enough to get a successful build? I don't think we should put the mozconfig in macosx-universal if we're not doing a universal build. Keep in mind mozconfig.common.override should go as late as possible.
Attachment #8793958 - Flags: feedback?(cmanchester) → feedback+
I think the Mac build completed, however, it seems to fail to gather some metrics at the end: > 10:10:00 INFO - setting properties set by mach build. Looking in path: /builds/slave/try-m64-0000000000000000000000/build/src/obj-firefox/x86_64/dist/mach_build_properties.json > 10:10:00 INFO - No mach_build_properties.json found - not importing properties. > 10:10:00 FATAL - Can't set the following properties: buildid, sourcestamp, appVersion, and appName. Required paths missing. Verify both /builds/slave/try-m64-0000000000000000000000/build/src/config/printconfigsetting.py and /builds/slave/try-m64-0000000000000000000000/build/src/obj-firefox/x86_64/dist/bin/application.ini exist. These paths require the 'build' action to be run prior to this https://treeherder.mozilla.org/#/jobs?repo=try&revision=a47920a3a64b It took 14 minutes.
(In reply to Armen Zambrano [:armenzg] (EDT/UTC-4) from comment #19) > I think the Mac build completed, however, it seems to fail to gather some > metrics at the end: > > 10:10:00 INFO - setting properties set by mach build. Looking in path: /builds/slave/try-m64-0000000000000000000000/build/src/obj-firefox/x86_64/dist/mach_build_properties.json > > 10:10:00 INFO - No mach_build_properties.json found - not importing properties. > > 10:10:00 FATAL - Can't set the following properties: buildid, sourcestamp, appVersion, and appName. Required paths missing. Verify both /builds/slave/try-m64-0000000000000000000000/build/src/config/printconfigsetting.py and /builds/slave/try-m64-0000000000000000000000/build/src/obj-firefox/x86_64/dist/bin/application.ini exist. These paths require the 'build' action to be run prior to this > > https://treeherder.mozilla.org/#/jobs?repo=try&revision=a47920a3a64b It looks like this is finding the wrong objdir. The config has "obj-firefox/x86_64" at https://hg.mozilla.org/try/rev/6c1b839b1c93#l2.29 but the build seems to be using just "obj-firefox". > > It took 14 minutes. That's slower than we'd like, and we should figure out where we're spending that time, but these builds are taking more like 1.5 hours on trunk, so a worthwhile improvement!
I don't see why the objdir could be affecting this. From our artifact build: 09:56:12 INFO - 'MOZ_OBJDIR': 'obj-firefox', 09:56:12 INFO - 'objdir': 'obj-firefox/x86_64', 10:10:00 INFO - setting properties set by mach build. Looking in path: /builds/slave/try-m64-0000000000000000000000/build/src/obj-firefox/x86_64/dist/mach_build_properties.json From a normal Mac build: 03:16:23 INFO - 'MOZ_OBJDIR': 'obj-firefox', 03:16:23 INFO - 'objdir': 'obj-firefox/x86_64', 05:13:05 INFO - setting properties set by mach build. Looking in path: /builds/slave/m-cen-m64-00000000000000000000/build/src/obj-firefox/x86_64/dist/mach_build_properties.json
Green win64 build + tests: https://treeherder.mozilla.org/#/jobs?repo=try&revision=a47920a3a64b Green mac64 build + tests: https://treeherder.mozilla.org/#/jobs?repo=try&revision=fa5dddf6c283 Running win32 build: https://treeherder.mozilla.org/#/jobs?repo=try&revision=efba32fa96c1 I'm seeing some of these [1] We should remove the 'download_symbols' config variable the same way we do for Asan test jobs. [1] 13:25:24 WARNING - Can't figure out symbols_url from installer_url: https://archive.mozilla.org/pub/firefox/try-builds/armenzg@mozilla.com-fa5dddf6c2834ffa12857213d31ef636eb1b0b59/try-macosx64/firefox-52.0a1.en-US.mac64.dmg! 13:25:24 WARNING - Traceback (most recent call last): 13:25:24 WARNING - 13:25:24 WARNING - File "/builds/slave/test/scripts/mozharness/mozilla/testing/testbase.py", line 200, in query_symbols_url 13:25:24 WARNING - self._urlopen(symbols_url, timeout=120) 13:25:24 WARNING - 13:25:24 WARNING - File "/builds/slave/test/scripts/mozharness/mozilla/testing/testbase.py", line 292, in _urlopen 13:25:24 WARNING - return urllib2.urlopen(url, **kwargs) 13:25:24 WARNING - 13:25:24 WARNING - File "/tools/python27/lib/python2.7/urllib2.py", line 126, in urlopen 13:25:24 WARNING - return _opener.open(url, data, timeout) 13:25:24 WARNING - 13:25:24 WARNING - File "/tools/python27/lib/python2.7/urllib2.py", line 406, in open 13:25:24 WARNING - response = meth(req, response) 13:25:24 WARNING - 13:25:24 WARNING - File "/tools/python27/lib/python2.7/urllib2.py", line 519, in http_response 13:25:24 WARNING - 'http', request, response, code, msg, hdrs) 13:25:24 WARNING - 13:25:24 WARNING - File "/tools/python27/lib/python2.7/urllib2.py", line 444, in error 13:25:24 WARNING - return self._call_chain(*args) 13:25:24 WARNING - 13:25:24 WARNING - File "/tools/python27/lib/python2.7/urllib2.py", line 378, in _call_chain 13:25:24 WARNING - result = func(*args) 13:25:24 WARNING - 13:25:24 WARNING - File "/tools/python27/lib/python2.7/urllib2.py", line 527, in http_error_default 13:25:24 WARNING - raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) 13:25:24 WARNING - 13:25:24 WARNING - HTTPError: HTTP Error 404: Not Found 13:25:24 WARNING - 13:25:24 WARNING - No symbols_url found. Let minidump_stackwalk query for symbols.
Summary: Define mozconfigs and mozharness configs for artifact builds on platforms other than Linux64 → Add Linux32, Win32, Win64 and OSX artifact builds
Attachment #8794375 - Flags: review?(cmanchester)
Attachment #8794376 - Flags: review?(cmanchester)
Attachment #8794377 - Flags: review?(cmanchester)
Attachment #8795298 - Flags: review?(cmanchester)
Attachment #8793958 - Attachment is obsolete: true
Comment on attachment 8794375 [details] Bug 1299702 - Add linux32 artifact configs https://reviewboard.mozilla.org/r/80856/#review80128 Looks good. My review on the mozharness configs is going to mostly be a rubber stamp based on the builds succeeding, if you're concerned at all about them you might want to get review from someone more familiar.
Attachment #8794375 - Flags: review?(cmanchester) → review+
Comment on attachment 8794376 [details] Bug 1299702 - Add win64 artifact configs https://reviewboard.mozilla.org/r/80858/#review80132 ::: browser/config/mozconfigs/win64/artifact:7 (Diff revision 2) > +MOZ_AUTOMATION_L10N_CHECK=0 > + > +. "$topsrcdir/build/mozconfig.win-common" > +. "$topsrcdir/browser/config/mozconfigs/win64/common-win64" > + > +# Beginning from win64\common-opt I'm a little worried about this duplication causing things to fall out of sync... can you add a more descriptive comment about this (saying why we're doing it, essentially for "--enable-jemalloc"), and file a follow up so we can find a cleaner solution? ::: browser/config/mozconfigs/win64/artifact:24 (Diff revision 2) > +# Package js shell. > +export MOZ_PACKAGE_JSSHELL=1 Let's leave this out, we're not building a jsshell, so there wont be anything to package.
Attachment #8794376 - Flags: review?(cmanchester) → review+
Comment on attachment 8794377 [details] Bug 1299702 - Add mac64 artifact configs https://reviewboard.mozilla.org/r/80860/#review80138 ::: browser/config/mozconfigs/macosx64/artifact:4 (Diff revision 2) > +. "$topsrcdir/build/macosx/mozconfig.common" > +. "$topsrcdir/build/mozconfig.common.override" This doesn't seem consistent with the way we did it for Windows... we don't end up with things like MOZILLA_OFFICIAL or MOZ_TELEMETRY_REPORTING. These are test only (really automation only) builds, so this might not matter, can we try a shorter mozconfig for the windows builds? ::: browser/config/mozconfigs/macosx64/artifact:7 (Diff revision 2) > +ac_add_options --disable-install-strip > +ac_add_options --enable-instruments > +ac_add_options --enable-dtrace These three are related to compiled code stuff, we can leave these out.
Attachment #8794377 - Flags: review?(cmanchester)
Comment on attachment 8795298 [details] Bug 1299702 - Add win32 artifact configs https://reviewboard.mozilla.org/r/81400/#review80140 Let's see this again once we work out the question from the OSX mozconfig.
Attachment #8795298 - Flags: review?(cmanchester)
chmanchester: I might have not re-flagged you saying that the patch is ready to review. Could you please have a look at it?
Flags: needinfo?(cmanchester)
I was flagged, I just didn't get a chance to look at it yesterday. I will do so this morning, sorry for the delay!
Flags: needinfo?(cmanchester)
Comment on attachment 8794377 [details] Bug 1299702 - Add mac64 artifact configs https://reviewboard.mozilla.org/r/80860/#review80758
Attachment #8794377 - Flags: review?(cmanchester) → review+
Comment on attachment 8795298 [details] Bug 1299702 - Add win32 artifact configs https://reviewboard.mozilla.org/r/81400/#review80762 This is much cleaner. There's a chance we'll find tests that rely on things like MOZ_OFFICIAL being set, but I think we should go with this for now.
Attachment #8795298 - Flags: review?(cmanchester) → review+
Armen, do we need something here like in Maja's patch in bug 1302152? Let's check before landing this.
Flags: needinfo?(armenzg)
Pushed by armenzg@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/c0122746911c Add linux32 artifact configs r=chmanchester https://hg.mozilla.org/integration/autoland/rev/8c12fa62d501 Add win64 artifact configs r=chmanchester https://hg.mozilla.org/integration/autoland/rev/a0de50ddb8f8 Add mac64 artifact configs r=chmanchester https://hg.mozilla.org/integration/autoland/rev/cf9d0c3c6f73 Add win32 artifact configs r=chmanchester
Flags: needinfo?(armenzg)
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: