1031085 - Get Mulet builds running on OSX on fig

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Jonathan Griffin (:jgriffin)]

Currently, we have Mulet builds working on fig on linux64 only; we should add OSX Mulet builds to the mix.

Comment 1

[Jonathan Griffin (:jgriffin)]

Attachment: Add mozconfig for osx mulet,

Comment 2

[Jonathan Griffin (:jgriffin)]

Attachment: buildbot-configs updates for osx Mulet,

This is mostly cargo-culted from Armen's work in bug 997096, so should be checked carefully.  I think this is all we need other than adding the build to production-masters.json.

Comment 3

[Jonathan Griffin (:jgriffin)]

Attachment: Add osx mulet to production-masters,

Comment 4

[Jonathan Griffin (:jgriffin)]

Attachment: buildbot-configs updates for osx Mulet,

Fixed a typo.

Comment 5

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8449900 [details] [diff] [review]
buildbot-configs updates for osx Mulet,

Review of attachment 8449900 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/config.py
@@ +109,5 @@
> +PLATFORMS['macosx64-mulet']['slave_platforms'] = ['snowleopard']
> +PLATFORMS['macosx64-mulet']['snowleopard'] = {
> +    'name': 'Rev4 MacOSX Mulet Snow Leopard 10.6',
> +    'build_dir_prefix': 'snowleopard_mulet',
> +    'scheduler_slave_platform_identifier': 'snowleopard_mulet'

Not sure if these two lines are needed; I got them from the linux64-mulet config.

::: mozilla/config.py
@@ +787,5 @@
> +            'slaves': SLAVES['macosx64-lion'],
> +            'platform_objdir': "%s/i386" % OBJDIR,
> +            'stage_product': 'firefox',
> +            'stage_platform': 'macosx64-mulet',
> +            'update_platform': 'Darwin_x86_64-gcc3',

Not sure if we need this line.

@@ +800,5 @@
> +                'SYMBOL_SERVER_USER': 'ffxbld',
> +                'SYMBOL_SERVER_PATH': SYMBOL_SERVER_PATH,
> +                'POST_SYMBOL_UPLOAD_CMD': SYMBOL_SERVER_POST_UPLOAD_CMD,
> +                'SYMBOL_SERVER_SSH_KEY': "/Users/cltbld/.ssh/ffxbld_dsa",
> +                'MOZ_SYMBOLS_EXTRA_BUILDID': 'macosx64-mulet',

Not sure if this value should be 'macosx64' or 'macosx64-mulet'.

@@ +821,5 @@
> +            # must be overridden explicitly.
> +            'nightly_signing_servers': 'mac-dep-signing',
> +            'dep_signing_servers': 'mac-dep-signing',
> +            'tooltool_manifest_src': 'browser/config/tooltool-manifests/macosx64/releng.manifest',
> +            'tooltool_l10n_manifest_src': 'browser/config/tooltool-manifests/macosx64/l10n.manifest',

Can we remove the above 4 lines?

Comment 6

[Armen [:armenzg]]

(In reply to Jonathan Griffin (:jgriffin) from comment #5)
> Comment on attachment 8449900 [details] [diff] [review]
> buildbot-configs updates for osx Mulet,
> 
> Review of attachment 8449900 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: mozilla-tests/config.py
> @@ +109,5 @@
> > +PLATFORMS['macosx64-mulet']['slave_platforms'] = ['snowleopard']
> > +PLATFORMS['macosx64-mulet']['snowleopard'] = {
> > +    'name': 'Rev4 MacOSX Mulet Snow Leopard 10.6',
> > +    'build_dir_prefix': 'snowleopard_mulet',
> > +    'scheduler_slave_platform_identifier': 'snowleopard_mulet'
> 
> Not sure if these two lines are needed; I got them from the linux64-mulet
> config.
> 
The last two are intended to reduce touching other places like BuildSlaves.py and production_config.py
I wished they also removed the need to touch production-masters.json but it did not :(

> ::: mozilla/config.py
> @@ +787,5 @@
> > +            'slaves': SLAVES['macosx64-lion'],
> > +            'platform_objdir': "%s/i386" % OBJDIR,
> > +            'stage_product': 'firefox',
> > +            'stage_platform': 'macosx64-mulet',
> > +            'update_platform': 'Darwin_x86_64-gcc3',
> 
> Not sure if we need this line.
> 
Unfortunately misc.py or factory.py requires it even if we say "enable_updates=False" (IIUC).

> @@ +800,5 @@
> > +                'SYMBOL_SERVER_USER': 'ffxbld',
> > +                'SYMBOL_SERVER_PATH': SYMBOL_SERVER_PATH,
> > +                'POST_SYMBOL_UPLOAD_CMD': SYMBOL_SERVER_POST_UPLOAD_CMD,
> > +                'SYMBOL_SERVER_SSH_KEY': "/Users/cltbld/.ssh/ffxbld_dsa",
> > +                'MOZ_SYMBOLS_EXTRA_BUILDID': 'macosx64-mulet',
> 
> Not sure if this value should be 'macosx64' or 'macosx64-mulet'.
> 
Good to differentiate it if we ever start uploading symbols.

> @@ +821,5 @@
> > +            # must be overridden explicitly.
> > +            'nightly_signing_servers': 'mac-dep-signing',
> > +            'dep_signing_servers': 'mac-dep-signing',
> > +            'tooltool_manifest_src': 'browser/config/tooltool-manifests/macosx64/releng.manifest',
> > +            'tooltool_l10n_manifest_src': 'browser/config/tooltool-manifests/macosx64/l10n.manifest',
> 
> Can we remove the above 4 lines?

I wish too!
The tooltool lines are for downloading tools like gcc and others.

Comment 7

[Michael Shal [:mshal]]

Comment on attachment 8449874 [details] [diff] [review]
Add mozconfig for osx mulet,

>+++ b/b2g/dev/config/mozconfigs/macosx-universal/mulet	Wed Jul 02 17:15:24 2014 -0700
>\ No newline at end of file

Mind adding the end-of-file newline? :)

Comment 8

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8449874 [details] [diff] [review]
Add mozconfig for osx mulet,

Added newline, pushed as https://hg.mozilla.org/integration/b2g-inbound/rev/ffa3484b48c9

Comment 9

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/ffa3484b48c9

Comment 10

[Jonathan Griffin (:jgriffin)]

Forgot to [leave-open].

Comment 11

[Chris AtLee [:catlee]]

Comment on attachment 8449900 [details] [diff] [review]
buildbot-configs updates for osx Mulet,

Review of attachment 8449900 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/config.py
@@ +800,5 @@
> +                'SYMBOL_SERVER_USER': 'ffxbld',
> +                'SYMBOL_SERVER_PATH': SYMBOL_SERVER_PATH,
> +                'POST_SYMBOL_UPLOAD_CMD': SYMBOL_SERVER_POST_UPLOAD_CMD,
> +                'SYMBOL_SERVER_SSH_KEY': "/Users/cltbld/.ssh/ffxbld_dsa",
> +                'MOZ_SYMBOLS_EXTRA_BUILDID': 'macosx64-mulet',

I think this is ok.

@@ +821,5 @@
> +            # must be overridden explicitly.
> +            'nightly_signing_servers': 'mac-dep-signing',
> +            'dep_signing_servers': 'mac-dep-signing',
> +            'tooltool_manifest_src': 'browser/config/tooltool-manifests/macosx64/releng.manifest',
> +            'tooltool_l10n_manifest_src': 'browser/config/tooltool-manifests/macosx64/l10n.manifest',

Do you want mulet builds to be signed?

And what compiler will they use? The tooltool manifest pulls down the clang compiler for the build I think.

Comment 12

[Jonathan Griffin (:jgriffin)]

(In reply to Chris AtLee [:catlee] from comment #11)
> 
> Do you want mulet builds to be signed?
> 

Yes, I think so.

> And what compiler will they use? The tooltool manifest pulls down the clang
> compiler for the build I think.

Same compiler as is used for Firefox, so I think the above should work OK.

Comment 13

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8449900 [details] [diff] [review]
buildbot-configs updates for osx Mulet,

https://hg.mozilla.org/build/buildbot-configs/rev/c29dea9cd76e

Comment 14

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8449900 [details] [diff] [review]
buildbot-configs updates for osx Mulet,

backed out for failing tests; can someone with Super Decoder Ring clearance tell me what http://10.134.48.37:8080/job/buildbot-configs_tests/2273/ says?

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

It failed 'buildbot checkconfig' for bm81-tests_scheduler:

INFO  - created  "bm81-tests_scheduler" master, running checkconfig
INFO  - starting to print log file '/run/shm/buildbot/bm81-tests_scheduler-90LgSw-checkconfig.log'
INFO  - Traceback (most recent call last):
INFO  -   File "/var/lib/jenkins/jobs/buildbot-configs_tests/workspace/.pyenv/local/lib/python2.7/site-packages/buildbot-0.8.2_hg_18d5488c8185_production_0.8-py2.7.egg/buildbot/scripts/runner.py", line 1042, in doCheckConfig
INFO  -     ConfigLoader(configFileName=configFileName)
INFO  -   File "/var/lib/jenkins/jobs/buildbot-configs_tests/workspace/.pyenv/local/lib/python2.7/site-packages/buildbot-0.8.2_hg_18d5488c8185_production_0.8-py2.7.egg/buildbot/scripts/checkconfig.py", line 31, in __init__
INFO  -     self.loadConfig(configFile, check_synchronously_only=True)
INFO  -   File "/var/lib/jenkins/jobs/buildbot-configs_tests/workspace/.pyenv/local/lib/python2.7/site-packages/buildbot-0.8.2_hg_18d5488c8185_production_0.8-py2.7.egg/buildbot/master.py", line 652, in loadConfig
INFO  -     exec f in localDict
INFO  -   File "/run/shm/buildbot/bm81-tests_scheduler-90LgSw/master.cfg", line 71, in <module>
INFO  -     BRANCH_UNITTEST_VARS['platforms'])
INFO  -   File "/var/lib/jenkins/jobs/buildbot-configs_tests/workspace/buildbotcustom/misc.py", line 2522, in generateTalosBranchObjects
INFO  -     "envName": platform_config['env_name'],
INFO  - KeyError: 'env_name'
INFO  - finished printing log file '/run/shm/buildbot/bm81-tests_scheduler-90LgSw-checkconfig.log'
ERROR - TEST-FAIL bm81-tests_scheduler failed to run checkconfig

https://wiki.mozilla.org/ReleaseEngineering:TestingTechniques#test-masters.sh

Comment 16

[Jonathan Griffin (:jgriffin)]

Attachment: buildbot-configs updates for osx Mulet,

Fixed the missing env_name problem.

Comment 17

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8452478 [details] [diff] [review]
buildbot-configs updates for osx Mulet,

https://hg.mozilla.org/build/buildbot-configs/rev/1fa3dbcce772

Comment 18

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8452478 [details] [diff] [review]
buildbot-configs updates for osx Mulet,

..and backed out again!  Will run the tests...

Comment 19

[Jonathan Griffin (:jgriffin)]

Finally got test-masters.sh to run for me; I had fat-fingered env_name again!  Fixed and re-landed: 

https://hg.mozilla.org/build/buildbot-configs/rev/25e48c6d0f2c

Comment 20

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8449899 [details] [diff] [review]
Add osx mulet to production-masters,

https://hg.mozilla.org/build/tools/rev/ac3398893658

Comment 21

[Jonathan Griffin (:jgriffin)]

The production-masters change caused a test failure, apparently.  Backing out:

https://hg.mozilla.org/build/tools/rev/aefd18f890da

Comment 22

[Jonathan Griffin (:jgriffin)]

Current error:

    INFO  - created  "bm106-tests1-macosx" master, running checkconfig
    INFO  - starting to print log file '/run/shm/buildbot/bm106-tests1-macosx-XmYkOt-checkconfig.log'
    INFO  - Traceback (most recent call last):
    INFO  -   File "/var/lib/jenkins/jobs/buildbot-configs_tests/workspace/.pyenv/local/lib/python2.7/site-packages/buildbot-0.8.2_hg_5484a944940e_production_0.8-py2.7.egg/buildbot/scripts/runner.py", line 1042, in doCheckConfig
    INFO  -     ConfigLoader(configFileName=configFileName)
    INFO  -   File "/var/lib/jenkins/jobs/buildbot-configs_tests/workspace/.pyenv/local/lib/python2.7/site-packages/buildbot-0.8.2_hg_5484a944940e_production_0.8-py2.7.egg/buildbot/scripts/checkconfig.py", line 31, in __init__
    INFO  -     self.loadConfig(configFile, check_synchronously_only=True)
    INFO  -   File "/var/lib/jenkins/jobs/buildbot-configs_tests/workspace/.pyenv/local/lib/python2.7/site-packages/buildbot-0.8.2_hg_5484a944940e_production_0.8-py2.7.egg/buildbot/master.py", line 831, in loadConfig
    INFO  -     % (b['name'], b['builddir']))
    INFO  - ValueError: builder Rev4 MacOSX Snow Leopard 10.6 try talos g1 reuses builddir try_snowleopard_test-g1
    INFO  - finished printing log file '/run/shm/buildbot/bm106-tests1-macosx-XmYkOt-checkconfig.log'
    ERROR - TEST-FAIL bm106-tests1-macosx failed to run checkconfig

Comment 23

[Pete Moore [:pmoore][:pete]]

This part is now running in production: https://hg.mozilla.org/build/buildbot-configs/rev/25e48c6d0f2c

Comment 24

[Pete Moore [:pmoore][:pete]]

Glandium pointed out we are getting runs of Mulet on try, and suggested removing from try. Catlee said we need it on try if it runs on trunk. I wondered if it might make sense to disable it on try by default. Glandium queried whether it really is neccesary to run on trunk too. Coop also suggested we should avoid running on try if we can.

How to proceed?

Thanks,
Pete

Comment 25

[Armen [:armenzg]]

I think we should disable OSX Mulet builds everwhere except Fig until they're tier-1 (which they're not).
I also don't see any work going on Fig for the last two weeks and they're broken there as wel..

See bug 1037861 as well.

Comment 26

[Jonathan Griffin (:jgriffin)]

Yes, I agree we can/should disable these on try until they're stable.

Comment 27

[Jonathan Griffin (:jgriffin)]

Attachment: Turn off Mulet on try; turn off Talos on macosx64-mulet,

This turns off mulet on try, and turns off talos_masters for macosx64-mulet, where we don't need it.

Comment 28

[Pete Moore [:pmoore][:pete]]

Thanks guys!

Comment 29

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8455522 [details] [diff] [review]
Turn off Mulet on try; turn off Talos on macosx64-mulet,

https://hg.mozilla.org/build/buildbot-configs/rev/f7dad836bece

Comment 30

[Jonathan Griffin (:jgriffin)]

(In reply to Jonathan Griffin (:jgriffin) from comment #29)
> Comment on attachment 8455522 [details] [diff] [review]
> Turn off Mulet on try; turn off Talos on macosx64-mulet,
> 
> https://hg.mozilla.org/build/buildbot-configs/rev/f7dad836bece

Backed out for test failure.

Comment 31

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8455522 [details] [diff] [review]
Turn off Mulet on try; turn off Talos on macosx64-mulet,

Once more, with feeling: https://hg.mozilla.org/build/buildbot-configs/rev/3acdb8ee14d6

Comment 32

[Aki Sasaki (not active)]

In production.

Comment 33

[Jonathan Griffin (:jgriffin)]

These builds are running now on fig, but failing with:

printf APPLMOZB > ../../dist/FirefoxNightly.app/Contents/PkgInfo
make[6]: Nothing to be done for `tools'.
set -e; \
	for mkfile in build/macosx/universal/flight.mk; do \
	  make -f /builds/slave/fig-osx64-mulet-00000000000000/build/$mkfile postflight_all TOPSRCDIR=/builds/slave/fig-osx64-mulet-00000000000000/build MOZ_OBJDIR=/builds/slave/fig-osx64-mulet-00000000000000/build/obj-firefox MOZ_BUILD_PROJECTS='i386 x86_64'; \
	done
mkdir -p /builds/slave/fig-osx64-mulet-00000000000000/build/obj-firefox/i386/dist/universal/
rm -f /builds/slave/fig-osx64-mulet-00000000000000/build/obj-firefox/x86_64/dist/universal
ln -s /builds/slave/fig-osx64-mulet-00000000000000/build/obj-firefox/i386/dist/universal /builds/slave/fig-osx64-mulet-00000000000000/build/obj-firefox/x86_64/dist/universal
make -C /builds/slave/fig-osx64-mulet-00000000000000/build/obj-firefox/i386/b2g/dev/installer \
	   PKG_SKIP_STRIP=1 stage-package
make: *** /builds/slave/fig-osx64-mulet-00000000000000/build/obj-firefox/i386/b2g/dev/installer: No such file or directory.  Stop.

I think this is probably not a problem with infrastructure, but a problem with the build on OSX, so I'm going to close this and open a new bug for this problem.