Closed Bug 1031085 Opened 6 years ago Closed 6 years ago

Get Mulet builds running on OSX on fig

Categories

(Release Engineering :: General, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: jgriffin, Assigned: jgriffin)

References

Details

Attachments

(4 files, 2 obsolete files)

Currently, we have Mulet builds working on fig on linux64 only; we should add OSX Mulet builds to the mix.
Attachment #8449874 - Flags: review?(mshal)
Assignee: nobody → jgriffin
This is mostly cargo-culted from Armen's work in bug 997096, so should be checked carefully.  I think this is all we need other than adding the build to production-masters.json.
Attachment #8449897 - Flags: review?(catlee)
See Also: → 997096
Fixed a typo.
Attachment #8449900 - Flags: review?(catlee)
Attachment #8449897 - Attachment is obsolete: true
Attachment #8449897 - Flags: review?(catlee)
Comment on attachment 8449900 [details] [diff] [review]
buildbot-configs updates for osx Mulet,

Review of attachment 8449900 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/config.py
@@ +109,5 @@
> +PLATFORMS['macosx64-mulet']['slave_platforms'] = ['snowleopard']
> +PLATFORMS['macosx64-mulet']['snowleopard'] = {
> +    'name': 'Rev4 MacOSX Mulet Snow Leopard 10.6',
> +    'build_dir_prefix': 'snowleopard_mulet',
> +    'scheduler_slave_platform_identifier': 'snowleopard_mulet'

Not sure if these two lines are needed; I got them from the linux64-mulet config.

::: mozilla/config.py
@@ +787,5 @@
> +            'slaves': SLAVES['macosx64-lion'],
> +            'platform_objdir': "%s/i386" % OBJDIR,
> +            'stage_product': 'firefox',
> +            'stage_platform': 'macosx64-mulet',
> +            'update_platform': 'Darwin_x86_64-gcc3',

Not sure if we need this line.

@@ +800,5 @@
> +                'SYMBOL_SERVER_USER': 'ffxbld',
> +                'SYMBOL_SERVER_PATH': SYMBOL_SERVER_PATH,
> +                'POST_SYMBOL_UPLOAD_CMD': SYMBOL_SERVER_POST_UPLOAD_CMD,
> +                'SYMBOL_SERVER_SSH_KEY': "/Users/cltbld/.ssh/ffxbld_dsa",
> +                'MOZ_SYMBOLS_EXTRA_BUILDID': 'macosx64-mulet',

Not sure if this value should be 'macosx64' or 'macosx64-mulet'.

@@ +821,5 @@
> +            # must be overridden explicitly.
> +            'nightly_signing_servers': 'mac-dep-signing',
> +            'dep_signing_servers': 'mac-dep-signing',
> +            'tooltool_manifest_src': 'browser/config/tooltool-manifests/macosx64/releng.manifest',
> +            'tooltool_l10n_manifest_src': 'browser/config/tooltool-manifests/macosx64/l10n.manifest',

Can we remove the above 4 lines?
(In reply to Jonathan Griffin (:jgriffin) from comment #5)
> Comment on attachment 8449900 [details] [diff] [review]
> buildbot-configs updates for osx Mulet,
> 
> Review of attachment 8449900 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: mozilla-tests/config.py
> @@ +109,5 @@
> > +PLATFORMS['macosx64-mulet']['slave_platforms'] = ['snowleopard']
> > +PLATFORMS['macosx64-mulet']['snowleopard'] = {
> > +    'name': 'Rev4 MacOSX Mulet Snow Leopard 10.6',
> > +    'build_dir_prefix': 'snowleopard_mulet',
> > +    'scheduler_slave_platform_identifier': 'snowleopard_mulet'
> 
> Not sure if these two lines are needed; I got them from the linux64-mulet
> config.
> 
The last two are intended to reduce touching other places like BuildSlaves.py and production_config.py
I wished they also removed the need to touch production-masters.json but it did not :(

> ::: mozilla/config.py
> @@ +787,5 @@
> > +            'slaves': SLAVES['macosx64-lion'],
> > +            'platform_objdir': "%s/i386" % OBJDIR,
> > +            'stage_product': 'firefox',
> > +            'stage_platform': 'macosx64-mulet',
> > +            'update_platform': 'Darwin_x86_64-gcc3',
> 
> Not sure if we need this line.
> 
Unfortunately misc.py or factory.py requires it even if we say "enable_updates=False" (IIUC).

> @@ +800,5 @@
> > +                'SYMBOL_SERVER_USER': 'ffxbld',
> > +                'SYMBOL_SERVER_PATH': SYMBOL_SERVER_PATH,
> > +                'POST_SYMBOL_UPLOAD_CMD': SYMBOL_SERVER_POST_UPLOAD_CMD,
> > +                'SYMBOL_SERVER_SSH_KEY': "/Users/cltbld/.ssh/ffxbld_dsa",
> > +                'MOZ_SYMBOLS_EXTRA_BUILDID': 'macosx64-mulet',
> 
> Not sure if this value should be 'macosx64' or 'macosx64-mulet'.
> 
Good to differentiate it if we ever start uploading symbols.

> @@ +821,5 @@
> > +            # must be overridden explicitly.
> > +            'nightly_signing_servers': 'mac-dep-signing',
> > +            'dep_signing_servers': 'mac-dep-signing',
> > +            'tooltool_manifest_src': 'browser/config/tooltool-manifests/macosx64/releng.manifest',
> > +            'tooltool_l10n_manifest_src': 'browser/config/tooltool-manifests/macosx64/l10n.manifest',
> 
> Can we remove the above 4 lines?

I wish too!
The tooltool lines are for downloading tools like gcc and others.
Comment on attachment 8449874 [details] [diff] [review]
Add mozconfig for osx mulet,

>+++ b/b2g/dev/config/mozconfigs/macosx-universal/mulet	Wed Jul 02 17:15:24 2014 -0700
>\ No newline at end of file

Mind adding the end-of-file newline? :)
Attachment #8449874 - Flags: review?(mshal) → review+
Comment on attachment 8449874 [details] [diff] [review]
Add mozconfig for osx mulet,

Added newline, pushed as https://hg.mozilla.org/integration/b2g-inbound/rev/ffa3484b48c9
Attachment #8449874 - Flags: checked-in+
https://hg.mozilla.org/mozilla-central/rev/ffa3484b48c9
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
Forgot to [leave-open].
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Comment on attachment 8449900 [details] [diff] [review]
buildbot-configs updates for osx Mulet,

Review of attachment 8449900 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/config.py
@@ +800,5 @@
> +                'SYMBOL_SERVER_USER': 'ffxbld',
> +                'SYMBOL_SERVER_PATH': SYMBOL_SERVER_PATH,
> +                'POST_SYMBOL_UPLOAD_CMD': SYMBOL_SERVER_POST_UPLOAD_CMD,
> +                'SYMBOL_SERVER_SSH_KEY': "/Users/cltbld/.ssh/ffxbld_dsa",
> +                'MOZ_SYMBOLS_EXTRA_BUILDID': 'macosx64-mulet',

I think this is ok.

@@ +821,5 @@
> +            # must be overridden explicitly.
> +            'nightly_signing_servers': 'mac-dep-signing',
> +            'dep_signing_servers': 'mac-dep-signing',
> +            'tooltool_manifest_src': 'browser/config/tooltool-manifests/macosx64/releng.manifest',
> +            'tooltool_l10n_manifest_src': 'browser/config/tooltool-manifests/macosx64/l10n.manifest',

Do you want mulet builds to be signed?

And what compiler will they use? The tooltool manifest pulls down the clang compiler for the build I think.
Attachment #8449900 - Flags: review?(catlee) → review+
(In reply to Chris AtLee [:catlee] from comment #11)
> 
> Do you want mulet builds to be signed?
> 

Yes, I think so.

> And what compiler will they use? The tooltool manifest pulls down the clang
> compiler for the build I think.

Same compiler as is used for Firefox, so I think the above should work OK.
Comment on attachment 8449900 [details] [diff] [review]
buildbot-configs updates for osx Mulet,

backed out for failing tests; can someone with Super Decoder Ring clearance tell me what http://10.134.48.37:8080/job/buildbot-configs_tests/2273/ says?
Attachment #8449900 - Flags: review-
Attachment #8449900 - Flags: review+
Attachment #8449900 - Flags: checked-in-
Attachment #8449900 - Flags: checked-in+
It failed 'buildbot checkconfig' for bm81-tests_scheduler:

INFO  - created  "bm81-tests_scheduler" master, running checkconfig
INFO  - starting to print log file '/run/shm/buildbot/bm81-tests_scheduler-90LgSw-checkconfig.log'
INFO  - Traceback (most recent call last):
INFO  -   File "/var/lib/jenkins/jobs/buildbot-configs_tests/workspace/.pyenv/local/lib/python2.7/site-packages/buildbot-0.8.2_hg_18d5488c8185_production_0.8-py2.7.egg/buildbot/scripts/runner.py", line 1042, in doCheckConfig
INFO  -     ConfigLoader(configFileName=configFileName)
INFO  -   File "/var/lib/jenkins/jobs/buildbot-configs_tests/workspace/.pyenv/local/lib/python2.7/site-packages/buildbot-0.8.2_hg_18d5488c8185_production_0.8-py2.7.egg/buildbot/scripts/checkconfig.py", line 31, in __init__
INFO  -     self.loadConfig(configFile, check_synchronously_only=True)
INFO  -   File "/var/lib/jenkins/jobs/buildbot-configs_tests/workspace/.pyenv/local/lib/python2.7/site-packages/buildbot-0.8.2_hg_18d5488c8185_production_0.8-py2.7.egg/buildbot/master.py", line 652, in loadConfig
INFO  -     exec f in localDict
INFO  -   File "/run/shm/buildbot/bm81-tests_scheduler-90LgSw/master.cfg", line 71, in <module>
INFO  -     BRANCH_UNITTEST_VARS['platforms'])
INFO  -   File "/var/lib/jenkins/jobs/buildbot-configs_tests/workspace/buildbotcustom/misc.py", line 2522, in generateTalosBranchObjects
INFO  -     "envName": platform_config['env_name'],
INFO  - KeyError: 'env_name'
INFO  - finished printing log file '/run/shm/buildbot/bm81-tests_scheduler-90LgSw-checkconfig.log'
ERROR - TEST-FAIL bm81-tests_scheduler failed to run checkconfig

https://wiki.mozilla.org/ReleaseEngineering:TestingTechniques#test-masters.sh
Fixed the missing env_name problem.
Attachment #8452478 - Flags: review?(catlee)
Attachment #8449900 - Attachment is obsolete: true
Attachment #8452478 - Flags: review?(catlee) → review+
Attachment #8449899 - Flags: review?(catlee) → review+
Comment on attachment 8452478 [details] [diff] [review]
buildbot-configs updates for osx Mulet,

..and backed out again!  Will run the tests...
Attachment #8452478 - Flags: checked-in+ → checked-in-
Finally got test-masters.sh to run for me; I had fat-fingered env_name again!  Fixed and re-landed: 

https://hg.mozilla.org/build/buildbot-configs/rev/25e48c6d0f2c
Attachment #8452478 - Flags: checked-in- → checked-in+
Comment on attachment 8449899 [details] [diff] [review]
Add osx mulet to production-masters,

https://hg.mozilla.org/build/tools/rev/ac3398893658
Attachment #8449899 - Flags: checked-in+
The production-masters change caused a test failure, apparently.  Backing out:

https://hg.mozilla.org/build/tools/rev/aefd18f890da
Attachment #8449899 - Flags: checked-in+ → checked-in-
Current error:

    INFO  - created  "bm106-tests1-macosx" master, running checkconfig
    INFO  - starting to print log file '/run/shm/buildbot/bm106-tests1-macosx-XmYkOt-checkconfig.log'
    INFO  - Traceback (most recent call last):
    INFO  -   File "/var/lib/jenkins/jobs/buildbot-configs_tests/workspace/.pyenv/local/lib/python2.7/site-packages/buildbot-0.8.2_hg_5484a944940e_production_0.8-py2.7.egg/buildbot/scripts/runner.py", line 1042, in doCheckConfig
    INFO  -     ConfigLoader(configFileName=configFileName)
    INFO  -   File "/var/lib/jenkins/jobs/buildbot-configs_tests/workspace/.pyenv/local/lib/python2.7/site-packages/buildbot-0.8.2_hg_5484a944940e_production_0.8-py2.7.egg/buildbot/scripts/checkconfig.py", line 31, in __init__
    INFO  -     self.loadConfig(configFile, check_synchronously_only=True)
    INFO  -   File "/var/lib/jenkins/jobs/buildbot-configs_tests/workspace/.pyenv/local/lib/python2.7/site-packages/buildbot-0.8.2_hg_5484a944940e_production_0.8-py2.7.egg/buildbot/master.py", line 831, in loadConfig
    INFO  -     % (b['name'], b['builddir']))
    INFO  - ValueError: builder Rev4 MacOSX Snow Leopard 10.6 try talos g1 reuses builddir try_snowleopard_test-g1
    INFO  - finished printing log file '/run/shm/buildbot/bm106-tests1-macosx-XmYkOt-checkconfig.log'
    ERROR - TEST-FAIL bm106-tests1-macosx failed to run checkconfig
Glandium pointed out we are getting runs of Mulet on try, and suggested removing from try. Catlee said we need it on try if it runs on trunk. I wondered if it might make sense to disable it on try by default. Glandium queried whether it really is neccesary to run on trunk too. Coop also suggested we should avoid running on try if we can.

How to proceed?

Thanks,
Pete
Flags: needinfo?(mh+mozilla)
Flags: needinfo?(catlee)
I think we should disable OSX Mulet builds everwhere except Fig until they're tier-1 (which they're not).
I also don't see any work going on Fig for the last two weeks and they're broken there as wel..

See bug 1037861 as well.
Yes, I agree we can/should disable these on try until they're stable.
This turns off mulet on try, and turns off talos_masters for macosx64-mulet, where we don't need it.
Attachment #8455522 - Flags: review?(catlee)
Attachment #8455522 - Flags: review?(catlee) → review+
Flags: needinfo?(catlee)
Thanks guys!
Flags: needinfo?(mh+mozilla)
Comment on attachment 8455522 [details] [diff] [review]
Turn off Mulet on try; turn off Talos on macosx64-mulet,

https://hg.mozilla.org/build/buildbot-configs/rev/f7dad836bece
Attachment #8455522 - Flags: checked-in+
(In reply to Jonathan Griffin (:jgriffin) from comment #29)
> Comment on attachment 8455522 [details] [diff] [review]
> Turn off Mulet on try; turn off Talos on macosx64-mulet,
> 
> https://hg.mozilla.org/build/buildbot-configs/rev/f7dad836bece

Backed out for test failure.
Attachment #8455522 - Flags: checked-in+ → checked-in-
No longer blocks: 1039076
Comment on attachment 8455522 [details] [diff] [review]
Turn off Mulet on try; turn off Talos on macosx64-mulet,

Once more, with feeling: https://hg.mozilla.org/build/buildbot-configs/rev/3acdb8ee14d6
Attachment #8455522 - Flags: checked-in- → checked-in+
Depends on: 1039834
No longer depends on: 1039834
In production.
These builds are running now on fig, but failing with:

printf APPLMOZB > ../../dist/FirefoxNightly.app/Contents/PkgInfo
make[6]: Nothing to be done for `tools'.
set -e; \
	for mkfile in build/macosx/universal/flight.mk; do \
	  make -f /builds/slave/fig-osx64-mulet-00000000000000/build/$mkfile postflight_all TOPSRCDIR=/builds/slave/fig-osx64-mulet-00000000000000/build MOZ_OBJDIR=/builds/slave/fig-osx64-mulet-00000000000000/build/obj-firefox MOZ_BUILD_PROJECTS='i386 x86_64'; \
	done
mkdir -p /builds/slave/fig-osx64-mulet-00000000000000/build/obj-firefox/i386/dist/universal/
rm -f /builds/slave/fig-osx64-mulet-00000000000000/build/obj-firefox/x86_64/dist/universal
ln -s /builds/slave/fig-osx64-mulet-00000000000000/build/obj-firefox/i386/dist/universal /builds/slave/fig-osx64-mulet-00000000000000/build/obj-firefox/x86_64/dist/universal
make -C /builds/slave/fig-osx64-mulet-00000000000000/build/obj-firefox/i386/b2g/dev/installer \
	   PKG_SKIP_STRIP=1 stage-package
make: *** /builds/slave/fig-osx64-mulet-00000000000000/build/obj-firefox/i386/b2g/dev/installer: No such file or directory.  Stop.

I think this is probably not a problem with infrastructure, but a problem with the build on OSX, so I'm going to close this and open a new bug for this problem.
Status: REOPENED → RESOLVED
Closed: 6 years ago6 years ago
Resolution: --- → FIXED
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.