The default bug view has changed. See this FAQ.

Get Mulet builds running on OSX on fig

RESOLVED FIXED

Status

Release Engineering
General Automation
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: jgriffin, Assigned: jgriffin)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(4 attachments, 2 obsolete attachments)

(Assignee)

Description

3 years ago
Currently, we have Mulet builds working on fig on linux64 only; we should add OSX Mulet builds to the mix.
(Assignee)

Comment 1

3 years ago
Created attachment 8449874 [details] [diff] [review]
Add mozconfig for osx mulet,
Attachment #8449874 - Flags: review?(mshal)
(Assignee)

Updated

3 years ago
Assignee: nobody → jgriffin
(Assignee)

Comment 2

3 years ago
Created attachment 8449897 [details] [diff] [review]
buildbot-configs updates for osx Mulet,

This is mostly cargo-culted from Armen's work in bug 997096, so should be checked carefully.  I think this is all we need other than adding the build to production-masters.json.
Attachment #8449897 - Flags: review?(catlee)
(Assignee)

Updated

3 years ago
See Also: → bug 997096
(Assignee)

Comment 3

3 years ago
Created attachment 8449899 [details] [diff] [review]
Add osx mulet to production-masters,
Attachment #8449899 - Flags: review?(catlee)
(Assignee)

Comment 4

3 years ago
Created attachment 8449900 [details] [diff] [review]
buildbot-configs updates for osx Mulet,

Fixed a typo.
Attachment #8449900 - Flags: review?(catlee)
(Assignee)

Updated

3 years ago
Attachment #8449897 - Attachment is obsolete: true
Attachment #8449897 - Flags: review?(catlee)
(Assignee)

Comment 5

3 years ago
Comment on attachment 8449900 [details] [diff] [review]
buildbot-configs updates for osx Mulet,

Review of attachment 8449900 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla-tests/config.py
@@ +109,5 @@
> +PLATFORMS['macosx64-mulet']['slave_platforms'] = ['snowleopard']
> +PLATFORMS['macosx64-mulet']['snowleopard'] = {
> +    'name': 'Rev4 MacOSX Mulet Snow Leopard 10.6',
> +    'build_dir_prefix': 'snowleopard_mulet',
> +    'scheduler_slave_platform_identifier': 'snowleopard_mulet'

Not sure if these two lines are needed; I got them from the linux64-mulet config.

::: mozilla/config.py
@@ +787,5 @@
> +            'slaves': SLAVES['macosx64-lion'],
> +            'platform_objdir': "%s/i386" % OBJDIR,
> +            'stage_product': 'firefox',
> +            'stage_platform': 'macosx64-mulet',
> +            'update_platform': 'Darwin_x86_64-gcc3',

Not sure if we need this line.

@@ +800,5 @@
> +                'SYMBOL_SERVER_USER': 'ffxbld',
> +                'SYMBOL_SERVER_PATH': SYMBOL_SERVER_PATH,
> +                'POST_SYMBOL_UPLOAD_CMD': SYMBOL_SERVER_POST_UPLOAD_CMD,
> +                'SYMBOL_SERVER_SSH_KEY': "/Users/cltbld/.ssh/ffxbld_dsa",
> +                'MOZ_SYMBOLS_EXTRA_BUILDID': 'macosx64-mulet',

Not sure if this value should be 'macosx64' or 'macosx64-mulet'.

@@ +821,5 @@
> +            # must be overridden explicitly.
> +            'nightly_signing_servers': 'mac-dep-signing',
> +            'dep_signing_servers': 'mac-dep-signing',
> +            'tooltool_manifest_src': 'browser/config/tooltool-manifests/macosx64/releng.manifest',
> +            'tooltool_l10n_manifest_src': 'browser/config/tooltool-manifests/macosx64/l10n.manifest',

Can we remove the above 4 lines?
(In reply to Jonathan Griffin (:jgriffin) from comment #5)
> Comment on attachment 8449900 [details] [diff] [review]
> buildbot-configs updates for osx Mulet,
> 
> Review of attachment 8449900 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: mozilla-tests/config.py
> @@ +109,5 @@
> > +PLATFORMS['macosx64-mulet']['slave_platforms'] = ['snowleopard']
> > +PLATFORMS['macosx64-mulet']['snowleopard'] = {
> > +    'name': 'Rev4 MacOSX Mulet Snow Leopard 10.6',
> > +    'build_dir_prefix': 'snowleopard_mulet',
> > +    'scheduler_slave_platform_identifier': 'snowleopard_mulet'
> 
> Not sure if these two lines are needed; I got them from the linux64-mulet
> config.
> 
The last two are intended to reduce touching other places like BuildSlaves.py and production_config.py
I wished they also removed the need to touch production-masters.json but it did not :(

> ::: mozilla/config.py
> @@ +787,5 @@
> > +            'slaves': SLAVES['macosx64-lion'],
> > +            'platform_objdir': "%s/i386" % OBJDIR,
> > +            'stage_product': 'firefox',
> > +            'stage_platform': 'macosx64-mulet',
> > +            'update_platform': 'Darwin_x86_64-gcc3',
> 
> Not sure if we need this line.
> 
Unfortunately misc.py or factory.py requires it even if we say "enable_updates=False" (IIUC).

> @@ +800,5 @@
> > +                'SYMBOL_SERVER_USER': 'ffxbld',
> > +                'SYMBOL_SERVER_PATH': SYMBOL_SERVER_PATH,
> > +                'POST_SYMBOL_UPLOAD_CMD': SYMBOL_SERVER_POST_UPLOAD_CMD,
> > +                'SYMBOL_SERVER_SSH_KEY': "/Users/cltbld/.ssh/ffxbld_dsa",
> > +                'MOZ_SYMBOLS_EXTRA_BUILDID': 'macosx64-mulet',
> 
> Not sure if this value should be 'macosx64' or 'macosx64-mulet'.
> 
Good to differentiate it if we ever start uploading symbols.

> @@ +821,5 @@
> > +            # must be overridden explicitly.
> > +            'nightly_signing_servers': 'mac-dep-signing',
> > +            'dep_signing_servers': 'mac-dep-signing',
> > +            'tooltool_manifest_src': 'browser/config/tooltool-manifests/macosx64/releng.manifest',
> > +            'tooltool_l10n_manifest_src': 'browser/config/tooltool-manifests/macosx64/l10n.manifest',
> 
> Can we remove the above 4 lines?

I wish too!
The tooltool lines are for downloading tools like gcc and others.
Comment on attachment 8449874 [details] [diff] [review]
Add mozconfig for osx mulet,

>+++ b/b2g/dev/config/mozconfigs/macosx-universal/mulet	Wed Jul 02 17:15:24 2014 -0700
>\ No newline at end of file

Mind adding the end-of-file newline? :)
Attachment #8449874 - Flags: review?(mshal) → review+
(Assignee)

Comment 8

3 years ago
Comment on attachment 8449874 [details] [diff] [review]
Add mozconfig for osx mulet,

Added newline, pushed as https://hg.mozilla.org/integration/b2g-inbound/rev/ffa3484b48c9
Attachment #8449874 - Flags: checked-in+
https://hg.mozilla.org/mozilla-central/rev/ffa3484b48c9
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
(Assignee)

Comment 10

3 years ago
Forgot to [leave-open].
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Comment on attachment 8449900 [details] [diff] [review]
buildbot-configs updates for osx Mulet,

Review of attachment 8449900 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/config.py
@@ +800,5 @@
> +                'SYMBOL_SERVER_USER': 'ffxbld',
> +                'SYMBOL_SERVER_PATH': SYMBOL_SERVER_PATH,
> +                'POST_SYMBOL_UPLOAD_CMD': SYMBOL_SERVER_POST_UPLOAD_CMD,
> +                'SYMBOL_SERVER_SSH_KEY': "/Users/cltbld/.ssh/ffxbld_dsa",
> +                'MOZ_SYMBOLS_EXTRA_BUILDID': 'macosx64-mulet',

I think this is ok.

@@ +821,5 @@
> +            # must be overridden explicitly.
> +            'nightly_signing_servers': 'mac-dep-signing',
> +            'dep_signing_servers': 'mac-dep-signing',
> +            'tooltool_manifest_src': 'browser/config/tooltool-manifests/macosx64/releng.manifest',
> +            'tooltool_l10n_manifest_src': 'browser/config/tooltool-manifests/macosx64/l10n.manifest',

Do you want mulet builds to be signed?

And what compiler will they use? The tooltool manifest pulls down the clang compiler for the build I think.
Attachment #8449900 - Flags: review?(catlee) → review+
(Assignee)

Comment 12

3 years ago
(In reply to Chris AtLee [:catlee] from comment #11)
> 
> Do you want mulet builds to be signed?
> 

Yes, I think so.

> And what compiler will they use? The tooltool manifest pulls down the clang
> compiler for the build I think.

Same compiler as is used for Firefox, so I think the above should work OK.
(Assignee)

Comment 13

3 years ago
Comment on attachment 8449900 [details] [diff] [review]
buildbot-configs updates for osx Mulet,

https://hg.mozilla.org/build/buildbot-configs/rev/c29dea9cd76e
Attachment #8449900 - Flags: checked-in+
(Assignee)

Comment 14

3 years ago
Comment on attachment 8449900 [details] [diff] [review]
buildbot-configs updates for osx Mulet,

backed out for failing tests; can someone with Super Decoder Ring clearance tell me what http://10.134.48.37:8080/job/buildbot-configs_tests/2273/ says?
Attachment #8449900 - Flags: review-
Attachment #8449900 - Flags: review+
Attachment #8449900 - Flags: checked-in-
Attachment #8449900 - Flags: checked-in+
It failed 'buildbot checkconfig' for bm81-tests_scheduler:

INFO  - created  "bm81-tests_scheduler" master, running checkconfig
INFO  - starting to print log file '/run/shm/buildbot/bm81-tests_scheduler-90LgSw-checkconfig.log'
INFO  - Traceback (most recent call last):
INFO  -   File "/var/lib/jenkins/jobs/buildbot-configs_tests/workspace/.pyenv/local/lib/python2.7/site-packages/buildbot-0.8.2_hg_18d5488c8185_production_0.8-py2.7.egg/buildbot/scripts/runner.py", line 1042, in doCheckConfig
INFO  -     ConfigLoader(configFileName=configFileName)
INFO  -   File "/var/lib/jenkins/jobs/buildbot-configs_tests/workspace/.pyenv/local/lib/python2.7/site-packages/buildbot-0.8.2_hg_18d5488c8185_production_0.8-py2.7.egg/buildbot/scripts/checkconfig.py", line 31, in __init__
INFO  -     self.loadConfig(configFile, check_synchronously_only=True)
INFO  -   File "/var/lib/jenkins/jobs/buildbot-configs_tests/workspace/.pyenv/local/lib/python2.7/site-packages/buildbot-0.8.2_hg_18d5488c8185_production_0.8-py2.7.egg/buildbot/master.py", line 652, in loadConfig
INFO  -     exec f in localDict
INFO  -   File "/run/shm/buildbot/bm81-tests_scheduler-90LgSw/master.cfg", line 71, in <module>
INFO  -     BRANCH_UNITTEST_VARS['platforms'])
INFO  -   File "/var/lib/jenkins/jobs/buildbot-configs_tests/workspace/buildbotcustom/misc.py", line 2522, in generateTalosBranchObjects
INFO  -     "envName": platform_config['env_name'],
INFO  - KeyError: 'env_name'
INFO  - finished printing log file '/run/shm/buildbot/bm81-tests_scheduler-90LgSw-checkconfig.log'
ERROR - TEST-FAIL bm81-tests_scheduler failed to run checkconfig

https://wiki.mozilla.org/ReleaseEngineering:TestingTechniques#test-masters.sh
(Assignee)

Comment 16

3 years ago
Created attachment 8452478 [details] [diff] [review]
buildbot-configs updates for osx Mulet,

Fixed the missing env_name problem.
Attachment #8452478 - Flags: review?(catlee)
(Assignee)

Updated

3 years ago
Attachment #8449900 - Attachment is obsolete: true

Updated

3 years ago
Attachment #8452478 - Flags: review?(catlee) → review+

Updated

3 years ago
Attachment #8449899 - Flags: review?(catlee) → review+
(Assignee)

Comment 17

3 years ago
Comment on attachment 8452478 [details] [diff] [review]
buildbot-configs updates for osx Mulet,

https://hg.mozilla.org/build/buildbot-configs/rev/1fa3dbcce772
Attachment #8452478 - Flags: checked-in+
(Assignee)

Comment 18

3 years ago
Comment on attachment 8452478 [details] [diff] [review]
buildbot-configs updates for osx Mulet,

..and backed out again!  Will run the tests...
Attachment #8452478 - Flags: checked-in+ → checked-in-
(Assignee)

Comment 19

3 years ago
Finally got test-masters.sh to run for me; I had fat-fingered env_name again!  Fixed and re-landed: 

https://hg.mozilla.org/build/buildbot-configs/rev/25e48c6d0f2c
(Assignee)

Updated

3 years ago
Attachment #8452478 - Flags: checked-in- → checked-in+
(Assignee)

Comment 20

3 years ago
Comment on attachment 8449899 [details] [diff] [review]
Add osx mulet to production-masters,

https://hg.mozilla.org/build/tools/rev/ac3398893658
Attachment #8449899 - Flags: checked-in+
(Assignee)

Comment 21

3 years ago
The production-masters change caused a test failure, apparently.  Backing out:

https://hg.mozilla.org/build/tools/rev/aefd18f890da
(Assignee)

Updated

3 years ago
Attachment #8449899 - Flags: checked-in+ → checked-in-
(Assignee)

Comment 22

3 years ago
Current error:

    INFO  - created  "bm106-tests1-macosx" master, running checkconfig
    INFO  - starting to print log file '/run/shm/buildbot/bm106-tests1-macosx-XmYkOt-checkconfig.log'
    INFO  - Traceback (most recent call last):
    INFO  -   File "/var/lib/jenkins/jobs/buildbot-configs_tests/workspace/.pyenv/local/lib/python2.7/site-packages/buildbot-0.8.2_hg_5484a944940e_production_0.8-py2.7.egg/buildbot/scripts/runner.py", line 1042, in doCheckConfig
    INFO  -     ConfigLoader(configFileName=configFileName)
    INFO  -   File "/var/lib/jenkins/jobs/buildbot-configs_tests/workspace/.pyenv/local/lib/python2.7/site-packages/buildbot-0.8.2_hg_5484a944940e_production_0.8-py2.7.egg/buildbot/scripts/checkconfig.py", line 31, in __init__
    INFO  -     self.loadConfig(configFile, check_synchronously_only=True)
    INFO  -   File "/var/lib/jenkins/jobs/buildbot-configs_tests/workspace/.pyenv/local/lib/python2.7/site-packages/buildbot-0.8.2_hg_5484a944940e_production_0.8-py2.7.egg/buildbot/master.py", line 831, in loadConfig
    INFO  -     % (b['name'], b['builddir']))
    INFO  - ValueError: builder Rev4 MacOSX Snow Leopard 10.6 try talos g1 reuses builddir try_snowleopard_test-g1
    INFO  - finished printing log file '/run/shm/buildbot/bm106-tests1-macosx-XmYkOt-checkconfig.log'
    ERROR - TEST-FAIL bm106-tests1-macosx failed to run checkconfig
This part is now running in production: https://hg.mozilla.org/build/buildbot-configs/rev/25e48c6d0f2c
Glandium pointed out we are getting runs of Mulet on try, and suggested removing from try. Catlee said we need it on try if it runs on trunk. I wondered if it might make sense to disable it on try by default. Glandium queried whether it really is neccesary to run on trunk too. Coop also suggested we should avoid running on try if we can.

How to proceed?

Thanks,
Pete
Flags: needinfo?(mh+mozilla)
Flags: needinfo?(catlee)
I think we should disable OSX Mulet builds everwhere except Fig until they're tier-1 (which they're not).
I also don't see any work going on Fig for the last two weeks and they're broken there as wel..

See bug 1037861 as well.
(Assignee)

Comment 26

3 years ago
Yes, I agree we can/should disable these on try until they're stable.
(Assignee)

Comment 27

3 years ago
Created attachment 8455522 [details] [diff] [review]
Turn off Mulet on try; turn off Talos on macosx64-mulet,

This turns off mulet on try, and turns off talos_masters for macosx64-mulet, where we don't need it.
Attachment #8455522 - Flags: review?(catlee)

Updated

3 years ago
Attachment #8455522 - Flags: review?(catlee) → review+

Updated

3 years ago
Flags: needinfo?(catlee)
Thanks guys!
(Assignee)

Updated

3 years ago
Flags: needinfo?(mh+mozilla)
(Assignee)

Comment 29

3 years ago
Comment on attachment 8455522 [details] [diff] [review]
Turn off Mulet on try; turn off Talos on macosx64-mulet,

https://hg.mozilla.org/build/buildbot-configs/rev/f7dad836bece
Attachment #8455522 - Flags: checked-in+
(Assignee)

Comment 30

3 years ago
(In reply to Jonathan Griffin (:jgriffin) from comment #29)
> Comment on attachment 8455522 [details] [diff] [review]
> Turn off Mulet on try; turn off Talos on macosx64-mulet,
> 
> https://hg.mozilla.org/build/buildbot-configs/rev/f7dad836bece

Backed out for test failure.
(Assignee)

Updated

3 years ago
Attachment #8455522 - Flags: checked-in+ → checked-in-

Updated

3 years ago
Blocks: 1039076

Updated

3 years ago
No longer blocks: 1039076
(Assignee)

Comment 31

3 years ago
Comment on attachment 8455522 [details] [diff] [review]
Turn off Mulet on try; turn off Talos on macosx64-mulet,

Once more, with feeling: https://hg.mozilla.org/build/buildbot-configs/rev/3acdb8ee14d6
Attachment #8455522 - Flags: checked-in- → checked-in+
(Assignee)

Updated

3 years ago
Depends on: 1039834
(Assignee)

Updated

3 years ago
No longer depends on: 1039834

Comment 32

3 years ago
In production.
(Assignee)

Comment 33

3 years ago
These builds are running now on fig, but failing with:

printf APPLMOZB > ../../dist/FirefoxNightly.app/Contents/PkgInfo
make[6]: Nothing to be done for `tools'.
set -e; \
	for mkfile in build/macosx/universal/flight.mk; do \
	  make -f /builds/slave/fig-osx64-mulet-00000000000000/build/$mkfile postflight_all TOPSRCDIR=/builds/slave/fig-osx64-mulet-00000000000000/build MOZ_OBJDIR=/builds/slave/fig-osx64-mulet-00000000000000/build/obj-firefox MOZ_BUILD_PROJECTS='i386 x86_64'; \
	done
mkdir -p /builds/slave/fig-osx64-mulet-00000000000000/build/obj-firefox/i386/dist/universal/
rm -f /builds/slave/fig-osx64-mulet-00000000000000/build/obj-firefox/x86_64/dist/universal
ln -s /builds/slave/fig-osx64-mulet-00000000000000/build/obj-firefox/i386/dist/universal /builds/slave/fig-osx64-mulet-00000000000000/build/obj-firefox/x86_64/dist/universal
make -C /builds/slave/fig-osx64-mulet-00000000000000/build/obj-firefox/i386/b2g/dev/installer \
	   PKG_SKIP_STRIP=1 stage-package
make: *** /builds/slave/fig-osx64-mulet-00000000000000/build/obj-firefox/i386/b2g/dev/installer: No such file or directory.  Stop.

I think this is probably not a problem with infrastructure, but a problem with the build on OSX, so I'm going to close this and open a new bug for this problem.
Status: REOPENED → RESOLVED
Last Resolved: 3 years ago3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.