1052544 - Add nightlies for linux64 Mulet

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Jonathan Griffin (:jgriffin)]

According to Alex, it would be good to produce nightlies for Mulet.

Comment 1

[Alexandre Poirot [:ochameau]]

Yes, that would be a big step forward for mulet.
It would allow gaia developers to start playing with it to see how usable it is.

Is there anything I can do to help spinning nightlies?
Do we need more tests running?

Comment 2

[Jonathan Griffin (:jgriffin)]

I'll take care of this.

Comment 3

[Jonathan Griffin (:jgriffin)]

Attachment: Add nightlies for linux64_mulet,

It looks like this is all that's needed.....

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Looks like we are uploading Mulet builds into /pub/mozilla.org/firefox/, while the older gecko builds go into .../b2g/. If Mulet is going to the wrong place it would be good to fix that up before we enable nightly builds.

Comment 5

[Alexandre Poirot [:ochameau]]

Yes, that would be great to see it appear on: http://ftp.mozilla.org/pub/mozilla.org/b2g/nightly/ with b2g-desktop and the simulator. That to not confuse people looking for firefox, the browser!

Comment 6

[Jordan Lund (:jlund)]

Comment on attachment 8474888 [details] [diff] [review]
Add nightlies for linux64_mulet,

Review of attachment 8474888 [details] [diff] [review]:
-----------------------------------------------------------------

I think we also need: 'update_platform': 'Linux_x86_64-gcc3',

because our nightly builder init expects it here: http://mxr.mozilla.org/build/source/buildbotcustom/misc.py#2098

BTW - this patch will AFAIK create a mar/snippets, partials, and do updates?

Do we want this to be like our linux64 asan nightlies where we don't 'make uploadsymbols', create mars, do updates, or worry about signing?

Is this bug mainly about clobbering and running with PGO (nightly build)

hopefully this info helps. I'll be around if I've just added more confusion :)

Comment 7

[Jonathan Griffin (:jgriffin)]

I don't know the answers to most of these questions.  Alexandre, what are the nightlies going to be used for?  Do we want them to do updates, ala Firefox?

Comment 8

[Alexandre Poirot [:ochameau]]

Nightlies are going to be used like b2g desktop is.
Mostly by gaia developers to test their patches against latest gecko,
but also integration tools are going to pull mulet instead of b2g desktop to run gaia tests.

Having mar and automated update could help gaia developers to use latest version,
it may also open opportunities for integration tools to download mar file intead of complete updates;
but that's not something we have to support. I would consider this as a "nice to have".

Comment 9

[Jonathan Griffin (:jgriffin)]

Attachment: Add nightlies for linux64_mulet,

Addressed review comments; also switched stage_platform from firefox to b2g.

Comment 10

[Jordan Lund (:jlund)]

Comment on attachment 8476896 [details] [diff] [review]
Add nightlies for linux64_mulet,

Review of attachment 8476896 [details] [diff] [review]:
-----------------------------------------------------------------

Apologies for the delay. I am on PTO atm until wed. If this is left to me, I may look at it before then but I can't guarantee it :)

Comment 11

[Jordan Lund (:jlund)]

Comment on attachment 8476896 [details] [diff] [review]
Add nightlies for linux64_mulet,

Review of attachment 8476896 [details] [diff] [review]:
-----------------------------------------------------------------

huh, strange. I ran this and it looks like all mulet builds disappear after this applies (incl non nightlies). I couldn't figure out why until I took a look at stage_product. It has a strange side effect.

we add all our builders in generateBranchObjects. And, we add them only if: http://mxr.mozilla.org/build/source/buildbotcustom/misc.py#1211

b2g_config.py has b2g in that list: http://mxr.mozilla.org/build/source/buildbot-configs/mozilla/b2g_config.py#60

config.py (where mulet currently lives) we don't have it: http://mxr.mozilla.org/build/source/buildbot-configs/mozilla/config.py#87

so i think for this to work, we will need to rip out this whole item from config.py and put it in b2g_config.py. Further, I think changing the stage_product will also change the builddir (where we keep history of individual build types on our slaves). I think this is ok. clobberer should rm the old dir if we change it but we will want to confirm that.

Comment 12

[Jonathan Griffin (:jgriffin)]

Thanks for the review!

> so i think for this to work, we will need to rip out this whole item from config.py and put it in 
> b2g_config.py

Ugh!  Ok, I'll try that.

Comment 13

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Any reason we'd just build linux binaries? We have most Gaia developers (which is the initial target audience) on Mac, and probably a few on Windows as well.

Also, is there a reason to do 64bit builds rather than 32bit? I don't think it matters much, but being more similar to device builds is always a good thing.

Comment 14

[Jonathan Griffin (:jgriffin)]

Mac builds are busted on fig, but it's been reset recently so we can't see it.  I'll cause a retrigger.

Comment 15

[Jonathan Griffin (:jgriffin)]

Attachment: Move mulet to b2g_config and enable nightlies,

This is a pretty big, complicated patch.  I had to move all the Mulet stuff from config.py to b2g_config.py, for both builds and tests.  At the same time, I also dropped all the fig-specific things; I think we can do remaining greening work on cedar.

There's a build-tools patch that goes along with this.  I don't know how to test these in tandem, so it's possible I've missed something.

Comment 16

[Jonathan Griffin (:jgriffin)]

Attachment: Move mulet to b2g_platforms in production-masters.json,

Comment 17

[Jonathan Griffin (:jgriffin)]

Attachment: Move mulet to b2g_config and enable nightlies,

Missed the OSX mochitests

Comment 18

[Jordan Lund (:jlund)]

Comment on attachment 8485090 [details] [diff] [review]
Move mulet to b2g_config and enable nightlies,

Review of attachment 8485090 [details] [diff] [review]:
-----------------------------------------------------------------

whoa I feel bad for you. I think with patches like these you are becoming a buildbot-config guru. It is a shame we have to change so much to enable Mulet nightlys and switch the product name to B2G

1 side effect I found while going through this is I think we are going to hit:
http://mxr.mozilla.org/build/source/buildbotcustom/misc.py#1732

which will call that script as part of the build steps. This is currently done by all our gecko desktop builds. we didn't hit this before. not sure what it does.


I think we are also going to need to remove the fig refs to mulet here(used by mozilla/config.py and mozilla-tests/config.py ): http://mxr.mozilla.org/build/source/buildbot-configs/mozilla/project_branches.py#181

and put them in here: http://mxr.mozilla.org/build/source/buildbot-configs/mozilla/b2g_project_branches.py#59

finally: I ran this agaiest test-masters.sh and I am not sure why but it looks like 1 test master fell over. I ran out of time looking today but it might be due to production-masters.json patch. or something entirely different :)

looks like master 51 which is a test linux 64 master:
  bm51-tests1-linux64
  bm51-tests1-linux64-universal

log snippet from test masters:
(build-master)[[e-mail] buildbot-configs]$ cat /builds/buildbot/jlund/build-master/buildbot-configs/test-output/bm51-tests1-linux64-VEJDmg-checkconfig.log
/builds/buildbot/jlund/build-master/lib/python2.6/site-packages/twisted/mail/smtp.py:10: DeprecationWarning: the MimeWriter module is deprecated; use the email package instead
  import MimeWriter, tempfile, rfc822
Traceback (most recent call last):
  File "/builds/buildbot/jlund/build-master/lib/python2.6/site-packages/buildbot-0.8.2_hg_c3434463c089_production_0.8-py2.6.egg/buildbot/scripts/runner.py", line 1042, in doCheckConfig
    ConfigLoader(configFileName=configFileName)
  File "/builds/buildbot/jlund/build-master/lib/python2.6/site-packages/buildbot-0.8.2_hg_c3434463c089_production_0.8-py2.6.egg/buildbot/scripts/checkconfig.py", line 31, in __init__
    self.loadConfig(configFile, check_synchronously_only=True)
  File "/builds/buildbot/jlund/build-master/lib/python2.6/site-packages/buildbot-0.8.2_hg_c3434463c089_production_0.8-py2.6.egg/buildbot/master.py", line 652, in loadConfig
    exec f in localDict
  File "/builds/buildbot/jlund/build-master/buildbot-configs/test-output/bm51-tests1-linux64-VEJDmg/master.cfg", line 70, in <module>
    ACTIVE_PLATFORMS[p] = deepcopy(PLATFORMS[p])
KeyError: u'linux64-mulet'


r- till we work out the small kinks

::: mozilla-tests/b2g_config.py
@@ +275,5 @@
> +    ('mochitest-mulet-plain-1', {'suite': 'mochitest-mulet-plain',
> +                                 'use_mozharness': True,
> +                                 'script_path': 'scripts/desktop_unittest.py',
> +                                 'blob_upload': True,
> +                                 'script_maxtime': 7200,

do we need maxtime here? 2 hours seems long.

@@ +798,5 @@
> +            'suite_config': {
> +                'mochitest-mulet-plain-1': {
> +                    'extra_args': [
> +                      '--cfg', 'unittests/linux_unittest.py',
> +                      '--total-chunks', 5, '--this-chunk', 1,

I'm having a hard time understanding this part. It looks like we only define mochi chunk 1 for linux and Mac. But we put the plumbing in for all the chunks above.

@@ +816,5 @@
> +            "NO_EM_RESTART": '1',
> +            "XPCOM_DEBUG_BREAK": 'warn',
> +            "MOZ_CRASHREPORTER_NO_REPORT": '1',
> +            # for extracting dmg's
> +            "PAGER": '/bin/cat',

missing closing } to env

::: mozilla-tests/config.py
@@ -129,5 @@
> -    'mozharness_python': '/tools/buildbot/bin/python',
> -    'hg_bin': 'hg',
> -    'reboot_command': ['/tools/buildbot/bin/python'] + MOZHARNESS_REBOOT_CMD,
> -    'system_bits': '64',
> -    'config_file': 'talos/mac_config.py',

did we run talos tests on mulet or is this a mistake that wasn't being used anywhere?

@@ -1571,5 @@
> -                    'config_files': ["web_platform_tests/prod_config.py"],
> -                },
> -                'mozbase': {
> -                    'config_files': ["unittests/mac_unittest.py"],
> -                },

there are so many suites here. I'm guessing we only cared about mochitests and the rest was just copied from other platforms?

::: mozilla/b2g_config.py
@@ +396,5 @@
> +            'script_timeout': 3 * 3600,
> +            'script_maxtime': int(5.5 * 3600),
> +        },
> +
> +        'consider_for_nightly': False,

hmm I think this is fine. I think linux64-asan (another nightly builder) sets this to False too.

the desc I found for this:
            # The status of this build doesn't affect the last good revision
            # algorithm for nightlies
and:
    # Which builders should we consider when looking at per-checkin results and
    # determining what revision we should do a nightly build on

::: mozilla/config.py
@@ -2620,5 @@
>  
> -# Mulet landed in gecko 34
> -for name, branch in items_before(BRANCHES, 'gecko_version', 34):
> -    if 'linux64-mulet' in branch['platforms']:
> -        del branch['platforms']['linux64-mulet']

huh, so I guess macosx64-mulet was never used? But it was defined above?

Comment 19

[Jordan Lund (:jlund)]

Comment on attachment 8485085 [details] [diff] [review]
Move mulet to b2g_platforms in production-masters.json,

Review of attachment 8485085 [details] [diff] [review]:
-----------------------------------------------------------------

I can't see anything immediately wrong with this but since this fails along side test-masters.sh I'm going to reset this to r-. feel free to request same patch again if we need to change nothing.

Comment 20

[Jonathan Griffin (:jgriffin)]

> 1 side effect I found while going through this is I think we are going to hit:
> http://mxr.mozilla.org/build/source/buildbotcustom/misc.py#1732

What other things is 'product' used for?  I want the builds to go into the /b2g/ dir on the ftp site, but I guess that's set by 'stage_product'.  We might be able to use 'mulet' for 'product'.

Comment 21

[Jonathan Griffin (:jgriffin)]

(In reply to Jordan Lund (:jlund) from comment #18)
> Comment on attachment 8485090 [details] [diff] [review]
> Move mulet to b2g_config and enable nightlies,
> 
> Review of attachment 8485090 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> whoa I feel bad for you. I think with patches like these you are becoming a
> buildbot-config guru. It is a shame we have to change so much to enable
> Mulet nightlys and switch the product name to B2G
> 
> 1 side effect I found while going through this is I think we are going to
> hit:
> http://mxr.mozilla.org/build/source/buildbotcustom/misc.py#1732
> 
> which will call that script as part of the build steps. This is currently
> done by all our gecko desktop builds. we didn't hit this before. not sure
> what it does.

I'll try changing product name to 'mulet'.  Whatever that b2gdesktop script does, I don't think it would work with Mulet.

> 
> 
> I think we are also going to need to remove the fig refs to mulet here(used
> by mozilla/config.py and mozilla-tests/config.py ):
> http://mxr.mozilla.org/build/source/buildbot-configs/mozilla/
> project_branches.py#181
> 
> and put them in here:
> http://mxr.mozilla.org/build/source/buildbot-configs/mozilla/
> b2g_project_branches.py#59
> 
> finally: I ran this agaiest test-masters.sh and I am not sure why but it
> looks like 1 test master fell over. I ran out of time looking today but it
> might be due to production-masters.json patch. or something entirely
> different :)
> 
> looks like master 51 which is a test linux 64 master:
>   bm51-tests1-linux64
>   bm51-tests1-linux64-universal
> 
> log snippet from test masters:
> (build-master)[[e-mail]
> buildbot-configs]$ cat
> /builds/buildbot/jlund/build-master/buildbot-configs/test-output/bm51-tests1-
> linux64-VEJDmg-checkconfig.log
> /builds/buildbot/jlund/build-master/lib/python2.6/site-packages/twisted/mail/
> smtp.py:10: DeprecationWarning: the MimeWriter module is deprecated; use the
> email package instead
>   import MimeWriter, tempfile, rfc822
> Traceback (most recent call last):
>   File
> "/builds/buildbot/jlund/build-master/lib/python2.6/site-packages/buildbot-0.
> 8.2_hg_c3434463c089_production_0.8-py2.6.egg/buildbot/scripts/runner.py",
> line 1042, in doCheckConfig
>     ConfigLoader(configFileName=configFileName)
>   File
> "/builds/buildbot/jlund/build-master/lib/python2.6/site-packages/buildbot-0.
> 8.2_hg_c3434463c089_production_0.8-py2.6.egg/buildbot/scripts/checkconfig.
> py", line 31, in __init__
>     self.loadConfig(configFile, check_synchronously_only=True)
>   File
> "/builds/buildbot/jlund/build-master/lib/python2.6/site-packages/buildbot-0.
> 8.2_hg_c3434463c089_production_0.8-py2.6.egg/buildbot/master.py", line 652,
> in loadConfig
>     exec f in localDict
>   File
> "/builds/buildbot/jlund/build-master/buildbot-configs/test-output/bm51-
> tests1-linux64-VEJDmg/master.cfg", line 70, in <module>
>     ACTIVE_PLATFORMS[p] = deepcopy(PLATFORMS[p])
> KeyError: u'linux64-mulet'
> 
> 
> r- till we work out the small kinks
> 
> ::: mozilla-tests/b2g_config.py
> @@ +275,5 @@
> > +    ('mochitest-mulet-plain-1', {'suite': 'mochitest-mulet-plain',
> > +                                 'use_mozharness': True,
> > +                                 'script_path': 'scripts/desktop_unittest.py',
> > +                                 'blob_upload': True,
> > +                                 'script_maxtime': 7200,
> 
> do we need maxtime here? 2 hours seems long.

I'll remove this and let it revert to the default.

> 
> @@ +798,5 @@
> > +            'suite_config': {
> > +                'mochitest-mulet-plain-1': {
> > +                    'extra_args': [
> > +                      '--cfg', 'unittests/linux_unittest.py',
> > +                      '--total-chunks', 5, '--this-chunk', 1,
> 
> I'm having a hard time understanding this part. It looks like we only define
> mochi chunk 1 for linux and Mac. But we put the plumbing in for all the
> chunks above.

For config.py, the chunking is handled semi-dynamically by buildbot; for b2g_config.py, chunks are always specified explicitly.  I just converted the format from the former to the latter, without a change in meaniing.

> 
> @@ +816,5 @@
> > +            "NO_EM_RESTART": '1',
> > +            "XPCOM_DEBUG_BREAK": 'warn',
> > +            "MOZ_CRASHREPORTER_NO_REPORT": '1',
> > +            # for extracting dmg's
> > +            "PAGER": '/bin/cat',
> 
> missing closing } to env
> 

Thanks, will fix.

> ::: mozilla-tests/config.py
> @@ -129,5 @@
> > -    'mozharness_python': '/tools/buildbot/bin/python',
> > -    'hg_bin': 'hg',
> > -    'reboot_command': ['/tools/buildbot/bin/python'] + MOZHARNESS_REBOOT_CMD,
> > -    'system_bits': '64',
> > -    'config_file': 'talos/mac_config.py',
> 
> did we run talos tests on mulet or is this a mistake that wasn't being used
> anywhere?

We didn't run talos on Mulet, and don't want to.

> 
> @@ -1571,5 @@
> > -                    'config_files': ["web_platform_tests/prod_config.py"],
> > -                },
> > -                'mozbase': {
> > -                    'config_files': ["unittests/mac_unittest.py"],
> > -                },
> 
> there are so many suites here. I'm guessing we only cared about mochitests
> and the rest was just copied from other platforms?

Initially, Mulet was created as a desktop build and all the desktop tests were run against it on Fig.  mochitest-plain is the only one that works, though, and I don't think we'll worry about the others, at least for the time being, so I removed them.

> 
> ::: mozilla/b2g_config.py
> @@ +396,5 @@
> > +            'script_timeout': 3 * 3600,
> > +            'script_maxtime': int(5.5 * 3600),
> > +        },
> > +
> > +        'consider_for_nightly': False,
> 
> hmm I think this is fine. I think linux64-asan (another nightly builder)
> sets this to False too.
> 
> the desc I found for this:
>             # The status of this build doesn't affect the last good revision
>             # algorithm for nightlies
> and:
>     # Which builders should we consider when looking at per-checkin results
> and
>     # determining what revision we should do a nightly build on
> 
> ::: mozilla/config.py
> @@ -2620,5 @@
> >  
> > -# Mulet landed in gecko 34
> > -for name, branch in items_before(BRANCHES, 'gecko_version', 34):
> > -    if 'linux64-mulet' in branch['platforms']:
> > -        del branch['platforms']['linux64-mulet']
> 
> huh, so I guess macosx64-mulet was never used? But it was defined above?

macosx64-mulet was only running on fig, and special-cased there, so it wasn't necessary to add it to this block.

Comment 22

[Jonathan Griffin (:jgriffin)]

Attachment: Move mulet to b2g_config and enable nightlies,

Addressed all your review comments, I believe.  This passes test-masters.sh for me locally, with the caveat that you have to hack test-masters.sh to use the copy of production-masters.json in the related patch; otherwise it uses the tip in hg, which won't work.  I've changed 'product' to 'mulet'; hopefully that won't cause problems.  I've left 'stage_product' as b2g, hoping that's what causes it to appear in the /b2g/ directory on ftp.

Comment 23

[Jonathan Griffin (:jgriffin)]

Attachment: Move mulet to b2g_platforms in production-masters.json,

Unbitrot production-masters.sh

Comment 24

[Jordan Lund (:jlund)]

> > 
> > @@ +798,5 @@
> > > +            'suite_config': {
> > > +                'mochitest-mulet-plain-1': {
> > > +                    'extra_args': [
> > > +                      '--cfg', 'unittests/linux_unittest.py',
> > > +                      '--total-chunks', 5, '--this-chunk', 1,
> > 
> > I'm having a hard time understanding this part. It looks like we only define
> > mochi chunk 1 for linux and Mac. But we put the plumbing in for all the
> > chunks above.
> 
> For config.py, the chunking is handled semi-dynamically by buildbot; for
> b2g_config.py, chunks are always specified explicitly.  I just converted the
> format from the former to the latter, without a change in meaniing.

I'm being slow today. You mentioned b2g_config has chunks explicitly defined which I agree with. But unless I'm mistaken we need to define more than just chunk one. It looks like we do chunks three and five in production.

Comment 25

[Jordan Lund (:jlund)]

Comment on attachment 8486806 [details] [diff] [review]
Move mulet to b2g_config and enable nightlies,

Review of attachment 8486806 [details] [diff] [review]:
-----------------------------------------------------------------

We spoke a irc and there will be a new patch

Comment 26

[Jonathan Griffin (:jgriffin)]

Attachment: Move mulet to b2g_config and enable nightlies,

Here's the updated patch with the right mix of product and stage names.

Comment 27

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8487491 [details] [diff] [review]
Move mulet to b2g_config and enable nightlies,

Carrying r+ forward, but feel free to take another whack at it if you want.

Comment 28

[Jonathan Griffin (:jgriffin)]

Attachment: Move mulet to b2g_config and enable nightlies,

Updated to set multi_locale to False, and to fix some problems with the Mac testers.  Will post an interdiff separately.

Comment 29

[Jonathan Griffin (:jgriffin)]

Attachment: interdiff of last two buildbot-configs patches

Comment 30

[Jonathan Griffin (:jgriffin)]

Attachment: Move mulet to b2g_config and enable nightlies,

Reverted an unintentional change to macosx64_gecko

Comment 31

[Jonathan Griffin (:jgriffin)]

Attachment: interdiff.diff

New interdiff

Comment 32

[Jonathan Griffin (:jgriffin)]

Attachment: interdiff.diff

Properly formatted interdiff

Comment 33

[Jonathan Griffin (:jgriffin)]

Attachment: interdiff.diff

Comment 34

[Simone Bruno [:simone]]

In prod with reconfig on 2014-09-15 08:30 PT

Comment 35

[Jordan Lund (:jlund)]

Comment on attachment 8488915 [details] [diff] [review]
Move mulet to b2g_config and enable nightlies,

Review of attachment 8488915 [details] [diff] [review]:
-----------------------------------------------------------------

So I had a think about this. WRT to the mountainlion-mulet discussion we had on friday, I would like to try one more thing though (see in line)

everything else from the interdiff lgtm :)

::: mozilla-tests/b2g_config.py
@@ +134,4 @@
>      'reboot_command': ['/tools/buildbot/bin/python'] + MOZHARNESS_REBOOT_CMD,
>  }
>  
> +PLATFORMS['macosx64-mulet']['slave_platforms'] = ['mountainlion-mulet']

I think we can do 'snowleopard' here. you mentioned this failed with dupe builddir errors but I think we can solve that with the below:

@@ +135,5 @@
>  }
>  
> +PLATFORMS['macosx64-mulet']['slave_platforms'] = ['mountainlion-mulet']
> +PLATFORMS['macosx64-mulet']['env_name'] = 'mac-perf'
> +PLATFORMS['macosx64-mulet']['mountainlion-mulet'] = {'name': builder_prefix + "_macosx64"}

here, like mozilla-tests/config.py, I *think* we can replace mountainlion-mulet with the following and misc.py will do the right thing:

+PLATFORMS['macosx64-mulet']['snowleopard'] = {
+    'name': builder_prefix + "_macosx64"
+    'build_dir_prefix': 'snowleopard_mulet',
+    'scheduler_slave_platform_identifier': 'snowleopard_mulet'
+}


this way we don't need another fake slave pf. can you try that against test-masters.sh?

Comment 36

[Jonathan Griffin (:jgriffin)]

Attachment: Move mulet to b2g_config and enable nightlies,

You're right, that works.  Latest patch attached; passes test-masters.sh

Comment 37

[Jonathan Griffin (:jgriffin)]

Attachment: interdiff.diff

New interdiff

Comment 38

[Jordan Lund (:jlund)]

Comment on attachment 8489736 [details] [diff] [review]
Move mulet to b2g_config and enable nightlies,

Review of attachment 8489736 [details] [diff] [review]:
-----------------------------------------------------------------

interdiff was scrambled.

but assuming that snowleopard change was the only new diff, lgtm :)

Comment 39

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8486807 [details] [diff] [review]
Move mulet to b2g_platforms in production-masters.json,

https://hg.mozilla.org/build/tools/rev/62be94af9566

Comment 40

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8489736 [details] [diff] [review]
Move mulet to b2g_config and enable nightlies,

https://hg.mozilla.org/build/buildbot-configs/rev/57ad434b91dd

Comment 41

[Jonathan Griffin (:jgriffin)]

Heads up for sheriffs:  this patch is pretty complicated, so if Mulet jobs blow up at the next reconfig, this patch is probably at fault.

Comment 42

[Justin Wood (:Callek)]

Backed out both changes --- this blew up due to no puppet patch for BuildSlaves*.py

Comment 43

[Jonathan Griffin (:jgriffin)]

Attachment: Add mulet to linux passwords,

Comment 44

[Jordan Lund (:jlund)]

Comment on attachment 8491034 [details] [diff] [review]
Add mulet to linux passwords,

Review of attachment 8491034 [details] [diff] [review]:
-----------------------------------------------------------------

I started poking around this and how things work and I think this is right.

I'm still confused about the BuildSlaves-try.py.erb file but I think that should just be build slaves (but BuildSlaves-builds.py.erb has more keys than try). At any rate, IIUC, BuildSlaves-tests.py.erb is shared everywhere and like ubuntu64_* this is the only file that fits our added slave.

Comment 45

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8491034 [details] [diff] [review]
Add mulet to linux passwords,

https://hg.mozilla.org/build/puppet/rev/5479a8704301

Comment 46

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8489736 [details] [diff] [review]
Move mulet to b2g_config and enable nightlies,

https://hg.mozilla.org/build/buildbot-configs/rev/4d2ec7a85bd0

Comment 47

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8486807 [details] [diff] [review]
Move mulet to b2g_platforms in production-masters.json,

https://hg.mozilla.org/build/tools/rev/8c58c8a9b1d8

Comment 48

[Simone Bruno [:simone]]

In prod with reconfig on 2014-09-18 08:14 PT

Comment 49

[Jonathan Griffin (:jgriffin)]

Attachment: Remove mulet from suite names,

This removes mulet from the suite name strings, which is inconsistent according to the sheriffs.

Comment 50

[Ed Morley [:emorley]]

Comment on attachment 8491592 [details] [diff] [review]
Remove mulet from suite names,

For all other jobs we have:
<unique-platform-identifer> <repo + foo> <test-name>

The current test buildername is:
b2g_ubuntu64_vm mozilla-inbound opt test mochitest-mulet-plain-1

With this patch, it would presumably be:
b2g_ubuntu64_vm mozilla-inbound opt test mochitest-plain-1

Whereas we really need:
b2g_mulet_ubuntu64_vm mozilla-inbound opt test mochitest-plain-1
or
mulet_ubuntu64_vm mozilla-inbound opt test mochitest-plain-1
...or any other variation that includes "mulet" prior to the repo name in that string.

Comment 51

[Ed Morley [:emorley]]

(In reply to Ed Morley [:edmorley] from comment #50)
> With this patch, it would presumably be:
> b2g_ubuntu64_vm mozilla-inbound opt test mochitest-plain-1

Which would overlap with the old style B2G desktop jobs. Unless this doesn't matter?

Comment 52

[Jonathan Griffin (:jgriffin)]

According to RyanVM, we'd like the test names to be e.g., Ubuntu VM 12.04 x64 Mulet mozilla-aurora opt test mochitest-4, as they were before this change.

Comment 53

[Ed Morley [:emorley]]

That wfm; happy to defer to him on this :-)

Comment 54

[Ryan VanderMeulen [:RyanVM]]

(In reply to Jonathan Griffin (:jgriffin) from comment #52)
> According to RyanVM, we'd like the test names to be e.g., Ubuntu VM 12.04
> x64 Mulet mozilla-aurora opt test mochitest-4, as they were before this
> change.

++

Thanks! That'll make life much easier vs. having to support two different naming conventions in an already-messy regex.

Comment 55

[Jonathan Griffin (:jgriffin)]

Attachment: Remove mulet from suite names,

This makes the job names consistent with their earlier versions.

Comment 56

[Jonathan Griffin (:jgriffin)]

The effects of this change:

Builders added:
+ Ubuntu VM 12.04 x64 Mulet ash opt test mochitest-1
+ Ubuntu VM 12.04 x64 Mulet ash opt test mochitest-2
+ Ubuntu VM 12.04 x64 Mulet ash opt test mochitest-3
+ Ubuntu VM 12.04 x64 Mulet ash opt test mochitest-4
+ Ubuntu VM 12.04 x64 Mulet ash opt test mochitest-5
+ Ubuntu VM 12.04 x64 Mulet b2g-inbound opt test mochitest-1
+ Ubuntu VM 12.04 x64 Mulet b2g-inbound opt test mochitest-2
+ Ubuntu VM 12.04 x64 Mulet b2g-inbound opt test mochitest-3
+ Ubuntu VM 12.04 x64 Mulet b2g-inbound opt test mochitest-4
+ Ubuntu VM 12.04 x64 Mulet b2g-inbound opt test mochitest-5
+ Ubuntu VM 12.04 x64 Mulet cedar opt test mochitest-1
+ Ubuntu VM 12.04 x64 Mulet cedar opt test mochitest-2
+ Ubuntu VM 12.04 x64 Mulet cedar opt test mochitest-3
+ Ubuntu VM 12.04 x64 Mulet cedar opt test mochitest-4
+ Ubuntu VM 12.04 x64 Mulet cedar opt test mochitest-5
+ Ubuntu VM 12.04 x64 Mulet cypress opt test mochitest-1
+ Ubuntu VM 12.04 x64 Mulet cypress opt test mochitest-2
+ Ubuntu VM 12.04 x64 Mulet cypress opt test mochitest-3
+ Ubuntu VM 12.04 x64 Mulet cypress opt test mochitest-4
+ Ubuntu VM 12.04 x64 Mulet cypress opt test mochitest-5
+ Ubuntu VM 12.04 x64 Mulet fx-team opt test mochitest-1
+ Ubuntu VM 12.04 x64 Mulet fx-team opt test mochitest-2
+ Ubuntu VM 12.04 x64 Mulet fx-team opt test mochitest-3
+ Ubuntu VM 12.04 x64 Mulet fx-team opt test mochitest-4
+ Ubuntu VM 12.04 x64 Mulet fx-team opt test mochitest-5
+ Ubuntu VM 12.04 x64 Mulet graphics opt test mochitest-1
+ Ubuntu VM 12.04 x64 Mulet graphics opt test mochitest-2
+ Ubuntu VM 12.04 x64 Mulet graphics opt test mochitest-3
+ Ubuntu VM 12.04 x64 Mulet graphics opt test mochitest-4
+ Ubuntu VM 12.04 x64 Mulet graphics opt test mochitest-5
+ Ubuntu VM 12.04 x64 Mulet jamun opt test mochitest-1
+ Ubuntu VM 12.04 x64 Mulet jamun opt test mochitest-2
+ Ubuntu VM 12.04 x64 Mulet jamun opt test mochitest-3
+ Ubuntu VM 12.04 x64 Mulet jamun opt test mochitest-4
+ Ubuntu VM 12.04 x64 Mulet jamun opt test mochitest-5
+ Ubuntu VM 12.04 x64 Mulet mozilla-aurora opt test mochitest-1
+ Ubuntu VM 12.04 x64 Mulet mozilla-aurora opt test mochitest-2
+ Ubuntu VM 12.04 x64 Mulet mozilla-aurora opt test mochitest-3
+ Ubuntu VM 12.04 x64 Mulet mozilla-aurora opt test mochitest-4
+ Ubuntu VM 12.04 x64 Mulet mozilla-aurora opt test mochitest-5
+ Ubuntu VM 12.04 x64 Mulet mozilla-central opt test mochitest-1
+ Ubuntu VM 12.04 x64 Mulet mozilla-central opt test mochitest-2
+ Ubuntu VM 12.04 x64 Mulet mozilla-central opt test mochitest-3
+ Ubuntu VM 12.04 x64 Mulet mozilla-central opt test mochitest-4
+ Ubuntu VM 12.04 x64 Mulet mozilla-central opt test mochitest-5
+ Ubuntu VM 12.04 x64 Mulet mozilla-inbound opt test mochitest-1
+ Ubuntu VM 12.04 x64 Mulet mozilla-inbound opt test mochitest-2
+ Ubuntu VM 12.04 x64 Mulet mozilla-inbound opt test mochitest-3
+ Ubuntu VM 12.04 x64 Mulet mozilla-inbound opt test mochitest-4
+ Ubuntu VM 12.04 x64 Mulet mozilla-inbound opt test mochitest-5
+ Ubuntu VM 12.04 x64 Mulet pine opt test mochitest-1
+ Ubuntu VM 12.04 x64 Mulet pine opt test mochitest-2
+ Ubuntu VM 12.04 x64 Mulet pine opt test mochitest-3
+ Ubuntu VM 12.04 x64 Mulet pine opt test mochitest-4
+ Ubuntu VM 12.04 x64 Mulet pine opt test mochitest-5
+ Ubuntu VM 12.04 x64 Mulet try opt test mochitest-1
+ Ubuntu VM 12.04 x64 Mulet try opt test mochitest-2
+ Ubuntu VM 12.04 x64 Mulet try opt test mochitest-3
+ Ubuntu VM 12.04 x64 Mulet try opt test mochitest-4
+ Ubuntu VM 12.04 x64 Mulet try opt test mochitest-5
Builders removed
- b2g_ubuntu64_vm ash opt test mochitest-mulet-plain-1
- b2g_ubuntu64_vm ash opt test mochitest-mulet-plain-2
- b2g_ubuntu64_vm ash opt test mochitest-mulet-plain-3
- b2g_ubuntu64_vm ash opt test mochitest-mulet-plain-4
- b2g_ubuntu64_vm ash opt test mochitest-mulet-plain-5
- b2g_ubuntu64_vm b2g-inbound opt test mochitest-mulet-plain-1
- b2g_ubuntu64_vm b2g-inbound opt test mochitest-mulet-plain-2
- b2g_ubuntu64_vm b2g-inbound opt test mochitest-mulet-plain-3
- b2g_ubuntu64_vm b2g-inbound opt test mochitest-mulet-plain-4
- b2g_ubuntu64_vm b2g-inbound opt test mochitest-mulet-plain-5
- b2g_ubuntu64_vm cedar opt test mochitest-mulet-plain-1
- b2g_ubuntu64_vm cedar opt test mochitest-mulet-plain-2
- b2g_ubuntu64_vm cedar opt test mochitest-mulet-plain-3
- b2g_ubuntu64_vm cedar opt test mochitest-mulet-plain-4
- b2g_ubuntu64_vm cedar opt test mochitest-mulet-plain-5
- b2g_ubuntu64_vm cypress opt test mochitest-mulet-plain-1
- b2g_ubuntu64_vm cypress opt test mochitest-mulet-plain-2
- b2g_ubuntu64_vm cypress opt test mochitest-mulet-plain-3
- b2g_ubuntu64_vm cypress opt test mochitest-mulet-plain-4
- b2g_ubuntu64_vm cypress opt test mochitest-mulet-plain-5
- b2g_ubuntu64_vm fx-team opt test mochitest-mulet-plain-1
- b2g_ubuntu64_vm fx-team opt test mochitest-mulet-plain-2
- b2g_ubuntu64_vm fx-team opt test mochitest-mulet-plain-3
- b2g_ubuntu64_vm fx-team opt test mochitest-mulet-plain-4
- b2g_ubuntu64_vm fx-team opt test mochitest-mulet-plain-5
- b2g_ubuntu64_vm graphics opt test mochitest-mulet-plain-1
- b2g_ubuntu64_vm graphics opt test mochitest-mulet-plain-2
- b2g_ubuntu64_vm graphics opt test mochitest-mulet-plain-3
- b2g_ubuntu64_vm graphics opt test mochitest-mulet-plain-4
- b2g_ubuntu64_vm graphics opt test mochitest-mulet-plain-5
- b2g_ubuntu64_vm jamun opt test mochitest-mulet-plain-1
- b2g_ubuntu64_vm jamun opt test mochitest-mulet-plain-2
- b2g_ubuntu64_vm jamun opt test mochitest-mulet-plain-3
- b2g_ubuntu64_vm jamun opt test mochitest-mulet-plain-4
- b2g_ubuntu64_vm jamun opt test mochitest-mulet-plain-5
- b2g_ubuntu64_vm mozilla-aurora opt test mochitest-mulet-plain-1
- b2g_ubuntu64_vm mozilla-aurora opt test mochitest-mulet-plain-2
- b2g_ubuntu64_vm mozilla-aurora opt test mochitest-mulet-plain-3
- b2g_ubuntu64_vm mozilla-aurora opt test mochitest-mulet-plain-4
- b2g_ubuntu64_vm mozilla-aurora opt test mochitest-mulet-plain-5
- b2g_ubuntu64_vm mozilla-central opt test mochitest-mulet-plain-1
- b2g_ubuntu64_vm mozilla-central opt test mochitest-mulet-plain-2
- b2g_ubuntu64_vm mozilla-central opt test mochitest-mulet-plain-3
- b2g_ubuntu64_vm mozilla-central opt test mochitest-mulet-plain-4
- b2g_ubuntu64_vm mozilla-central opt test mochitest-mulet-plain-5
- b2g_ubuntu64_vm mozilla-inbound opt test mochitest-mulet-plain-1
- b2g_ubuntu64_vm mozilla-inbound opt test mochitest-mulet-plain-2
- b2g_ubuntu64_vm mozilla-inbound opt test mochitest-mulet-plain-3
- b2g_ubuntu64_vm mozilla-inbound opt test mochitest-mulet-plain-4
- b2g_ubuntu64_vm mozilla-inbound opt test mochitest-mulet-plain-5
- b2g_ubuntu64_vm pine opt test mochitest-mulet-plain-1
- b2g_ubuntu64_vm pine opt test mochitest-mulet-plain-2
- b2g_ubuntu64_vm pine opt test mochitest-mulet-plain-3
- b2g_ubuntu64_vm pine opt test mochitest-mulet-plain-4
- b2g_ubuntu64_vm pine opt test mochitest-mulet-plain-5
- b2g_ubuntu64_vm try opt test mochitest-mulet-plain-1
- b2g_ubuntu64_vm try opt test mochitest-mulet-plain-2
- b2g_ubuntu64_vm try opt test mochitest-mulet-plain-3
- b2g_ubuntu64_vm try opt test mochitest-mulet-plain-4
- b2g_ubuntu64_vm try opt test mochitest-mulet-plain-5

Comment 57

[Jordan Lund (:jlund)]

Comment on attachment 8491647 [details] [diff] [review]
Remove mulet from suite names,

Review of attachment 8491647 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm with one fix

::: mozilla-tests/b2g_config.py
@@ +114,4 @@
>  
>  PLATFORMS['linux64-mulet']['slave_platforms'] = ['ubuntu64_vm-mulet']
>  PLATFORMS['linux64-mulet']['env_name'] = 'linux-perf'
> +PLATFORMS['linux64-mulet']['ubuntu64_vm-mulet'] = {'name': 'Ubuntu VM 12.04 x64 Mulet'}

I think we need the same for osx as we do have tests running there: http://mxr.mozilla.org/build/source/buildbot-configs/mozilla-tests/b2g_config.py#2156

otherwise bbot complains about dupe builder names

I suppose like linux, we can change it to something like it was before: http://hg.mozilla.org/build/buildbot-configs/rev/4d2ec7a85bd0#l3.39

doing so at least passes for me so feel free to r+ if u do that one change

Comment 58

[Jonathan Griffin (:jgriffin)]

Attachment: Remove mulet from suite names,

With suggested fix for OSX name

Comment 59

[Jordan Lund (:jlund)]

Comment on attachment 8492251 [details] [diff] [review]
Remove mulet from suite names,

Review of attachment 8492251 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm :)

Comment 60

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8492251 [details] [diff] [review]
Remove mulet from suite names,

https://hg.mozilla.org/build/buildbot-configs/rev/df3039f78a1f

Comment 61

[Simone Bruno [:simone]]

In prod with reconfig on 2014-09-22 08:20 PT

Comment 62

[Jonathan Griffin (:jgriffin)]

I declare this bug vanquished!