997096 - Start doing mulet builds on Fig

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Alexandre Poirot [:ochameau]]

Once bug 961745 lands, we can start building Firefox Mulet out of mozilla central.
Firefox Mulet is a special build of Firefox with various b2g specifics.
It is meant to replace b2g desktop.
If bug 961745's patch doesn't change much, you just have to build the firefox target, with --enable-mulet.

I don't really know how release automation works,
but a good first step would be to allow us to build the Mulet on try.
Just building it.
Then we can iterate on it by enabling each test suite, still only on try.
We would start with xpcshell, then mochitest-plain.
That's where it may be complicated. Even if that's mostly Firefox,
many tests will fail because of the b2g specifics, so, we would like
to run only the xpcshell and mochitest-plain tests that are currently run on b2g desktop.

I would like to use this bug as a meta bug regarding the mulet and automation. We would discuss in this bug about what action items we can start working on.

First iteration:

  Expose a "mulet (linux64)" platform target for try builds

Second:

  Start running b2g desktop xpcshell tests on this new target

Third:

  Start running b2g desktop mochitest-plain tests on this new target

Comment 1

[Alexandre Poirot [:ochameau]]

Here is a try run of the Mulet:
  https://tbpl.mozilla.org/?tree=Try&rev=b1cbfd59cd98

Which comes with very good results!
The list of failing tests looks quite small given how special Firefox Mulet is...
We shouldn't run mochitest-browser/mochitest-e10s as I don't think we really care about Firefox features that much. (I'm not sure it is really worth spending time in ensuring that browser test pass)
Similar thing may apply to mochitest-chrome. Even if we would like to see mochitest-chrome running on b2g, as we don't run them on device/emulator, I'm not sure it is really worth running them on desktop.

But the good news is that all xpcshell tests just work!
Then the only thing we would need is either:
 - fix the failing mochitest-plain,
 - somehow disable them for the mulet,
    How does mochitest.ini filter works?
    Can we easily add a "skip-if = mulet"?
    (the best way to identify a mulet is to check for MOZ_MULET env/config variable),
 - only run current b2g desktop whitelist.

Gregor, can you find whoever can help us on this (expose mulet target on try and filter mochitests)?

Comment 2

[Gregor Wagner [:gwagner]]

(In reply to Alexandre Poirot (:ochameau) from comment #1)
> Here is a try run of the Mulet:
>   https://tbpl.mozilla.org/?tree=Try&rev=b1cbfd59cd98
> 
> Which comes with very good results!
> The list of failing tests looks quite small given how special Firefox Mulet
> is...
> We shouldn't run mochitest-browser/mochitest-e10s as I don't think we really
> care about Firefox features that much. (I'm not sure it is really worth
> spending time in ensuring that browser test pass)
> Similar thing may apply to mochitest-chrome. Even if we would like to see
> mochitest-chrome running on b2g, as we don't run them on device/emulator,
> I'm not sure it is really worth running them on desktop.
> 
> But the good news is that all xpcshell tests just work!
> Then the only thing we would need is either:
>  - fix the failing mochitest-plain,
>  - somehow disable them for the mulet,
>     How does mochitest.ini filter works?
>     Can we easily add a "skip-if = mulet"?
>     (the best way to identify a mulet is to check for MOZ_MULET env/config
> variable),
>  - only run current b2g desktop whitelist.
> 
> Gregor, can you find whoever can help us on this (expose mulet target on try
> and filter mochitests)?

Thats great!
I think jgriffin is the best person to ask here. As soon as we see that 961745 is ready to land we should start the automation process.

Comment 3

[Andrew Halberstadt [:ahal]]

(In reply to Alexandre Poirot (:ochameau) from comment #1)
> But the good news is that all xpcshell tests just work!
> Then the only thing we would need is either:
>  - fix the failing mochitest-plain,
>  - somehow disable them for the mulet,
>     How does mochitest.ini filter works?
>     Can we easily add a "skip-if = mulet"?
>     (the best way to identify a mulet is to check for MOZ_MULET env/config
> variable),
>  - only run current b2g desktop whitelist.

The variables the manifests know about are written by the build here:
http://mxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/mozinfo.py

Though if this is going to replace b2g desktop, can't we just use 'buildapp == b2g && toolkit != gonk' again? Or do we plan to run mulet alongside b2g desktop?

Comment 4

[Andrew Halberstadt [:ahal]]

Also note, I'd guess that try run is running the same set of mochitest-plain that b2g_desktop is currently running, since they probably hit the same skip-if conditions.

Comment 5

[Jonathan Griffin (:jgriffin)]

They're actually running the same tests as regular desktop Firefox, not the B2G desktop build.  I'm not sure whether this is what we want, or not.

Comment 6

[Jonathan Griffin (:jgriffin)]

And yes, as soon as bug 961745 lands, I can start the process of having builds and tests added to buildbot.

Comment 7

[Andrew Halberstadt [:ahal]]

Are you sure? They are chunked the same as desktop Firefox, but I don't think they are running the same set of tests. Right now we skip b2g_desktop tests with:
skip-if = buildapp == 'b2g' && toolkit != 'gonk'

Aiui, that should catch mulet as well.

Comment 8

[Jonathan Griffin (:jgriffin)]

Not positive, but the aggregate number of tests seems to be much larger than what we run for b2g-desktop.

Comment 9

[Alexandre Poirot [:ochameau]]

(In reply to Jonathan Griffin (:jgriffin) from comment #5)
> They're actually running the same tests as regular desktop Firefox, not the
> B2G desktop build.  I'm not sure whether this is what we want, or not.

Yes we are running firefox test suites, as the Mulet is closer to Firefox than b2g-desktop.
If the build system patch doesn't change, to build a Mulet, you just need to build firefox with --enable-mulet.

I think that's a good thing. We are running tests through the same codepath than Firefox desktop.
I imagine it will be easier to maintain! But we have to find a way to filter tests.


(In reply to Andrew Halberstadt [:ahal] from comment #3)
> Though if this is going to replace b2g desktop, can't we just use 'buildapp
> == b2g && toolkit != gonk' again? Or do we plan to run mulet alongside b2g
> desktop?

We will most likely have them running alonside, until the Mulet is mature enough to replace b2g desktop.
So we no longer have a white/black-list for b2g, right?
We just run all and rely on manual filter in each mochitest.ini?
If that's the case, we can get the correct filtering quite easily by tuning mozinfo.py to make it so that buildappname = appname = b2g? (and/or MOZ_APP_NAME = MOZ_BUILD_APP = b2g)

Comment 10

[Alexandre Poirot [:ochameau]]

I hacked mozinfo.py to expose "mulet" variable. I'm not sure that the best way to go, but that allows to see how many tests we have to disable just for the mulet and it looks like we don't have to disable many of them:
All (but e10s) mochitests green:
  https://tbpl.mozilla.org/?tree=Try&rev=50519e76486e
Disabled tests:
  https://github.com/ochameau/mozilla-central/commit/0edbde06c03e0e7e09b5fbf219440393524752db

Comment 11

[Alexandre Poirot [:ochameau]]

Bug 961745 is about to reach master.
It means that we can start building mulets. The build system patch changed quite a bit, so here is the updated instruction do build a mulet:

You just need a mozconfig with:

  ac_add_options --enable-application=b2g/dev

You may pass any other flag that Firefox is used to have, but nothing else is required at this stage.
Once we get it build with just that, we can iterate and start looking at shipping a gaia profile, like what we are doing for b2g desktop. But let's just build it as-is in a first round.

Comment 12

[Gregor Wagner [:gwagner]]

(In reply to Jonathan Griffin (:jgriffin) (away until May 11) from comment #6)
> And yes, as soon as bug 961745 lands, I can start the process of having
> builds and tests added to buildbot.

\o/

jmaher, armen, can we get some help here? jgriffin mentioned that you can help out when he is away.

Comment 13

[Armen [:armenzg]]

I will see what I can do.

Comment 14

[Alexandre Poirot [:ochameau]]

Here is a try run with various patches that still need to land on central,
but proves that we can get mochitests and xpcshell up and running green against b2g/dev!
  https://tbpl.mozilla.org/?tree=Try&rev=b620c9971a60

Comment 15

[Alexandre Poirot [:ochameau]]

Just to be clear, as requested on first comment, we don't need to build that on every branch right away.
What would help is having it built only on try and allow running xpcshell, mochitest-plain, mochitest-chrome and may be also mochitest-browser. I don't know what kind of options you need to specify; but regarding test runners, Mulet should work as firefox. Test runners should be run as if it was Firefox.
Also, to save resources, you can prevent building it if not explicitely asked, or if that's not something we support, we should keep it hidden as it is not ready for prime time (yet).

Comment 16

[Armen [:armenzg]]

If I can't find anyone from Releng to grab it I will try to help next week when I'm out of buildduty.

Comment 17

[Armen [:armenzg]]

Hi catlee,
I've reviewed the different comments on this bug and we're ready to add a new type of build which uses a new mozconfig with this extra line:
  ac_add_options --enable-application=b2g/dev

FYI: Mulet behaves more like Firefox desktop than B2g desktop.

The scope right now is to enable the build on try plus few suites (xpcshell, mochitest-plain, mochitest-chrome and may be also mochitest-browse). 

Is there anyone that has time to look into it this week?
If not I hope to be able to make time next week for this.

Comment 18

[Chris AtLee [:catlee]]

We probably won't have time this week to get to this. Who is the intended audience for these builds?

Comment 19

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

(In reply to Chris AtLee [:catlee] from comment #18)
> Who is the intended audience for these builds?

The intended audience for these builds is all of FirefoxOS development. There will be a few teams that won't use this, like teams that need to work directly against hardware that we're not emulating (like telephony/sms), however most frontend and gecko teams will be using this.

We're also planning on migrating 3rd party app developers to using this.

See https://groups.google.com/d/msg/mozilla.dev.b2g/2WUg10993OM/VSceN9i2w2EJ

Comment 20

[Jonathan Griffin (:jgriffin)]

i.e. we will eventually deprecate B2G desktop builds in favor of Mulet

Comment 21

[Jonathan Griffin (:jgriffin)]

We've decided cedar has diverged too far from m-c for this work; let's schedule these on fig and try (we'll hide them initially on try).

Comment 22

[Armen [:armenzg]]

Attachment: Add in-tree mulet mozconfig

I wonder if it should be named "nightly-mulet" or if it should just "mulet".

Is it the right place to have it under "browser/config/mozconfigs"?

Comment 23

[[e-mail] (:bhearsum)]

Comment on attachment 8425642 [details] [diff] [review]
Add in-tree mulet mozconfig

Review of attachment 8425642 [details] [diff] [review]:
-----------------------------------------------------------------

I'm only reviewing this for organization - I'll leave it to someone else to review it for correctness.

I think this is the right place to put, we have no precedent for putting anything except target platforms directly in "mozconfigs".

"mulet" is probably a better name. There's nothing nightly-specific about this config. r=me with that change.

Comment 24

[Michael Shal [:mshal]]

Comment on attachment 8425642 [details] [diff] [review]
Add in-tree mulet mozconfig

Isn't it nightly specific since it includes linux64/nightly?

Also I will defer to the mulet folks on where they want it, but IMO it makes more sense to put this under b2g/dev/ instead of browser/. It looks like for all the other applications, we have config/mozconfigs/ at the same level as the corresponding app.mozbuild file. Eg:

b2g/app.mozbuild
b2g/config/mozconfigs/

mobile/android/app.mozbuild
mobile/android/config/mozconfigs/

browser/app.mozbuild
browser/config/mozconfigs/

xulrunner/app.mozbuild
xulrunner/config/mozconfigs/

Since mulet has b2g/dev/app.mozbuild, I think it should be b2g/dev/config/mozconfigs/...

Comment 25

[Alexandre Poirot [:ochameau]]

Comment on attachment 8425642 [details] [diff] [review]
Add in-tree mulet mozconfig

Review of attachment 8425642 [details] [diff] [review]:
-----------------------------------------------------------------

(In reply to Michael Shal [:mshal] from comment #24)
> Comment on attachment 8425642 [details] [diff] [review]
> Add in-tree mulet mozconfig
> 
> Isn't it nightly specific since it includes linux64/nightly?

+1, I would imagine this mozconfig is to build the mulet against mozilla central tip,
and in future we may have a mozconfig for beta and release?

> Since mulet has b2g/dev/app.mozbuild, I think it should be
> b2g/dev/config/mozconfigs/...

+1

Comment 26

[Armen [:armenzg]]

Attachment: Add in-tree mulet mozconfig

Comment 27

[Michael Shal [:mshal]]

Comment on attachment 8425716 [details] [diff] [review]
Add in-tree mulet mozconfig

Looks good!

Comment 28

[Armen [:armenzg]]

Which other builds besides the mulet builds should we enable on Fig?

Comment 29

[Jonathan Griffin (:jgriffin)]

I think we can get away with having only Mulet builds on fig; we can use m-c and cedar for comparison with other builds, as necessary.

Comment 30

[Armen [:armenzg]]

Attachment: [wip] add mulet builds and tests to fig

catlee: how is this shaping?

Comment 31

[Armen [:armenzg]]

Is the packaging going to look like a Firefox build?

I've pushed to try to see where we currently are:
https://tbpl.mozilla.org/?tree=Try&rev=e8b436c152a8

Comment 32

[Alexandre Poirot [:ochameau]]

(In reply to Armen Zambrano [:armenzg] (EDT/UTC-4) from comment #31)
> Is the packaging going to look like a Firefox build?

Yes. You should find a firefox-VERSION-... tarball/zip after make package.

Comment 33

[Chris AtLee [:catlee]]

Comment on attachment 8425807 [details] [diff] [review]
[wip] add mulet builds and tests to fig

Review of attachment 8425807 [details] [diff] [review]:
-----------------------------------------------------------------

looks like you're on the right track!

::: mozilla/config.py
@@ +651,5 @@
> +        'linux64-mulet': {
> +            'mozharness_config': {
> +                'script_name': 'scripts/fx_desktop_build.py',
> +                'extra_args': [
> +                    '--config', 'builds/releng_base_linux_64_builds.py',

does this need --custom-build-variant-cfg mulet here? I see that in the asan configs.

@@ +659,5 @@
> +            },
> +
> +            'unittest_platform': 'linux64-mulet',
> +            'stage_product': 'firefox',
> +            'stage_platform': 'mulet',

linux64-mulet maybe?

Comment 34

[Armen [:armenzg]]

Attachment: Add mulet to fig and try

Comment 35

[Armen [:armenzg]]

(In reply to Chris AtLee [:catlee] from comment #33)
> Comment on attachment 8425807 [details] [diff] [review]
> [wip] add mulet builds and tests to fig
> 
> Review of attachment 8425807 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> looks like you're on the right track!
> 
> ::: mozilla/config.py
> @@ +651,5 @@
> > +        'linux64-mulet': {
> > +            'mozharness_config': {
> > +                'script_name': 'scripts/fx_desktop_build.py',
> > +                'extra_args': [
> > +                    '--config', 'builds/releng_base_linux_64_builds.py',
> 
> does this need --custom-build-variant-cfg mulet here? I see that in the asan
> configs.
> 
It's the mozconfig that makes it different. No mozharness changes afaik.

Comment 36

[Armen [:armenzg]]

Alexandre, would you please review the results in here? Thanks!
https://tbpl.mozilla.org/?tree=Try&rev=e8b436c152a8

Comment 37

[Jonathan Griffin (:jgriffin)]

It's interesting that debug tests are greener than opt...we usually see the reverse!

When we actually schedule these tests, I imagine we should skip M-e10s (since all tests are OOP by default on Mulet, unless I'm mistaken).

Comment 38

[Alexandre Poirot [:ochameau]]

This is really weird, I downloaded the opt tarball and this looks like a regular firefox, even if I can see --enable-application=b2g/dev in about:buildconfig, I tried to search for all the mulet specifics and found nothing.
But what is really suspicious is that the test failures we see on tbpl for opt are the expected mulet's one! (I have a patch with a new mulet specific blacklist for xpcshell and mochitests)
The only thing I can think of is that "make package" strip all mulet specifics.
I'll look into this, if that's it, the opt build is just fine, it will be matter of landing a fix for make package.

But I think there is something wrong with mozconfig on debug build. I also downloaded the mulet and I don't even see --enable-application=b2g/dev in about:buildconfig, and given that all tests are green, I think debug is really not a mulet and just a regular firefox!

Comment 39

[Armen [:armenzg]]

My apologies.
The debug build was triggered by mistake based on a normal Firefox build.

Comment 40

[Armen [:armenzg]]

Comment on attachment 8426520 [details] [diff] [review]
Add mulet to fig and try

https://hg.mozilla.org/build/buildbot-configs/rev/b339f91b754c

Comment 41

[Armen [:armenzg]]

Comment on attachment 8425716 [details] [diff] [review]
Add in-tree mulet mozconfig

https://hg.mozilla.org/integration/mozilla-inbound/rev/b873c10c208d

Comment 42

[Armen [:armenzg]]

Once this code goes into the buildbot masters we should be able to trigger Linux 64-bit mulet jobs on Fig and Try (assuming the m-i code gets into the pushes of those trees).

ochameau: have you had a chance to file a bug for the issues you saw in comment 38?

Comment 43

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/b873c10c208d

Comment 44

[Alexandre Poirot [:ochameau]]

I opened bug 1015949 for the packaging issue.
Otherwise, I was wondering what string should I used to build b2g/dev on Try?
Also, do you know who is maintaining http://trychooser.pub.build.mozilla.org/ so that we can get it updated?

Comment 45

[Armen [:armenzg]]

You should use 'linux64-mulet' as the platform.
However, the changes are not yet live in production.

The TryChooser code lives in here:
http://hg.mozilla.org/build/tools/file/tip/trychooser

Comment 46

[Chris Cooper [:coop] (he/him)]

Merged and deployed to production.

Comment 47

[Armen [:armenzg]]

I've pushed m-i to fig so we can gets builds:
https://tbpl.mozilla.org/?tree=Fig

Comment 48

[Armen [:armenzg]]

Unfortunately, I made a lot of assumptions (like we had already switch to mozharness for builds) and I assumed I would only need minor tweaks.

I have setup a local build master and will be testing until I get this right.

Comment 49

[Armen [:armenzg]]

Which builds besides Linux 64 are we going to have?
Are we also going to do debug? or are we fine with opt?

I assume we won't be running nightly builds of this.

Comment 50

[Armen [:armenzg]]

I'm getting this build issue:

updating cache ../.././config.cache
creating ./config.status
Reticulating splines...
Traceback (most recent call last):
  File "./config.status", line 979, in <module>
    config_status(**args)
  File "/builds/slave/fig-l64-mulet-0000000000000000/build/python/mozbuild/mozbuild/config_status.py", line 148, in config_status
    summary = the_backend.consume(definitions)
  File "/builds/slave/fig-l64-mulet-0000000000000000/build/python/mozbuild/mozbuild/backend/base.py", line 186, in consume
    for obj in objs:
  File "/builds/slave/fig-l64-mulet-0000000000000000/build/python/mozbuild/mozbuild/frontend/emitter.py", line 105, in emit
    for out in output:
  File "/builds/slave/fig-l64-mulet-0000000000000000/build/python/mozbuild/mozbuild/frontend/reader.py", line 726, in read_mozbuild
    raise bre
mozbuild.frontend.reader.BuildReaderError: ==============================
ERROR PROCESSING MOZBUILD FILE
==============================

The error occurred while processing the following file:

    /builds/slave/fig-l64-mulet-0000000000000000/build/moz.build

The underlying problem is we referenced a path that does not exist. That path is:

    /builds/slave/fig-l64-mulet-0000000000000000/build/b2g/dev/app/moz.build

Either create the file if it needs to exist or do not reference it.

Comment 51

[Armen [:armenzg]]

I've pushed to try to see if we hit the same issue:
https://tbpl.mozilla.org/?tree=Try&rev=ca8025b48d5d

Comment 52

[Armen [:armenzg]]

Attachment: switch mulet builds to non-mozharness, change builder name plus disable from release branches

Comment 53

[Jonathan Griffin (:jgriffin)]

(In reply to Armen Zambrano [:armenzg] (EDT/UTC-4) from comment #49)
> Which builds besides Linux 64 are we going to have?
> Are we also going to do debug? or are we fine with opt?
> 
> I assume we won't be running nightly builds of this.

We'll want debug as well as opt, and we'll want the same platforms that we're currently building b2gdesktop against...OSX, linux32/64, and win32.

We won't need nightlies for now, but that could change later.

Comment 54

[Jonathan Griffin (:jgriffin)]

It does seem like moz.build is missing.  Any ideas Alexandre?

Comment 55

[Chris AtLee [:catlee]]

Comment on attachment 8430231 [details] [diff] [review]
switch mulet builds to non-mozharness, change builder name plus disable from release branches

Review of attachment 8430231 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/config.py
@@ -64,5 @@
>          'linux64-debug': {},
>          'linux64-asan': {},
>          'linux64-asan-debug': {},
>          'linux64-st-an-debug': {},
> -        'linux64-mulet': {},

isn't this still needed?

@@ -2278,5 @@
> -for name, branch in items_at_least(BRANCHES, 'gecko_version', 32):
> -    if name in ('try', 'fig'):
> -        continue
> -    if 'linux64-mulet' in branch['platforms']:
> -        del branch['platforms']['linux64-mulet']

you're ready to enable on all branches?

Comment 56

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Michael, is there any help you can provide here? See comment 50 to 54.

Comment 57

[Armen [:armenzg]]

(In reply to Chris AtLee [:catlee] from comment #55)
> Comment on attachment 8430231 [details] [diff] [review]
> switch mulet builds to non-mozharness, change builder name plus disable from
> release branches
> 
> Review of attachment 8430231 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: mozilla/config.py
> @@ -64,5 @@
> >          'linux64-debug': {},
> >          'linux64-asan': {},
> >          'linux64-asan-debug': {},
> >          'linux64-st-an-debug': {},
> > -        'linux64-mulet': {},
> 
> isn't this still needed?

That line is to enable across the board.

> 
> @@ -2278,5 @@
> > -for name, branch in items_at_least(BRANCHES, 'gecko_version', 32):
> > -    if name in ('try', 'fig'):
> > -        continue
> > -    if 'linux64-mulet' in branch['platforms']:
> > -        del branch['platforms']['linux64-mulet']
> 
> you're ready to enable on all branches?

I won't need this block because I'm removing the previous line.

Comment 58

[Michael Shal [:mshal]]

(In reply to Jonas Sicking (:sicking) from comment #56)
> Michael, is there any help you can provide here? See comment 50 to 54.

Looks like I missed that in reviewing bug 1015949 where it adds b2g/dev/app as a tier dir. We can probably just copy a blank moz.build file (like b2g/installer/moz.build) into b2g/dev/app.

Comment 59

[Jonathan Griffin (:jgriffin)]

try build with empty moz.build in b2g/dev/app:  https://tbpl.mozilla.org/?tree=Try&rev=15331561588b

Comment 60

[Armen [:armenzg]]

Catlee, did comment 57 answer your questions?

Comment 61

[Alexandre Poirot [:ochameau]]

Oops, I was out of town. I submitted a patch for the missing moz.build in bug 1015949.

Comment 62

[Armen [:armenzg]]

(In reply to Armen Zambrano [:armenzg] (EDT/UTC-4) from comment #60)
> Catlee, did comment 57 answer your questions?

Adding NI. I think the answer should satisfy the review concerns.

Comment 63

[Aki Sasaki (not active)]

I see an r+

Comment 64

[Armen [:armenzg]]

Comment on attachment 8430231 [details] [diff] [review]
switch mulet builds to non-mozharness, change builder name plus disable from release branches

https://hg.mozilla.org/build/buildbot-configs/rev/b11352bd5d07

Comment 65

[Simone Bruno [:simone]]

In production with reconfig on 2014-06-04 02:19 PT

Comment 66

[Armen [:armenzg]]

Some configure issues:
https://tbpl.mozilla.org/php/getParsedLog.php?id=41067418&tree=Fig&full=1#error0

I will look into it later today.

checking whether the C compiler (/builds/slave/fig-l64-mulet-0000000000000000/build/gcc/bin/gcc  ) works... no
configure: error: installation or configuration problem: C compiler cannot create executables.
------ config.log ------
This file contains any messages produced by compilers while
running configure, to aid debugging if configure makes a mistake.

Comment 67

[Armen [:armenzg]]

ochameau, can you please check that you can still build mulet?

I've loaned a machine for staging and I will be working to make it work next week.
I want to make sure that we have not been regressed by other changes. Thanks.

Comment 68

[Alexandre Poirot [:ochameau]]

Yes nothing regressed. mach build, package, run. Everything works.
There is just one thing. If you want to play with it, see gaia running, you have to manually open chrome://b2g/content/shell.html in a tab.

Comment 69

[Aki Sasaki (not active)]

If we can pull gaia from git rather than hg for Mulet, we can get rid of a whole host of issues syncing gaia to hg.m.o and then translating between the two revisions (e.g. bug 1022864).
This is blocked on getting git installed on our windows builders, however.

Comment 70

[Armen [:armenzg]]

Attachment: add tooltool configuration

Comment 71

[Armen [:armenzg]]

Comment on attachment 8437188 [details] [diff] [review]
add tooltool configuration

https://hg.mozilla.org/build/buildbot-configs/rev/e371c888a099

Comment 72

[Simone Bruno [:simone]]

Live with reconfig on 2014-06-11 08:47 PT

Comment 73

[Armen [:armenzg]]

Triggered new build on Fig:
https://tbpl.mozilla.org/?tree=Fig

Comment 74

[Armen [:armenzg]]

Attachment: add sendchanges

I forgot that talos_masters are needed to activate sendchanges.

Diff of master dumps:
--- old_dump.txt	2014-06-11 17:08:57.325008345 -0400
+++ new_dump.txt	2014-06-11 16:39:17.929087679 -0400
@@ -6689,0 +6690 @@ Linux Mulet x86-64 fig build NightlyBuil
+    SendChangeStep {'branch': 'fig-linux64-mulet-talos', 'command': None, 'comments': <WithProperties "%(comments:-)s">, 'description': None, 'descriptionDone': None, 'env': {'CCACHE_COMPRESS': '1',  'CCACHE_DIR': '/builds/ccache',  'CCACHE_UMASK': '002',  'DISPLAY': ':2',  'HG_SHARE_BASE_DIR': '/builds/hg-shared',  'LC_ALL': 'C',  'MOZ_AUTOMATION': '1',  'MOZ_CRASHREPORTER_NO_REPORT': '1',  'MOZ_OBJDIR': 'obj-firefox',  'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin',  'TINDERBOX_OUTPUT': '1',  'TOOLTOOL_CACHE': '/builds/tooltool_cache',  'TOOLTOOL_HOME': '/builds'}, 'files': [<WithProperties "%(packageUrl)s">], 'log_eval_func': None, 'logfiles': {}, 'master': 'buildbot-master81.build.mozilla.org:9301', 'name': 'sendchange_buildbot-master81.build.mozilla.org:9301', 'retries': 5, 'revision': <WithProperties "%(got_revision)s">, 'sendchange_props': {'buildid': <WithProperties "%(buildid:-)s">,  'builduid': <WithProperties "%(builduid:-)s">,  'pgo_build': False}, 'timeout': 1800, 'usePTY': 'slave-config', 'user': 'sendchange', 'warnOnFailure': True, 'workdir': None} {}

Comment 75

[Armen [:armenzg]]

FYI the builds can now run on Fig.
We now need to add the tests.

Comment 76

[Aki Sasaki (not active)]

Comment on attachment 8438699 [details] [diff] [review]
add sendchanges

Uh, do you really want to run talos against this?
I'd be surprised.
I think you want unittest masters, which should already be set, and packageTests.  Am I wrong?
Clearing review for now.

Comment 77

[Jonathan Griffin (:jgriffin)]

We don't need talos on these builds.

Comment 78

[Armen [:armenzg]]

Attachment: add unit test sendchanges

I think everything I said or did today came out wrong.

This activates the unit tests sendchanges as well as make package-tests:
http://hg.mozilla.org/build/buildbotcustom/file/default/process/factory.py#l2050

--- old_dump.txt	2014-06-11 17:08:57.325008345 -0400
+++ new_dump.txt	2014-06-11 18:51:38.303819795 -0400
@@ -6679,0 +6680,2 @@ Linux Mulet x86-64 fig build NightlyBuil
+    MockCommand {'command': ['make', 'buildsymbols'], 'description': None, 'descriptionDone': None, 'env': {'CCACHE_COMPRESS': '1',  'CCACHE_DIR': '/builds/ccache',  'CCACHE_UMASK': '002',  'DISPLAY': ':2',  'HG_SHARE_BASE_DIR': '/builds/hg-shared',  'LC_ALL': 'C',  'MOZ_AUTOMATION': '1',  'MOZ_CRASHREPORTER_NO_REPORT': '1',  'MOZ_OBJDIR': 'obj-firefox',  'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin',  'TINDERBOX_OUTPUT': '1',  'TOOLTOOL_CACHE': '/builds/tooltool_cache',  'TOOLTOOL_HOME': '/builds'}, 'haltOnFailure': True, 'log_eval_func': None, 'logfiles': {}, 'mock': True, 'mock_args': ['--unpriv'], 'mock_login': 'mock_mozilla', 'mock_workdir_mutator': <function <lambda>>, 'mock_workdir_prefix': None, 'name': 'make_buildsymbols', 'target': 'mozilla-centos6-x86_64', 'timeout': 3600, 'usePTY': 'slave-config', 'workdir': <WithProperties "%(basedir)s/build/obj-firefox">} {}
+    MockCommand {'command': ['make', 'package-tests'], 'description': None, 'descriptionDone': None, 'env': {'CCACHE_COMPRESS': '1',  'CCACHE_DIR': '/builds/ccache',  'CCACHE_UMASK': '002',  'DISPLAY': ':2',  'HG_SHARE_BASE_DIR': '/builds/hg-shared',  'LC_ALL': 'C',  'MOZ_AUTOMATION': '1',  'MOZ_CRASHREPORTER_NO_REPORT': '1',  'MOZ_OBJDIR': 'obj-firefox',  'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin',  'TINDERBOX_OUTPUT': '1',  'TOOLTOOL_CACHE': '/builds/tooltool_cache',  'TOOLTOOL_HOME': '/builds'}, 'haltOnFailure': True, 'log_eval_func': None, 'logfiles': {}, 'mock': True, 'mock_args': ['--unpriv'], 'mock_login': 'mock_mozilla', 'mock_workdir_mutator': <function <lambda>>, 'mock_workdir_prefix': None, 'name': 'make_pkg_tests', 'target': 'mozilla-centos6-x86_64', 'usePTY': 'slave-config', 'workdir': <WithProperties "%(basedir)s/build/obj-firefox">} {}
@@ -6689,0 +6692 @@ Linux Mulet x86-64 fig build NightlyBuil
+    SendChangeStep {'branch': 'fig-linux64-mulet-unittest', 'command': None, 'comments': <WithProperties "%(comments:-)s">, 'description': None, 'descriptionDone': None, 'env': {'CCACHE_COMPRESS': '1',  'CCACHE_DIR': '/builds/ccache',  'CCACHE_UMASK': '002',  'DISPLAY': ':2',  'HG_SHARE_BASE_DIR': '/builds/hg-shared',  'LC_ALL': 'C',  'MOZ_AUTOMATION': '1',  'MOZ_CRASHREPORTER_NO_REPORT': '1',  'MOZ_OBJDIR': 'obj-firefox',  'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin',  'TINDERBOX_OUTPUT': '1',  'TOOLTOOL_CACHE': '/builds/tooltool_cache',  'TOOLTOOL_HOME': '/builds'}, 'files': [<WithProperties "%(packageUrl)s">, <WithProperties "%(testsUrl)s">], 'log_eval_func': None, 'logfiles': {}, 'master': 'buildbot-master81.build.mozilla.org:9301', 'name': 'sendchange_buildbot-master81.build.mozilla.org:9301', 'retries': 5, 'revision': <WithProperties "%(got_revision)s">, 'sendchange_props': {'buildid': <WithProperties "%(buildid:-)s">,  'builduid': <WithProperties "%(builduid:-)s">,  'pgo_build': False}, 'timeout': 1800, 'usePTY': 'slave-config', 'user': 'sendchange-unittest', 'warnOnFailure': True, 'workdir': None} {}

Comment 79

[Aki Sasaki (not active)]

Comment on attachment 8438775 [details] [diff] [review]
add unit test sendchanges

a) awesome, and thanks for taking these on

b) I noticed this is in mozilla/config.py; we have the rest of the b2g builds in mozilla/b2g_config.py .  I don't want to delay these, but if you have some spare cycles to try moving this into b2g_config.py, that will help keep people from getting [more] confused when looking at our configs.  At the very least I'd like a followup bug to move mulet into b2g_config.py; it would be even better if that bug has details about what breaks if we try moving it, for posterity.  And even better still if it Just Works.

Comment 80

[Armen [:armenzg]]

Comment on attachment 8438775 [details] [diff] [review]
add unit test sendchanges

https://hg.mozilla.org/build/buildbot-configs/rev/c4ab10c33bfa

I don't think enough spare cycles as the "all tests" work has taking me way longer than expected. I'm hoping to focus now on blobber discoverability.

Comment 81

[Simone Bruno [:simone]]

In prod with reconfig on 2014-06-12 10:46 PT

Comment 82

[Armen [:armenzg]]

Thanks Simone!
I've triggered another build.

We should see tests being triggered after it:
https://tbpl.mozilla.org/?tree=Fig

Comment 83

[Armen [:armenzg]]

We have sendchanges but don't trigger anything.
I will grab a master and two set of slaves to make sure that this works beyond my local master.

I think something like this is needed:
+++ b/buildfarm/maintenance/production-masters.json
@@ -307,4 +307,5 @@
     "limit_fx_platforms": [
       "linux64",
+      "linux64-mulet",
       "linux64-asan"
     ],
@@ -313,4 +314,7 @@
         "ubuntu64_vm"
       ],
+      "linux64-mulet": [
+        "ubuntu64_vm"
+      ],
       "linux64-asan": [
         "ubuntu64-asan_vm"

Comment 84

[Armen [:armenzg]]

Attachment: Add linux64-mulet to the list of platforms for linux64 masters

I guess it worked for me when I didn't have limiting platforms for my development master.

Comment 85

[Armen [:armenzg]]

Attachment: fix the unittest branch

Comment 86

[Armen [:armenzg]]

Comment on attachment 8440765 [details] [diff] [review]
Add linux64-mulet to the list of platforms for linux64 masters

https://hg.mozilla.org/build/tools/rev/0e1aa44efd68

Comment 87

[Armen [:armenzg]]

Comment on attachment 8440766 [details] [diff] [review]
fix the unittest branch

https://hg.mozilla.org/build/buildbot-configs/rev/dd5829e1d377

Comment 88

[Jordan Lund (:jlund)]

buildbot-config patch live in production :)

Comment 89

[Armen [:armenzg]]

The test results are showing.

Time to green them up? Should we file a new bug for that?

Comment 90

[Alexandre Poirot [:ochameau]]

\o/

I already have patches for that but I need some feedback from someone, I don't know who.
We need to filter out tests that shouldn't be running on mulet.
In my patch I introduce a new magic keyword `mulet` within *.ini test manifest files,
but I'm not sure that's the right way to proceed.

https://github.com/ochameau/mozilla-central/commit/3b9e0fc310ef45dabcd1a9c752ad055001c6b214

Do you know who should I bug about that and in which component should I fill a bug and attach such patch?

Comment 91

[Jonathan Griffin (:jgriffin)]

Let's file a new bug for greening up the tests.

ochameau, you can file a bug under Testing:General for the manifest changes (since they span multiple harnesses) and flag :ahal for review.

Comment 92

[Armen [:armenzg]]

Filed bug 1027242 for it.

Comment 93

[Armen [:armenzg]]

Attachment: Fix naming

r=RyanVM
https://hg.mozilla.org/build/buildbot-configs/rev/442454f765a3

As per https://bugzilla.mozilla.org/show_bug.cgi?id=1016448#c3
-    'name': 'Ubuntu Mulet VM 12.04 x64',
+    'name': 'Ubuntu VM 12.04 x64 Mulet',

-            'base_name': 'Linux Mulet x86-64 %(branch)s',
+            'base_name': 'Linux x86-64 Mulet %(branch)s',

Comment 94

[Armen [:armenzg]]

Waiting for reconfig.

Comment 95

[Chris Cooper [:coop] (he/him)]

Merged to production, and deployed.

Comment 96

[Armen [:armenzg]]

Naming fixed!