The default bug view has changed. See this FAQ.

Start doing mulet builds on Fig

RESOLVED FIXED in B2G C4 (2jan on)

Status

Release Engineering
General Automation
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: ochameau, Assigned: armenzg)

Tracking

(Blocks: 1 bug)

unspecified
B2G C4 (2jan on)
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(8 attachments, 3 obsolete attachments)

550 bytes, patch
mshal
: review+
armenzg
: checked-in+
Details | Diff | Splinter Review
13.47 KB, patch
catlee
: review+
armenzg
: checked-in+
Details | Diff | Splinter Review
7.17 KB, patch
catlee
: review+
armenzg
: checked-in+
Details | Diff | Splinter Review
1.25 KB, patch
aki
: review+
armenzg
: checked-in+
Details | Diff | Splinter Review
805 bytes, patch
aki
: review+
armenzg
: checked-in+
Details | Diff | Splinter Review
7.18 KB, patch
aki
: review+
armenzg
: checked-in+
Details | Diff | Splinter Review
927 bytes, patch
aki
: review+
armenzg
: checked-in+
Details | Diff | Splinter Review
1.81 KB, patch
armenzg
: review+
Details | Diff | Splinter Review
(Reporter)

Description

3 years ago
Once bug 961745 lands, we can start building Firefox Mulet out of mozilla central.
Firefox Mulet is a special build of Firefox with various b2g specifics.
It is meant to replace b2g desktop.
If bug 961745's patch doesn't change much, you just have to build the firefox target, with --enable-mulet.

I don't really know how release automation works,
but a good first step would be to allow us to build the Mulet on try.
Just building it.
Then we can iterate on it by enabling each test suite, still only on try.
We would start with xpcshell, then mochitest-plain.
That's where it may be complicated. Even if that's mostly Firefox,
many tests will fail because of the b2g specifics, so, we would like
to run only the xpcshell and mochitest-plain tests that are currently run on b2g desktop.

I would like to use this bug as a meta bug regarding the mulet and automation. We would discuss in this bug about what action items we can start working on.

First iteration:

  Expose a "mulet (linux64)" platform target for try builds

Second:

  Start running b2g desktop xpcshell tests on this new target

Third:

  Start running b2g desktop mochitest-plain tests on this new target
(Reporter)

Comment 1

3 years ago
Here is a try run of the Mulet:
  https://tbpl.mozilla.org/?tree=Try&rev=b1cbfd59cd98

Which comes with very good results!
The list of failing tests looks quite small given how special Firefox Mulet is...
We shouldn't run mochitest-browser/mochitest-e10s as I don't think we really care about Firefox features that much. (I'm not sure it is really worth spending time in ensuring that browser test pass)
Similar thing may apply to mochitest-chrome. Even if we would like to see mochitest-chrome running on b2g, as we don't run them on device/emulator, I'm not sure it is really worth running them on desktop.

But the good news is that all xpcshell tests just work!
Then the only thing we would need is either:
 - fix the failing mochitest-plain,
 - somehow disable them for the mulet,
    How does mochitest.ini filter works?
    Can we easily add a "skip-if = mulet"?
    (the best way to identify a mulet is to check for MOZ_MULET env/config variable),
 - only run current b2g desktop whitelist.

Gregor, can you find whoever can help us on this (expose mulet target on try and filter mochitests)?
Flags: needinfo?(anygregor)
(In reply to Alexandre Poirot (:ochameau) from comment #1)
> Here is a try run of the Mulet:
>   https://tbpl.mozilla.org/?tree=Try&rev=b1cbfd59cd98
> 
> Which comes with very good results!
> The list of failing tests looks quite small given how special Firefox Mulet
> is...
> We shouldn't run mochitest-browser/mochitest-e10s as I don't think we really
> care about Firefox features that much. (I'm not sure it is really worth
> spending time in ensuring that browser test pass)
> Similar thing may apply to mochitest-chrome. Even if we would like to see
> mochitest-chrome running on b2g, as we don't run them on device/emulator,
> I'm not sure it is really worth running them on desktop.
> 
> But the good news is that all xpcshell tests just work!
> Then the only thing we would need is either:
>  - fix the failing mochitest-plain,
>  - somehow disable them for the mulet,
>     How does mochitest.ini filter works?
>     Can we easily add a "skip-if = mulet"?
>     (the best way to identify a mulet is to check for MOZ_MULET env/config
> variable),
>  - only run current b2g desktop whitelist.
> 
> Gregor, can you find whoever can help us on this (expose mulet target on try
> and filter mochitests)?

Thats great!
I think jgriffin is the best person to ask here. As soon as we see that 961745 is ready to land we should start the automation process.
Flags: needinfo?(anygregor) → needinfo?(jgriffin)
(In reply to Alexandre Poirot (:ochameau) from comment #1)
> But the good news is that all xpcshell tests just work!
> Then the only thing we would need is either:
>  - fix the failing mochitest-plain,
>  - somehow disable them for the mulet,
>     How does mochitest.ini filter works?
>     Can we easily add a "skip-if = mulet"?
>     (the best way to identify a mulet is to check for MOZ_MULET env/config
> variable),
>  - only run current b2g desktop whitelist.

The variables the manifests know about are written by the build here:
http://mxr.mozilla.org/mozilla-central/source/python/mozbuild/mozbuild/mozinfo.py

Though if this is going to replace b2g desktop, can't we just use 'buildapp == b2g && toolkit != gonk' again? Or do we plan to run mulet alongside b2g desktop?
Also note, I'd guess that try run is running the same set of mochitest-plain that b2g_desktop is currently running, since they probably hit the same skip-if conditions.
They're actually running the same tests as regular desktop Firefox, not the B2G desktop build.  I'm not sure whether this is what we want, or not.
Flags: needinfo?(jgriffin)
And yes, as soon as bug 961745 lands, I can start the process of having builds and tests added to buildbot.
Are you sure? They are chunked the same as desktop Firefox, but I don't think they are running the same set of tests. Right now we skip b2g_desktop tests with:
skip-if = buildapp == 'b2g' && toolkit != 'gonk'

Aiui, that should catch mulet as well.
Not positive, but the aggregate number of tests seems to be much larger than what we run for b2g-desktop.
(Reporter)

Comment 9

3 years ago
(In reply to Jonathan Griffin (:jgriffin) from comment #5)
> They're actually running the same tests as regular desktop Firefox, not the
> B2G desktop build.  I'm not sure whether this is what we want, or not.

Yes we are running firefox test suites, as the Mulet is closer to Firefox than b2g-desktop.
If the build system patch doesn't change, to build a Mulet, you just need to build firefox with --enable-mulet.

I think that's a good thing. We are running tests through the same codepath than Firefox desktop.
I imagine it will be easier to maintain! But we have to find a way to filter tests.


(In reply to Andrew Halberstadt [:ahal] from comment #3)
> Though if this is going to replace b2g desktop, can't we just use 'buildapp
> == b2g && toolkit != gonk' again? Or do we plan to run mulet alongside b2g
> desktop?

We will most likely have them running alonside, until the Mulet is mature enough to replace b2g desktop.
So we no longer have a white/black-list for b2g, right?
We just run all and rely on manual filter in each mochitest.ini?
If that's the case, we can get the correct filtering quite easily by tuning mozinfo.py to make it so that buildappname = appname = b2g? (and/or MOZ_APP_NAME = MOZ_BUILD_APP = b2g)
(Reporter)

Comment 10

3 years ago
I hacked mozinfo.py to expose "mulet" variable. I'm not sure that the best way to go, but that allows to see how many tests we have to disable just for the mulet and it looks like we don't have to disable many of them:
All (but e10s) mochitests green:
  https://tbpl.mozilla.org/?tree=Try&rev=50519e76486e
Disabled tests:
  https://github.com/ochameau/mozilla-central/commit/0edbde06c03e0e7e09b5fbf219440393524752db
(Reporter)

Comment 11

3 years ago
Bug 961745 is about to reach master.
It means that we can start building mulets. The build system patch changed quite a bit, so here is the updated instruction do build a mulet:

You just need a mozconfig with:

  ac_add_options --enable-application=b2g/dev

You may pass any other flag that Firefox is used to have, but nothing else is required at this stage.
Once we get it build with just that, we can iterate and start looking at shipping a gaia profile, like what we are doing for b2g desktop. But let's just build it as-is in a first round.
(In reply to Jonathan Griffin (:jgriffin) (away until May 11) from comment #6)
> And yes, as soon as bug 961745 lands, I can start the process of having
> builds and tests added to buildbot.

\o/

jmaher, armen, can we get some help here? jgriffin mentioned that you can help out when he is away.
Flags: needinfo?(jmaher)
Flags: needinfo?(armenzg)
I will see what I can do.
Assignee: nobody → armenzg
Flags: needinfo?(jmaher)
Flags: needinfo?(armenzg)
(Reporter)

Comment 14

3 years ago
Here is a try run with various patches that still need to land on central,
but proves that we can get mochitests and xpcshell up and running green against b2g/dev!
  https://tbpl.mozilla.org/?tree=Try&rev=b620c9971a60
(Reporter)

Comment 15

3 years ago
Just to be clear, as requested on first comment, we don't need to build that on every branch right away.
What would help is having it built only on try and allow running xpcshell, mochitest-plain, mochitest-chrome and may be also mochitest-browser. I don't know what kind of options you need to specify; but regarding test runners, Mulet should work as firefox. Test runners should be run as if it was Firefox.
Also, to save resources, you can prevent building it if not explicitely asked, or if that's not something we support, we should keep it hidden as it is not ready for prime time (yet).
If I can't find anyone from Releng to grab it I will try to help next week when I'm out of buildduty.
Hi catlee,
I've reviewed the different comments on this bug and we're ready to add a new type of build which uses a new mozconfig with this extra line:
  ac_add_options --enable-application=b2g/dev

FYI: Mulet behaves more like Firefox desktop than B2g desktop.

The scope right now is to enable the build on try plus few suites (xpcshell, mochitest-plain, mochitest-chrome and may be also mochitest-browse). 

Is there anyone that has time to look into it this week?
If not I hope to be able to make time next week for this.
Flags: needinfo?(catlee)
We probably won't have time this week to get to this. Who is the intended audience for these builds?
Flags: needinfo?(catlee)

Updated

3 years ago
Summary: Starts building Mulet target → Start doing mulet builds
(In reply to Chris AtLee [:catlee] from comment #18)
> Who is the intended audience for these builds?

The intended audience for these builds is all of FirefoxOS development. There will be a few teams that won't use this, like teams that need to work directly against hardware that we're not emulating (like telephony/sms), however most frontend and gecko teams will be using this.

We're also planning on migrating 3rd party app developers to using this.

See https://groups.google.com/d/msg/mozilla.dev.b2g/2WUg10993OM/VSceN9i2w2EJ
i.e. we will eventually deprecate B2G desktop builds in favor of Mulet
We've decided cedar has diverged too far from m-c for this work; let's schedule these on fig and try (we'll hide them initially on try).
Depends on: 1009974
Blocks: 1011767
Created attachment 8425642 [details] [diff] [review]
Add in-tree mulet mozconfig

I wonder if it should be named "nightly-mulet" or if it should just "mulet".

Is it the right place to have it under "browser/config/mozconfigs"?
Attachment #8425642 - Flags: review?(mshal)
Attachment #8425642 - Flags: review?(bhearsum)
Attachment #8425642 - Flags: feedback?(poirot.alex)
Comment on attachment 8425642 [details] [diff] [review]
Add in-tree mulet mozconfig

Review of attachment 8425642 [details] [diff] [review]:
-----------------------------------------------------------------

I'm only reviewing this for organization - I'll leave it to someone else to review it for correctness.

I think this is the right place to put, we have no precedent for putting anything except target platforms directly in "mozconfigs".

"mulet" is probably a better name. There's nothing nightly-specific about this config. r=me with that change.
Attachment #8425642 - Flags: review?(bhearsum) → review-
Comment on attachment 8425642 [details] [diff] [review]
Add in-tree mulet mozconfig

Isn't it nightly specific since it includes linux64/nightly?

Also I will defer to the mulet folks on where they want it, but IMO it makes more sense to put this under b2g/dev/ instead of browser/. It looks like for all the other applications, we have config/mozconfigs/ at the same level as the corresponding app.mozbuild file. Eg:

b2g/app.mozbuild
b2g/config/mozconfigs/

mobile/android/app.mozbuild
mobile/android/config/mozconfigs/

browser/app.mozbuild
browser/config/mozconfigs/

xulrunner/app.mozbuild
xulrunner/config/mozconfigs/

Since mulet has b2g/dev/app.mozbuild, I think it should be b2g/dev/config/mozconfigs/...
Attachment #8425642 - Flags: review?(mshal) → review+
(Reporter)

Comment 25

3 years ago
Comment on attachment 8425642 [details] [diff] [review]
Add in-tree mulet mozconfig

Review of attachment 8425642 [details] [diff] [review]:
-----------------------------------------------------------------

(In reply to Michael Shal [:mshal] from comment #24)
> Comment on attachment 8425642 [details] [diff] [review]
> Add in-tree mulet mozconfig
> 
> Isn't it nightly specific since it includes linux64/nightly?

+1, I would imagine this mozconfig is to build the mulet against mozilla central tip,
and in future we may have a mozconfig for beta and release?

> Since mulet has b2g/dev/app.mozbuild, I think it should be
> b2g/dev/config/mozconfigs/...

+1
Attachment #8425642 - Flags: feedback?(poirot.alex) → feedback+
Created attachment 8425716 [details] [diff] [review]
Add in-tree mulet mozconfig
Attachment #8425716 - Flags: review?(mshal)
(Assignee)

Updated

3 years ago
Attachment #8425642 - Attachment is obsolete: true
Comment on attachment 8425716 [details] [diff] [review]
Add in-tree mulet mozconfig

Looks good!
Attachment #8425716 - Flags: review?(mshal) → review+
Which other builds besides the mulet builds should we enable on Fig?
I think we can get away with having only Mulet builds on fig; we can use m-c and cedar for comparison with other builds, as necessary.
Created attachment 8425807 [details] [diff] [review]
[wip] add mulet builds and tests to fig

catlee: how is this shaping?
Attachment #8425807 - Flags: feedback?(catlee)
Is the packaging going to look like a Firefox build?

I've pushed to try to see where we currently are:
https://tbpl.mozilla.org/?tree=Try&rev=e8b436c152a8
(Reporter)

Comment 32

3 years ago
(In reply to Armen Zambrano [:armenzg] (EDT/UTC-4) from comment #31)
> Is the packaging going to look like a Firefox build?

Yes. You should find a firefox-VERSION-... tarball/zip after make package.
Comment on attachment 8425807 [details] [diff] [review]
[wip] add mulet builds and tests to fig

Review of attachment 8425807 [details] [diff] [review]:
-----------------------------------------------------------------

looks like you're on the right track!

::: mozilla/config.py
@@ +651,5 @@
> +        'linux64-mulet': {
> +            'mozharness_config': {
> +                'script_name': 'scripts/fx_desktop_build.py',
> +                'extra_args': [
> +                    '--config', 'builds/releng_base_linux_64_builds.py',

does this need --custom-build-variant-cfg mulet here? I see that in the asan configs.

@@ +659,5 @@
> +            },
> +
> +            'unittest_platform': 'linux64-mulet',
> +            'stage_product': 'firefox',
> +            'stage_platform': 'mulet',

linux64-mulet maybe?
Attachment #8425807 - Flags: feedback?(catlee) → feedback+
(Assignee)

Updated

3 years ago
Depends on: 1014212
Created attachment 8426520 [details] [diff] [review]
Add mulet to fig and try
Attachment #8425807 - Attachment is obsolete: true
Attachment #8426520 - Flags: review?(catlee)
(In reply to Chris AtLee [:catlee] from comment #33)
> Comment on attachment 8425807 [details] [diff] [review]
> [wip] add mulet builds and tests to fig
> 
> Review of attachment 8425807 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> looks like you're on the right track!
> 
> ::: mozilla/config.py
> @@ +651,5 @@
> > +        'linux64-mulet': {
> > +            'mozharness_config': {
> > +                'script_name': 'scripts/fx_desktop_build.py',
> > +                'extra_args': [
> > +                    '--config', 'builds/releng_base_linux_64_builds.py',
> 
> does this need --custom-build-variant-cfg mulet here? I see that in the asan
> configs.
> 
It's the mozconfig that makes it different. No mozharness changes afaik.
Alexandre, would you please review the results in here? Thanks!
https://tbpl.mozilla.org/?tree=Try&rev=e8b436c152a8
Flags: needinfo?(poirot.alex)
It's interesting that debug tests are greener than opt...we usually see the reverse!

When we actually schedule these tests, I imagine we should skip M-e10s (since all tests are OOP by default on Mulet, unless I'm mistaken).
(Reporter)

Comment 38

3 years ago
This is really weird, I downloaded the opt tarball and this looks like a regular firefox, even if I can see --enable-application=b2g/dev in about:buildconfig, I tried to search for all the mulet specifics and found nothing.
But what is really suspicious is that the test failures we see on tbpl for opt are the expected mulet's one! (I have a patch with a new mulet specific blacklist for xpcshell and mochitests)
The only thing I can think of is that "make package" strip all mulet specifics.
I'll look into this, if that's it, the opt build is just fine, it will be matter of landing a fix for make package.

But I think there is something wrong with mozconfig on debug build. I also downloaded the mulet and I don't even see --enable-application=b2g/dev in about:buildconfig, and given that all tests are green, I think debug is really not a mulet and just a regular firefox!
Flags: needinfo?(poirot.alex)
My apologies.
The debug build was triggered by mistake based on a normal Firefox build.

Updated

3 years ago
Attachment #8426520 - Flags: review?(catlee) → review+
Comment on attachment 8426520 [details] [diff] [review]
Add mulet to fig and try

https://hg.mozilla.org/build/buildbot-configs/rev/b339f91b754c
Attachment #8426520 - Flags: checked-in+
Comment on attachment 8425716 [details] [diff] [review]
Add in-tree mulet mozconfig

https://hg.mozilla.org/integration/mozilla-inbound/rev/b873c10c208d
Attachment #8425716 - Flags: checked-in+
Once this code goes into the buildbot masters we should be able to trigger Linux 64-bit mulet jobs on Fig and Try (assuming the m-i code gets into the pushes of those trees).

ochameau: have you had a chance to file a bug for the issues you saw in comment 38?
Flags: needinfo?(poirot.alex)
https://hg.mozilla.org/mozilla-central/rev/b873c10c208d
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → B2G C4 (2jan on)
(Reporter)

Updated

3 years ago
Depends on: 1015949
(Reporter)

Comment 44

3 years ago
I opened bug 1015949 for the packaging issue.
Otherwise, I was wondering what string should I used to build b2g/dev on Try?
Also, do you know who is maintaining http://trychooser.pub.build.mozilla.org/ so that we can get it updated?
Flags: needinfo?(poirot.alex)
You should use 'linux64-mulet' as the platform.
However, the changes are not yet live in production.

The TryChooser code lives in here:
http://hg.mozilla.org/build/tools/file/tip/trychooser
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Whiteboard: [leave open]
Merged and deployed to production.
(Assignee)

Updated

3 years ago
Depends on: 1016448
I've pushed m-i to fig so we can gets builds:
https://tbpl.mozilla.org/?tree=Fig
Unfortunately, I made a lot of assumptions (like we had already switch to mozharness for builds) and I assumed I would only need minor tweaks.

I have setup a local build master and will be testing until I get this right.
Which builds besides Linux 64 are we going to have?
Are we also going to do debug? or are we fine with opt?

I assume we won't be running nightly builds of this.
I'm getting this build issue:

updating cache ../.././config.cache
creating ./config.status
Reticulating splines...
Traceback (most recent call last):
  File "./config.status", line 979, in <module>
    config_status(**args)
  File "/builds/slave/fig-l64-mulet-0000000000000000/build/python/mozbuild/mozbuild/config_status.py", line 148, in config_status
    summary = the_backend.consume(definitions)
  File "/builds/slave/fig-l64-mulet-0000000000000000/build/python/mozbuild/mozbuild/backend/base.py", line 186, in consume
    for obj in objs:
  File "/builds/slave/fig-l64-mulet-0000000000000000/build/python/mozbuild/mozbuild/frontend/emitter.py", line 105, in emit
    for out in output:
  File "/builds/slave/fig-l64-mulet-0000000000000000/build/python/mozbuild/mozbuild/frontend/reader.py", line 726, in read_mozbuild
    raise bre
mozbuild.frontend.reader.BuildReaderError: ==============================
ERROR PROCESSING MOZBUILD FILE
==============================

The error occurred while processing the following file:

    /builds/slave/fig-l64-mulet-0000000000000000/build/moz.build

The underlying problem is we referenced a path that does not exist. That path is:

    /builds/slave/fig-l64-mulet-0000000000000000/build/b2g/dev/app/moz.build

Either create the file if it needs to exist or do not reference it.
I've pushed to try to see if we hit the same issue:
https://tbpl.mozilla.org/?tree=Try&rev=ca8025b48d5d
Created attachment 8430231 [details] [diff] [review]
switch mulet builds to non-mozharness, change builder name plus disable from release branches
Attachment #8430231 - Flags: review?(catlee)
(In reply to Armen Zambrano [:armenzg] (EDT/UTC-4) from comment #49)
> Which builds besides Linux 64 are we going to have?
> Are we also going to do debug? or are we fine with opt?
> 
> I assume we won't be running nightly builds of this.

We'll want debug as well as opt, and we'll want the same platforms that we're currently building b2gdesktop against...OSX, linux32/64, and win32.

We won't need nightlies for now, but that could change later.
It does seem like moz.build is missing.  Any ideas Alexandre?
Flags: needinfo?(poirot.alex)
Comment on attachment 8430231 [details] [diff] [review]
switch mulet builds to non-mozharness, change builder name plus disable from release branches

Review of attachment 8430231 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/config.py
@@ -64,5 @@
>          'linux64-debug': {},
>          'linux64-asan': {},
>          'linux64-asan-debug': {},
>          'linux64-st-an-debug': {},
> -        'linux64-mulet': {},

isn't this still needed?

@@ -2278,5 @@
> -for name, branch in items_at_least(BRANCHES, 'gecko_version', 32):
> -    if name in ('try', 'fig'):
> -        continue
> -    if 'linux64-mulet' in branch['platforms']:
> -        del branch['platforms']['linux64-mulet']

you're ready to enable on all branches?
Michael, is there any help you can provide here? See comment 50 to 54.
Flags: needinfo?(mshal)
(In reply to Chris AtLee [:catlee] from comment #55)
> Comment on attachment 8430231 [details] [diff] [review]
> switch mulet builds to non-mozharness, change builder name plus disable from
> release branches
> 
> Review of attachment 8430231 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: mozilla/config.py
> @@ -64,5 @@
> >          'linux64-debug': {},
> >          'linux64-asan': {},
> >          'linux64-asan-debug': {},
> >          'linux64-st-an-debug': {},
> > -        'linux64-mulet': {},
> 
> isn't this still needed?

That line is to enable across the board.

> 
> @@ -2278,5 @@
> > -for name, branch in items_at_least(BRANCHES, 'gecko_version', 32):
> > -    if name in ('try', 'fig'):
> > -        continue
> > -    if 'linux64-mulet' in branch['platforms']:
> > -        del branch['platforms']['linux64-mulet']
> 
> you're ready to enable on all branches?

I won't need this block because I'm removing the previous line.
(In reply to Jonas Sicking (:sicking) from comment #56)
> Michael, is there any help you can provide here? See comment 50 to 54.

Looks like I missed that in reviewing bug 1015949 where it adds b2g/dev/app as a tier dir. We can probably just copy a blank moz.build file (like b2g/installer/moz.build) into b2g/dev/app.
Flags: needinfo?(mshal)
try build with empty moz.build in b2g/dev/app:  https://tbpl.mozilla.org/?tree=Try&rev=15331561588b
Catlee, did comment 57 answer your questions?
(Reporter)

Comment 61

3 years ago
Oops, I was out of town. I submitted a patch for the missing moz.build in bug 1015949.
Flags: needinfo?(poirot.alex)
(In reply to Armen Zambrano [:armenzg] (EDT/UTC-4) from comment #60)
> Catlee, did comment 57 answer your questions?

Adding NI. I think the answer should satisfy the review concerns.
Flags: needinfo?(catlee)

Updated

3 years ago
Attachment #8430231 - Flags: review?(catlee) → review+
(Assignee)

Updated

3 years ago
Flags: needinfo?(catlee) → needinfo?(aki)

Comment 63

3 years ago
I see an r+
Flags: needinfo?(aki)
Comment on attachment 8430231 [details] [diff] [review]
switch mulet builds to non-mozharness, change builder name plus disable from release branches

https://hg.mozilla.org/build/buildbot-configs/rev/b11352bd5d07
Attachment #8430231 - Flags: checked-in+
In production with reconfig on 2014-06-04 02:19 PT
Some configure issues:
https://tbpl.mozilla.org/php/getParsedLog.php?id=41067418&tree=Fig&full=1#error0

I will look into it later today.

checking whether the C compiler (/builds/slave/fig-l64-mulet-0000000000000000/build/gcc/bin/gcc  ) works... no
configure: error: installation or configuration problem: C compiler cannot create executables.
------ config.log ------
This file contains any messages produced by compilers while
running configure, to aid debugging if configure makes a mistake.
ochameau, can you please check that you can still build mulet?

I've loaned a machine for staging and I will be working to make it work next week.
I want to make sure that we have not been regressed by other changes. Thanks.
(Reporter)

Comment 68

3 years ago
Yes nothing regressed. mach build, package, run. Everything works.
There is just one thing. If you want to play with it, see gaia running, you have to manually open chrome://b2g/content/shell.html in a tab.

Comment 69

3 years ago
If we can pull gaia from git rather than hg for Mulet, we can get rid of a whole host of issues syncing gaia to hg.m.o and then translating between the two revisions (e.g. bug 1022864).
This is blocked on getting git installed on our windows builders, however.
Blocks: 1022864
Created attachment 8437188 [details] [diff] [review]
add tooltool configuration
Attachment #8437188 - Flags: review?(aki)

Updated

3 years ago
Attachment #8437188 - Flags: review?(aki) → review+
Comment on attachment 8437188 [details] [diff] [review]
add tooltool configuration

https://hg.mozilla.org/build/buildbot-configs/rev/e371c888a099
Attachment #8437188 - Flags: checked-in+
Live with reconfig on 2014-06-11 08:47 PT
Triggered new build on Fig:
https://tbpl.mozilla.org/?tree=Fig
Created attachment 8438699 [details] [diff] [review]
add sendchanges

I forgot that talos_masters are needed to activate sendchanges.

Diff of master dumps:
--- old_dump.txt	2014-06-11 17:08:57.325008345 -0400
+++ new_dump.txt	2014-06-11 16:39:17.929087679 -0400
@@ -6689,0 +6690 @@ Linux Mulet x86-64 fig build NightlyBuil
+    SendChangeStep {'branch': 'fig-linux64-mulet-talos', 'command': None, 'comments': <WithProperties "%(comments:-)s">, 'description': None, 'descriptionDone': None, 'env': {'CCACHE_COMPRESS': '1',  'CCACHE_DIR': '/builds/ccache',  'CCACHE_UMASK': '002',  'DISPLAY': ':2',  'HG_SHARE_BASE_DIR': '/builds/hg-shared',  'LC_ALL': 'C',  'MOZ_AUTOMATION': '1',  'MOZ_CRASHREPORTER_NO_REPORT': '1',  'MOZ_OBJDIR': 'obj-firefox',  'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin',  'TINDERBOX_OUTPUT': '1',  'TOOLTOOL_CACHE': '/builds/tooltool_cache',  'TOOLTOOL_HOME': '/builds'}, 'files': [<WithProperties "%(packageUrl)s">], 'log_eval_func': None, 'logfiles': {}, 'master': 'buildbot-master81.build.mozilla.org:9301', 'name': 'sendchange_buildbot-master81.build.mozilla.org:9301', 'retries': 5, 'revision': <WithProperties "%(got_revision)s">, 'sendchange_props': {'buildid': <WithProperties "%(buildid:-)s">,  'builduid': <WithProperties "%(builduid:-)s">,  'pgo_build': False}, 'timeout': 1800, 'usePTY': 'slave-config', 'user': 'sendchange', 'warnOnFailure': True, 'workdir': None} {}
Attachment #8438699 - Flags: review?(aki)
FYI the builds can now run on Fig.
We now need to add the tests.

Comment 76

3 years ago
Comment on attachment 8438699 [details] [diff] [review]
add sendchanges

Uh, do you really want to run talos against this?
I'd be surprised.
I think you want unittest masters, which should already be set, and packageTests.  Am I wrong?
Clearing review for now.
Attachment #8438699 - Flags: review?(aki)
We don't need talos on these builds.
Created attachment 8438775 [details] [diff] [review]
add unit test sendchanges

I think everything I said or did today came out wrong.

This activates the unit tests sendchanges as well as make package-tests:
http://hg.mozilla.org/build/buildbotcustom/file/default/process/factory.py#l2050

--- old_dump.txt	2014-06-11 17:08:57.325008345 -0400
+++ new_dump.txt	2014-06-11 18:51:38.303819795 -0400
@@ -6679,0 +6680,2 @@ Linux Mulet x86-64 fig build NightlyBuil
+    MockCommand {'command': ['make', 'buildsymbols'], 'description': None, 'descriptionDone': None, 'env': {'CCACHE_COMPRESS': '1',  'CCACHE_DIR': '/builds/ccache',  'CCACHE_UMASK': '002',  'DISPLAY': ':2',  'HG_SHARE_BASE_DIR': '/builds/hg-shared',  'LC_ALL': 'C',  'MOZ_AUTOMATION': '1',  'MOZ_CRASHREPORTER_NO_REPORT': '1',  'MOZ_OBJDIR': 'obj-firefox',  'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin',  'TINDERBOX_OUTPUT': '1',  'TOOLTOOL_CACHE': '/builds/tooltool_cache',  'TOOLTOOL_HOME': '/builds'}, 'haltOnFailure': True, 'log_eval_func': None, 'logfiles': {}, 'mock': True, 'mock_args': ['--unpriv'], 'mock_login': 'mock_mozilla', 'mock_workdir_mutator': <function <lambda>>, 'mock_workdir_prefix': None, 'name': 'make_buildsymbols', 'target': 'mozilla-centos6-x86_64', 'timeout': 3600, 'usePTY': 'slave-config', 'workdir': <WithProperties "%(basedir)s/build/obj-firefox">} {}
+    MockCommand {'command': ['make', 'package-tests'], 'description': None, 'descriptionDone': None, 'env': {'CCACHE_COMPRESS': '1',  'CCACHE_DIR': '/builds/ccache',  'CCACHE_UMASK': '002',  'DISPLAY': ':2',  'HG_SHARE_BASE_DIR': '/builds/hg-shared',  'LC_ALL': 'C',  'MOZ_AUTOMATION': '1',  'MOZ_CRASHREPORTER_NO_REPORT': '1',  'MOZ_OBJDIR': 'obj-firefox',  'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin',  'TINDERBOX_OUTPUT': '1',  'TOOLTOOL_CACHE': '/builds/tooltool_cache',  'TOOLTOOL_HOME': '/builds'}, 'haltOnFailure': True, 'log_eval_func': None, 'logfiles': {}, 'mock': True, 'mock_args': ['--unpriv'], 'mock_login': 'mock_mozilla', 'mock_workdir_mutator': <function <lambda>>, 'mock_workdir_prefix': None, 'name': 'make_pkg_tests', 'target': 'mozilla-centos6-x86_64', 'usePTY': 'slave-config', 'workdir': <WithProperties "%(basedir)s/build/obj-firefox">} {}
@@ -6689,0 +6692 @@ Linux Mulet x86-64 fig build NightlyBuil
+    SendChangeStep {'branch': 'fig-linux64-mulet-unittest', 'command': None, 'comments': <WithProperties "%(comments:-)s">, 'description': None, 'descriptionDone': None, 'env': {'CCACHE_COMPRESS': '1',  'CCACHE_DIR': '/builds/ccache',  'CCACHE_UMASK': '002',  'DISPLAY': ':2',  'HG_SHARE_BASE_DIR': '/builds/hg-shared',  'LC_ALL': 'C',  'MOZ_AUTOMATION': '1',  'MOZ_CRASHREPORTER_NO_REPORT': '1',  'MOZ_OBJDIR': 'obj-firefox',  'PATH': '/tools/buildbot/bin:/usr/local/bin:/usr/lib64/ccache:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/tools/git/bin:/tools/python27/bin:/tools/python27-mercurial/bin:/home/cltbld/bin',  'TINDERBOX_OUTPUT': '1',  'TOOLTOOL_CACHE': '/builds/tooltool_cache',  'TOOLTOOL_HOME': '/builds'}, 'files': [<WithProperties "%(packageUrl)s">, <WithProperties "%(testsUrl)s">], 'log_eval_func': None, 'logfiles': {}, 'master': 'buildbot-master81.build.mozilla.org:9301', 'name': 'sendchange_buildbot-master81.build.mozilla.org:9301', 'retries': 5, 'revision': <WithProperties "%(got_revision)s">, 'sendchange_props': {'buildid': <WithProperties "%(buildid:-)s">,  'builduid': <WithProperties "%(builduid:-)s">,  'pgo_build': False}, 'timeout': 1800, 'usePTY': 'slave-config', 'user': 'sendchange-unittest', 'warnOnFailure': True, 'workdir': None} {}
Attachment #8438699 - Attachment is obsolete: true
Attachment #8438775 - Flags: review?(aki)

Comment 79

3 years ago
Comment on attachment 8438775 [details] [diff] [review]
add unit test sendchanges

a) awesome, and thanks for taking these on

b) I noticed this is in mozilla/config.py; we have the rest of the b2g builds in mozilla/b2g_config.py .  I don't want to delay these, but if you have some spare cycles to try moving this into b2g_config.py, that will help keep people from getting [more] confused when looking at our configs.  At the very least I'd like a followup bug to move mulet into b2g_config.py; it would be even better if that bug has details about what breaks if we try moving it, for posterity.  And even better still if it Just Works.
Attachment #8438775 - Flags: review?(aki) → review+
Comment on attachment 8438775 [details] [diff] [review]
add unit test sendchanges

https://hg.mozilla.org/build/buildbot-configs/rev/c4ab10c33bfa

I don't think enough spare cycles as the "all tests" work has taking me way longer than expected. I'm hoping to focus now on blobber discoverability.
Attachment #8438775 - Flags: checked-in+
In prod with reconfig on 2014-06-12 10:46 PT
Thanks Simone!
I've triggered another build.

We should see tests being triggered after it:
https://tbpl.mozilla.org/?tree=Fig
We have sendchanges but don't trigger anything.
I will grab a master and two set of slaves to make sure that this works beyond my local master.

I think something like this is needed:
+++ b/buildfarm/maintenance/production-masters.json
@@ -307,4 +307,5 @@
     "limit_fx_platforms": [
       "linux64",
+      "linux64-mulet",
       "linux64-asan"
     ],
@@ -313,4 +314,7 @@
         "ubuntu64_vm"
       ],
+      "linux64-mulet": [
+        "ubuntu64_vm"
+      ],
       "linux64-asan": [
         "ubuntu64-asan_vm"
Created attachment 8440765 [details] [diff] [review]
Add linux64-mulet to the list of platforms for linux64 masters

I guess it worked for me when I didn't have limiting platforms for my development master.
Attachment #8440765 - Flags: review?(aki)
Created attachment 8440766 [details] [diff] [review]
fix the unittest branch
Attachment #8440766 - Flags: review?(aki)

Updated

3 years ago
Attachment #8440765 - Flags: review?(aki) → review+

Updated

3 years ago
Attachment #8440766 - Flags: review?(aki) → review+
Comment on attachment 8440765 [details] [diff] [review]
Add linux64-mulet to the list of platforms for linux64 masters

https://hg.mozilla.org/build/tools/rev/0e1aa44efd68
Attachment #8440765 - Flags: checked-in+
Comment on attachment 8440766 [details] [diff] [review]
fix the unittest branch

https://hg.mozilla.org/build/buildbot-configs/rev/dd5829e1d377
Attachment #8440766 - Flags: checked-in+
buildbot-config patch live in production :)
The test results are showing.

Time to green them up? Should we file a new bug for that?
(Reporter)

Comment 90

3 years ago
\o/

I already have patches for that but I need some feedback from someone, I don't know who.
We need to filter out tests that shouldn't be running on mulet.
In my patch I introduce a new magic keyword `mulet` within *.ini test manifest files,
but I'm not sure that's the right way to proceed.

https://github.com/ochameau/mozilla-central/commit/3b9e0fc310ef45dabcd1a9c752ad055001c6b214

Do you know who should I bug about that and in which component should I fill a bug and attach such patch?
Let's file a new bug for greening up the tests.

ochameau, you can file a bug under Testing:General for the manifest changes (since they span multiple harnesses) and flag :ahal for review.
(Assignee)

Updated

3 years ago
Blocks: 1027242
Filed bug 1027242 for it.
Status: REOPENED → RESOLVED
Last Resolved: 3 years ago3 years ago
Resolution: --- → FIXED
Summary: Start doing mulet builds → Start doing mulet builds on Fig
See Also: → bug 1031085
Created attachment 8450210 [details] [diff] [review]
Fix naming

r=RyanVM
https://hg.mozilla.org/build/buildbot-configs/rev/442454f765a3

As per https://bugzilla.mozilla.org/show_bug.cgi?id=1016448#c3
-    'name': 'Ubuntu Mulet VM 12.04 x64',
+    'name': 'Ubuntu VM 12.04 x64 Mulet',

-            'base_name': 'Linux Mulet x86-64 %(branch)s',
+            'base_name': 'Linux x86-64 Mulet %(branch)s',
Attachment #8450210 - Flags: review+
Waiting for reconfig.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Whiteboard: [leave open]
Merged to production, and deployed.
Naming fixed!
Status: REOPENED → RESOLVED
Last Resolved: 3 years ago3 years ago
Resolution: --- → FIXED
(Assignee)

Updated

3 years ago
Depends on: 1062501
You need to log in before you can comment on or make changes to this bug.