916111 - Do debug B2G desktop builds

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Chris AtLee [:catlee]]

We need to build debug variants of all our desktop B2G builds.

Comment 1

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

+1, I was about to file a new bug when b.m.o's duplicate check feature found this.

This should allow folks to run Marionette using a debug Gaia profile present in these debug B2G builds.

Comment 2

[Ryan VanderMeulen [:RyanVM]]

Is the plan for just builds or tests as well?

Comment 3

[Chris AtLee [:catlee]]

Tests too

Comment 5

[Laura Thomson :laura]

Simone, can you update this bug for status?

Comment 6

[Simone Bruno [:simone]]

Sure! Final patches for linux OS's and Mac OSX upcoming today; plan is to complete the windows one as well by the end of the week.

Comment 9

[Simone Bruno [:simone]]

Attachment: buildbot-configs_WIP.diff

Work In Progress - Patch to buildbot-configs doing the following:
- add linux32_gecko-debug and win32_gecko-debug platforms and relative configuration sections
- explicitly disable nightly builds on m-c, aurora, mozilla-b2g28_v1_3t and mozilla-b2g28_v1_3
- remove code which enabled linux64 and macOS builds on m-c and aurora only (violating visibility rules); now they should run on the same branches where the regular desktop builds run

WIP since I need to run tests on the two newly added platforms in staging environment.

For Aki:
- What repos should I add the mozconfig debug files to, in order not to break any of the builds?

Comment 10

[Aki Sasaki (not active)]

(In reply to Simone Bruno [:simone] from comment #9)
> Created attachment 8408340 [details] [diff] [review]
> buildbot-configs_WIP.diff
> 
> Work In Progress - Patch to buildbot-configs doing the following:
> - add linux32_gecko-debug and win32_gecko-debug platforms and relative
> configuration sections
> - explicitly disable nightly builds on m-c, aurora, mozilla-b2g28_v1_3t and
> mozilla-b2g28_v1_3

You might be able to do that via enable_nightly=False inside the *_gecko-debug config blocks (you're setting enable_dep=True; you can set enable_nightly=False).

> - remove code which enabled linux64 and macOS builds on m-c and aurora only
> (violating visibility rules); now they should run on the same branches where
> the regular desktop builds run

You probably want to remove them, but with an if statement like


    for name, branch in items_before(BRANCHES, 'gecko_version', 30):

which will get everything older than mozilla-aurora.

> 
> WIP since I need to run tests on the two newly added platforms in staging
> environment.
> 
> For Aki:
> - What repos should I add the mozconfig debug files to, in order not to
> break any of the builds?

b2g-inbound will get all the trunk-level repos after some time.  You'll explicitly need to uplift to mozilla-aurora.

Comment 11

[Simone Bruno [:simone]]

Attachment: buildbot-configs.diff

Given the changes occurred since last week in the underlining code-base, this patch became simpler: just adding sections for debug builds, disabling nightlies in the main configuration section for each, and enabling only for gecko_version >= 30.

Comment 12

[Simone Bruno [:simone]]

Attachment: win32_debug_mozconfig

mozconfig file to be used for win32 gecko debug build

Comment 13

[Simone Bruno [:simone]]

Attachment: mozconfig_linux32_gecko-debug

Comment 14

[Aki Sasaki (not active)]

Comment on attachment 8410483 [details] [diff] [review]
buildbot-configs.diff

As long as you land the mozconfigs this week, aurora is good.  If it's after Monday, you'll need to land them in mozilla-b2g30_v1_4.

This should [hopefully] handle all the builds.  Now we need tests.
That'll involve setting enable_debug_unittests to True:
http://hg.mozilla.org/build/buildbot-configs/file/e1778eb13d63/mozilla-tests/b2g_config.py#l620
ideally we also turn that off on the older-than-1.4 branches to avoid creating unnecessary builders.  We also need to populate the debug_unittest_suites: http://hg.mozilla.org/build/buildbot-configs/file/e1778eb13d63/mozilla-tests/b2g_config.py#l623

Let me know if you have questions about that.

Comment 15

[Aki Sasaki (not active)]

Jonathan: same unittest suites for desktop debug builds?

Comment 16

[Jonathan Griffin (:jgriffin)]

(In reply to Aki Sasaki [:aki] from comment #15)
> Jonathan: same unittest suites for desktop debug builds?

Yes, please.  We may have to disable some tests to get green, but that's a good starting point.

Comment 17

[Simone Bruno [:simone]]

Thanks Aki, Jonathan.
I will add a new patch to enable unittests first thing tomorrow morning.

Comment 18

[Simone Bruno [:simone]]

Attachment: buildbot-configs_II_unittests.diff

- enabling unittests
- defining unittest suites
- disabling unit test suites on older branches

Comment 19

[Aki Sasaki (not active)]

Comment on attachment 8410890 [details] [diff] [review]
buildbot-configs_II_unittests.diff

Thanks!  I wasn't able to apply this due to bitrot -- if you can run test-master.sh before landing, great.  If not, a) we should get you able to do so in the future, and b) watch releng-ci in #releng for a green result.

Comment 20

[Simone Bruno [:simone]]

mozconfigs for linux32 and win32 pushed to aurora, now pushing to b2g-inbound

Comment 21

[Simone Bruno [:simone]]

mozconfig files checked in with commits:
http://hg.mozilla.org/integration/b2g-inbound/rev/6c67c3dec3d0
http://hg.mozilla.org/releases/mozilla-aurora/rev/4dc5063a4574

Comment 22

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/6c67c3dec3d0

Comment 23

[Aki Sasaki (not active)]

We still need the buildbot bits aiui.

Comment 24

[Simone Bruno [:simone]]

Buildbot part pushed to default branch of buildbot-configs as well:
https://hg.mozilla.org/build/buildbot-configs/rev/18feb0e40ad8
plus extra bug fix:
https://hg.mozilla.org/build/buildbot-configs/rev/67c339f110df

Comment 25

[Simone Bruno [:simone]]

Attachment: enable_tests_on_cedar_only.diff

As agreed with RyanVM and coop (in #releng), these tests need to be enabled on cedar only until they green up before enabling them in prod.

Comment 26

[Simone Bruno [:simone]]

Comment on attachment 8416041 [details] [diff] [review]
enable_tests_on_cedar_only.diff

https://hg.mozilla.org/build/buildbot-configs/rev/f6924a6fa84a

Comment 27

[Chris Cooper [:coop] (he/him)]

Merged and deployed to production.

Comment 28

[Ed Morley [:emorley]]

Per bug 1004610 comment 9, we're now ok to enable Linux64 (only) debug gaia-ui-test on all trunk trees (or at the least: {mozilla-central, mozilla-inbound, fx-team, b2g-inbound, try}).

Comment 30

[Simone Bruno [:simone]]

Attachment: enable_gaia_ui_tests_for_linux64

Comment 31

[Simone Bruno [:simone]]

Attachment: enable_gaia_ui_tests_for_linux64

Comment 32

[Simone Bruno [:simone]]

Comment on attachment 8421845 [details] [diff] [review]
enable_gaia_ui_tests_for_linux64

Checked in: https://hg.mozilla.org/build/buildbot-configs/rev/3fc51abcc61a

Comment 33

[[e-mail] (:bhearsum)]

something here is in production

Comment 34

[Phil Ringnalda (:philor)]

Simone: bad news, that patch didn't turn back on just gaia-ui-tests, it turned everything back on. We're now back to running all of the still-broken jobs on every tree.

(Worse news, though you couldn't have expected it: turning something back on that quickly, before the thing that cleans up old schedulers gets a chance to remove the old ones, causes us to schedule all the jobs between the time they were turned off and when they were turned back on, so we're currently trying to run the intervening 3000-odd jobs.)

Comment 35

[Aki Sasaki (not active)]

Ugh, I should have (or you could have) run builder_list.py and diffed:
https://wiki.mozilla.org/ReleaseEngineering:TestingTechniques#builder_list.py_.2F_dump_master.py
https://bugzilla.mozilla.org/show_bug.cgi?id=1009517#c3

Comment 36

[Aki Sasaki (not active)]

Comment on attachment 8421845 [details] [diff] [review]
enable_gaia_ui_tests_for_linux64

Backed out: https://hg.mozilla.org/build/buildbot-configs/rev/86a72a6bb486

Comment 37

[Simone Bruno [:simone]]

The problem was... a missing comma while looping on slave_platforms :-|

I am now testing the patch as recommended by aki in comment 35 before submitting for review.

Comment 38

[Simone Bruno [:simone]]

Attachment: enable_gaia_ui_tests_for_linux64_take_02

This time I tested this using braindump/buildbot-related/dump_masters.sh to verify that only "gaia-ui-test" tests are added on non-cedar branches.

Comment 39

[Aki Sasaki (not active)]

Comment on attachment 8423229 [details] [diff] [review]
enable_gaia_ui_tests_for_linux64_take_02

I also verified we're only adding gaia-ui-tests, across linux, linux64, and macosx test masters :)

>+            for slave_platform in (('linux64_gecko', 'ubuntu64_vm-b2gdt'),):

This is an odd construct for a single tuple (i.e., why are we running a for-loop once?)
I'm currently guessing we might want to add more slave_platforms in here later?

If so, can you put a space after the final comma for pep8-niceness?

    ... 'ubuntu64_vm-b2gdt'), ):

Comment 40

[Simone Bruno [:simone]]

Thanks aki!
That was my (arguable) idea behind the looping on a one-item tuple: keep the code ready for simple changes when we will enable more tests as soon as they are fixed on cedar.

Checked in: https://hg.mozilla.org/build/buildbot-configs/rev/0f6a6991ad9e

Comment 41

[Aki Sasaki (not active)]

Simone: jgriffin says we can mark this bug RESO FIXED.

* I see debug b2g desktop builds here: https://tbpl.mozilla.org/?showall=1&rev=9d8d16695f6a
* we can have a separate bug to enable desktop debug tests
* we may be shifting gears to emphasize mulet builds+tests instead of b2g desktop

Thanks!  (I'm leaving it REOPENED in case you wanted to resolve it yourself.)

Comment 42

[Simone Bruno [:simone]]

Thanks a lot Aki!

I create Bug 1013914 (Enable tests for debug B2G desktop builds as soon as they are green on Cedar) to track enabling of tests on platfroms other than linux64.

Comment 43

[Firefox Bug Husbandry Bot]

Removing leave-open keyword from resolved bugs, per :sylvestre.