760168 - buildbot changes for desktop b2g builds

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P2)

Description

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: buildbot-configs v1

There are some changes that need to be made for B2G desktop builds.

I have attached a patch that I think is all that is needed to do this.  The mozconfigs that are needed to build are being added to the tree in bug 744008, so testing will require the mozconfigs from that bug.  The structure of the patch is to add '-b2g' platforms that have the customizations needed for b2g to build under that platform.  Also worth mentioning is that Windows will be permanently red until 744008 is fixed.  Please leave the builds enabled so work can be done to fix those failures.

Some differences between the '-b2g' and it's corresponding firefox platform:

1) No sendchanges.  There are no test builders (currently) set up for these b2g builds.  Once tests are set up for b2g builds on the test masters, sendchanges should be sent

2) I don't know if signing is expected to work or not.  Signing with the same credentials (or at least the same type of cert) as desktop Firefox would be nice, but not strictly needed for now

3) Nightly updates are desirable, not sure if this patch will enable them or if work is needed for follow up AUS changes.  I also don't know if the product is set up to use the plain AUS.  Please review this patch with extra attention to it not breaking desktop nightly updates

4) I don't know whether to upload symbols to the symbols server.  Let's start with uploading to the symbols server disabled for now, while building the tarballs and uploading them to FTP.  My understanding is that the tarballs will still be built and uploaded to ftp because packageTests is set to true [1]

I am not able to test these is staging but do get a clean ./setup-masters.py -t run.  Can someone please validate these changes in staging so I can land them on buildbot-configs please?  I am marking this bug as critical because it is very important that we get something that Gaia and App developers can easily test locally with.

[1] http://hg.mozilla.org/build/buildbotcustom/file/7d0cd546fb77/process/factory.py#l985

Comment 1

[Chris AtLee [:catlee]]

Comment on attachment 628780 [details] [diff] [review]
buildbot-configs v1

Review of attachment 628780 [details] [diff] [review]:
-----------------------------------------------------------------

as discussed yesterday, this is better done as a separate product like Thunderbird. At the very lease the aus base directory needs to be different.

Comment 2

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: buildbot-configs v2

This patch has nightly updates disabled.

Comment 3

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Chris AtLee [:catlee] from comment #1)
> Comment on attachment 628780 [details] [diff] [review]
> buildbot-configs v1
> 
> Review of attachment 628780 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> as discussed yesterday, this is better done as a separate product like
> Thunderbird. At the very lease the aus base directory needs to be different.

Chris, I think the pragmatic approach is to use a patch structured like the one attached.  It follows a pattern we'vebeen using for a long time for adding variant builds.  We have used this pattern to add Fennec builds,QT builds and RPM builds to the automation.  My understanding for the thunderbird structure was due to the necessity of comm-central integration which is not a concern for B2G desktop builds.

I also don't know how you'd want to structure a separate project to best take Release Engineering maintainability concerns into account.  Would you be doing the same for Fennec?  Desktop B2G builds relate to desktop Firefox builds in the same way that Fennec builds relate to desktop Firefox builds.

It's very important that we have these desktop builds generated as soon as possible to allow Web, Gaia and App developers a place to develop and validate their contributions without having to build and flash an expensive phone

I have attached a patch that removes the nightly update portion of the patch, which seems to be the sticking point.  Let's file a different bug for nightly updates should those become a priority

Comment 4

[Dietrich Ayala (:dietrich)]

Hey Chris, moving this bug forward is a huge part of getting B2G in the hands of nightly testers. We need to do the minimum possible to get nightly builds of desktop-b2g available asap. What is actually blocking the landing of this now?

Also, what's the sticking point for nightly updates? That makes the feedback for the nightly tester very easy, which is what we want. Can we figure that out in this bug? If that requires some massive other piece of work then I guess we could split it out, but I'd rather get it all resolved now if possible.

Comment 5

[Dietrich Ayala (:dietrich)]

Talked with Joduinn, who clarified that the current patch requires re-testing of all product builds whenever the config changes (IIUC?), which seems costly. John, what amount of effort is required to make B2G builds more independent from the other product builds?

Comment 6

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Dietrich Ayala (:dietrich) from comment #5)
> Talked with Joduinn, who clarified that the current patch requires
> re-testing of all product builds whenever the config changes (IIUC?), which
> seems costly. John, what amount of effort is required to make B2G builds
> more independent from the other product builds?

That is not correct.  The patch, as is, makes B2G completely independent from non-B2G product builds.  Only branch level settings (like where the code is, whether to do nightlies) would be shared.  Sharing branch level changes is a feature not a defect.  This approach is exactly the same approach that we took for fennec-desktop builds, so we already know that it works just fine and wasn't a major time-sink.

I've looked at the product approach taken for thunderbird.  This approach makes sense for Thunderbird because it is different automation running with different repositories.  If we took this approach for B2G, almost every single branch-level variable would be identical to the Firefox/Fennec ones.  This means that there is either a lot more overhead required from Releng or that B2G will always lag behind Firefox/Fennec.

Because of a namespace collision, we'd also need to call the B2G branches something like "mozilla-central2", "maple2", etc.  There would be duplication of all the schedulers and change sources.  This would add lots of extra load to the scheduling masters and on the already way-overloaded hg.mozilla.org installation.

If Releng does not want my contribution, that's fine.  I am trying to help out since Releng is prioritizing non-b2g projects.

Comment 7

[Chris AtLee [:catlee]]

Comment on attachment 629844 [details] [diff] [review]
buildbot-configs v2

Review of attachment 629844 [details] [diff] [review]:
-----------------------------------------------------------------

Just because mobile does it this way doesn't mean it's a good idea. If we had the time, we would love to refactor the mobile code to be more like thunderbird configs.

Your point about having name collisions with branch names is a valid one, I'm not sure how to handle that. However, b2g is a separate product, and so mixing it in with the firefox configs is going to cause us lots of future maintenance issues. We should get this right from the start.

Comment 8

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Chris AtLee [:catlee] from comment #7)
> Comment on attachment 629844 [details] [diff] [review]
> buildbot-configs v2
> 
> Review of attachment 629844 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Just because mobile does it this way doesn't mean it's a good idea. If we
> had the time, we would love to refactor the mobile code to be more like
> thunderbird configs.
> 
> Your point about having name collisions with branch names is a valid one,
> I'm not sure how to handle that. However, b2g is a separate product, and so
> mixing it in with the firefox configs is going to cause us lots of future
> maintenance issues. We should get this right from the start.

I'm the first to say that this is a pragmatic approach, not the most theoretically 'clean' approach.  The Fennec builds are just as 'mixed in' as these b2g builds would be and as you acknowledge, the approach you've suggested has a known issue with unknown remedy.  We also *already* have b2g builds going with this exact structure!

Comment 9

[Chris Cooper [:coop] (he/him)]

(In reply to John Ford [:jhford] from comment #6)
> If Releng does not want my contribution, that's fine.  I am trying to help
> out since Releng is prioritizing non-b2g projects.

To be frank, we're not. We see this train coming a mile away and are trying to build some tracks that we won't need to tear up and re-lay in August.

jhopkins is still reeling from having to do this for Thunderbird, but I'm going to leverage his experience to try to do the config split for b2g.

Comment 10

[[e-mail] (:bhearsum)]

We did a bunch of ground work in bug 758425 that should make this easier.

Comment 11

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Ben Hearsum [:bhearsum] from comment #10)
> We did a bunch of ground work in bug 758425 that should make this easier.

Actually, it makes the patch slightly more work.  Patch for desktop builds will be posted shortly.  Who should I flag for review?

Comment 12

[John Ford [:jhford] CET/CEST Berlin Time]

Attachment: modify patch to use new b2g configuration

Enable B2G desktop builds.  I have tested this as best as I can, which is ./setup-master.py -t and starting up the buildbot master looking at the twistd.log and the webui 'builders' page.

I would have named the platforms without '-b2g', except that there is a collision between them and the firefox builder directories.  This does not have nightly updates enabled.  Let's get builds going then look at updates.  I think there are product code changes that are required before updates will do something useful (at least the app.update.url needs to be changed).

Ben, I've asked you for review because it looks like you are most familiar with this code.

Comment 13

[John Ford [:jhford] CET/CEST Berlin Time]

Ping?

Comment 14

[Chris Cooper [:coop] (he/him)]

Re-assigning based on who is actually submitting patches here.

Comment 15

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

per irc, jhford is focusing on bug#769315 instead so reassigning.

Comment 16

[[e-mail] (:bhearsum)]

Attachment: add desktop build support to buildbotcustom

Comment 17

[[e-mail] (:bhearsum)]

Attachment: turn on windows and mac desktop builds

I also removed the SLAVES/TRY_SLAVES definitions because I realized they fully duplicated the ones from the Firefox configs. May as well just them instead. Linux is still to come, after we figure out how to build them in mock.

Windows won't work until bug 760138 is fixed.

Also depends on mozconfig re-arrangement happening in bug 744008.

Comment 18

[[e-mail] (:bhearsum)]

Attachment: enable desktop platforms; create enableInstaller parameter

Comment 19

[[e-mail] (:bhearsum)]

Having some trouble with buildsymbols on Linux, not sure if it's because I'm using a user repo or what....

./dist/bin/triplescat, section '.eh_frame': the call frame entry at offset 0x18 uses a DWARF expression to describe how to recover register '.cfa',  but this translator cannot yet translate DWARF expressions to Breakpad postfix expressions
abort: .hg/sharedpath points to nonexistent directory /builds/hg-shared/users/catlee_mozilla.com/mozilla-central/.hg!
Failed to get HG Repo for /builds/slave/m-cen-linux32-gecko-ntly/build
Could not determine repo info for /builds/slave/m-cen-linux32-gecko-ntly/build.  This is either not a clone of the web-based
repository, or you have not specified SRCSRV_ROOT, or the clone is corrupt.
Unexpected error:  <type 'exceptions.SystemExit'>
error writing symbol file: Broken pipe
make: *** [buildsymbols] Error 1
Failed to write symbol file.


[cltbld@bld-centos6-hp-003 build]$ cd /builds/slave/m-cen-linux32-gecko-ntly/build
[cltbld@bld-centos6-hp-003 build]$ cat .hg/
branch      cache/      dirstate    hgrc        requires    sharedpath  
[cltbld@bld-centos6-hp-003 build]$ cat .hg/hgrc
[paths]
default = http://hg.mozilla.org/users/catlee_mozilla.com/mozilla-central
[cltbld@bld-centos6-hp-003 build]$ cat .hg/sharedpath 
/builds/hg-shared/users/catlee_mozilla.com/mozilla-central/.hg[cltbld@bld-centos6-hp-003 build]$ cat /builds/hg-shared/users/catlee_mozilla.com/mozilla-central/.hg/hgrc
[paths]
default = http://hg.mozilla.org/users/catlee_mozilla.com/mozilla-central

Comment 20

[[e-mail] (:bhearsum)]

Attachment: enable linux32, mac, windows desktop b2g+gecko builds

Pretty straightforward, I think:
* Remove unnecessary SLAVES/TRY_SLAVES creation in favour of re-using the ones created by the other localconfigs.
* Enable b2g+gecko builds on linux32, macosx64, and win32. The Linux ones use Mock, we'll need to deploy ssh keys to those slaves before landing.
* Set enable_installer where necessary.

I ran builds in staging. The mac one was fully green, the Windows one burnt because of bad ssh config, and the Linux one failed at 'make buildsymbols'. I don't think either of those problems will be an issue in production.

Comment 21

[[e-mail] (:bhearsum)]

Comment on attachment 638763 [details] [diff] [review]
enable linux32, mac, windows desktop b2g+gecko builds

Landed on default and merged to production. These should be enabled very soon.

Comment 22

[[e-mail] (:bhearsum)]

These are enabled and mostly working. There's a Linux compilation failure tracked in bug 770624, and there may be a follow-up issue with Mac. Let's track those in individual bugs.