793628 - Produce Lightning Nightly builds during Thunderbird build process

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Philipp Kewisch [:Fallen][📱📆]]

At the Mozcamp in Warsaw we discussed that to lessen the burden of building Lightning, we should attempt to integrate Lightning builds into Thunderbird.

In an ideal world, all that is needed is adding --enable-calendar and setting DISABLE_LIGHTNING_INSTALL=1. Since bug 724257 the make upload target should also be uploading Lightning if --enable-calendar is specified.

Open questions, some of these to be handled in dependent bugs:

* Where do we run unit and mozmill tests so that failing Lightning tests don't
  disrupt Thunderbird?
* Lightning build issues are not very common, but is it ok that Lightning build
  failures will also fail a Thunderbird build?
* Lightning l10n works a bit differently
  * We can repack single l10n builds just like Thunderbird (needs hookup)
  * Release builds will need a separate job that merges all single l10n builds
* Merging 32/64 bit builds still needs to happen in a separate job

As a first step, I would appreciate if someone (John?) could comment on how to best get this rolling. Personally I think we should start with the nightly builds and see how this works out, then advance to the beta builds and then to the release builds.

Comment 1

[Mark Banner (:standard8)]

I think this would be better tracked in releng land, as it is their infrastructure that's affected here.

Comment 2

[John Hopkins (:jhopkins)]

(In reply to Philipp Kewisch [:Fallen] from comment #0)
...
> Open questions, some of these to be handled in dependent bugs:
> 
> * Where do we run unit and mozmill tests so that failing Lightning tests
> don't disrupt Thunderbird?

Can you explain 'disrupt Thunderbird' a bit more?

> * Lightning build issues are not very common, but is it ok that Lightning
> build failures will also fail a Thunderbird build?

If Lightning is going to be a "first class" addon for Thunderbird (in that Lightning bustage blocks a Thunderbird release) then I believe the answer is yes.  Has that been decided?

> * Lightning l10n works a bit differently
>   * We can repack single l10n builds just like Thunderbird (needs hookup)
>   * Release builds will need a separate job that merges all single l10n
> builds
> * Merging 32/64 bit builds still needs to happen in a separate job

Are there existing Buildbot factories that can do this?

> As a first step, I would appreciate if someone (John?) could comment on how
> to best get this rolling. Personally I think we should start with the
> nightly builds and see how this works out, then advance to the beta builds
> and then to the release builds.

That sounds like a good way to start.  Also, if possible it would be good if we could get most of this done with updated 'make' targets so as not to over-complicate the buildbot automation or take too much releng time (which is scarce with all the other work going on right now).  Where that isn't possible, I'd like for us to come up with a list of things that need to be done on the buildbot automation side.

Comment 3

[Philipp Kewisch [:Fallen][📱📆]]

(In reply to John Hopkins (:jhopkins) from comment #2)
> (In reply to Philipp Kewisch [:Fallen] from comment #0)
> ...
> > Open questions, some of these to be handled in dependent bugs:
> > 
> > * Where do we run unit and mozmill tests so that failing Lightning tests
> > don't disrupt Thunderbird?
> 
> Can you explain 'disrupt Thunderbird' a bit more?

Disrupt may be a bit harsh. I just mean that a failing Lightning test will make the build go orange, even if all Thunderbird tests were successful. I talked to Mark about this and he said it would be fine.

> 
> > * Lightning build issues are not very common, but is it ok that Lightning
> > build failures will also fail a Thunderbird build?
> 
> If Lightning is going to be a "first class" addon for Thunderbird (in that
> Lightning bustage blocks a Thunderbird release) then I believe the answer is
> yes.  Has that been decided?
I guess thats up to us and the remaining community to decide. Given most of our build failures are because of build config issues and not actual code issues, I would decide yes.

> > * Lightning l10n works a bit differently
> >   * We can repack single l10n builds just like Thunderbird (needs hookup)
> >   * Release builds will need a separate job that merges all single l10n
> > builds
> > * Merging 32/64 bit builds still needs to happen in a separate job
> 
> Are there existing Buildbot factories that can do this?
Not at the moment. I can hack up some factories to do this though, right now the 32/64 bit linux builds are being merged by a simple script.


> 
> > As a first step, I would appreciate if someone (John?) could comment on how
> > to best get this rolling. Personally I think we should start with the
> > nightly builds and see how this works out, then advance to the beta builds
> > and then to the release builds.
> 
> That sounds like a good way to start.  Also, if possible it would be good if
> we could get most of this done with updated 'make' targets so as not to
> over-complicate the buildbot automation or take too much releng time (which
> is scarce with all the other work going on right now).  Where that isn't
> possible, I'd like for us to come up with a list of things that need to be
> done on the buildbot automation side.

Indeed, I think we can get most of this working using the standard make targets. make upload is already hooked up, so if tests and l10n is ignored, then creating a nightly en-US should already be possible.

I can probably get a try build running to proove it.

Comment 4

[Philipp Kewisch [:Fallen][📱📆]]

Attachment: Patch for comm-central mozconfigs - v1

John had this run through on aurora and there were no notable differences. I did a compare of the current nightly lightning and the produced Lightning and all that is different is build id and some paths in preprocess comments. The same goes for Thunderbird, with some minor differences:

* I couldn't check actual differences in binary code, I assume it will be ok
* In thunderbird/chrome/toolkit/content/global/buildconfig.html the option
  --enable-calendar is shown.

I think this is enough to get comm-central and comm-aurora nightly builds running on moco machines. I'd appreciate a quick review since our builds are broken.

I suggest enabling for central first and waiting for all platforms to go green, then doing the same for aurora.

Comment 5

[Philipp Kewisch [:Fallen][📱📆]]

Attachment: Patch for comm-aurora mozconfigs - v1

[Approval Request Comment]
Bug caused by (feature/regressing bug #): bug 793634 
User impact if declined: Red builds
Testing completed (on m-c, etc.): tested builds by jhopkins on comm-aurora

Comment 6

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 668425 [details] [diff] [review]
Patch for comm-aurora mozconfigs - v1

you want comm-approval here, not approval-mozilla-aurora but I can't set the flag here for some reason probably cause it's a releng bug.

Comment 7

[Philipp Kewisch [:Fallen][📱📆]]

Attachment: Fix upload location for nightly and tinderbox builds - v1

This patch has run through staging and puts lightning at the right spot. The gdata parts are new, but follow the same concept.

I think this is ready to land, lets start with comm-central. Mark, can you take a look?

Comment 8

[Mark Banner (:standard8)]

Comment on attachment 670954 [details] [diff] [review]
Fix upload location for nightly and tinderbox builds - v1

Review of attachment 670954 [details] [diff] [review]:
-----------------------------------------------------------------

So before we can deploy these patches, we need to sort out about ssh keys. I've just tried this on the try server, and upload failed on Mac & Linux as it was attempting to use the Thunderbird ssh keys to upload to calendar/lightning, which just won't work.

We're either going to need to change the permissions there, or find a way to use a different set of ssh keys...

::: calendar/lightning/lightning-packager.mk
@@ +134,4 @@
>  
> +# Lightning uses Thunderbird's build machinery, so we need to hack the post
> +# upload command to use Lightning's directories.
> +upload: POST_UPLOAD_CMD := $(subst thunderbird,calendar/lightning,$(POST_UPLOAD_CMD))

This doesn't work on Windows, probably because of the "/".

Comment 9

[Philipp Kewisch [:Fallen][📱📆]]

For

Comment 10

[Philipp Kewisch [:Fallen][📱📆]]

For the ssh keys issue, see bugs in previous comment.

I've submitted a new try build with a possible fix for the windows directory issue.

https://tbpl.mozilla.org/?tree=Thunderbird-Try&rev=40afc721c97d

Comment 11

[Philipp Kewisch [:Fallen][📱📆]]

Attachment: Fix upload location for nightly and tinderbox builds - v2

This patch works for windows. Probably since windows starts a new shell, the env var isn't updated. This patch ensures that the changed variable is passed.

Comment 12

[Philipp Kewisch [:Fallen][📱📆]]

Here is the try run: https://tbpl.mozilla.org/?tree=Thunderbird-Try&rev=40afc721c97d

Comment 13

[Philipp Kewisch [:Fallen][📱📆]]

Comment on attachment 674377 [details] [diff] [review]
Fix upload location for nightly and tinderbox builds - v2


>+upload: stage_upload
>+	echo $(POST_UPLOAD_CMD)
>+	POST_UPLOAD_CMD="$(POST_UPLOAD_CMD)" \

I have removed that extra "echo" command locally.


Mark, any chance you could take a look at this again?

Comment 14

[Philipp Kewisch [:Fallen][📱📆]]

Comment on attachment 674377 [details] [diff] [review]
Fix upload location for nightly and tinderbox builds - v2


Pushed upload location patch:
https://hg.mozilla.org/comm-central/rev/4fa5f8fb07e7

Comment 15

[Mark Banner (:standard8)]

Comment on attachment 668423 [details] [diff] [review]
Patch for comm-central mozconfigs - v1

Ok, lets try this on trunk to being with and give it a few days to check on how it all goes.

Comment 16

[Philipp Kewisch [:Fallen][📱📆]]

Comment on attachment 668423 [details] [diff] [review]
Patch for comm-central mozconfigs - v1

https://hg.mozilla.org/comm-central/rev/8285f35f9fd5

*crossing fingers*

Comment 17

[Philipp Kewisch [:Fallen][📱📆]]

Comment on attachment 668425 [details] [diff] [review]
Patch for comm-aurora mozconfigs - v1

Taking this from your queue until its relevant

Comment 18

[Philipp Kewisch [:Fallen][📱📆]]

Temporarily disabled comm-central calendar builders here: 

http://hg.mozilla.org/build/buildbot-configs/rev/4f433037c0aa

Comment 19

[Stefan Sitter [:ssitter]]

Seems to work to some degree. Lot of new comm-central-* folders and builds have appeared on https://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/tinderbox-builds/

Were the changes to the .xpi name done on purpose?
> .../tinderbox-builds/lightning/tinderbox-builds/comm-central-win32/1351755294
> gdata-provider-.en-US.win32.xpi
> lightning-2.1a1.en-US.win32.xpi

Comment 20

[Philipp Kewisch [:Fallen][📱📆]]

Yes, the xpi name changes were done on purpose to line up with what Thunderbird has. I believe I also had another reason I don't recall right now.

There was an intermediate error due to missing directory permissions and I am not quite sure yet if the nightly bustage is because of a missing rebuild or a real error.

Comment 21

[Stefan Sitter [:ssitter]]

Ok, it just means you fixed Bug 331274 although you resolved that bug as invalid before :)

As you can see in comment 19 the version number for win32 gdata-provider is not set at all, probably the same regression as reported from linux builds in Bug 799643.

It seems that nightly builds were produced and are stored in root of https://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/nightly/latest-comm-central/ instead of platform dependent folders.

Comment 22

[Mark Banner (:standard8)]

(In reply to Stefan Sitter from comment #21)
> It seems that nightly builds were produced and are stored in root of
> https://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/nightly/latest-
> comm-central/ instead of platform dependent folders.

That doesn't really matter now that they have the platform in their names, does it? It means they are quicker to find if you're navigating directories (although external links will need to be updated, of course).

Comment 23

[Philipp Kewisch [:Fallen][📱📆]]

(In reply to Stefan Sitter from comment #21)
> Ok, it just means you fixed Bug 331274 although you resolved that bug as
> invalid before :)
You have brilliant memory :-) I think one of the reasons was that it was extra work to get those conventions followed, and now we get that for free.


> As you can see in comment 19 the version number for win32 gdata-provider is
> not set at all, probably the same regression as reported from linux builds
> in Bug 799643.
Taken care of.

Comment 24

[Stefan Sitter [:ssitter]]

Could you remove the now obsolete subfolders in https://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/nightly/latest-comm-central/ to not cause confusion?

Comment 25

[Stefan Sitter [:ssitter]]

There seems to be a problem with multi-platform builds.

The old package at latest-comm-central/macosx-xpi/lightning.xpi supports both Darwin_x86-gcc3 and Darwin_x86_64-gcc3 platform.

The new package at latest-comm-central/lightning-2.1a1.en-US.mac.xpi supports only Darwin_x86-gcc3.

Should I file a new bug report?

Comment 26

[Philipp Kewisch [:Fallen][📱📆]]

(In reply to Stefan Sitter from comment #25)
> There seems to be a problem with multi-platform builds.
> 
> The old package at latest-comm-central/macosx-xpi/lightning.xpi supports
> both Darwin_x86-gcc3 and Darwin_x86_64-gcc3 platform.
> 
> The new package at latest-comm-central/lightning-2.1a1.en-US.mac.xpi
> supports only Darwin_x86-gcc3.
> 
> Should I file a new bug report?

I'll take care of this here. The problem is that I did't add MOZ_POSTFLIGHT_ALL variables for Lightning in the mac mozconfigs. I'll also need to adapt our universal.mk so it works with the new package names.

Comment 27

[Philipp Kewisch [:Fallen][📱📆]]

Attachment: Fix mac universal builds - v1

This patch should take care of the universal builds. I've tested this on comm-central, not using the original process but something vaguely similar. Not asking for review yet since I haven't been able to test it on stage, but will get back to this when my other stage tasks are done.

Comment 28

[Darren Kenny]

Any timeline on when this might be resolved in the nightly builds?

Comment 29

[Philipp Kewisch [:Fallen][📱📆]]

Although it might not look like anything is happening by this bug, I am doing new test builds every day to try to fix the issues remaining. This needs to be done before the next release after Nov 20th.

Aside from that, this is really just a technical issue, there is no user-visible advantage from doing this.

Comment 30

[Guillermo Huerta [Disabled MoCo account]]

Lightning does not work with Daily on OS X without this.

Comment 31

[Philipp Kewisch [:Fallen][📱📆]]

Yes, because I broke it by enabling the new builds :-) I'm working on it, believe me. If you really want this now, I can enable the old comm-central machines again so at least a compatible mac build shows up.

Comment 32

[Guillermo Huerta [Disabled MoCo account]]

I'm using release as a stop-gap, but is there anything I can do to help?  I've been looking for a way to get involved in Thunderbird.

Comment 33

[Philipp Kewisch [:Fallen][📱📆]]

Indeed there are some steps that still need to be done. Here is the short version:

These steps are pretty much independent of the work I am doing:

* We need a buildbot step (or at least a script) that can merge N single-locale xpi into one multi-locale xpi
* We need a buildbot step to merge at least the linux i686 and x86_64 binary components into one xpi. There is a shell script already available, but doing this using buildbot automation and its scheduler would be nicer.
* To extend on that, it would probably be nice to have a more general script that can merge any sort of xpi, i.e merge the windows, linux and mac xpis into one all-platform xpi. I have started a such script in python at one point, but it never passed some initial fiddling.


These are my next steps, I am working on the first two, the others are not done yet. Taking care of these might require buildvpn access or pinging me or #build to trigger a build on staging to find out if the changed code works:

* Hook up Lightning l10n to Thunderbird's targets so it is called during the build process (this create single-language xpis)
* Fix the mac platform packaging issue
* Hook up xpcshell and mozmill tests to the Thunderbird targets
* Hook up code signing so that the windows dll is signed


Let me know what you are interested in and I can give you more details.

Comment 34

[Philipp Kewisch [:Fallen][📱📆]]

Attachment: Fix mac universal builds - v2

This is ready for review, I've tested it on the staging machines.

Comment 35

[Philipp Kewisch [:Fallen][📱📆]]

Attachment: Fix mac universal builds - v3

Remove gdata universal stuff as per IRC discussion

Comment 36

[Philipp Kewisch [:Fallen][📱📆]]

Comment on attachment 668425 [details] [diff] [review]
Patch for comm-aurora mozconfigs - v1

Aurora will be taken care of automatically by the merge

Comment 37

[Philipp Kewisch [:Fallen][📱📆]]

Comment on attachment 681904 [details] [diff] [review]
Fix mac universal builds - v3

comm-central changeset f9cec7b19696

Comment 38

[Philipp Kewisch [:Fallen][📱📆]]

I'm going to take care of the remaining topics in dependent issues, unless it is something directly related to the patches already on this bug.

Comment 39

[Philipp Kewisch [:Fallen][📱📆]]

Marking as fixed since the nightly builds are running. As mentioned, dependent bugs for remaining issues.