Closed Bug 793628 Opened 12 years ago Closed 12 years ago

Produce Lightning Nightly builds during Thunderbird build process

Categories

(Release Engineering :: General, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: Fallen, Assigned: Fallen)

References

Details

Attachments

(3 files, 4 obsolete files)

At the Mozcamp in Warsaw we discussed that to lessen the burden of building Lightning, we should attempt to integrate Lightning builds into Thunderbird.

In an ideal world, all that is needed is adding --enable-calendar and setting DISABLE_LIGHTNING_INSTALL=1. Since bug 724257 the make upload target should also be uploading Lightning if --enable-calendar is specified.

Open questions, some of these to be handled in dependent bugs:

* Where do we run unit and mozmill tests so that failing Lightning tests don't
  disrupt Thunderbird?
* Lightning build issues are not very common, but is it ok that Lightning build
  failures will also fail a Thunderbird build?
* Lightning l10n works a bit differently
  * We can repack single l10n builds just like Thunderbird (needs hookup)
  * Release builds will need a separate job that merges all single l10n builds
* Merging 32/64 bit builds still needs to happen in a separate job

As a first step, I would appreciate if someone (John?) could comment on how to best get this rolling. Personally I think we should start with the nightly builds and see how this works out, then advance to the beta builds and then to the release builds.
I think this would be better tracked in releng land, as it is their infrastructure that's affected here.
Component: Build Config → Release Engineering
Product: Thunderbird → mozilla.org
Version: unspecified → other
(In reply to Philipp Kewisch [:Fallen] from comment #0)
...
> Open questions, some of these to be handled in dependent bugs:
> 
> * Where do we run unit and mozmill tests so that failing Lightning tests
> don't disrupt Thunderbird?

Can you explain 'disrupt Thunderbird' a bit more?

> * Lightning build issues are not very common, but is it ok that Lightning
> build failures will also fail a Thunderbird build?

If Lightning is going to be a "first class" addon for Thunderbird (in that Lightning bustage blocks a Thunderbird release) then I believe the answer is yes.  Has that been decided?

> * Lightning l10n works a bit differently
>   * We can repack single l10n builds just like Thunderbird (needs hookup)
>   * Release builds will need a separate job that merges all single l10n
> builds
> * Merging 32/64 bit builds still needs to happen in a separate job

Are there existing Buildbot factories that can do this?

> As a first step, I would appreciate if someone (John?) could comment on how
> to best get this rolling. Personally I think we should start with the
> nightly builds and see how this works out, then advance to the beta builds
> and then to the release builds.

That sounds like a good way to start.  Also, if possible it would be good if we could get most of this done with updated 'make' targets so as not to over-complicate the buildbot automation or take too much releng time (which is scarce with all the other work going on right now).  Where that isn't possible, I'd like for us to come up with a list of things that need to be done on the buildbot automation side.
(In reply to John Hopkins (:jhopkins) from comment #2)
> (In reply to Philipp Kewisch [:Fallen] from comment #0)
> ...
> > Open questions, some of these to be handled in dependent bugs:
> > 
> > * Where do we run unit and mozmill tests so that failing Lightning tests
> > don't disrupt Thunderbird?
> 
> Can you explain 'disrupt Thunderbird' a bit more?

Disrupt may be a bit harsh. I just mean that a failing Lightning test will make the build go orange, even if all Thunderbird tests were successful. I talked to Mark about this and he said it would be fine.

> 
> > * Lightning build issues are not very common, but is it ok that Lightning
> > build failures will also fail a Thunderbird build?
> 
> If Lightning is going to be a "first class" addon for Thunderbird (in that
> Lightning bustage blocks a Thunderbird release) then I believe the answer is
> yes.  Has that been decided?
I guess thats up to us and the remaining community to decide. Given most of our build failures are because of build config issues and not actual code issues, I would decide yes.

> > * Lightning l10n works a bit differently
> >   * We can repack single l10n builds just like Thunderbird (needs hookup)
> >   * Release builds will need a separate job that merges all single l10n
> > builds
> > * Merging 32/64 bit builds still needs to happen in a separate job
> 
> Are there existing Buildbot factories that can do this?
Not at the moment. I can hack up some factories to do this though, right now the 32/64 bit linux builds are being merged by a simple script.


> 
> > As a first step, I would appreciate if someone (John?) could comment on how
> > to best get this rolling. Personally I think we should start with the
> > nightly builds and see how this works out, then advance to the beta builds
> > and then to the release builds.
> 
> That sounds like a good way to start.  Also, if possible it would be good if
> we could get most of this done with updated 'make' targets so as not to
> over-complicate the buildbot automation or take too much releng time (which
> is scarce with all the other work going on right now).  Where that isn't
> possible, I'd like for us to come up with a list of things that need to be
> done on the buildbot automation side.

Indeed, I think we can get most of this working using the standard make targets. make upload is already hooked up, so if tests and l10n is ignored, then creating a nightly en-US should already be possible.

I can probably get a try build running to proove it.
John had this run through on aurora and there were no notable differences. I did a compare of the current nightly lightning and the produced Lightning and all that is different is build id and some paths in preprocess comments. The same goes for Thunderbird, with some minor differences:

* I couldn't check actual differences in binary code, I assume it will be ok
* In thunderbird/chrome/toolkit/content/global/buildconfig.html the option
  --enable-calendar is shown.

I think this is enough to get comm-central and comm-aurora nightly builds running on moco machines. I'd appreciate a quick review since our builds are broken.

I suggest enabling for central first and waiting for all platforms to go green, then doing the same for aurora.
Assignee: nobody → philipp
Status: NEW → ASSIGNED
Attachment #668423 - Flags: review?(mbanner)
Attachment #668423 - Flags: feedback?
Attachment #668423 - Flags: feedback? → feedback?(jhopkins)
[Approval Request Comment]
Bug caused by (feature/regressing bug #): bug 793634 
User impact if declined: Red builds
Testing completed (on m-c, etc.): tested builds by jhopkins on comm-aurora
Attachment #668425 - Flags: review?(mbanner)
Attachment #668425 - Flags: approval-mozilla-aurora?
Attachment #668423 - Flags: feedback?(jhopkins) → feedback+
Comment on attachment 668425 [details] [diff] [review]
Patch for comm-aurora mozconfigs - v1

you want comm-approval here, not approval-mozilla-aurora but I can't set the flag here for some reason probably cause it's a releng bug.
Attachment #668425 - Flags: approval-mozilla-aurora?
This patch has run through staging and puts lightning at the right spot. The gdata parts are new, but follow the same concept.

I think this is ready to land, lets start with comm-central. Mark, can you take a look?
Attachment #670954 - Flags: review?(mbanner)
Comment on attachment 670954 [details] [diff] [review]
Fix upload location for nightly and tinderbox builds - v1

Review of attachment 670954 [details] [diff] [review]:
-----------------------------------------------------------------

So before we can deploy these patches, we need to sort out about ssh keys. I've just tried this on the try server, and upload failed on Mac & Linux as it was attempting to use the Thunderbird ssh keys to upload to calendar/lightning, which just won't work.

We're either going to need to change the permissions there, or find a way to use a different set of ssh keys...

::: calendar/lightning/lightning-packager.mk
@@ +134,4 @@
>  
> +# Lightning uses Thunderbird's build machinery, so we need to hack the post
> +# upload command to use Lightning's directories.
> +upload: POST_UPLOAD_CMD := $(subst thunderbird,calendar/lightning,$(POST_UPLOAD_CMD))

This doesn't work on Windows, probably because of the "/".
Attachment #670954 - Flags: review?(mbanner) → review-
For
Depends on: 799587, 804138
For the ssh keys issue, see bugs in previous comment.

I've submitted a new try build with a possible fix for the windows directory issue.

https://tbpl.mozilla.org/?tree=Thunderbird-Try&rev=40afc721c97d
This patch works for windows. Probably since windows starts a new shell, the env var isn't updated. This patch ensures that the changed variable is passed.
Attachment #670954 - Attachment is obsolete: true
Attachment #674377 - Flags: review?(mbanner)
Comment on attachment 674377 [details] [diff] [review]
Fix upload location for nightly and tinderbox builds - v2


>+upload: stage_upload
>+	echo $(POST_UPLOAD_CMD)
>+	POST_UPLOAD_CMD="$(POST_UPLOAD_CMD)" \

I have removed that extra "echo" command locally.


Mark, any chance you could take a look at this again?
Attachment #674377 - Flags: review?(mbanner) → review+
Comment on attachment 674377 [details] [diff] [review]
Fix upload location for nightly and tinderbox builds - v2


Pushed upload location patch:
https://hg.mozilla.org/comm-central/rev/4fa5f8fb07e7
Attachment #674377 - Flags: checked-in+
Comment on attachment 668423 [details] [diff] [review]
Patch for comm-central mozconfigs - v1

Ok, lets try this on trunk to being with and give it a few days to check on how it all goes.
Attachment #668423 - Flags: review?(mbanner) → review+
Comment on attachment 668423 [details] [diff] [review]
Patch for comm-central mozconfigs - v1

https://hg.mozilla.org/comm-central/rev/8285f35f9fd5

*crossing fingers*
Attachment #668423 - Flags: checked-in+
Comment on attachment 668425 [details] [diff] [review]
Patch for comm-aurora mozconfigs - v1

Taking this from your queue until its relevant
Attachment #668425 - Flags: review?(mbanner)
Temporarily disabled comm-central calendar builders here: 

http://hg.mozilla.org/build/buildbot-configs/rev/4f433037c0aa
Seems to work to some degree. Lot of new comm-central-* folders and builds have appeared on https://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/tinderbox-builds/

Were the changes to the .xpi name done on purpose?
> .../tinderbox-builds/lightning/tinderbox-builds/comm-central-win32/1351755294
> gdata-provider-.en-US.win32.xpi
> lightning-2.1a1.en-US.win32.xpi
Yes, the xpi name changes were done on purpose to line up with what Thunderbird has. I believe I also had another reason I don't recall right now.

There was an intermediate error due to missing directory permissions and I am not quite sure yet if the nightly bustage is because of a missing rebuild or a real error.
Ok, it just means you fixed Bug 331274 although you resolved that bug as invalid before :)

As you can see in comment 19 the version number for win32 gdata-provider is not set at all, probably the same regression as reported from linux builds in Bug 799643.

It seems that nightly builds were produced and are stored in root of https://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/nightly/latest-comm-central/ instead of platform dependent folders.
(In reply to Stefan Sitter from comment #21)
> It seems that nightly builds were produced and are stored in root of
> https://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/nightly/latest-
> comm-central/ instead of platform dependent folders.

That doesn't really matter now that they have the platform in their names, does it? It means they are quicker to find if you're navigating directories (although external links will need to be updated, of course).
(In reply to Stefan Sitter from comment #21)
> Ok, it just means you fixed Bug 331274 although you resolved that bug as
> invalid before :)
You have brilliant memory :-) I think one of the reasons was that it was extra work to get those conventions followed, and now we get that for free.


> As you can see in comment 19 the version number for win32 gdata-provider is
> not set at all, probably the same regression as reported from linux builds
> in Bug 799643.
Taken care of.
Blocks: 808668
Could you remove the now obsolete subfolders in https://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/nightly/latest-comm-central/ to not cause confusion?
There seems to be a problem with multi-platform builds.

The old package at latest-comm-central/macosx-xpi/lightning.xpi supports both Darwin_x86-gcc3 and Darwin_x86_64-gcc3 platform.

The new package at latest-comm-central/lightning-2.1a1.en-US.mac.xpi supports only Darwin_x86-gcc3.

Should I file a new bug report?
(In reply to Stefan Sitter from comment #25)
> There seems to be a problem with multi-platform builds.
> 
> The old package at latest-comm-central/macosx-xpi/lightning.xpi supports
> both Darwin_x86-gcc3 and Darwin_x86_64-gcc3 platform.
> 
> The new package at latest-comm-central/lightning-2.1a1.en-US.mac.xpi
> supports only Darwin_x86-gcc3.
> 
> Should I file a new bug report?

I'll take care of this here. The problem is that I did't add MOZ_POSTFLIGHT_ALL variables for Lightning in the mac mozconfigs. I'll also need to adapt our universal.mk so it works with the new package names.
Attached patch Fix mac universal builds - v1 (obsolete) — Splinter Review
This patch should take care of the universal builds. I've tested this on comm-central, not using the original process but something vaguely similar. Not asking for review yet since I haven't been able to test it on stage, but will get back to this when my other stage tasks are done.
Blocks: 808123
Any timeline on when this might be resolved in the nightly builds?
Although it might not look like anything is happening by this bug, I am doing new test builds every day to try to fix the issues remaining. This needs to be done before the next release after Nov 20th.

Aside from that, this is really just a technical issue, there is no user-visible advantage from doing this.
Lightning does not work with Daily on OS X without this.
Yes, because I broke it by enabling the new builds :-) I'm working on it, believe me. If you really want this now, I can enable the old comm-central machines again so at least a compatible mac build shows up.
I'm using release as a stop-gap, but is there anything I can do to help?  I've been looking for a way to get involved in Thunderbird.
Indeed there are some steps that still need to be done. Here is the short version:

These steps are pretty much independent of the work I am doing:

* We need a buildbot step (or at least a script) that can merge N single-locale xpi into one multi-locale xpi
* We need a buildbot step to merge at least the linux i686 and x86_64 binary components into one xpi. There is a shell script already available, but doing this using buildbot automation and its scheduler would be nicer.
* To extend on that, it would probably be nice to have a more general script that can merge any sort of xpi, i.e merge the windows, linux and mac xpis into one all-platform xpi. I have started a such script in python at one point, but it never passed some initial fiddling.


These are my next steps, I am working on the first two, the others are not done yet. Taking care of these might require buildvpn access or pinging me or #build to trigger a build on staging to find out if the changed code works:

* Hook up Lightning l10n to Thunderbird's targets so it is called during the build process (this create single-language xpis)
* Fix the mac platform packaging issue
* Hook up xpcshell and mozmill tests to the Thunderbird targets
* Hook up code signing so that the windows dll is signed


Let me know what you are interested in and I can give you more details.
Attached patch Fix mac universal builds - v2 (obsolete) — Splinter Review
This is ready for review, I've tested it on the staging machines.
Attachment #678831 - Attachment is obsolete: true
Attachment #681894 - Flags: review?(mbanner)
Remove gdata universal stuff as per IRC discussion
Attachment #681894 - Attachment is obsolete: true
Attachment #681894 - Flags: review?(mbanner)
Attachment #681904 - Flags: review?(mbanner)
Attachment #681904 - Flags: review?(mbanner) → review+
Comment on attachment 668425 [details] [diff] [review]
Patch for comm-aurora mozconfigs - v1

Aurora will be taken care of automatically by the merge
Attachment #668425 - Attachment is obsolete: true
I'm going to take care of the remaining topics in dependent issues, unless it is something directly related to the patches already on this bug.
Depends on: 812299
Marking as fixed since the nightly builds are running. As mentioned, dependent bugs for remaining issues.
Status: ASSIGNED → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.