Closed Bug 1160805 Opened 9 years ago Closed 9 years ago

Missing Windows and Linux nightly builds, build step set props: previous_buildid fails

Categories

(Thunderbird :: Build Config, defect)

39 Branch
x86
Windows 7
defect
Not set
blocker

Tracking

(thunderbird39 fixed, thunderbird_esr38 fixed)

RESOLVED FIXED
Thunderbird 41.0
Tracking Status
thunderbird39 --- fixed
thunderbird_esr38 --- fixed

People

(Reporter: ssitter, Assigned: Fallen)

References

Details

Attachments

(1 file)

There are no Windows and Linux nightly builds since 29-Apr-2015. Looking at the build logs its seems that compilation is OK, but later there is an error or problem that prevents the upload.

The last executed steps is "set props: previous_buildid", following steps are skipped.

This is extract from the last working build http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/2015/04/2015-04-29-03-02-05-comm-central/comm-central-win32-nightly-bm85-build1-build5.txt.gz:

> ========= Started set props: previous_buildid (results: 0, elapsed: 0 secs) (at 2015-04-29 04:31:14.524165) =========
> 'python' 'build/mozilla/config/printconfigsetting.py' 'build/objdir-tb/previous/application.ini' 'App' 'BuildID'
> [...]
> 20150428030213
> program finished with exit code 0
> elapsedTime=0.108000
> previous_buildid: '20150428030213'
> ========= Finished set props: previous_buildid (results: 0, elapsed: 0 secs) (at 2015-04-29 04:31:14.646325) =========
> 
> ========= Started 'find . ...' (results: 0, elapsed: 0 secs) (at 2015-04-29 04:31:14.647297) =========

This is extract from the current failing build http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/2015/05/2015-05-02-03-02-11-comm-central/comm-central-win32-nightly-bm85-build1-build6.txt.gz:

> ========= Started set props: previous_buildid (results: 2, elapsed: 0 secs) (at 2015-05-02 04:37:41.844605) =========
> 'python' 'build/mozilla/config/printconfigsetting.py' 'build/objdir-tb/previous/application.ini\nprevious/extensions/{e2fda1a4-762b-4020-b5ad-a41df1933103}/application.ini' 'App' 'BuildID'
> [...]
> Usage: printconfigsetting.py <file> <section> <setting>
> program finished with exit code 1
> elapsedTime=0.109000
> previous_buildid: 'Usage: printconfigsetting.py <file> <section> <setting>'
> ========= Finished set props: previous_buildid (results: 2, elapsed: 0 secs) (at 2015-05-02 04:37:41.965110) =========
> 
> ========= Skipped  (results: not started, elapsed: not started) =========

Note that the step now fails, probably due to the changed filename passed to printconfigsetting.py. The changed filename contains reference to Lightning extension.
I think the step is triggered here: https://mxr.mozilla.org/build/source/buildbotcustom/process/factory.py#2124

In step "find inipath" it looks for application.ini in the unpacked previous complete mar and passes the result to step "set previous buildid". Before Bug 1130854 the first step would return
> previous/application.ini
but now it returns
> previous/application.ini
> previous/extensions/{e2fda1a4-762b-4020-b5ad-a41df1933103}/application.ini
causing the error in the next step.
Same for comm-aurora.
Severity: major → blocker
Version: 40 → 39
Ok, we should figure out if application.ini in Lightning is still needed and/or fix the build step to ignore extensions.
Assignee: nobody → philipp
Status: NEW → ASSIGNED
This would also happen for beta builds if accessing previous mar's wasn't broken due to missing ssh keys.

I checked automation, and both mozharness and the automation-partial-patch target use:

> find previous -maxdepth 4 -type f -name application.ini

to find the application.ini in the extracted mar file. I have no idea why they don't just use a predefined path, I guess it might take mac into account that has previous/XXX.app/Contents/Resources/application.ini.

We use the application.ini only to get the build id for repacks. From a quick look we might just be able to use MOZ_BUILD_DATE instead. Right now it seems broken anyway, I see a lot of -DGRE_BUILDID= in preprocessor calls, missing the date. When fixing this we need to make sure that mac unify doesn't break because it will see the differences between the build ids.

For a quick fix we can just rename the file. Removing the application.ini will require a few try runs and some staging runs on beta (which takes so much time :-/).

Fixing it in build automation is fairly simple, but it won't go down to beta. I might be able to request for aurora, but that doesn't fix beta builds when partial generation is fixed.

I think renaming on beta/aurora is our best bet, and then removing application.ini and/or fixing automation for central.
renaming works on try, but then again it doesn't do l10n repacks.

https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=0a5f05da5dea
This is my version of Philipps patch. I did not note problems during my local Windows build. Maybe we could give it a try on the comm-aurora server to restore nightly builds?
Flags: needinfo?(philipp)
Looks like there needs to be done more.

During build it will fetch the *.complete.mar from http://stage.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-comm-aurora/. And this package contains Lightning with application.ini file. And it will not be replaced by a new *.complete.mar with fixed Lightning due to this bug here.

Maybe someone could remove the affected files from the server? Or is there some way to skip those steps once?
Branch uplift took place today and now the fix is not available on comm-aurora anymore. Up to now no new broken 41.0a1 or 40.0a2 builds with Lightnings application.ini exists on the server. Maybe use the chance and re-apply the patch to ensure that the next successful nightly builds will not introduce the same problem again?
Pushed to central and aurora so we have it on all branches.

https://hg.mozilla.org/releases/comm-aurora/rev/dc0abd44e250
https://hg.mozilla.org/comm-central/rev/d1979f04df5a

I bet we could get releng to do some deletes, but I'm not sure how that will affect the update server. I'll see if I can either push forward the patch to fix it in the build system or get releng to do soemthing else about it.
Unfortunately nightly builds are still broken due to this problem.
Wouldn't it be enough to add

@DIR_MACOS@extensions/{e2fda1a4-762b-4020-b5ad-a41df1933103}/application.ini

to /mail/installer/removed-files.in to make sure this gets removed?
It would not help. Problem is that the nightly build process fails when trying to create the partial patch from last successful nightly build package to current nightly build package. And it was the last successful nightly build package that contained the two application.ini files causing the problem.

I just saw that the old nightly builds were removed from the server. Maybe tomorrows build will succeed.
How nice, it seems that sometimes ignoring the problem does make it go away ;-) I assume this can be closed now?
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Target Milestone: --- → Thunderbird 41.0
I was looking at <https://treeherder.mozilla.org/#/jobs?repo=comm-esr38> and noticed that it shows this error for at least all nightly builds after merge from comm-beta on June 3rd.

Did we missed to port this fix to all affected versions?
Flags: needinfo?(philipp)
By now we only need it on comm-esr38's default branch, the other branches are obsolete.

https://hg.mozilla.org/releases/comm-esr38/rev/4b3c82df2117
You need to log in before you can comment on or make changes to this bug.