Missing Windows and Linux nightly builds, build step set props: previous_buildid fails

RESOLVED FIXED in Thunderbird 41.0

Status

--
blocker
RESOLVED FIXED
4 years ago
3 years ago

People

(Reporter: ssitter, Assigned: Fallen)

Tracking

39 Branch
Thunderbird 41.0
x86
Windows 7
Dependency tree / graph

Thunderbird Tracking Flags

(thunderbird39 fixed, thunderbird_esr38 fixed)

Details

Attachments

(1 attachment)

(Reporter)

Description

4 years ago
There are no Windows and Linux nightly builds since 29-Apr-2015. Looking at the build logs its seems that compilation is OK, but later there is an error or problem that prevents the upload.

The last executed steps is "set props: previous_buildid", following steps are skipped.

This is extract from the last working build http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/2015/04/2015-04-29-03-02-05-comm-central/comm-central-win32-nightly-bm85-build1-build5.txt.gz:

> ========= Started set props: previous_buildid (results: 0, elapsed: 0 secs) (at 2015-04-29 04:31:14.524165) =========
> 'python' 'build/mozilla/config/printconfigsetting.py' 'build/objdir-tb/previous/application.ini' 'App' 'BuildID'
> [...]
> 20150428030213
> program finished with exit code 0
> elapsedTime=0.108000
> previous_buildid: '20150428030213'
> ========= Finished set props: previous_buildid (results: 0, elapsed: 0 secs) (at 2015-04-29 04:31:14.646325) =========
> 
> ========= Started 'find . ...' (results: 0, elapsed: 0 secs) (at 2015-04-29 04:31:14.647297) =========

This is extract from the current failing build http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/2015/05/2015-05-02-03-02-11-comm-central/comm-central-win32-nightly-bm85-build1-build6.txt.gz:

> ========= Started set props: previous_buildid (results: 2, elapsed: 0 secs) (at 2015-05-02 04:37:41.844605) =========
> 'python' 'build/mozilla/config/printconfigsetting.py' 'build/objdir-tb/previous/application.ini\nprevious/extensions/{e2fda1a4-762b-4020-b5ad-a41df1933103}/application.ini' 'App' 'BuildID'
> [...]
> Usage: printconfigsetting.py <file> <section> <setting>
> program finished with exit code 1
> elapsedTime=0.109000
> previous_buildid: 'Usage: printconfigsetting.py <file> <section> <setting>'
> ========= Finished set props: previous_buildid (results: 2, elapsed: 0 secs) (at 2015-05-02 04:37:41.965110) =========
> 
> ========= Skipped  (results: not started, elapsed: not started) =========

Note that the step now fails, probably due to the changed filename passed to printconfigsetting.py. The changed filename contains reference to Lightning extension.
(Reporter)

Comment 1

4 years ago
I think the step is triggered here: https://mxr.mozilla.org/build/source/buildbotcustom/process/factory.py#2124

In step "find inipath" it looks for application.ini in the unpacked previous complete mar and passes the result to step "set previous buildid". Before Bug 1130854 the first step would return
> previous/application.ini
but now it returns
> previous/application.ini
> previous/extensions/{e2fda1a4-762b-4020-b5ad-a41df1933103}/application.ini
causing the error in the next step.
(Reporter)

Comment 2

4 years ago
Same for comm-aurora.
Severity: major → blocker
Version: 40 → 39
Ok, we should figure out if application.ini in Lightning is still needed and/or fix the build step to ignore extensions.
Assignee: nobody → philipp
Status: NEW → ASSIGNED
This would also happen for beta builds if accessing previous mar's wasn't broken due to missing ssh keys.

I checked automation, and both mozharness and the automation-partial-patch target use:

> find previous -maxdepth 4 -type f -name application.ini

to find the application.ini in the extracted mar file. I have no idea why they don't just use a predefined path, I guess it might take mac into account that has previous/XXX.app/Contents/Resources/application.ini.

We use the application.ini only to get the build id for repacks. From a quick look we might just be able to use MOZ_BUILD_DATE instead. Right now it seems broken anyway, I see a lot of -DGRE_BUILDID= in preprocessor calls, missing the date. When fixing this we need to make sure that mac unify doesn't break because it will see the differences between the build ids.

For a quick fix we can just rename the file. Removing the application.ini will require a few try runs and some staging runs on beta (which takes so much time :-/).

Fixing it in build automation is fairly simple, but it won't go down to beta. I might be able to request for aurora, but that doesn't fix beta builds when partial generation is fixed.

I think renaming on beta/aurora is our best bet, and then removing application.ini and/or fixing automation for central.
renaming works on try, but then again it doesn't do l10n repacks.

https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=0a5f05da5dea
(Assignee)

Updated

4 years ago
Depends on: 1160899
(Assignee)

Updated

4 years ago
Blocks: 1153752
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
Comment hidden (Treeherder Robot)
(Reporter)

Comment 30

4 years ago
Created attachment 8603711 [details] [diff] [review]
rename Lightnings application.ini to app.ini

This is my version of Philipps patch. I did not note problems during my local Windows build. Maybe we could give it a try on the comm-aurora server to restore nightly builds?
Flags: needinfo?(philipp)
Ok, lets do it: https://hg.mozilla.org/releases/comm-aurora/rev/5f331d8e5a9f
Flags: needinfo?(philipp)
(Reporter)

Comment 32

4 years ago
Looks like there needs to be done more.

During build it will fetch the *.complete.mar from http://stage.mozilla.org/pub/mozilla.org/thunderbird/nightly/latest-comm-aurora/. And this package contains Lightning with application.ini file. And it will not be replaced by a new *.complete.mar with fixed Lightning due to this bug here.

Maybe someone could remove the affected files from the server? Or is there some way to skip those steps once?
(Reporter)

Comment 33

4 years ago
Branch uplift took place today and now the fix is not available on comm-aurora anymore. Up to now no new broken 41.0a1 or 40.0a2 builds with Lightnings application.ini exists on the server. Maybe use the chance and re-apply the patch to ensure that the next successful nightly builds will not introduce the same problem again?
Pushed to central and aurora so we have it on all branches.

https://hg.mozilla.org/releases/comm-aurora/rev/dc0abd44e250
https://hg.mozilla.org/comm-central/rev/d1979f04df5a

I bet we could get releng to do some deletes, but I'm not sure how that will affect the update server. I'll see if I can either push forward the patch to fix it in the build system or get releng to do soemthing else about it.
(Reporter)

Comment 35

4 years ago
Unfortunately nightly builds are still broken due to this problem.

Comment 36

4 years ago
Wouldn't it be enough to add

@DIR_MACOS@extensions/{e2fda1a4-762b-4020-b5ad-a41df1933103}/application.ini

to /mail/installer/removed-files.in to make sure this gets removed?
(Reporter)

Comment 37

4 years ago
It would not help. Problem is that the nightly build process fails when trying to create the partial patch from last successful nightly build package to current nightly build package. And it was the last successful nightly build package that contained the two application.ini files causing the problem.

I just saw that the old nightly builds were removed from the server. Maybe tomorrows build will succeed.
How nice, it seems that sometimes ignoring the problem does make it go away ;-) I assume this can be closed now?
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → Thunderbird 41.0
(Reporter)

Comment 40

3 years ago
I was looking at <https://treeherder.mozilla.org/#/jobs?repo=comm-esr38> and noticed that it shows this error for at least all nightly builds after merge from comm-beta on June 3rd.

Did we missed to port this fix to all affected versions?
Flags: needinfo?(philipp)
(Reporter)

Updated

3 years ago
status-thunderbird39: --- → fixed
status-thunderbird_esr38: --- → affected
By now we only need it on comm-esr38's default branch, the other branches are obsolete.

https://hg.mozilla.org/releases/comm-esr38/rev/4b3c82df2117
status-thunderbird_esr38: affected → fixed
You need to log in before you can comment on or make changes to this bug.