1356562 - [Thunderbird] update_verify: Linux updates fail because libgtk-3 is missing

Status: RESOLVED WONTFIX

(Release Engineering :: Release Automation: Other, defect, P5)

Description

[Johan Lorenzo [:jlorenzo]]

Thunderbird 52.0 -> 52.0.1 failed[1] with this error:

> Found updater at updater
> ../../update/updater: error while loading shared libraries: libgtk-3.so.0: cannot open shared object file: No such file or directory
> cat: update/update.log: No such file or directory
> cat: update/update.status: No such file or directory
> FAIL: update status was not successful: 
> FAIL: check_updates returned failure for Linux_x86_64-gcc3 downloads/thunderbird-52.0b4.tar.bz2 vs. downloads/thunderbird-53.0b1.tar.bz2: 1

In the same batch, 45.8.0 -> 52.0.1 passed[2]. 52.0 -> 53.0b1 is also busted[3].

I suspect this is because libgtk3 wasn't a requirement on 45.0[4], but now is in 52.0[5]. It should a matter of installing gtk3 on the buildbot workers (aka slaves). I'm not sure where this bug should belong.

[1] https://archive.mozilla.org/pub/thunderbird/candidates/52.0.1-candidates/build3/logs/release-comm-esr52-linux64_release_update_verify_1-bm91-build1-build0.txt.gz 
[2] https://archive.mozilla.org/pub/thunderbird/candidates/52.0.1-candidates/build3/logs/release-comm-esr52-linux64_release_update_verify_6-bm70-build1-build0.txt.gz
[3] https://archive.mozilla.org/pub/thunderbird/candidates/53.0b1-candidates/build3/logs/release-comm-beta-linux64_beta_update_verify_1-bm72-build1-build6.txt.gz
[4] https://www.mozilla.org/en-US/thunderbird/45.0/system-requirements/
[5] https://www.mozilla.org/en-US/thunderbird/52.0/system-requirements/

Comment 1

[Wayne Mery (:wsmwk)]

to be expected .. failed also for 52.1.0

Comment 2

[Tom Prince [:tomprince]]

I don't think there is any good way to fix this before we migrate to taskcluster. However, we can probably do something that violates a bunch of abstractions:

1. Add some tootltool manifests with just gtk-3 in tools repo (at scripts/release/updates/thunderbird-{linux,linux64}.manifest perhaps)
2. Add code to buildbotcustom.process.release to
   a) add tooltool options to the update_verify ScriptFactory
   b) add gtk3 to LD_LIBRARY_PATH
   if some configuration is set in the configs.
3. Add that configuration to buildbot-config.

I know this is ugly and hacky, but the update-verify step is basically useless to without libgtk3 available, and I don't see any other way that isn't more invasive.

The only other vaguely reasonable option I can see is to create RPMs of the gtk3 bits and get them to be installed in the mock environment.

Nick, would either of these be acceptable changes?

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

The tooltool option would be OK, and relatively self-contained from a risk point of view. Creating RPMs and meeting dependencies is likely a nasty place, I would not buy a ticket there.

Do you think you'll need a staging environment and slave to work out all the details ?

Comment 4

[Tom Prince [:tomprince]]

> Creating RPMs and meeting dependencies is likely a nasty place, I would not buy a ticket there.

My plan was simply to unpack the tooltool gtk3 tarball(s) and repack the `.so`s from them in /usr/local/lib or /usr/lib as RPMs. I'm actually leaning this way, as it seems like it would be a less invasive change.

> Do you think you'll need a staging environment and slave to work out all the details ?

I think there is enough detail in the logs and code to develop at least a first patch. But I do think testing this in a staging environment, rather than waiting for a release will definitely be worthwhile. Once I've got something to test, we can figure out how to stage the changes.

Comment 5

[Tom Prince [:tomprince]]

Attachment: Spec file for repacking tooltool gtk3 archives.

Here is an RPM spec to repack the tooltool archives as installable RPMS

Comment 6

[Tom Prince [:tomprince]]

Attachment: 32-bit RPM

32 RPM

Comment 8

[Tom Prince [:tomprince]]

Attachment: Bug 1356562 - Install gtk3 for thunderbird on linux/linux64; .

This is needed for the update_verify steps as part of releases.

Review commit: https://reviewboard.mozilla.org/r/168824/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/168824/

Comment 9

[Tom Prince [:tomprince]]

This should probably be tested in staging:
1) Verify that the update_verify steps do actually succeed (or at least fail in ways unrelated to gtk3)
2) Verify that installing just the `.so`s doesn't break building or testing on linux or linux64.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8897555 [details]
Bug 1356562 - Install gtk3 for thunderbird on linux/linux64; .

https://reviewboard.mozilla.org/r/168824/#review175216

I don't like this approach because it affects any builder using these platform configs, including Firefox builders on esr52. It would be preferable to extend mock_packages just for the update verify builders at https://hg.mozilla.org/build/buildbotcustom/file/default/process/release.py#l1150, based on config which is only set for Thunderbird.

Comment 11

[Tom Prince [:tomprince]]

Comment on attachment 8897555 [details]
Bug 1356562 - Install gtk3 for thunderbird on linux/linux64; .

https://reviewboard.mozilla.org/r/168824/#review175216

This is set only for thunderbird (although not just for the verify step). I don't think this should be an issue since it only installs .so.VERSION, not .so, files. This seems less likely to affect Firefox ESR than adding more logic to https://hg.mozilla.org/build/buildbotcustom/file/default/process/release.py#l1150 to conditionally add extra config. If it does end up affecting TB ESR builds, I'm happy to revist and do the more complicated thing.

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8897555 [details]
Bug 1356562 - Install gtk3 for thunderbird on linux/linux64; .

https://reviewboard.mozilla.org/r/168824/#review195196

Huh, I must have misread the patch originally, sorry about that.

Comment 13

[Tom Prince [:tomprince]]

https://hg.mozilla.org/build/buildbot-configs/rev/035726acee9528a1457d74b6d7e4574034787102

Comment 14

[Tom Prince [:tomprince]]

It looks like the RPMs never got uploaded.

The file names they should have are (bugzilla truncates them; and the 64bit one is apparently to large to be attached):

tooltool-gtk3-1-1.3915f8ec396c56a8a92e6f9695b70f09ce9d1582359d1258e37e3fd43a143bc974410e4cfc27f500e095f34a8956206e0ebf799b7287f0f38def0d5e34ed71c9.x86_64.rpm
tooltool-gtk3-1-1.18bc52b0599b1308b667e282abb45f47597bfc98a5140cfcab8da71dacf89dd76d0dee22a04ce26fe7ad1f04e2d6596991f9e5b01fd2aaaab5542965f596b0e6.i686.rpm


To do this, it looks like a new stanza needs to be added to 
https://dxr.mozilla.org/build-central/source/puppet/modules/runner/files/mockbuild-config-templates/mozilla-centos6-x86_64.cfg
and yum repos created with them at the places listed in 
https://dxr.mozilla.org/build-central/source/puppet/modules/runner/templates/tasks/config_mockbuild.erb


At this point, perhaps we don't bother with fixing this.

Comment 15

[Tom Prince [:tomprince]]

I'm going to see if I can port the linux taskcluster update-verify taskcluster tasks to comm-central.  We should be able to run them manually even before the rest of the release process is ported.