[Thunderbird] update_verify: Linux updates fail because libgtk-3 is missing

RESOLVED WONTFIX

Status

Release Engineering
Release Automation: Other
P5
normal
RESOLVED WONTFIX
a year ago
3 months ago

People

(Reporter: jlorenzo, Assigned: tomprince)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

MozReview Requests

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(3 attachments)

(Reporter)

Description

a year ago
Thunderbird 52.0 -> 52.0.1 failed[1] with this error:

> Found updater at updater
> ../../update/updater: error while loading shared libraries: libgtk-3.so.0: cannot open shared object file: No such file or directory
> cat: update/update.log: No such file or directory
> cat: update/update.status: No such file or directory
> FAIL: update status was not successful: 
> FAIL: check_updates returned failure for Linux_x86_64-gcc3 downloads/thunderbird-52.0b4.tar.bz2 vs. downloads/thunderbird-53.0b1.tar.bz2: 1

In the same batch, 45.8.0 -> 52.0.1 passed[2]. 52.0 -> 53.0b1 is also busted[3].

I suspect this is because libgtk3 wasn't a requirement on 45.0[4], but now is in 52.0[5]. It should a matter of installing gtk3 on the buildbot workers (aka slaves). I'm not sure where this bug should belong.

[1] https://archive.mozilla.org/pub/thunderbird/candidates/52.0.1-candidates/build3/logs/release-comm-esr52-linux64_release_update_verify_1-bm91-build1-build0.txt.gz 
[2] https://archive.mozilla.org/pub/thunderbird/candidates/52.0.1-candidates/build3/logs/release-comm-esr52-linux64_release_update_verify_6-bm70-build1-build0.txt.gz
[3] https://archive.mozilla.org/pub/thunderbird/candidates/53.0b1-candidates/build3/logs/release-comm-beta-linux64_beta_update_verify_1-bm72-build1-build6.txt.gz
[4] https://www.mozilla.org/en-US/thunderbird/45.0/system-requirements/
[5] https://www.mozilla.org/en-US/thunderbird/52.0/system-requirements/
to be expected .. failed also for 52.1.0
Priority: -- → P5
(Assignee)

Updated

10 months ago
Assignee: nobody → mozilla
(Assignee)

Comment 2

10 months ago
I don't think there is any good way to fix this before we migrate to taskcluster. However, we can probably do something that violates a bunch of abstractions:

1. Add some tootltool manifests with just gtk-3 in tools repo (at scripts/release/updates/thunderbird-{linux,linux64}.manifest perhaps)
2. Add code to buildbotcustom.process.release to
   a) add tooltool options to the update_verify ScriptFactory
   b) add gtk3 to LD_LIBRARY_PATH
   if some configuration is set in the configs.
3. Add that configuration to buildbot-config.

I know this is ugly and hacky, but the update-verify step is basically useless to without libgtk3 available, and I don't see any other way that isn't more invasive.

The only other vaguely reasonable option I can see is to create RPMs of the gtk3 bits and get them to be installed in the mock environment.

Nick, would either of these be acceptable changes?
Flags: needinfo?(nthomas)
The tooltool option would be OK, and relatively self-contained from a risk point of view. Creating RPMs and meeting dependencies is likely a nasty place, I would not buy a ticket there.

Do you think you'll need a staging environment and slave to work out all the details ?
Flags: needinfo?(nthomas)
(Assignee)

Comment 4

9 months ago
> Creating RPMs and meeting dependencies is likely a nasty place, I would not buy a ticket there.

My plan was simply to unpack the tooltool gtk3 tarball(s) and repack the `.so`s from them in /usr/local/lib or /usr/lib as RPMs. I'm actually leaning this way, as it seems like it would be a less invasive change.

> Do you think you'll need a staging environment and slave to work out all the details ?

I think there is enough detail in the logs and code to develop at least a first patch. But I do think testing this in a staging environment, rather than waiting for a release will definitely be worthwhile. Once I've got something to test, we can figure out how to stage the changes.
Flags: needinfo?(mozilla)
(Assignee)

Comment 5

9 months ago
Created attachment 8897538 [details]
Spec file for repacking tooltool gtk3 archives.

Here is an RPM spec to repack the tooltool archives as installable RPMS
(Assignee)

Comment 6

9 months ago
Created attachment 8897541 [details]
32-bit RPM

32 RPM

Updated

9 months ago
Duplicate of this bug: 1390604
Comment hidden (mozreview-request)
(Assignee)

Comment 9

9 months ago
This should probably be tested in staging:
1) Verify that the update_verify steps do actually succeed (or at least fail in ways unrelated to gtk3)
2) Verify that installing just the `.so`s doesn't break building or testing on linux or linux64.
Flags: needinfo?(mozilla)

Comment 10

9 months ago
mozreview-review
Comment on attachment 8897555 [details]
Bug 1356562 - Install gtk3 for thunderbird on linux/linux64; .

https://reviewboard.mozilla.org/r/168824/#review175216

I don't like this approach because it affects any builder using these platform configs, including Firefox builders on esr52. It would be preferable to extend mock_packages just for the update verify builders at https://hg.mozilla.org/build/buildbotcustom/file/default/process/release.py#l1150, based on config which is only set for Thunderbird.
Attachment #8897555 - Flags: review?(nthomas) → review-
(Assignee)

Comment 11

8 months ago
mozreview-review-reply
Comment on attachment 8897555 [details]
Bug 1356562 - Install gtk3 for thunderbird on linux/linux64; .

https://reviewboard.mozilla.org/r/168824/#review175216

This is set only for thunderbird (although not just for the verify step). I don't think this should be an issue since it only installs .so.VERSION, not .so, files. This seems less likely to affect Firefox ESR than adding more logic to https://hg.mozilla.org/build/buildbotcustom/file/default/process/release.py#l1150 to conditionally add extra config. If it does end up affecting TB ESR builds, I'm happy to revist and do the more complicated thing.
(Assignee)

Updated

8 months ago
Attachment #8897555 - Flags: review- → review?
(Assignee)

Updated

7 months ago
Flags: needinfo?(nthomas)

Comment 12

7 months ago
mozreview-review
Comment on attachment 8897555 [details]
Bug 1356562 - Install gtk3 for thunderbird on linux/linux64; .

https://reviewboard.mozilla.org/r/168824/#review195196

Huh, I must have misread the patch originally, sorry about that.
Attachment #8897555 - Flags: review+
Flags: needinfo?(nthomas)
(Assignee)

Updated

7 months ago
Attachment #8897555 - Flags: review?
(Assignee)

Comment 14

6 months ago
It looks like the RPMs never got uploaded.

The file names they should have are (bugzilla truncates them; and the 64bit one is apparently to large to be attached):

tooltool-gtk3-1-1.3915f8ec396c56a8a92e6f9695b70f09ce9d1582359d1258e37e3fd43a143bc974410e4cfc27f500e095f34a8956206e0ebf799b7287f0f38def0d5e34ed71c9.x86_64.rpm
tooltool-gtk3-1-1.18bc52b0599b1308b667e282abb45f47597bfc98a5140cfcab8da71dacf89dd76d0dee22a04ce26fe7ad1f04e2d6596991f9e5b01fd2aaaab5542965f596b0e6.i686.rpm


To do this, it looks like a new stanza needs to be added to 
https://dxr.mozilla.org/build-central/source/puppet/modules/runner/files/mockbuild-config-templates/mozilla-centos6-x86_64.cfg
and yum repos created with them at the places listed in 
https://dxr.mozilla.org/build-central/source/puppet/modules/runner/templates/tasks/config_mockbuild.erb


At this point, perhaps we don't bother with fixing this.
(Assignee)

Comment 15

6 months ago
I'm going to see if I can port the linux taskcluster update-verify taskcluster tasks to comm-central.  We should be able to run them manually even before the rest of the release process is ported.
(Assignee)

Updated

3 months ago
Status: NEW → RESOLVED
Last Resolved: 3 months ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.