803664 - Remove screensaver package from talos build slaves

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[John Hopkins (:jhopkins)]

mconley reported Thunderbird test runs having input focus taken away (thus failing tests) and upon investigation I found there was a gnome-screensaver process running on talos-r3-fed64-055.

I suggest we remove gnome-screensaver package altogether via puppet.  This is a bit tricky because of circular RPM dependencies.

See also: bug 604465 and bug 585286

Comment 1

[John Hopkins (:jhopkins)]

Attachment: [puppet-manifests] remove gnome-screensaver

Comment 2

[John Hopkins (:jhopkins)]

I won't be around Monday so if the patch is r+'d someone can deploy before I get back.

Comment 3

[Kim Moir [:kmoir] ET]

Comment on attachment 673395 [details] [diff] [review]
[puppet-manifests] remove gnome-screensaver

lgtm if it tested okay in staging.  Circular dependencies are a PITA :-)

Comment 4

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

triaged to correct component.

Comment 5

[John Hopkins (:jhopkins)]

Comment on attachment 673395 [details] [diff] [review]
[puppet-manifests] remove gnome-screensaver

Landed in http://hg.mozilla.org/build/puppet-manifests/rev/4f4a43cd1497

Needs to be deployed on puppet masters.

Comment 6

[John Hopkins (:jhopkins)]

Deployed to puppet masters and monitored slave uptake.

These all behaved as expected: 

 talos-r3-fed64-004
 talos-r3-fed64-007
 talos-r3-fed-017
 talos-r3-fed-004
 talos-r3-fed-003

One anomaly:

 talos-r3-fed-007 only had 1 of 3 expected packages present which I then had to remove manually.

Comment 7

[John Hopkins (:jhopkins)]

Another spot check of about a dozen machines looks good.

Comment 9

[Daniel Holbert [:dholbert]]

We've been hitting Mochitest-Oth oranges on Mozilla-Beta since around when this landed:
  https://tbpl.mozilla.org/?tree=Mozilla-Beta

Last good run was from a push around 6am this morning.
First bad run started at 10:13 am: https://tbpl.mozilla.org/php/getParsedLog.php?id=16421740&tree=Mozilla-Beta

The failures all seem to be from waiting for paints and not receiving them, which seems like it could be related to a screensaver-config-change. (The "guilty" push appears to be unrelated -- it's a sync services change.)  So I suspect this might be causing the trouble...

Comment 10

[Daniel Holbert [:dholbert]]

Also, as noted in bug 805135, the desktop-screenshots from the failure logs are fully-black -- so maybe we're still blanking the screen for some reason?

Comment 11

[Daniel Holbert [:dholbert]]

(Filed bug 805170 for the test-timeouts mentioned in comment 9 - 10.   likely will require a backout of this bug or a followup; figured I'd file a new bug for it, though, in case it ends up being independent and/or so we can track it without cluttering up this bug too much)

Comment 12

[[e-mail] (:bhearsum)]

This caused a problem on a newly reimaged Fedora 64 machine. This was in the Puppet log:
Oct 30 10:41:17 talos-r3-fed64-002 puppetd[2357]: (//Node[talos-r3-fed64-002]/talosslave/talos_fedora/Exec[rpm -e fedora-screensaver-theme fedorainfinity-screensaver-theme gnome-screensaver]/returns) change from notrun to 0 failed: rpm -e fedora-screensaver-theme fedorainfinity-screensaver-theme gnome-screensaver returned 1 instead of 0 at /etc/puppet/manifests/os/talos_fedora.pp:10

And this came back when I ran the command by hand:
error: package fedora-screensaver-theme is not installed

I ran this to remove the packages manually:
rpm -e fedorainfinity-screensaver-theme gnome-screensaver

I think the "onlyif" needs some adjustment.

Comment 13

[John Hopkins (:jhopkins)]

This stems from the problem that not all machines are starting with the same set of packages installed.  The puppet recipe expects all three screensaver RPMs to be present.  If they aren't, the logic doesn't work.  In my testing I only found 1 machine that didn't have all 3 RPMs installed and considered it an anomaly.  So apparently fresh installs now only include 2 of the 3 screensaver packages?

Comment 14

[[e-mail] (:bhearsum)]

Maybe we can separate out the packages into 3 separate installs, so we can test individually?

Comment 15

[John Hopkins (:jhopkins)]

Attachment: [puppet-manifests] dynamically determine which of the three packages to remove

Patch needs testing on a fresh Fedora install.

Comment 16

[John Hopkins (:jhopkins)]

(In reply to Ben Hearsum [:bhearsum] from comment #14)
> Maybe we can separate out the packages into 3 separate installs, so we can
> test individually?

The packages can't be handled individually due to the circular dependency they share.  eg. If you try and remove one package, that will fail due to one or more other packages depending on it.  The same is true for the other two packages.  Therefore they all have to be removed in a single transaction.

Comment 17

[[e-mail] (:bhearsum)]

(In reply to John Hopkins (:jhopkins) from comment #16)
> (In reply to Ben Hearsum [:bhearsum] from comment #14)
> > Maybe we can separate out the packages into 3 separate installs, so we can
> > test individually?
> 
> The packages can't be handled individually due to the circular dependency
> they share.  eg. If you try and remove one package, that will fail due to
> one or more other packages depending on it.  The same is true for the other
> two packages.  Therefore they all have to be removed in a single transaction.

You can probably use --nodeps to workaround that during removal.

Comment 18

[John Hopkins (:jhopkins)]

(In reply to Ben Hearsum [:bhearsum] from comment #17)
> You can probably use --nodeps to workaround that during removal.

Yes, that's possible, but it would leave the package database (at least temporarily) in an inconsistent state.  We should try and avoid that.

Comment 19

[[e-mail] (:bhearsum)]

(In reply to John Hopkins (:jhopkins) from comment #18)
> (In reply to Ben Hearsum [:bhearsum] from comment #17)
> > You can probably use --nodeps to workaround that during removal.
> 
> Yes, that's possible, but it would leave the package database (at least
> temporarily) in an inconsistent state.  We should try and avoid that.

I'm not sure that's a big issue, because Puppet will run in a loop until they all succeed, but I agree it's less than ideal. If you can make the current approach work, WFM. For testing, I suggest putting a machine in that state by hand rather than waiting for one to get there.

Comment 20

[:aceman]

What is the state of this work?
I still get strange failures on talos-r3-fed64-* machines as if clicks are not delivered. Was the fix backed up?

Comment 21

[John Hopkins (:jhopkins)]

aceman: can you list some specific machine names that you are seeing missing clicks on?

I'm going to be testing and deploying the attached patch this week.

Comment 22

[:aceman]

talos-r3-fed64-041
talos-r3-fed64-028
talos-r3-fed64-006

talos-r3-fed-055
talos-r3-fed64-031
talos-r3-fed-032

Bug 803604 and bug 726966 are suspected to be caused by this problem. You can find more machines listed there.

Comment 23

[John Hopkins (:jhopkins)]

None of the machines listed above have gnome screensaver RPMs installed.

The other change we made was in https://bug585286.bugzilla.mozilla.org/attachment.cgi?id=674821.  This patch was landed and is still in place.

Comment 24

[John Hopkins (:jhopkins)]

Comment on attachment 677418 [details] [diff] [review]
[puppet-manifests] dynamically determine which of the three packages to remove

bhearsum: I've tested this patch with 0..3 screensaver RPMs and verified that it works in all of those cases on 32-bit and 64-bit talos machines.  Once you've reviewed the patch I will land it.  Thanks.

Comment 25

[John Hopkins (:jhopkins)]

Comment on attachment 677418 [details] [diff] [review]
[puppet-manifests] dynamically determine which of the three packages to remove

Landed in http://hg.mozilla.org/build/puppet-manifests/rev/26d24411a10a