Linux Puppet slaves are pulling down all RPMs on every boot

RESOLVED FIXED

Status

RESOLVED FIXED
8 years ago
5 years ago

People

(Reporter: bhearsum, Assigned: bhearsum)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [puppet])

Attachments

(1 attachment)

(Assignee)

Description

8 years ago
Nick discovered that despite attempts to avoid it, all of our Linux machines are pulling down all of the RPMs every time they run. This is happening because the package {} check in functions.pp checks for existence of the package on every run. I tried to avoid this by making it require the "check-for-$name" check before running the package, but that doesn't prevent the package {} check from running as I thought it did.

Using ensure => installed instead of ensure => latest on the package {} check may fix this. If that doesn't work we can try messing with the slave-side Puppet code.
(Assignee)

Comment 1

8 years ago
ensure => latest is the culprit here.

ensure => installed works around the issue, but doesn't allow us to upgrade.

ensure => $version_number worked well for both not downloading on each run and allowing upgrades to happen, in my limited test. I'm going to do a more extensive test before posting a patch.
(Assignee)

Comment 2

8 years ago
Created attachment 451725 [details] [diff] [review]
fix rpm checks

Here's the fix, using ensure => version.

Here's some output with --evaltrace --debug:
info: //Node[moz2-linux-slave17.build.mozilla.org]/staging-buildslave/devtools/Install_rpm[jdk1.6]/Exec[check-for-jdk1.6]: Evaluated in 0.76 seconds
info: //Node[moz2-linux-slave17.build.mozilla.org]/staging-buildslave/devtools/Install_rpm[jdk1.6]/Package[jdk1.6]: Evaluated in 0.11 seconds
debug: //Node[moz2-linux-slave17.build.mozilla.org]/staging-buildslave/devtools/Install_rpm[python26]/Exec[check-for-python26]: Executing check '/bin/bash -c '! /bin/rpm -ql python26''
debug: Executing '/bin/bash -c '! /bin/rpm -ql python26''
info: //Node[moz2-linux-slave17.build.mozilla.org]/staging-buildslave/devtools/Install_rpm[python26]/Exec[check-for-python26]: Evaluated in 0.44 seconds
info: //Node[moz2-linux-slave17.build.mozilla.org]/staging-buildslave/devtools/Install_rpm[python26]/Package[python26]: Evaluated in 0.03 seconds
debug: //Node[moz2-linux-slave17.build.mozilla.org]/staging-buildslave/devtools/Install_rpm[mercurial-py26]/Exec[check-for-mercurial-py26]: Executing check '/bin/bash -c '! /bin/rpm -ql mercurial-py26''
debug: Executing '/bin/bash -c '! /bin/rpm -ql mercurial-py26''
info: //Node[moz2-linux-slave17.build.mozilla.org]/staging-buildslave/devtools/Install_rpm[mercurial-py26]/Exec[check-for-mercurial-py26]: Evaluated in 0.61 seconds
info: //Node[moz2-linux-slave17.build.mozilla.org]/staging-buildslave/devtools/Install_rpm[mercurial-py26]/Package[mercurial-py26]: Evaluated in 0.04 seconds
debug: //Node[moz2-linux-slave17.build.mozilla.org]/staging-buildslave/devtools/Install_rpm[virtualenv]/Exec[check-for-virtualenv]: Executing check '/bin/bash -c '! /bin/rpm -ql virtualenv''
debug: Executing '/bin/bash -c '! /bin/rpm -ql virtualenv''
info: //Node[moz2-linux-slave17.build.mozilla.org]/staging-buildslave/devtools/Install_rpm[virtualenv]/Exec[check-for-virtualenv]: Evaluated in 0.55 seconds
info: //Node[moz2-linux-slave17.build.mozilla.org]/staging-buildslave/devtools/Install_rpm[virtualenv]/Package[virtualenv]: Evaluated in 0.03 seconds
debug: //Node[moz2-linux-slave17.build.mozilla.org]/staging-buildslave/devtools/Install_rpm[gcc433]/Exec[check-for-gcc433]: Executing check '/bin/bash -c '! /bin/rpm -ql gcc433''
debug: Executing '/bin/bash -c '! /bin/rpm -ql gcc433''
info: //Node[moz2-linux-slave17.build.mozilla.org]/staging-buildslave/devtools/Install_rpm[gcc433]/Exec[check-for-gcc433]: Evaluated in 2.68 seconds
info: //Node[moz2-linux-slave17.build.mozilla.org]/staging-buildslave/devtools/Install_rpm[gcc433]/Package[gcc433]: Evaluated in 0.03 seconds
debug: //Node[moz2-linux-slave17.build.mozilla.org]/staging-buildslave/devtools/Install_rpm[gcc411]/Exec[check-for-gcc411]: Executing check '/bin/bash -c '! /bin/rpm -ql gcc411''
debug: Executing '/bin/bash -c '! /bin/rpm -ql gcc411''
info: //Node[moz2-linux-slave17.build.mozilla.org]/staging-buildslave/devtools/Install_rpm[gcc411]/Exec[check-for-gcc411]: Evaluated in 0.48 seconds
info: //Node[moz2-linux-slave17.build.mozilla.org]/staging-buildslave/devtools/Install_rpm[gcc411]/Package[gcc411]: Evaluated in 0.04 seconds


I tested by upgrading and then downgrading android-ndk.

So, the package checks still run, but they don't do any downloading; they just check the local package database (not shown in the output, unfortunately). I double checked this by looking at the access http access logs on staging-puppet. The only entries were for the upgrade and downgrade of android.
Attachment #451725 - Flags: review?(bear)

Updated

8 years ago
Attachment #451725 - Flags: review?(bear) → review+

Comment 3

8 years ago
Comment on attachment 451725 [details] [diff] [review]
fix rpm checks

looks good.  great find on nick's part!
(Assignee)

Updated

8 years ago
Whiteboard: [puppet]
(Assignee)

Comment 4

8 years ago
Comment on attachment 451725 [details] [diff] [review]
fix rpm checks

changeset:   180:f0a34521189c

I had a bit of merging in devtools.pp due to Android changes.
Attachment #451725 - Flags: checked-in+
(Assignee)

Comment 5

8 years ago
This landed, I tested a slave, and have been watching the logs on production-puppet. Haven't seen any issues.
Status: NEW → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.