add a mechanism to flush yum metadata caches

RESOLVED FIXED

Status

Infrastructure & Operations
RelOps
RESOLVED FIXED
6 years ago
5 years ago

People

(Reporter: dustin, Assigned: dustin)

Tracking

Details

Attachments

(1 attachment)

(Assignee)

Description

6 years ago
Yum caches repo metadata for 6 hours by default.  We should have a way to force that to update.

My thinking is to add a file to the packages module with an integer that we just increment in hg to force an update by running
  yum clean expire-cache
during the packagesetup stage.
(Assignee)

Comment 1

6 years ago
Docs for the wiki:

== Forcing Metadata Updates ==

By default, yum caches repository metadata for 6 hours, which is perfectly reasonable for a distro, but may cause puppet to not see an updated package in a repository for up to six hours.  If updated manifests depend on that updated package, this can cause significant problems!

There is a simple mechanism in place to invalidate the cached metadata on every host: increment the contents of ''/etc/.repo-flag'' in ''modules/packages/manifests/setup.pp''.  The changed file will trigger a run of 'yum clean expire-cache', which causes the metadata to be re-fetched next time yum is run.  Include the increment of this value in your patch for review.  It can't hurt to increment this value!
(Assignee)

Comment 2

6 years ago
Created attachment 640889 [details] [diff] [review]
bug772599.patch

I did the following to test that this will affect yum invocations in the same puppet run:

* created mozilla-python26-2.6.7-6.el6 (same content as -5)
* added 'include packages::python26' to relabs08's node temporarily
* ran puppet, saw that nothing happened
* put the new RPM on releng-puppet1.build.mtv1.mozilla.com temporarily
* ran createrepo --update .
* ran puppet, saw that nothing happened (the bug we're trying to fix)
> [root@relabs08 ~]# puppet agent --test --server=relabs-puppet.build.mtv1.mozilla.com --environment=dmitchell
> info: Caching catalog for relabs08.build.mtv1.mozilla.com
> info: Applying configuration version '420394838d95'
> notice: Finished catalog run in 3.36 seconds
* bumped the $repoflag
* ran puppet, saw that the new version of python26 was installed:
> [root@relabs08 ~]# puppet agent --test --server=relabs-puppet.build.mtv1.mozilla.com --environment=dmitchell
> info: Caching catalog for relabs08.build.mtv1.mozilla.com
> info: Applying configuration version '420394838d95'
> notice: /Stage[packagesetup]/Packages::Setup/File[/etc/.repo-flag]/content:
> --- /etc/.repo-flag     2012-07-10 18:34:55.421892053 -0700
> +++ /tmp/puppet-file20120710-39868-cq5blc-0     2012-07-10 18:48:23.601883470 -0700
> @@ -1,2 +1,2 @@
>  # see $repoflag in modules/packages/manifests/setup.pp
> -2
> +3
>
> info: FileBucket adding {md5}5a166919d5b4d675649291c09dd200e8
> info: /Stage[packagesetup]/Packages::Setup/File[/etc/.repo-flag]: Filebucketed /etc/.repo-flag to puppet with sum 5a166919d5b4d675649291c09dd200e8
> notice: /Stage[packagesetup]/Packages::Setup/File[/etc/.repo-flag]/content: content changed '{md5}5a166919d5b4d675649291c09dd200e8' to '{md5}ce4ecd02e02f7cfb1f0d0613b07fda92'
> info: /Stage[packagesetup]/Packages::Setup/File[/etc/.repo-flag]: Scheduling refresh of Exec[yum-clean-expire-cache]
> notice: /Stage[packagesetup]/Packages::Setup/Exec[yum-clean-expire-cache]: Triggered 'refresh' from 1 events
> notice: /Stage[main]/Packages::Mozilla::Python26/Package[mozilla-python26]/ensure: ensure changed '2.6.7-5.el6' to '0:2.6.7-6.el6'
> notice: Finished catalog run in 10.06 seconds
* used rsync to overwrite the temporary changes to the repo
Assignee: server-ops-releng → dustin
Attachment #640889 - Flags: review?(bugspam.Callek)
Comment on attachment 640889 [details] [diff] [review]
bug772599.patch

I'm a fan of yum clean all, but that might be more damaging here (take longer too) than you are looking for
Attachment #640889 - Flags: review?(bugspam.Callek) → review+
(Assignee)

Comment 4

6 years ago
This achieves the desired effect, and I *think* it uses if-modified-since requests or something like that to be even more efficient.  At any rate, I only see it re-download the changed repo data.
(Assignee)

Updated

6 years ago
Attachment #640889 - Flags: checked-in+
(Assignee)

Updated

6 years ago
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
Component: Server Operations: RelEng → RelOps
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.