puppet never reaches state with kernel-firmware package

RESOLVED FIXED

Status

Infrastructure & Operations
RelOps: Puppet
RESOLVED FIXED
2 months ago
a month ago

People

(Reporter: dividehex, Assigned: dragrom)

Tracking

Details

(Reporter)

Description

2 months ago
I've seen this for awhile now. No matter how many times you run puppet, it never reaches state with regards to the kernel-firmware

Notice: /Stage[main]/Packages::Kernel/Package[kernel-firmware-2.6.32-642.13.1.el6]/ensure: current_value absent, should be present (noop)
(Assignee)

Updated

a month ago
Assignee: relops → dcrisan
(Assignee)

Updated

a month ago
Status: NEW → ASSIGNED
(Assignee)

Comment 1

a month ago
I found the followings:

I puppet code we have the following:

$kernel_ver = "${current_kernel}.${::architecture}"
package {
    [ "kernel-${current_kernel}",
      "kernel-headers-${current_kernel}",
      "kernel-firmware-${current_kernel}"]:
          ensure  => present,
          require => Package['bfa-firmware'],
         }

on the server:
[dcrisan@rejh1.srv.releng.scl3.mozilla.com ~]$ facter architecture
x86_64

so, the package will have the name: kernel-firmware-2.6.32-642.13.1.el6.x86_64.rmp

On the yum repository,we have the following kernel-firmware packages:
[dcrisan@releng-puppet2.srv.releng.scl3.mozilla.com ~]$ ll /data/repos/yum/custom/kernel/x86_64/|grep "kernel-firmware"
-rw-r--r-- 1 puppetsync puppetsync 15128264 2014-12-17 04:04 kernel-firmware-2.6.32-504.3.3.el6.noarch.rpm
-rw-r--r-- 1 puppetsync puppetsync 29432056 2017-02-08 09:45 kernel-firmware-2.6.32-642.13.1.el6.noarch.rpm
-rw-r--r-- 1 puppetsync puppetsync 29432364 2017-02-23 12:06 kernel-firmware-2.6.32-642.13.2.el6.noarch.rpm

So, the kernel package firmware have noarch as architecture instead on x86_64.

We have 2 choices: 
1. Add in puppet a logic to replace the architecture with noarch for kernel firmware
2. Rename the package to the repository and replace the noarch with architecture name
(Assignee)

Comment 2

a month ago
but if I looked on the rejh1.srv.releng.mdc1.mozilla.com and run sudo /usr/bin/yum -d 0 -e 0 -y install kernel-firmware-2.6.32-642.13.1.el6, I received: Package matching kernel-firmware-2.6.32-642.13.1.el6.noarch already installed. Checking for update.

So, looks like the package is installed, but puppet for some reason don't find this :), so ignore the first comment :)
(Assignee)

Comment 3

a month ago
looking into the yum log file (sudo cat /var/log/yum.log |grep kernel-firmware) I found this: Jun 28 14:56:07 Updated: kernel-firmware-2.6.32-642.13.2.el6.noarch. Looks like we try to install kernel-firmware-2.6.32-642.13.1.el6 which is older that kernel-firmware-2.6.32-642.13.2.el6. 

But something is strange: in /etc/grub.conf, we have:
title CentOS (2.6.32-642.13.1.el6.x86_64)
	root (hd0,0)
	kernel /vmlinuz-2.6.32-642.13.1.el6.x86_64 ro root=UUID=b43666e8-e254-412e-bc50-5b4f38646ec8 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto  KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet clocksource=pit
	initrd /initramfs-2.6.32-642.13.1.el6.x86_64.img
so looks like we use kernel-2.6.32-642.13.1.el6, but in /var/log/yum.log we have:

[dcrisan@rejh1.srv.releng.mdc1.mozilla.com ~]$ sudo cat /var/log/yum.log |grep kernel|awk -F: '{ print $4 }'|sort -r
 kernel-headers-2.6.32-642.13.1.el6.x86_64
 kernel-firmware-2.6.32-642.13.2.el6.noarch
 kernel-firmware-2.6.32-431.11.2.el6.noarch
 kernel-2.6.32-642.13.1.el6.x86_64
 kernel-2.6.32-431.11.2.el6.x86_64
 dracut-kernel-004-409.el6_8.2.noarch
 dracut-kernel-004-336.el6_5.2.noarch

It is strange, because we install a newest version of kernel-firmware that we specified
(Assignee)

Comment 4

a month ago
[dcrisan@rejh1.srv.releng.mdc1.mozilla.com ~]$ sudo /usr/bin/yum list installed kernel-firmware
Loaded plugins: security
Installed Packages
kernel-firmware.noarch                                                                              2.6.32-642.13.2.el6                                                                              @kernel
(Assignee)

Comment 5

a month ago
What I did:
 - I removed kernel-firmware-2.6.32-642.13.2.el6 package: sudo /usr/bin/yum -d 0 -e 0 -y remove kernel-firmware-2.6.32-642.13.2.el6
 - I run puppet: 
[dcrisan@rejh1.srv.releng.mdc1.mozilla.com ~]$ sudo puppet agent -t --environment=dcrisan --server=releng-puppet2.srv.releng.scl3.mozilla.com
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for rejh1.srv.releng.mdc1.mozilla.com
Info: Applying configuration version '8255684921e8+'
Notice: /Stage[main]/Packages::Kernel/Package[kernel-firmware-2.6.32-642.13.1.el6]/ensure: created
Notice: Finished catalog run in 24.60 seconds
 - On another puppet run:
[dcrisan@rejh1.srv.releng.mdc1.mozilla.com ~]$ sudo puppet agent -t --environment=dcrisan --server=releng-puppet2.srv.releng.scl3.mozilla.com
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for rejh1.srv.releng.mdc1.mozilla.com
Info: Applying configuration version '8255684921e8+'
Notice: Finished catalog run in 19.68 seconds
- Check what kerner-firmware package is installed:
[dcrisan@rejh1.srv.releng.mdc1.mozilla.com ~]$ sudo /usr/bin/yum list installed kernel-firmware
Loaded plugins: security
Installed Packages
kernel-firmware.noarch                                                                              2.6.32-642.13.1.el6                                                                              @kernel

Need to do: look why was installed kernel-firmware-2.6.32-642.13.2.el6 and not kernel-firmware-2.6.32-642.13.1.el6
(Assignee)

Comment 6

a month ago
The nice part on this bug: rejh[i] are VM's, into vmware. Looks like vmwaretools::kernel_upgrade and packages::kernel don't work well together.
(Assignee)

Comment 7

a month ago
Did you installed kernel kernel-2.6.32-642.13.2.el6.x86_64.rpm and then downgraded to kernel-2.6.32-642.13.1.el6.x86_64.rpm on jumphosts? I saw there are packages for kernel 2.6.32-642.13.2.el6.x86_64 in the yum repository. I didn't understood why on the jumphosts is installed kernel-firmware 2.6.32-642.13.2.el6 and the kernel is 2.6.32-642.13.1.el6?
Flags: needinfo?(dhouse)

Comment 8

a month ago
That is correct. I downgraded to kernel-2.6.32-642.13.1 after finding that some releng servers could not install a new kernel because they did not have enough space in /boot: https://bugzilla.mozilla.org/show_bug.cgi?id=1330695
The bugs to increase the space are:
1. bug 1342518 for aws, increase the size in the golden ami's (change made to create_ami script in March '17, so aws instances created off newer ami's have enough space. but I have not checked to see what servers have been built or rebuilt since then)
2. bug 1342521 for in-datacenter hardware and vm's. postponed for now as lower priority with an expectation that we will have to increase the /boot partition size when upgrading to CentOS 7. The mdc1 systems have been created with larger /boot partitions.
Flags: needinfo?(dhouse)
(Assignee)

Comment 9

a month ago
Manually uninstalled kernel-firmware-2.6.32-642.13.2.el6 on jumphost servers and installed using puppet. For now this bug can be closed and marked as resolved on jumphosts.
Status: ASSIGNED → RESOLVED
Last Resolved: a month ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.