Closed Bug 1436781 Opened 6 years ago Closed 6 years ago

nss-util package pinning not applying to treescriptworker

Categories

(Infrastructure & Operations :: RelOps: General, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: dhouse, Assigned: dhouse)

References

Details

Attachments

(1 file)

treescriptworker-dev1 is finding and trying to upgrade to the new nss-util package added to the nss custom repo for the security update bug 1433165.

It may be that we are pinning within a module not included for treescriptworker.
Callek is checking if the ordering matters for pinning nss-utils and nss. Original pin set in https://hg.mozilla.org/build/puppet/rev/12245f06a1c5

```
[dhouse@treescriptworker-dev1.srv.releng.usw2.mozilla.com ~]$ sudo yum list available nss-util
Loaded plugins: security
Available Packages
nss-util.i686                                3.27.1-3.el6                              nss
nss-util.x86_64                              3.27.1-3.el6                              nss
[dhouse@treescriptworker-dev1.srv.releng.usw2.mozilla.com ~]$ sudo /usr/bin/yum -d 0 -e 0
-y install nss-3.21.3-2.el6_8
Package matching nss-3.21.3-2.el6_8.x86_64 already installed. Checking for update.
Error: Package: nss-util-3.27.1-3.el6.x86_64 (nss)
           Requires: nspr >= 4.13.0-1
           Installed: nspr-4.11.0-1.el6.x86_64 (@security_update_1319455)
               nspr = 4.11.0-1.el6
           Available: nspr-4.9.2-1.el6.i686 (releng-public-CentOS6-x86_64)
               nspr = 4.9.2-1.el6
           Available: nspr-4.10.0-1.el6.i686 (base)
               nspr = 4.10.0-1.el6
           Available: nspr-4.10.2-1.el6_5.i686 (updates)
               nspr = 4.10.2-1.el6_5
Error: Package: nss-3.27.1-13.el6.x86_64 (nss)
           Requires: nspr >= 4.13.0
           Installed: nspr-4.11.0-1.el6.x86_64 (@security_update_1319455)
               nspr = 4.11.0-1.el6
           Available: nspr-4.9.2-1.el6.i686 (releng-public-CentOS6-x86_64)
               nspr = 4.9.2-1.el6
           Available: nspr-4.10.0-1.el6.i686 (base)
               nspr = 4.10.0-1.el6
           Available: nspr-4.10.2-1.el6_5.i686 (updates)
               nspr = 4.10.2-1.el6_5
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest
```
Callek is manually pinning and fixing the packages for treescriptworker-dev1. Changing the order did not fix the deps problem.

The package::yum module we're running matches https://github.com/puppetlabs/puppet/blob/3.x/lib/puppet/provider/package/yum.rb (minus the latest commit fixing the list command, which does not relate to this problem).
Attached file puppet failure log
Dragos, from preparing the security update, do you have insight on the nss-utils and nspr package dependency problem seen on treescriptworker? It looks like puppet is not restricting the nss-sysinit's dependency on nss-util to the pinned nss-util version.
Flags: needinfo?(dcrisan)
Yes, I put the newest version of nspr package into security update repository? But I don't know why the puppet try to install the latest version, beacuse the version is pinnet into nss_tools.pp
Flags: needinfo?(dcrisan)
So this is an issue because of the newer version in the nss repository, however nss-tools and nss-sysinit both have interdependencies.


specifically logs to support:

=== List of versions available ===

[root@treescriptworker1.srv.releng.use1.mozilla.com ~]# yum list --showduplicates | grep "nss-sysinit\|nss-tools"
nss-sysinit.x86_64                      3.15.1-15.el6                 @base/$releasever
nss-tools.x86_64                        3.15.1-15.el6                 @base/$releasever
nss-sysinit.x86_64                      3.15.1-15.el6                 base      
nss-sysinit.x86_64                      3.15.3-2.el6_5                updates   
nss-sysinit.x86_64                      3.15.3-3.el6_5                updates   
nss-sysinit.x86_64                      3.15.3-6.el6_5                updates   
nss-sysinit.x86_64                      3.21.3-2.el6_8                nss       
nss-sysinit.x86_64                      3.27.1-13.el6                 nss       
nss-tools.x86_64                        3.15.1-15.el6                 base      
nss-tools.x86_64                        3.15.3-2.el6_5                updates   
nss-tools.x86_64                        3.15.3-3.el6_5                updates   
nss-tools.x86_64                        3.15.3-6.el6_5                updates   
nss-tools.x86_64                        3.21.3-2.el6_8                nss       
nss-tools.x86_64                        3.27.1-13.el6                 nss       


=== Broken nss-tools ==

# yum install nss-tools-3.21.3-2.el6_8
Setting up Install Process
Package matching nss-tools-3.21.3-2.el6_8.x86_64 already installed. Checking for update.
Resolving Dependencies
--> Running transaction check
---> Package nss-tools.x86_64 0:3.15.1-15.el6 will be updated
---> Package nss-tools.x86_64 0:3.21.3-2.el6_8 will be an update
--> Processing Dependency: nss(x86-64) = 3.21.3-2.el6_8 for package: nss-tools-3.21.3-2.el6_8.x86_64
--> Processing Dependency: libssl3.so(NSS_3.20)(64bit) for package: nss-tools-3.21.3-2.el6_8.x86_64
--> Processing Dependency: libssl3.so(NSS_3.15.4)(64bit) for package: nss-tools-3.21.3-2.el6_8.x86_64
--> Processing Dependency: libnss3.so(NSS_3.18)(64bit) for package: nss-tools-3.21.3-2.el6_8.x86_64
--> Processing Dependency: libnss3.so(NSS_3.16.2)(64bit) for package: nss-tools-3.21.3-2.el6_8.x86_64
--> Processing Dependency: libnss3.so(NSS_3.16.1)(64bit) for package: nss-tools-3.21.3-2.el6_8.x86_64
--> Running transaction check
---> Package nss.x86_64 0:3.15.1-15.el6 will be updated
--> Processing Dependency: nss = 3.15.1-15.el6 for package: nss-sysinit-3.15.1-15.el6.x86_64
---> Package nss.x86_64 0:3.21.3-2.el6_8 will be an update
--> Processing Dependency: nss-softokn(x86-64) >= 3.14.3-22 for package: nss-3.21.3-2.el6_8.x86_64
--> Running transaction check
---> Package nss-softokn.x86_64 0:3.14.3-9.el6 will be updated
---> Package nss-softokn.x86_64 0:3.14.3-23.3.el6_8 will be an update
--> Processing Dependency: nss-softokn-freebl(x86-64) >= 3.14.3-23.3.el6_8 for package: nss-softokn-3.14.3-23.3.el6_8.x86_64
---> Package nss-sysinit.x86_64 0:3.15.1-15.el6 will be updated
---> Package nss-sysinit.x86_64 0:3.27.1-13.el6 will be an update
--> Processing Dependency: nss = 3.27.1-13.el6 for package: nss-sysinit-3.27.1-13.el6.x86_64
--> Running transaction check
---> Package nss.x86_64 0:3.15.1-15.el6 will be updated
---> Package nss.x86_64 0:3.15.1-15.el6 will be updated
---> Package nss.x86_64 0:3.21.3-2.el6_8 will be an update
---> Package nss.x86_64 0:3.27.1-13.el6 will be an update
--> Processing Dependency: nss-util >= 3.27.1 for package: nss-3.27.1-13.el6.x86_64
--> Processing Dependency: nspr >= 4.13.0 for package: nss-3.27.1-13.el6.x86_64
--> Processing Dependency: libnssutil3.so(NSSUTIL_3.24)(64bit) for package: nss-3.27.1-13.el6.x86_64
---> Package nss-softokn-freebl.x86_64 0:3.14.3-9.el6 will be updated
---> Package nss-softokn-freebl.x86_64 0:3.14.3-23.3.el6_8 will be an update
---> Package nss-tools.x86_64 0:3.15.1-15.el6 will be updated
---> Package nss-tools.x86_64 0:3.15.1-15.el6 will be updated
---> Package nss-tools.x86_64 0:3.21.3-2.el6_8 will be an update
---> Package nss-tools.x86_64 0:3.27.1-13.el6 will be an update
--> Running transaction check
---> Package nss.x86_64 0:3.27.1-13.el6 will be an update
--> Processing Dependency: nspr >= 4.13.0 for package: nss-3.27.1-13.el6.x86_64
---> Package nss-util.x86_64 0:3.21.3-1.el6_8 will be updated
---> Package nss-util.x86_64 0:3.27.1-3.el6 will be an update
--> Processing Dependency: nspr >= 4.13.0-1 for package: nss-util-3.27.1-3.el6.x86_64
--> Finished Dependency Resolution
Error: Package: nss-util-3.27.1-3.el6.x86_64 (nss)
           Requires: nspr >= 4.13.0-1
           Installed: nspr-4.11.0-1.el6.x86_64 (@security_update_1319455)
               nspr = 4.11.0-1.el6
           Available: nspr-4.9.2-1.el6.i686 (releng-public-CentOS6-x86_64)
               nspr = 4.9.2-1.el6
           Available: nspr-4.10.0-1.el6.i686 (base)
               nspr = 4.10.0-1.el6
           Available: nspr-4.10.2-1.el6_5.i686 (updates)
               nspr = 4.10.2-1.el6_5
Error: Package: nss-3.27.1-13.el6.x86_64 (nss)
           Requires: nspr >= 4.13.0
           Installed: nspr-4.11.0-1.el6.x86_64 (@security_update_1319455)
               nspr = 4.11.0-1.el6
           Available: nspr-4.9.2-1.el6.i686 (releng-public-CentOS6-x86_64)
               nspr = 4.9.2-1.el6
           Available: nspr-4.10.0-1.el6.i686 (base)
               nspr = 4.10.0-1.el6
           Available: nspr-4.10.2-1.el6_5.i686 (updates)
               nspr = 4.10.2-1.el6_5
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

=== Broken nss-sysinit ===

# yum install nss-sysinit-3.21.3-2.el6_8
Setting up Install Process
Package matching nss-sysinit-3.21.3-2.el6_8.x86_64 already installed. Checking for update.
Resolving Dependencies
--> Running transaction check
---> Package nss-sysinit.x86_64 0:3.15.1-15.el6 will be updated
---> Package nss-sysinit.x86_64 0:3.21.3-2.el6_8 will be an update
--> Processing Dependency: nss = 3.21.3-2.el6_8 for package: nss-sysinit-3.21.3-2.el6_8.x86_64
--> Running transaction check
---> Package nss.x86_64 0:3.15.1-15.el6 will be updated
--> Processing Dependency: nss(x86-64) = 3.15.1-15.el6 for package: nss-tools-3.15.1-15.el6.x86_64
---> Package nss.x86_64 0:3.21.3-2.el6_8 will be an update
--> Processing Dependency: nss-softokn(x86-64) >= 3.14.3-22 for package: nss-3.21.3-2.el6_8.x86_64
--> Running transaction check
---> Package nss-softokn.x86_64 0:3.14.3-9.el6 will be updated
---> Package nss-softokn.x86_64 0:3.14.3-23.3.el6_8 will be an update
--> Processing Dependency: nss-softokn-freebl(x86-64) >= 3.14.3-23.3.el6_8 for package: nss-softokn-3.14.3-23.3.el6_8.x86_64
---> Package nss-tools.x86_64 0:3.15.1-15.el6 will be updated
---> Package nss-tools.x86_64 0:3.27.1-13.el6 will be an update
--> Processing Dependency: nss(x86-64) = 3.27.1-13.el6 for package: nss-tools-3.27.1-13.el6.x86_64
--> Processing Dependency: libssl3.so(NSS_3.24)(64bit) for package: nss-tools-3.27.1-13.el6.x86_64
--> Processing Dependency: libssl3.so(NSS_3.22)(64bit) for package: nss-tools-3.27.1-13.el6.x86_64
--> Running transaction check
---> Package nss.x86_64 0:3.15.1-15.el6 will be updated
---> Package nss.x86_64 0:3.15.1-15.el6 will be updated
---> Package nss.x86_64 0:3.21.3-2.el6_8 will be an update
---> Package nss.x86_64 0:3.27.1-13.el6 will be an update
--> Processing Dependency: nss-util >= 3.27.1 for package: nss-3.27.1-13.el6.x86_64
--> Processing Dependency: nspr >= 4.13.0 for package: nss-3.27.1-13.el6.x86_64
--> Processing Dependency: libnssutil3.so(NSSUTIL_3.24)(64bit) for package: nss-3.27.1-13.el6.x86_64
---> Package nss-softokn-freebl.x86_64 0:3.14.3-9.el6 will be updated
---> Package nss-softokn-freebl.x86_64 0:3.14.3-23.3.el6_8 will be an update
---> Package nss-sysinit.x86_64 0:3.15.1-15.el6 will be updated
---> Package nss-sysinit.x86_64 0:3.15.1-15.el6 will be updated
---> Package nss-sysinit.x86_64 0:3.21.3-2.el6_8 will be an update
---> Package nss-sysinit.x86_64 0:3.27.1-13.el6 will be an update
--> Running transaction check
---> Package nss.x86_64 0:3.27.1-13.el6 will be an update
--> Processing Dependency: nspr >= 4.13.0 for package: nss-3.27.1-13.el6.x86_64
---> Package nss-util.x86_64 0:3.21.3-1.el6_8 will be updated
---> Package nss-util.x86_64 0:3.27.1-3.el6 will be an update
--> Processing Dependency: nspr >= 4.13.0-1 for package: nss-util-3.27.1-3.el6.x86_64
--> Finished Dependency Resolution
Error: Package: nss-util-3.27.1-3.el6.x86_64 (nss)
           Requires: nspr >= 4.13.0-1
           Installed: nspr-4.11.0-1.el6.x86_64 (@security_update_1319455)
               nspr = 4.11.0-1.el6
           Available: nspr-4.9.2-1.el6.i686 (releng-public-CentOS6-x86_64)
               nspr = 4.9.2-1.el6
           Available: nspr-4.10.0-1.el6.i686 (base)
               nspr = 4.10.0-1.el6
           Available: nspr-4.10.2-1.el6_5.i686 (updates)
               nspr = 4.10.2-1.el6_5
Error: Package: nss-3.27.1-13.el6.x86_64 (nss)
           Requires: nspr >= 4.13.0
           Installed: nspr-4.11.0-1.el6.x86_64 (@security_update_1319455)
               nspr = 4.11.0-1.el6
           Available: nspr-4.9.2-1.el6.i686 (releng-public-CentOS6-x86_64)
               nspr = 4.9.2-1.el6
           Available: nspr-4.10.0-1.el6.i686 (base)
               nspr = 4.10.0-1.el6
           Available: nspr-4.10.2-1.el6_5.i686 (updates)
               nspr = 4.10.2-1.el6_5
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest


=== Explanation ===

This is because both nss-tools and nss-sysinit depend on an exact version of nss, so when we update either one we get the updated nss, which then forces yum to look for an update to the other, which it finds the newer version, of course then it reruns the transaction checks and tries to update past the version we're pinned against and fails since there are conflicts.

The workaround is to install both nss packages together


=== Work around ===
[root@treescriptworker1.srv.releng.use1.mozilla.com ~]# yum install nss-sysinit-3.21.3-2.el6_8 nss-tools-3.21.3-2.el6_8
Setting up Install Process
Package matching nss-sysinit-3.21.3-2.el6_8.x86_64 already installed. Checking for update.
Package matching nss-tools-3.21.3-2.el6_8.x86_64 already installed. Checking for update.
Resolving Dependencies
--> Running transaction check
---> Package nss-sysinit.x86_64 0:3.15.1-15.el6 will be updated
---> Package nss-sysinit.x86_64 0:3.21.3-2.el6_8 will be an update
--> Processing Dependency: nss = 3.21.3-2.el6_8 for package: nss-sysinit-3.21.3-2.el6_8.x86_64
---> Package nss-tools.x86_64 0:3.15.1-15.el6 will be updated
---> Package nss-tools.x86_64 0:3.21.3-2.el6_8 will be an update
--> Running transaction check
---> Package nss.x86_64 0:3.15.1-15.el6 will be updated
---> Package nss.x86_64 0:3.21.3-2.el6_8 will be an update
--> Processing Dependency: nss-softokn(x86-64) >= 3.14.3-22 for package: nss-3.21.3-2.el6_8.x86_64
--> Running transaction check
---> Package nss-softokn.x86_64 0:3.14.3-9.el6 will be updated
---> Package nss-softokn.x86_64 0:3.14.3-23.3.el6_8 will be an update
--> Processing Dependency: nss-softokn-freebl(x86-64) >= 3.14.3-23.3.el6_8 for package: nss-softokn-3.14.3-23.3.el6_8.x86_64
--> Running transaction check
---> Package nss-softokn-freebl.x86_64 0:3.14.3-9.el6 will be updated
---> Package nss-softokn-freebl.x86_64 0:3.14.3-23.3.el6_8 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================
 Package                             Arch                    Version                               Repository            Size
==============================================================================================================================
Updating:
 nss-sysinit                         x86_64                  3.21.3-2.el6_8                        nss                   47 k
 nss-tools                           x86_64                  3.21.3-2.el6_8                        nss                  437 k
Updating for dependencies:
 nss                                 x86_64                  3.21.3-2.el6_8                        nss                  859 k
 nss-softokn                         x86_64                  3.14.3-23.3.el6_8                     nss                  262 k
 nss-softokn-freebl                  x86_64                  3.14.3-23.3.el6_8                     nss                  168 k

Transaction Summary
==============================================================================================================================
Upgrade       5 Package(s)

Total download size: 1.7 M
Is this ok [y/N]: y
Downloading Packages:
(1/5): nss-3.21.3-2.el6_8.x86_64.rpm                                                                   | 859 kB     00:00     
(2/5): nss-softokn-3.14.3-23.3.el6_8.x86_64.rpm                                                        | 262 kB     00:00     
(3/5): nss-softokn-freebl-3.14.3-23.3.el6_8.x86_64.rpm                                                 | 168 kB     00:00     
(4/5): nss-sysinit-3.21.3-2.el6_8.x86_64.rpm                                                           |  47 kB     00:00     
(5/5): nss-tools-3.21.3-2.el6_8.x86_64.rpm                                                             | 437 kB     00:00     
------------------------------------------------------------------------------------------------------------------------------
Total                                                                                         5.2 MB/s | 1.7 MB     00:00     
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Updating   : nss-softokn-freebl-3.14.3-23.3.el6_8.x86_64                                                               1/10 
  Updating   : nss-softokn-3.14.3-23.3.el6_8.x86_64                                                                      2/10 
  Updating   : nss-sysinit-3.21.3-2.el6_8.x86_64                                                                         3/10 
  Updating   : nss-3.21.3-2.el6_8.x86_64                                                                                 4/10 
  Updating   : nss-tools-3.21.3-2.el6_8.x86_64                                                                           5/10 
  Cleanup    : nss-tools-3.15.1-15.el6.x86_64                                                                            6/10 
  Cleanup    : nss-sysinit-3.15.1-15.el6.x86_64                                                                          7/10 
  Cleanup    : nss-3.15.1-15.el6.x86_64                                                                                  8/10 
  Cleanup    : nss-softokn-3.14.3-9.el6.x86_64                                                                           9/10 
  Cleanup    : nss-softokn-freebl-3.14.3-9.el6.x86_64                                                                   10/10 
  Verifying  : nss-softokn-3.14.3-23.3.el6_8.x86_64                                                                      1/10 
  Verifying  : nss-3.21.3-2.el6_8.x86_64                                                                                 2/10 
  Verifying  : nss-softokn-freebl-3.14.3-23.3.el6_8.x86_64                                                               3/10 
  Verifying  : nss-sysinit-3.21.3-2.el6_8.x86_64                                                                         4/10 
  Verifying  : nss-tools-3.21.3-2.el6_8.x86_64                                                                           5/10 
  Verifying  : nss-softokn-freebl-3.14.3-9.el6.x86_64                                                                    6/10 
  Verifying  : nss-softokn-3.14.3-9.el6.x86_64                                                                           7/10 
  Verifying  : nss-3.15.1-15.el6.x86_64                                                                                  8/10 
  Verifying  : nss-tools-3.15.1-15.el6.x86_64                                                                            9/10 
  Verifying  : nss-sysinit-3.15.1-15.el6.x86_64                                                                         10/10 

Updated:
  nss-sysinit.x86_64 0:3.21.3-2.el6_8                            nss-tools.x86_64 0:3.21.3-2.el6_8                           

Dependency Updated:
  nss.x86_64 0:3.21.3-2.el6_8     nss-softokn.x86_64 0:3.14.3-23.3.el6_8     nss-softokn-freebl.x86_64 0:3.14.3-23.3.el6_8    

Complete!


=== Real fix in puppet ===

There are two ways to fix this in our puppet version, one is to update the nss files to the security release now, so then they are always finding the latest version during a puppet run.

The other is to do an Exec or similar to run this as one yum transaction.
Thankyou Justin for the detailed notes! I haven't picked this back up yet, but I'll put together a patch tomorrow. I'll likely end up doing the exec like you suggested.
I removed the 3.27* packages from the nss custom repo to prevent production servers from hitting this problem.

I'll make a note in bug 1433165
I am not seeing the 3.27 nss packages now from the puppet master yum mirrors:
```
dhouse@releng-puppet2:/data/repos/yum/custom/nss/x86_64$ yum list --showduplicates | grep "nss-sysinit\|nss-tools"
[...]
nss-sysinit.x86_64                      3.21.3-2.el6_8                @nss
nss-tools.x86_64                        3.21.3-2.el6_8                @nss
nss-sysinit.x86_64                      3.15.1-15.el6                 base
nss-sysinit.x86_64                      3.15.3-2.el6_5                updates
nss-sysinit.x86_64                      3.15.3-3.el6_5                updates
nss-sysinit.x86_64                      3.15.3-6.el6_5                updates
nss-sysinit.x86_64                      3.21.3-2.el6_8                nss
nss-tools.x86_64                        3.15.1-15.el6                 base
nss-tools.x86_64                        3.15.3-2.el6_5                updates
nss-tools.x86_64                        3.15.3-3.el6_5                updates
nss-tools.x86_64                        3.15.3-6.el6_5                updates
nss-tools.x86_64                        3.21.3-2.el6_8                nss
```
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: