We currently cannot create EC2 builders without manual intervention

RESOLVED FIXED

Status

Release Engineering
Platform Support
RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: armenzg, Assigned: rail)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

4 years ago
I haven't figured out how to fix it even after reading https://bugzilla.mozilla.org/show_bug.cgi?id=870853#c24 and https://bugzilla.mozilla.org/show_bug.cgi?id=870853#c26

Anyone know how to work around it?

###############################
14:32 armenzg: rail: do you think you could help with this?
14:33 rail: armenzg: not before https://bugzilla.mozilla.org/show_bug.cgi?id=905322 is resolved, we can't use http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/centos/6/latest/os/i386/ with current scripts, because the yum groups are broken in that repo
14:34 rail: we use http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/centos/6/latest/os/x86_64-for-ks/

###############################

Thu Oct 17 11:04:09 -0700 2013 Puppet (err): Could not update: Execution of '/usr/bin/yum -d 0 -e 0 -y install collectd-libvirt-5.3.0-2.el6' returned 1: Error: Package: glibc-2.12-1.47.el6_2.5.i686 (updates)
           Requires: glibc-common = 2.12-1.47.el6_2.5
           Installed: glibc-common-2.12-1.47.el6_2.12.x86_64 (@updates/$releasever)
               glibc-common = 2.12-1.47.el6_2.12
           Available: glibc-common-2.12-1.47.el6.x86_64 (base)
               glibc-common = 2.12-1.47.el6
           Available: glibc-common-2.12-1.47.el6_2.5.x86_64 (updates)
               glibc-common = 2.12-1.47.el6_2.5
 You could try using --skip-broken to work around the problem
** Found 1 pre-existing rpmdb problem(s), 'yum check' output follows:
collectd-5.3.0-2.el6.x86_64 has installed conflicts collectd <= ('0', '5.3.0', '2.el6'): collectd-5.3.0-2.el6.x86_64

Thu Oct 17 11:04:09 -0700 2013 /Stage[main]/Packages::Collectd/Package[collectd-libvirt]/ensure (err): change from absent to 5.3.0-2.el6 failed: Could not update: Execution of '/usr/bin/yum -d 0 -e 0 -y install collectd-libvirt-5.3.0-2.el6' returned 1: Error: Package: glibc-2.12-1.47.el6_2.5.i686 (updates)
           Requires: glibc-common = 2.12-1.47.el6_2.5
           Installed: glibc-common-2.12-1.47.el6_2.12.x86_64 (@updates/$releasever)
               glibc-common = 2.12-1.47.el6_2.12
           Available: glibc-common-2.12-1.47.el6.x86_64 (base)
               glibc-common = 2.12-1.47.el6
           Available: glibc-common-2.12-1.47.el6_2.5.x86_64 (updates)
               glibc-common = 2.12-1.47.el6_2.5
 You could try using --skip-broken to work around the problem
** Found 1 pre-existing rpmdb problem(s), 'yum check' output follows:
collectd-5.3.0-2.el6.x86_64 has installed conflicts collectd <= ('0', '5.3.0', '2.el6'): collectd-5.3.0-2.el6.x86_64
Have you tried "yum downgrade glibc-common"?  Generally when your glibc versions are out of whack you're in pretty bad shape, but you might be able to recover with a bunch of downgrades.
(Reporter)

Comment 2

4 years ago
[root@dev-bld-linux64-ec2-dminor.dev.releng.use1.mozilla.com ~]# yum downgrade glibc-common
Loaded plugins: security
Setting up Downgrade Process
Resolving Dependencies
--> Running transaction check
---> Package glibc-common.x86_64 0:2.12-1.47.el6_2.5 will be a downgrade
--> Processing Dependency: glibc = 2.12-1.47.el6_2.5 for package: glibc-common-2.12-1.47.el6_2.5.x86_64
---> Package glibc-common.x86_64 0:2.12-1.47.el6_2.12 will be erased
--> Running transaction check
---> Package glibc.i686 0:2.12-1.47.el6_2.5 will be installed
--> Processing Dependency: libfreebl3.so(NSSRAWHASH_3.12.3) for package: glibc-2.12-1.47.el6_2.5.i686
--> Processing Dependency: libfreebl3.so for package: glibc-2.12-1.47.el6_2.5.i686
---> Package glibc-common.x86_64 0:2.12-1.47.el6_2.12 will be erased
--> Running transaction check
---> Package glibc-common.x86_64 0:2.12-1.47.el6_2.12 will be erased
---> Package nss-softokn-freebl.i686 0:3.12.9-11.el6 will be installed
--> Finished Dependency Resolution
Error: Package: glibc-2.12-1.47.el6_2.12.x86_64 (@updates/$releasever)
           Requires: glibc-common = 2.12-1.47.el6_2.12
           Removing: glibc-common-2.12-1.47.el6_2.12.x86_64 (@updates/$releasever)
               glibc-common = 2.12-1.47.el6_2.12
           Downgraded By: glibc-common-2.12-1.47.el6_2.5.x86_64 (updates)
               glibc-common = 2.12-1.47.el6_2.5
           Available: glibc-common-2.12-1.47.el6.x86_64 (base)
               glibc-common = 2.12-1.47.el6
 You could try using --skip-broken to work around the problem
** Found 1 pre-existing rpmdb problem(s), 'yum check' output follows:
collectd-5.3.0-2.el6.x86_64 has installed conflicts collectd <= ('0', '5.3.0', '2.el6'): collectd-5.3.0-2.el6.x86_64
(Reporter)

Comment 3

4 years ago
[root@dev-bld-linux64-ec2-dminor.dev.releng.use1.mozilla.com ~]# yum downgrade glibc glibc-common
Loaded plugins: security
Setting up Downgrade Process
Resolving Dependencies
--> Running transaction check
---> Package glibc.x86_64 0:2.12-1.47.el6_2.5 will be a downgrade
---> Package glibc.x86_64 0:2.12-1.47.el6_2.12 will be erased
---> Package glibc-common.x86_64 0:2.12-1.47.el6_2.5 will be a downgrade
---> Package glibc-common.x86_64 0:2.12-1.47.el6_2.12 will be erased
--> Finished Dependency Resolution

Dependencies Resolved

==========================================================================================================================================================================
 Package                                   Arch                                Version                                         Repository                            Size
==========================================================================================================================================================================
Downgrading:
 glibc                                     x86_64                              2.12-1.47.el6_2.5                               updates                              3.8 M
 glibc-common                              x86_64                              2.12-1.47.el6_2.5                               updates                               14 M

Transaction Summary
==========================================================================================================================================================================
Downgrade     2 Package(s)

Total download size: 18 M
Is this ok [y/N]: Y
Downloading Packages:
(1/2): glibc-2.12-1.47.el6_2.5.x86_64.rpm                                                                                                          | 3.8 MB     00:00     
(2/2): glibc-common-2.12-1.47.el6_2.5.x86_64.rpm                                                                                                   |  14 MB     00:00     
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                      17 MB/s |  18 MB     00:01     
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : glibc-2.12-1.47.el6_2.5.x86_64                                                                                                                         1/4 
  Installing : glibc-common-2.12-1.47.el6_2.5.x86_64                                                                                                                  2/4 
  Cleanup    : glibc-common-2.12-1.47.el6_2.12.x86_64                                                                                                                 3/4 
  Cleanup    : glibc-2.12-1.47.el6_2.12.x86_64                                                                                                                        4/4 

Removed:
  glibc.x86_64 0:2.12-1.47.el6_2.12                                                glibc-common.x86_64 0:2.12-1.47.el6_2.12                                               

Installed:
  glibc.x86_64 0:2.12-1.47.el6_2.5                                                 glibc-common.x86_64 0:2.12-1.47.el6_2.5                                                

Complete!
(Assignee)

Comment 4

4 years ago
It turns out that AMI ID was quite old in the configs. Pushed the fix: https://hg.mozilla.org/build/cloud-tools/rev/be85aae3b14b
(Reporter)

Comment 5

4 years ago
rail fixed it!

Thanks rail!
Assignee: nobody → rail
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.