Closed Bug 928000 Opened 12 years ago Closed 12 years ago

We currently cannot create EC2 builders without manual intervention

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

Product:
Infrastructure & Operations Graveyard
Component:
CIDuty
Platform:
x86
macOS
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: armenzg, Assigned: rail)

References

Details

I haven't figured out how to fix it even after reading https://bugzilla.mozilla.org/show_bug.cgi?id=870853#c24 and https://bugzilla.mozilla.org/show_bug.cgi?id=870853#c26 Anyone know how to work around it? ############################### 14:32 armenzg: rail: do you think you could help with this? 14:33 rail: armenzg: not before https://bugzilla.mozilla.org/show_bug.cgi?id=905322 is resolved, we can't use http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/centos/6/latest/os/i386/ with current scripts, because the yum groups are broken in that repo 14:34 rail: we use http://puppetagain.pub.build.mozilla.org/data/repos/yum/mirrors/centos/6/latest/os/x86_64-for-ks/ ############################### Thu Oct 17 11:04:09 -0700 2013 Puppet (err): Could not update: Execution of '/usr/bin/yum -d 0 -e 0 -y install collectd-libvirt-5.3.0-2.el6' returned 1: Error: Package: glibc-2.12-1.47.el6_2.5.i686 (updates) Requires: glibc-common = 2.12-1.47.el6_2.5 Installed: glibc-common-2.12-1.47.el6_2.12.x86_64 (@updates/$releasever) glibc-common = 2.12-1.47.el6_2.12 Available: glibc-common-2.12-1.47.el6.x86_64 (base) glibc-common = 2.12-1.47.el6 Available: glibc-common-2.12-1.47.el6_2.5.x86_64 (updates) glibc-common = 2.12-1.47.el6_2.5 You could try using --skip-broken to work around the problem ** Found 1 pre-existing rpmdb problem(s), 'yum check' output follows: collectd-5.3.0-2.el6.x86_64 has installed conflicts collectd <= ('0', '5.3.0', '2.el6'): collectd-5.3.0-2.el6.x86_64 Thu Oct 17 11:04:09 -0700 2013 /Stage[main]/Packages::Collectd/Package[collectd-libvirt]/ensure (err): change from absent to 5.3.0-2.el6 failed: Could not update: Execution of '/usr/bin/yum -d 0 -e 0 -y install collectd-libvirt-5.3.0-2.el6' returned 1: Error: Package: glibc-2.12-1.47.el6_2.5.i686 (updates) Requires: glibc-common = 2.12-1.47.el6_2.5 Installed: glibc-common-2.12-1.47.el6_2.12.x86_64 (@updates/$releasever) glibc-common = 2.12-1.47.el6_2.12 Available: glibc-common-2.12-1.47.el6.x86_64 (base) glibc-common = 2.12-1.47.el6 Available: glibc-common-2.12-1.47.el6_2.5.x86_64 (updates) glibc-common = 2.12-1.47.el6_2.5 You could try using --skip-broken to work around the problem ** Found 1 pre-existing rpmdb problem(s), 'yum check' output follows: collectd-5.3.0-2.el6.x86_64 has installed conflicts collectd <= ('0', '5.3.0', '2.el6'): collectd-5.3.0-2.el6.x86_64
Have you tried "yum downgrade glibc-common"? Generally when your glibc versions are out of whack you're in pretty bad shape, but you might be able to recover with a bunch of downgrades.
[root@dev-bld-linux64-ec2-dminor.dev.releng.use1.mozilla.com ~]# yum downgrade glibc-common Loaded plugins: security Setting up Downgrade Process Resolving Dependencies --> Running transaction check ---> Package glibc-common.x86_64 0:2.12-1.47.el6_2.5 will be a downgrade --> Processing Dependency: glibc = 2.12-1.47.el6_2.5 for package: glibc-common-2.12-1.47.el6_2.5.x86_64 ---> Package glibc-common.x86_64 0:2.12-1.47.el6_2.12 will be erased --> Running transaction check ---> Package glibc.i686 0:2.12-1.47.el6_2.5 will be installed --> Processing Dependency: libfreebl3.so(NSSRAWHASH_3.12.3) for package: glibc-2.12-1.47.el6_2.5.i686 --> Processing Dependency: libfreebl3.so for package: glibc-2.12-1.47.el6_2.5.i686 ---> Package glibc-common.x86_64 0:2.12-1.47.el6_2.12 will be erased --> Running transaction check ---> Package glibc-common.x86_64 0:2.12-1.47.el6_2.12 will be erased ---> Package nss-softokn-freebl.i686 0:3.12.9-11.el6 will be installed --> Finished Dependency Resolution Error: Package: glibc-2.12-1.47.el6_2.12.x86_64 (@updates/$releasever) Requires: glibc-common = 2.12-1.47.el6_2.12 Removing: glibc-common-2.12-1.47.el6_2.12.x86_64 (@updates/$releasever) glibc-common = 2.12-1.47.el6_2.12 Downgraded By: glibc-common-2.12-1.47.el6_2.5.x86_64 (updates) glibc-common = 2.12-1.47.el6_2.5 Available: glibc-common-2.12-1.47.el6.x86_64 (base) glibc-common = 2.12-1.47.el6 You could try using --skip-broken to work around the problem ** Found 1 pre-existing rpmdb problem(s), 'yum check' output follows: collectd-5.3.0-2.el6.x86_64 has installed conflicts collectd <= ('0', '5.3.0', '2.el6'): collectd-5.3.0-2.el6.x86_64
[root@dev-bld-linux64-ec2-dminor.dev.releng.use1.mozilla.com ~]# yum downgrade glibc glibc-common Loaded plugins: security Setting up Downgrade Process Resolving Dependencies --> Running transaction check ---> Package glibc.x86_64 0:2.12-1.47.el6_2.5 will be a downgrade ---> Package glibc.x86_64 0:2.12-1.47.el6_2.12 will be erased ---> Package glibc-common.x86_64 0:2.12-1.47.el6_2.5 will be a downgrade ---> Package glibc-common.x86_64 0:2.12-1.47.el6_2.12 will be erased --> Finished Dependency Resolution Dependencies Resolved ========================================================================================================================================================================== Package Arch Version Repository Size ========================================================================================================================================================================== Downgrading: glibc x86_64 2.12-1.47.el6_2.5 updates 3.8 M glibc-common x86_64 2.12-1.47.el6_2.5 updates 14 M Transaction Summary ========================================================================================================================================================================== Downgrade 2 Package(s) Total download size: 18 M Is this ok [y/N]: Y Downloading Packages: (1/2): glibc-2.12-1.47.el6_2.5.x86_64.rpm | 3.8 MB 00:00 (2/2): glibc-common-2.12-1.47.el6_2.5.x86_64.rpm | 14 MB 00:00 -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 17 MB/s | 18 MB 00:01 Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Installing : glibc-2.12-1.47.el6_2.5.x86_64 1/4 Installing : glibc-common-2.12-1.47.el6_2.5.x86_64 2/4 Cleanup : glibc-common-2.12-1.47.el6_2.12.x86_64 3/4 Cleanup : glibc-2.12-1.47.el6_2.12.x86_64 4/4 Removed: glibc.x86_64 0:2.12-1.47.el6_2.12 glibc-common.x86_64 0:2.12-1.47.el6_2.12 Installed: glibc.x86_64 0:2.12-1.47.el6_2.5 glibc-common.x86_64 0:2.12-1.47.el6_2.5 Complete!
It turns out that AMI ID was quite old in the configs. Pushed the fix: https://hg.mozilla.org/build/cloud-tools/rev/be85aae3b14b
rail fixed it! Thanks rail!
Assignee: nobody → rail
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Component: Platform Support → Buildduty
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.