872549 - Nominate ubuntu ix systems for upgrade to puppet 3.2.0

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task, P3)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

I'd like to test upgrading a few Ubuntu systems to Puppet-3.2.0, before upgrading the entire silo.

Please nominate two machines for upgrade.  Once they're upgraded, we'll put them in staging to make sure they behave properly, then move them back to production.

Aside from the need to add the proper .deb to the puppet configs, I don't anticipate any difficulties with this, but caution doesn't hurt.

Comment 1

[Armen [:armenzg]]

A couple of talos-linux64-ix=* and a couple of talos-linux32-ix-* hosts?

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

That sounds perfect, yes

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

I gave armen the wrong bug #:
> talos-linux*-ix-00{5,6} are yours.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Here's the full upgrade sequence from puppet:

root@talos-linux32-ix-006:~# puppet agent --test --server=releng-puppet2.srv.releng.scl3.mozilla.com --environment=test
info: Retrieving plugin
err: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate: Error 400 on SERVER: this master is not a CA
info: Loading facts in /var/lib/puppet/lib/facter/num_masters.rb
info: Caching catalog for talos-linux32-ix-006.test.releng.scl3.mozilla.com
info: /Stage[main]/Tweaks::Cleanup/Tidy[/home/cltbld/.mozilla/firefox/console.log]: File does not exist
info: Applying configuration version 'e9a1fe7d4131'
notice: /Stage[packagesetup]/Packages::Setup/Packages::Aptrepo[precise]/File[/etc/apt/sources.list.d/precise.list]/content: 
DIFFS DISABLED - https://bugzilla.mozilla.org/show_bug.cgi?id=791102 /etc/apt/sources.list.d/precise.list /tmp/puppet-file20130522-2821-szhg6s-0

notice: /Stage[packagesetup]/Packages::Setup/Packages::Aptrepo[precise]/File[/etc/apt/sources.list.d/precise.list]/content: content changed '{md5}001a8e0d54893e0c6bb45448bf03496b' to '{md5}681addfd35d31021e7e7882696210d4d'
info: /Stage[packagesetup]/Packages::Setup/Packages::Aptrepo[precise]/File[/etc/apt/sources.list.d/precise.list]: Scheduling refresh of Exec[apt-get-update]
notice: /Stage[packagesetup]/Packages::Setup/Packages::Aptrepo[xorg-edgers]/File[/etc/apt/sources.list.d/xorg-edgers.list]/content: 
DIFFS DISABLED - https://bugzilla.mozilla.org/show_bug.cgi?id=791102 /etc/apt/sources.list.d/xorg-edgers.list /tmp/puppet-file20130522-2821-rp5p6q-0

notice: /Stage[packagesetup]/Packages::Setup/Packages::Aptrepo[xorg-edgers]/File[/etc/apt/sources.list.d/xorg-edgers.list]/content: content changed '{md5}0189f25575cfe649363bbbcc6a297ecc' to '{md5}7210409aa90914629f5423206bc8e894'
info: /Stage[packagesetup]/Packages::Setup/Packages::Aptrepo[xorg-edgers]/File[/etc/apt/sources.list.d/xorg-edgers.list]: Scheduling refresh of Exec[apt-get-update]
notice: /Stage[packagesetup]/Packages::Setup/Packages::Aptrepo[releng]/File[/etc/apt/sources.list.d/releng.list]/content: 
DIFFS DISABLED - https://bugzilla.mozilla.org/show_bug.cgi?id=791102 /etc/apt/sources.list.d/releng.list /tmp/puppet-file20130522-2821-1rtl1zt-0

notice: /Stage[packagesetup]/Packages::Setup/Packages::Aptrepo[releng]/File[/etc/apt/sources.list.d/releng.list]/content: content changed '{md5}8582e64ef3b1f833d69de50635717c92' to '{md5}a22325ac76042df4e938619324a546b4'
info: /Stage[packagesetup]/Packages::Setup/Packages::Aptrepo[releng]/File[/etc/apt/sources.list.d/releng.list]: Scheduling refresh of Exec[apt-get-update]
notice: /Stage[packagesetup]/Packages::Setup/Packages::Aptrepo[precise-security]/File[/etc/apt/sources.list.d/precise-security.list]/content: 
DIFFS DISABLED - https://bugzilla.mozilla.org/show_bug.cgi?id=791102 /etc/apt/sources.list.d/precise-security.list /tmp/puppet-file20130522-2821-1od29cj-0

notice: /Stage[packagesetup]/Packages::Setup/Packages::Aptrepo[precise-security]/File[/etc/apt/sources.list.d/precise-security.list]/content: content changed '{md5}b877a7fdace7ae2ad28a33285576f0d3' to '{md5}1477ade6e82e583fad3e91609721e9ef'
info: /Stage[packagesetup]/Packages::Setup/Packages::Aptrepo[precise-security]/File[/etc/apt/sources.list.d/precise-security.list]: Scheduling refresh of Exec[apt-get-update]
notice: /Stage[packagesetup]/Packages::Setup/Packages::Aptrepo[releng-updates]/File[/etc/apt/sources.list.d/releng-updates.list]/content: 
DIFFS DISABLED - https://bugzilla.mozilla.org/show_bug.cgi?id=791102 /etc/apt/sources.list.d/releng-updates.list /tmp/puppet-file20130522-2821-7ig3ng-0

notice: /Stage[packagesetup]/Packages::Setup/Packages::Aptrepo[releng-updates]/File[/etc/apt/sources.list.d/releng-updates.list]/content: content changed '{md5}0728a61f71e9371b8503279c9369c556' to '{md5}bdc6a166197afa2d23726b25927b72ba'
info: /Stage[packagesetup]/Packages::Setup/Packages::Aptrepo[releng-updates]/File[/etc/apt/sources.list.d/releng-updates.list]: Scheduling refresh of Exec[apt-get-update]
notice: /Stage[packagesetup]/Packages::Setup/Packages::Aptrepo[puppetlabs]/File[/etc/apt/sources.list.d/puppetlabs.list]/ensure: defined content as '{md5}100ccf449d51afc8ba04259a907f5e1e'
info: /Stage[packagesetup]/Packages::Setup/Packages::Aptrepo[puppetlabs]/File[/etc/apt/sources.list.d/puppetlabs.list]: Scheduling refresh of Exec[apt-get-update]
notice: /Stage[packagesetup]/Packages::Setup/Exec[apt-get-update]: Triggered 'refresh' from 6 events
notice: /Stage[packagesetup]/Packages::Setup/Exec[apt-get-update-daily]/returns: executed successfully
notice: /Stage[main]/Tweaks::Cleanup/Exec[find /tmp/* -mmin +15 -print | xargs -n1 rm -rf]/returns: executed successfully
notice: /Stage[main]/Users::Builder::Setup/Python::User_pip_conf[cltbld]/File[/home/cltbld/.pip/pip.conf]/content: 
DIFFS DISABLED - https://bugzilla.mozilla.org/show_bug.cgi?id=791102 /home/cltbld/.pip/pip.conf /tmp/puppet-file20130522-2821-fuqqw-0

notice: /Stage[main]/Users::Builder::Setup/Python::User_pip_conf[cltbld]/File[/home/cltbld/.pip/pip.conf]/content: content changed '{md5}ef3a4af6a10c771e701334f1e1ad2156' to '{md5}0312d7f9ac6641e2cb17f9bc4862779c'
notice: /Stage[main]/Puppet::Atboot/File[/etc/puppet/puppetmasters.txt]/content: 
DIFFS DISABLED - https://bugzilla.mozilla.org/show_bug.cgi?id=791102 /etc/puppet/puppetmasters.txt /tmp/puppet-file20130522-2821-cfwo4r-0

notice: /Stage[main]/Puppet::Atboot/File[/etc/puppet/puppetmasters.txt]/content: content changed '{md5}f950a1b7115db3f0e29bf505184a229b' to '{md5}73695d159bd26c49f966374794c5fdca'
notice: /Stage[main]/Puppet::Atboot/File[/etc/init/puppet.conf]/content: 
DIFFS DISABLED - https://bugzilla.mozilla.org/show_bug.cgi?id=791102 /etc/init/puppet.conf /tmp/puppet-file20130522-2821-17f39q0-0

notice: /Stage[main]/Puppet::Atboot/File[/etc/init/puppet.conf]/content: content changed '{md5}1d18ca27820430caa641e17bbcc8426a' to '{md5}6cad5fb53f236527d808beef7e9b1f41'
err: /Stage[main]/Packages::Puppet/Package[puppet-common]/ensure: change from 2.7.17-1mozilla1 to 3.2.1-1puppetlabs1 failed: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold --force-yes install puppet-common=3.2.1-1puppetlabs1' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
The following extra packages will be installed:
  facter hiera libjson-ruby libruby puppet ruby-json ruby-rgen virt-what
Suggested packages:
  puppet-el vim-puppet ruby-selinux libselinux-ruby1.8 librrd-ruby1.9.1
  librrd-ruby1.8
The following NEW packages will be installed:
  hiera libjson-ruby libruby ruby-json ruby-rgen virt-what
The following packages will be upgraded:
  facter puppet puppet-common
3 upgraded, 6 newly installed, 0 to remove and 7 not upgraded.
Need to get 1290 kB of archives.
After this operation, 1200 kB of additional disk space will be used.
WARNING: The following packages cannot be authenticated!
  libruby virt-what facter ruby-json libjson-ruby hiera ruby-rgen puppet
  puppet-common
Authentication warning overridden.
Get:1 http://releng-puppet2.srv.releng.scl3.mozilla.com/repos/apt/ubuntu/ precise/main libruby all 4.8 [4766 B]
Get:2 http://releng-puppet2.srv.releng.scl3.mozilla.com/repos/apt/ubuntu/ precise/universe virt-what i386 1.11-1 [12.8 kB]
Get:3 http://releng-puppet2.srv.releng.scl3.mozilla.com/repos/apt/puppetlabs/ precise/main facter i386 1.7.1-1puppetlabs1 [73.3 kB]
Get:4 http://releng-puppet2.srv.releng.scl3.mozilla.com/repos/apt/ubuntu/ precise/universe ruby-json i386 1.6.3-1 [72.4 kB]
Get:5 http://releng-puppet2.srv.releng.scl3.mozilla.com/repos/apt/ubuntu/ precise/universe libjson-ruby all 1.6.3-1 [1588 B]
Get:6 http://releng-puppet2.srv.releng.scl3.mozilla.com/repos/apt/puppetlabs/ precise/main hiera all 1.2.1-1puppetlabs1 [18.7 kB]
Get:7 http://releng-puppet2.srv.releng.scl3.mozilla.com/repos/apt/puppetlabs/ precise/dependencies ruby-rgen all 0.6.2-1puppetlabs1 [81.8 kB]
Get:8 http://releng-puppet2.srv.releng.scl3.mozilla.com/repos/apt/puppetlabs/ precise/main puppet all 3.2.1-1puppetlabs1 [68.3 kB]
Get:9 http://releng-puppet2.srv.releng.scl3.mozilla.com/repos/apt/puppetlabs/ precise/main puppet-common all 3.2.1-1puppetlabs1 [957 kB]
Fetched 1290 kB in 0s (11.7 MB/s)
Selecting previously unselected package libruby.
(Reading database ... 149636 files and directories currently installed.)
Unpacking libruby (from .../archives/libruby_4.8_all.deb) ...
Selecting previously unselected package virt-what.
Unpacking virt-what (from .../virt-what_1.11-1_i386.deb) ...
Preparing to replace facter 1.6.5-1ubuntu1 (using .../facter_1.7.1-1puppetlabs1_i386.deb) ...
Unpacking replacement facter ...
Selecting previously unselected package ruby-json.
Unpacking ruby-json (from .../ruby-json_1.6.3-1_i386.deb) ...
Selecting previously unselected package libjson-ruby.
Unpacking libjson-ruby (from .../libjson-ruby_1.6.3-1_all.deb) ...
Selecting previously unselected package hiera.
Unpacking hiera (from .../hiera_1.2.1-1puppetlabs1_all.deb) ...
Selecting previously unselected package ruby-rgen.
Unpacking ruby-rgen (from .../ruby-rgen_0.6.2-1puppetlabs1_all.deb) ...
Preparing to replace puppet 2.7.17-1mozilla1 (using .../puppet_3.2.1-1puppetlabs1_all.deb) ...
Unpacking replacement puppet ...
dpkg: error processing /var/cache/apt/archives/puppet_3.2.1-1puppetlabs1_all.deb (--unpack):
 trying to overwrite '/usr/share/man/man8/puppet-help.8.gz', which is also in package puppet-common 2.7.17-1mozilla1
puppet stop/waiting
Preparing to replace puppet-common 2.7.17-1mozilla1 (using .../puppet-common_3.2.1-1puppetlabs1_all.deb) ...
Unpacking replacement puppet-common ...
Processing triggers for man-db ...
Processing triggers for ureadahead ...
ureadahead will be reprofiled on next reboot
Errors were encountered while processing:
 /var/cache/apt/archives/puppet_3.2.1-1puppetlabs1_all.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
 at /etc/puppet/environments/test/modules/packages/manifests/puppet.pp:29
notice: /Stage[main]/Puppet::Puppetize_sh/File[/root/puppetize.sh]/content: 
DIFFS DISABLED - https://bugzilla.mozilla.org/show_bug.cgi?id=791102 /root/puppetize.sh /tmp/puppet-file20130522-2821-19iuzv5-0

notice: /Stage[main]/Puppet::Puppetize_sh/File[/root/puppetize.sh]/content: content changed '{md5}accc03f7d52130b8eccfb929c5edf097' to '{md5}dec4143c085136d793b6145b4ab7314c'
notice: /Stage[main]/Packages::Puppet/Package[puppet]/ensure: ensure changed '2.7.17-1mozilla1' to '3.2.1-1puppetlabs1'
notice: /Stage[main]/Users::Root::Setup/File[/root/.hgrc]/ensure: defined content as '{md5}b10c8c3bf195f9cf22b191e119e8cb9f'
notice: /Stage[main]/Users::Root::Setup/Python::User_pip_conf[root]/File[/root/.pip/pip.conf]/content: 
DIFFS DISABLED - https://bugzilla.mozilla.org/show_bug.cgi?id=791102 /root/.pip/pip.conf /tmp/puppet-file20130522-2821-ug91op-0

notice: /Stage[main]/Users::Root::Setup/Python::User_pip_conf[root]/File[/root/.pip/pip.conf]/content: content changed '{md5}ef3a4af6a10c771e701334f1e1ad2156' to '{md5}0312d7f9ac6641e2cb17f9bc4862779c'
notice: /Stage[main]/Puppet::Config/File[/etc/puppet/puppet.conf]/content: 
DIFFS DISABLED - https://bugzilla.mozilla.org/show_bug.cgi?id=791102 /etc/puppet/puppet.conf /tmp/puppet-file20130522-2821-rx538g-0

notice: /Stage[main]/Puppet::Config/File[/etc/puppet/puppet.conf]/content: content changed '{md5}394daa41c4117ae6c2b029318029e52f' to '{md5}b8a55bff0f62ef3ff28bdd4e6333aae4'
notice: Finished catalog run in 39.60 seconds

----

I don't see anything to worry about there in terms of impacting production.  The cltbld pip.conf change is due to the changing puppetmasters.  I'll attach the corresponding patch shortly for review.  Once that's up, we can put these slaves into staging and see if anything weird happens.  If not, we upgrade the lot of 'em.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

er, sorry, I didn't mean to undo your assignment - weird that bugzilla didn't give me a midair warning.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: ubuntu-320.patch

rail, do you see anything wrong here?

The change to /etc/init/puppet.conf is hopefully explained by the comments.  It works around bug 825056 comment 29.

Comment 7

[Rail Aliiev [:rail]]

Comment on attachment 753012 [details] [diff] [review]
ubuntu-320.patch

lgtm

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

Armen -- these are ready to be tested in staging and, assuming that's OK, production, whichever you'd prefer.  Do you want to handle that in another bug, or here?

Once these hosts pass muster, I'll convert the remainder of the scl3 Ubuntu systems to puppet-3.2.1.  We're still waiting for an AWS master for the AWS Ubuntu hosts (bug 872545), but I expect that we won't need any updates to the puppet config to move them to puppet-3.2.1.

Comment 9

[Armen [:armenzg]]

I can use this. It will take a bit though.

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

These hosts are now having problems running 'apt-get update'.  I'll dig into that.

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

Ah, this was because one of the puppetmasters wasn't syncing its files (bug 876797).  I filed bug 876812 about the failure.

It's fixed now (the puppetmaster is sync'd), so back to releng for testing.  Armen, will you hand this off to someone else if you don't get to it by the time you leave?

Comment 12

[Armen [:armenzg]]

I'm attaching these hosts to my staging master and triggered with sendchanges:
* talos-linux32-ix-005
* talos-linux32-ix-006
* talos-linux64-ix-005
* talos-linux64-ix-006

The results should be able to be seen in here:
http://dev-master01.build.scl1.mozilla.com:8042/buildslaves/talos-linux32-ix-005?numbuilds=50
http://dev-master01.build.scl1.mozilla.com:8042/buildslaves/talos-linux32-ix-006?numbuilds=50
http://dev-master01.build.scl1.mozilla.com:8042/buildslaves/talos-linux64-ix-005?numbuilds=50
http://dev-master01.build.scl1.mozilla.com:8042/buildslaves/talos-linux64-ix-006?numbuilds=50

buildbot sendchange --master dev-master01.build.mozilla.org:9042 --username sendchange-unittest --branch mozilla-inbound-linux64-opt-unittest --revision default  http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux64/1370967217/firefox-24.0a1.en-US.linux-x86_64.tar.bz2 http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux64/1370967217/firefox-24.0a1.en-US.linux-x86_64.tests.zip
buildbot sendchange --master dev-master01.build.mozilla.org:9042 --username sendchange-unittest --branch mozilla-inbound-linux64-debug-unittest --revision default  http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux64-debug/1370966894/firefox-24.0a1.en-US.linux-x86_64.tar.bz2 http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux64-debug/1370966894/firefox-24.0a1.en-US.linux-x86_64.tests.zip
buildbot sendchange --master dev-master01.build.mozilla.org:9042 --username sendchange --branch mozilla-inbound-linux64-talos --revision default  http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux64/1370967217/firefox-24.0a1.en-US.linux-x86_64.tar.bz2
buildbot sendchange --master dev-master01.build.mozilla.org:9042 --username sendchange-unittest --branch mozilla-inbound-linux-opt-unittest --revision default  http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux/1370966894/firefox-24.0a1.en-US.linux-i686.tar.bz2 http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux/1370966894/firefox-24.0a1.en-US.linux-i686.tests.zip
buildbot sendchange --master dev-master01.build.mozilla.org:9042 --username sendchange-unittest --branch mozilla-inbound-linux-debug-unittest --revision default  http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux-debug/1370966894/firefox-24.0a1.en-US.linux-i686.tar.bz2 http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux-debug/1370966894/firefox-24.0a1.en-US.linux-i686.tests.zip
buildbot sendchange --master dev-master01.build.mozilla.org:9042 --username sendchange --branch mozilla-inbound-linux-talos --revision default  http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-inbound-linux/1370966894/firefox-24.0a1.en-US.linux-i686.tar.bz2

Comment 13

[Armen [:armenzg]]

dustin, what are we looking for in here?
The machines seem to have taken all jobs and are looking green.
I've triggered another set of sendchanges to draw more load on them.

> The results should be able to be seen in here:
> http://dev-master01.build.scl1.mozilla.com:8042/buildslaves/talos-linux32-ix-
> 005?numbuilds=50
> http://dev-master01.build.scl1.mozilla.com:8042/buildslaves/talos-linux32-ix-
> 006?numbuilds=50
> http://dev-master01.build.scl1.mozilla.com:8042/buildslaves/talos-linux64-ix-
> 005?numbuilds=50
> http://dev-master01.build.scl1.mozilla.com:8042/buildslaves/talos-linux64-ix-
> 006?numbuilds=50

Comment 14

[Dustin J. Mitchell [:dustin] (he/him)]

We're looking for anything that would indicate these can't be put into production this way.  I don't expect anything, so that's good news.  If you're not seeing anything to worry about, I'll work with Aki to upgrade them in batches.

Comment 15

[Armen [:armenzg]]

Sorry about how long it took.
I've moved those machines to production as they were doing well on staging.

Comment 16

[Dustin J. Mitchell [:dustin] (he/him)]

Per irc, these are good to go.  I filed a new bug to track deployment.