Closed Bug 882739 Opened 12 years ago Closed 12 years ago

Upgrade mountain lion hosts to puppet 3.2.0

Categories

(Infrastructure & Operations :: RelOps: General, task)

Product:
Infrastructure & Operations
Component:
RelOps: General
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: dustin, Assigned: arich)

References

Details

Good news! This is already 1/3 done! By accident! When I changed where puppet.test.releng.scl3.mozilla.com points, thinking that VLAN only had Ubuntu testers in it, about 1/3 of the mountain lion systems hit the new master and upgraded themselves. The logging looks like this: > (/File[/var/lib/puppet/lib]) Failed to generate additional resources using 'eval_generate: Error 400 on SERVER: this master is not a CA > (/Stage[main]//Resources[user]) Failed to generate additional resources using 'generate': undefined method `string' for nil:NilClass > (/Stage[main]/Users::Global/File[/etc/profile.d]/ensure) created > (/Stage[main]/Tweaks::Cleanup/Exec[find /tmp/* -mmin +15 -print | xargs -n1 rm -rf]/returns) executed successfully > (/Stage[main]/Disableservices::Common/File[/var/lib/puppet/.indexing-disabled]/ensure) defined content as '{md5}3e0603800a91ffe98248d5f5486e7ee9' > (/Stage[main]/Disableservices::Common/Exec[remove-index]) Triggered 'refresh' from 1 events > (/Stage[main]/Users::Global/File[/etc/profile]/content) content changed '{md5}3a3febb29e55ee975098903a31f8022a' to '{md5}a16b1e70bf770c4d03b9acc999e6f7e3' > (/Stage[main]/Users::Global/File[/etc/profile]/mode) mode changed '0444' to '0644' > (/Stage[main]/Users::Global/File[/etc/profile.d/ps1.sh]/ensure) defined content as '{md5}945ff42f3710e0ca60cdee0891c78ee2' > (/Stage[main]/Packages::Mozilla::Python27/File[/tools/python]/ensure) created > (/Stage[main]/Packages::Mozilla::Python27/File[/tools/python2]/ensure) created > (/Stage[main]/Packages::Puppet/Packages::Pkgdmg[puppet]/Package[puppet-3.2.0-rc1.dmg]/ensure) created > (/Stage[main]/Packages::Puppet/Packages::Pkgdmg[facter]/Package[facter-1.7.1.dmg]/ensure) created > (/Stage[main]/Ssh::Config/File[/etc/ssh_known_hosts]/ensure) defined content as '{md5}2bfe11a7587b5f268466f0e5c41431e4' > (/Stage[main]/Disableservices::Common/Exec[disable-indexing]) Triggered 'refresh' from 1 events > (/Stage[main]/Users::Builder::Setup/Ssh::Userconfig[cltbld]/File[/Users/cltbld/.ssh/known_hosts]/content) content changed '{md5}998ef6f565c6f6acfa810d9dfbb8a91b' to '{md5}2bfe11a7587b5f268466f0e5c41431e4' > (/Stage[main]/Puppet::Atboot/File[/etc/puppet/puppetmasters.txt]/content) content changed '{md5}f950a1b7115db3f0e29bf505184a229b' to '{md5}3e183891fad10a9886fceabfac6e1182' > (/Stage[main]/Puppet::Atboot/File[/usr/local/bin/run-puppet.sh]/content) content changed '{md5}c69b54a238db3086dd622dc58025f3d9' to '{md5}41a65252d5327181be58f2c31a0bb62c' > (/Stage[main]/Users::Builder::Setup/Python::User_pip_conf[cltbld]/File[/Users/cltbld/.pip/pip.conf]/content) content changed '{md5}ef3a4af6a10c771e701334f1e1ad2156' to '{md5}7afafd1669199be38c1d4cb7cf7396b6' > (/Stage[main]/Users::Builder::Setup/Ssh::Userconfig[cltbld]/Concat[/Users/cltbld/.ssh/authorized_keys]/File[/var/lib/puppet/concat/_Users_cltbld_.ssh_authorized_keys]/ensure) created > (/Stage[main]/Users::Builder::Setup/Ssh::Userconfig[cltbld]/Concat[/Users/cltbld/.ssh/authorized_keys]/File[/var/lib/puppet/concat/_Users_cltbld_.ssh_authorized_keys/fragments.concat]/ensure) created > (/Stage[main]/Users::Builder::Setup/Ssh::Userconfig[cltbld]/Concat[/Users/cltbld/.ssh/authorized_keys]/File[/var/lib/puppet/concat/_Users_cltbld_.ssh_authorized_keys/fragments]/ensure) created > (/Stage[main]/Users::Builder::Setup/Ssh::Userconfig[cltbld]/Concat::Fragment[/Users/cltbld::]/File[/var/lib/puppet/concat/_Users_cltbld_.ssh_authorized_keys/fragments/10__Users_cltbld::]/ensure) created > (/Stage[main]/Users::Builder::Setup/Ssh::Userconfig[cltbld]/Concat[/Users/cltbld/.ssh/authorized_keys]/Exec[concat_/Users/cltbld/.ssh/authorized_keys]/returns) executed successfully > (/Stage[main]/Users::Builder::Setup/Ssh::Userconfig[cltbld]/Concat[/Users/cltbld/.ssh/authorized_keys]/Exec[concat_/Users/cltbld/.ssh/authorized_keys]) Triggered 'refresh' from 3 events > (/Stage[main]/Users::Builder::Setup/Ssh::Userconfig[cltbld]/Concat[/Users/cltbld/.ssh/authorized_keys]/File[/Users/cltbld/.ssh/authorized_keys]/content) content changed '{md5}9b0bf1ef7e9e1dd8161f42d58970d53e' to '{md5}02af804ecda99078c95b2cc2bc25d09e' > (/Stage[main]/Puppet::Puppetize_sh/File[/var/root/puppetize.sh]/content) content changed '{md5}accc03f7d52130b8eccfb929c5edf097' to '{md5}dec4143c085136d793b6145b4ab7314c' > (/Stage[main]/Users::Root::Setup/File[/var/root/.hgrc]/ensure) defined content as '{md5}b10c8c3bf195f9cf22b191e119e8cb9f' > (/Stage[main]/Users::Root::Setup/Ssh::Userconfig[root]/Concat[/var/root/.ssh/authorized_keys]/File[/var/lib/puppet/concat/_var_root_.ssh_authorized_keys]/ensure) created > (/Stage[main]/Users::Root::Setup/Ssh::Userconfig[root]/Concat[/var/root/.ssh/authorized_keys]/File[/var/lib/puppet/concat/_var_root_.ssh_authorized_keys/fragments]/ensure) created > (/Stage[main]/Users::Root::Setup/Ssh::Userconfig[root]/Concat[/var/root/.ssh/authorized_keys]/File[/var/lib/puppet/concat/_var_root_.ssh_authorized_keys/fragments.concat.out]/ensure) created > (/Stage[main]/Users::Root::Setup/Ssh::Userconfig[root]/Concat::Fragment[/var/root::]/File[/var/lib/puppet/concat/_var_root_.ssh_authorized_keys/fragments/10__var_root::]/ensure) created > (/Stage[main]/Users::Root::Setup/Ssh::Userconfig[root]/File[/var/root/.ssh/known_hosts]/content) content changed '{md5}998ef6f565c6f6acfa810d9dfbb8a91b' to '{md5}2bfe11a7587b5f268466f0e5c41431e4' > (/Stage[main]/Users::Root::Setup/Ssh::Userconfig[root]/Concat[/var/root/.ssh/authorized_keys]/File[/var/lib/puppet/concat/_var_root_.ssh_authorized_keys/fragments.concat]/ensure) created > (/Stage[main]/Users::Root::Setup/Ssh::Userconfig[root]/Concat[/var/root/.ssh/authorized_keys]/Exec[concat_/var/root/.ssh/authorized_keys]/returns) executed successfully > (/Stage[main]/Users::Root::Setup/Ssh::Userconfig[root]/Concat[/var/root/.ssh/authorized_keys]/Exec[concat_/var/root/.ssh/authorized_keys]) Triggered 'refresh' from 3 events > (/Stage[main]/Users::Root::Setup/Ssh::Userconfig[root]/Concat[/var/root/.ssh/authorized_keys]/File[/var/root/.ssh/authorized_keys]/content) content changed '{md5}9b0bf1ef7e9e1dd8161f42d58970d53e' to '{md5}02af804ecda99078c95b2cc2bc25d09e' > (/Stage[main]/Users::Root::Setup/Python::User_pip_conf[root]/File[/var/root/.pip/pip.conf]/content) content changed '{md5}ef3a4af6a10c771e701334f1e1ad2156' to '{md5}7afafd1669199be38c1d4cb7cf7396b6' > (/Stage[main]/Puppet::Config/File[/etc/puppet/puppet.conf]/content) content changed '{md5}394daa41c4117ae6c2b029318029e52f' to '{md5}2dfd9a0a4f0b118e0aa64810ee72dc78' > Finished catalog run in 119.62 seconds > > Unable to fetch my node definition, but the agent run will continue: > Could not intern from pson: source '"#<Puppet::Node:0x7f' not in PSON! > (/Stage[main]/Tweaks::Cleanup/Exec[find /tmp/* -mmin +15 -print | xargs -n1 rm -rf]/returns) executed successfully > (/Stage[main]/Users::Builder::Setup/Ssh::Userconfig[cltbld]/File[/Users/cltbld/.ssh/authorized_keys]/content) content changed '{md5}02af804ecda99078c95b2cc2bc25d09e' to '{md5}9b0bf1ef7e9e1dd8161f42d58970d53e' > (/Stage[main]/Users::Builder::Setup/Ssh::Userconfig[cltbld]/File[/Users/cltbld/.ssh/known_hosts]/content) content changed '{md5}2bfe11a7587b5f268466f0e5c41431e4' to '{md5}998ef6f565c6f6acfa810d9dfbb8a91b' > (/Stage[main]/Puppet::Atboot/File[/etc/puppet/puppetmasters.txt]/content) content changed '{md5}3e183891fad10a9886fceabfac6e1182' to '{md5}f950a1b7115db3f0e29bf505184a229b' > (/Stage[main]/Puppet::Atboot/File[/usr/local/bin/run-puppet.sh]/content) content changed '{md5}41a65252d5327181be58f2c31a0bb62c' to '{md5}c69b54a238db3086dd622dc58025f3d9' > (/Stage[main]/Users::Builder::Setup/Python::User_pip_conf[cltbld]/File[/Users/cltbld/.pip/pip.conf]/content) content changed '{md5}7afafd1669199be38c1d4cb7cf7396b6' to '{md5}ef3a4af6a10c771e701334f1e1ad2156' > (/Stage[main]/Puppet::Puppetize_sh/File[/var/root/puppetize.sh]/content) content changed '{md5}dec4143c085136d793b6145b4ab7314c' to '{md5}accc03f7d52130b8eccfb929c5edf097' > (/Stage[main]/Users::Root::Setup/Ssh::Userconfig[root]/File[/var/root/.ssh/known_hosts]/content) content changed '{md5}2bfe11a7587b5f268466f0e5c41431e4' to '{md5}998ef6f565c6f6acfa810d9dfbb8a91b' > (/Stage[main]/Users::Root::Setup/Ssh::Userconfig[root]/File[/var/root/.ssh/authorized_keys]/content) content changed '{md5}02af804ecda99078c95b2cc2bc25d09e' to '{md5}9b0bf1ef7e9e1dd8161f42d58970d53e' > (/Stage[main]/Users::Root::Setup/Python::User_pip_conf[root]/File[/var/root/.pip/pip.conf]/content) content changed '{md5}7afafd1669199be38c1d4cb7cf7396b6' to '{md5}ef3a4af6a10c771e701334f1e1ad2156' > (/Stage[main]/Puppet::Config/File[/etc/puppet/puppet.conf]/content) content changed '{md5}2dfd9a0a4f0b118e0aa64810ee72dc78' to '{md5}394daa41c4117ae6c2b029318029e52f' > Finished catalog run in 29.31 seconds I don't see anything to worry about there, but I won't upgrade the rest until we're confident. Also, all hosts will need new certificates issued - they are still running with certs from the old puppetmaster.
Affected hosts: talos-mtnlion-r5-003.test.releng.scl3.mozilla.com talos-mtnlion-r5-005.test.releng.scl3.mozilla.com talos-mtnlion-r5-014.test.releng.scl3.mozilla.com talos-mtnlion-r5-018.test.releng.scl3.mozilla.com talos-mtnlion-r5-019.test.releng.scl3.mozilla.com talos-mtnlion-r5-024.test.releng.scl3.mozilla.com talos-mtnlion-r5-027.test.releng.scl3.mozilla.com talos-mtnlion-r5-036.test.releng.scl3.mozilla.com talos-mtnlion-r5-039.test.releng.scl3.mozilla.com talos-mtnlion-r5-042.test.releng.scl3.mozilla.com talos-mtnlion-r5-050.test.releng.scl3.mozilla.com talos-mtnlion-r5-051.test.releng.scl3.mozilla.com talos-mtnlion-r5-062.test.releng.scl3.mozilla.com talos-mtnlion-r5-073.test.releng.scl3.mozilla.com talos-mtnlion-r5-084.test.releng.scl3.mozilla.com talos-mtnlion-r5-086.test.releng.scl3.mozilla.com talos-mtnlion-r5-087.test.releng.scl3.mozilla.com talos-mtnlion-r5-089.test.releng.scl3.mozilla.com
It's a little more complicated than that. The second run quoted above was from the run-puppet.sh script, which detected a failure in the first run and re-ran puppet against the old (2.7.17) masters. So these hosts are running puppet-3.2.0, but running it against a 2.7.17 master. That's not supported in general, but seems to work here, aside from the "Could not intern from pson" error. So, these hosts are still using the 2.7.17 manifests - just running a different puppet version. So there's not a lot to worry about here, and essentially we'll need to upgrade the whole fleet, including the hosts in comment 1, when the time comes. That time should probably come soon, though.
Assignee: server-ops-releng → dustin
coop just loaned me talos-mtnlion-r5-002 to test this. notice: /Stage[main]/Users::Global/File[/etc/profile.d]/ensure: created notice: /Stage[main]/Tweaks::Cleanup/Exec[find /tmp/* -mmin +15 -print | xargs -n1 rm -rf]/returns: executed successfully notice: /Stage[main]/Disableservices::Common/File[/var/lib/puppet/.indexing-disabled]/ensure: defined content as '{md5}3e0603800a91ffe98248d5f5486e7ee9' info: /Stage[main]/Disableservices::Common/File[/var/lib/puppet/.indexing-disabled]: Scheduling refresh of Exec[disable-indexing] info: /Stage[main]/Disableservices::Common/File[/var/lib/puppet/.indexing-disabled]: Scheduling refresh of Exec[remove-index] notice: /Stage[main]/Disableservices::Common/Exec[remove-index]: Triggered 'refresh' from 1 events notice: /Stage[main]/Users::Global/File[/etc/profile]/content: DIFFS DISABLED - https://bugzilla.mozilla.org/show_bug.cgi?id=791102 /etc/profile /var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/puppet-file20130617-829-170pbdr-0 notice: /Stage[main]/Users::Global/File[/etc/profile]/content: content changed '{md5}3a3febb29e55ee975098903a31f8022a' to '{md5}a16b1e70bf770c4d03b9acc999e6f7e3' notice: /Stage[main]/Users::Global/File[/etc/profile]/mode: mode changed '0444' to '0644' notice: /Stage[main]/Users::Global/File[/etc/profile.d/ps1.sh]/ensure: defined content as '{md5}66d2903479699b7ff612a796850e74a7' notice: /Stage[main]/Packages::Mozilla::Python27/File[/tools/python]/ensure: created notice: /Stage[main]/Packages::Mozilla::Python27/File[/tools/python2]/ensure: created notice: /Stage[main]/Packages::Puppet/Packages::Pkgdmg[puppet]/Package[puppet-3.2.0-rc1.dmg]/ensure: created notice: /Stage[main]/Packages::Puppet/Packages::Pkgdmg[facter]/Package[facter-1.7.1.dmg]/ensure: created notice: /Stage[main]/Ssh::Config/File[/etc/ssh_known_hosts]/ensure: defined content as '{md5}2bfe11a7587b5f268466f0e5c41431e4' notice: /Stage[main]/Disableservices::Common/Exec[disable-indexing]: Triggered 'refresh' from 1 events notice: /Stage[main]/Users::Builder::Setup/Ssh::Userconfig[cltbld]/File[/Users/cltbld/.ssh/known_hosts]/content: DIFFS DISABLED - https://bugzilla.mozilla.org/show_bug.cgi?id=791102 /Users/cltbld/.ssh/known_hosts /var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/puppet-file20130617-829-wttvnt-0 notice: /Stage[main]/Users::Builder::Setup/Ssh::Userconfig[cltbld]/File[/Users/cltbld/.ssh/known_hosts]/content: content changed '{md5}998ef6f565c6f6acfa810d9dfbb8a91b' to '{md5}2bfe11a7587b5f268466f0e5c41431e4' notice: /Stage[main]/Puppet::Atboot/File[/etc/puppet/puppetmasters.txt]/content: DIFFS DISABLED - https://bugzilla.mozilla.org/show_bug.cgi?id=791102 /etc/puppet/puppetmasters.txt /var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/puppet-file20130617-829-p13muh-0 notice: /Stage[main]/Puppet::Atboot/File[/etc/puppet/puppetmasters.txt]/content: content changed '{md5}f950a1b7115db3f0e29bf505184a229b' to '{md5}58f7793fbb2d9c2b89184a839f14268a' notice: /Stage[main]/Puppet::Atboot/File[/usr/local/bin/run-puppet.sh]/content: DIFFS DISABLED - https://bugzilla.mozilla.org/show_bug.cgi?id=791102 /usr/local/bin/run-puppet.sh /var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/puppet-file20130617-829-ww3lz2-0 notice: /Stage[main]/Puppet::Atboot/File[/usr/local/bin/run-puppet.sh]/content: content changed '{md5}c69b54a238db3086dd622dc58025f3d9' to '{md5}41a65252d5327181be58f2c31a0bb62c' notice: /Stage[main]/Users::Builder::Setup/Python::User_pip_conf[cltbld]/File[/Users/cltbld/.pip/pip.conf]/content: DIFFS DISABLED - https://bugzilla.mozilla.org/show_bug.cgi?id=791102 /Users/cltbld/.pip/pip.conf /var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/puppet-file20130617-829-y1i4zp-0 notice: /Stage[main]/Users::Builder::Setup/Python::User_pip_conf[cltbld]/File[/Users/cltbld/.pip/pip.conf]/content: content changed '{md5}ef3a4af6a10c771e701334f1e1ad2156' to '{md5}153c86c79a6a867dee2cc5941bd5bb04' notice: /Stage[main]/Users::Builder::Setup/Ssh::Userconfig[cltbld]/Concat[/Users/cltbld/.ssh/authorized_keys]/File[/var/lib/puppet/concat/_Users_cltbld_.ssh_authorized_keys]/ensure: created info: /Stage[main]/Users::Builder::Setup/Ssh::Userconfig[cltbld]/Concat[/Users/cltbld/.ssh/authorized_keys]/File[/var/lib/puppet/concat/_Users_cltbld_.ssh_authorized_keys]: Scheduling refresh of Exec[concat_/Users/cltbld/.ssh/authorized_keys] notice: /Stage[main]/Users::Builder::Setup/Ssh::Userconfig[cltbld]/Concat[/Users/cltbld/.ssh/authorized_keys]/File[/var/lib/puppet/concat/_Users_cltbld_.ssh_authorized_keys/fragments.concat]/ensure: created notice: /Stage[main]/Users::Builder::Setup/Ssh::Userconfig[cltbld]/Concat[/Users/cltbld/.ssh/authorized_keys]/File[/var/lib/puppet/concat/_Users_cltbld_.ssh_authorized_keys/fragments]/ensure: created info: /Stage[main]/Users::Builder::Setup/Ssh::Userconfig[cltbld]/Concat[/Users/cltbld/.ssh/authorized_keys]/File[/var/lib/puppet/concat/_Users_cltbld_.ssh_authorized_keys/fragments]: Scheduling refresh of Exec[concat_/Users/cltbld/.ssh/authorized_keys] notice: /Stage[main]/Users::Builder::Setup/Ssh::Userconfig[cltbld]/Concat::Fragment[/Users/cltbld::]/File[/var/lib/puppet/concat/_Users_cltbld_.ssh_authorized_keys/fragments/10__Users_cltbld::]/ensure: created info: /Stage[main]/Users::Builder::Setup/Ssh::Userconfig[cltbld]/Concat::Fragment[/Users/cltbld::]/File[/var/lib/puppet/concat/_Users_cltbld_.ssh_authorized_keys/fragments/10__Users_cltbld::]: Scheduling refresh of Exec[concat_/Users/cltbld/.ssh/authorized_keys] notice: /Stage[main]/Users::Builder::Setup/Ssh::Userconfig[cltbld]/Concat[/Users/cltbld/.ssh/authorized_keys]/Exec[concat_/Users/cltbld/.ssh/authorized_keys]/returns: executed successfully notice: /Stage[main]/Users::Builder::Setup/Ssh::Userconfig[cltbld]/Concat[/Users/cltbld/.ssh/authorized_keys]/Exec[concat_/Users/cltbld/.ssh/authorized_keys]: Triggered 'refresh' from 3 events notice: /Stage[main]/Users::Builder::Setup/Ssh::Userconfig[cltbld]/Concat[/Users/cltbld/.ssh/authorized_keys]/File[/Users/cltbld/.ssh/authorized_keys]/content: DIFFS DISABLED - https://bugzilla.mozilla.org/show_bug.cgi?id=791102 /Users/cltbld/.ssh/authorized_keys /var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/puppet-file20130617-829-104yzl5-0 info: FileBucket adding {md5}9b0bf1ef7e9e1dd8161f42d58970d53e info: /Stage[main]/Users::Builder::Setup/Ssh::Userconfig[cltbld]/Concat[/Users/cltbld/.ssh/authorized_keys]/File[/Users/cltbld/.ssh/authorized_keys]: Filebucketed /Users/cltbld/.ssh/authorized_keys to puppet with sum 9b0bf1ef7e9e1dd8161f42d58970d53e notice: /Stage[main]/Users::Builder::Setup/Ssh::Userconfig[cltbld]/Concat[/Users/cltbld/.ssh/authorized_keys]/File[/Users/cltbld/.ssh/authorized_keys]/content: content changed '{md5}9b0bf1ef7e9e1dd8161f42d58970d53e' to '{md5}02af804ecda99078c95b2cc2bc25d09e' notice: /Stage[main]/Puppet::Puppetize_sh/File[/var/root/puppetize.sh]/content: DIFFS DISABLED - https://bugzilla.mozilla.org/show_bug.cgi?id=791102 /var/root/puppetize.sh /var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/puppet-file20130617-829-h6l4uk-0 notice: /Stage[main]/Puppet::Puppetize_sh/File[/var/root/puppetize.sh]/content: content changed '{md5}7711f372973c02e74d2207eb0a09216b' to '{md5}f808045949c4d56b0ddad0357b360ec6' notice: /Stage[main]/Users::Root::Setup/File[/var/root/.hgrc]/ensure: defined content as '{md5}b10c8c3bf195f9cf22b191e119e8cb9f' notice: /Stage[main]/Users::Root::Setup/Ssh::Userconfig[root]/Concat[/var/root/.ssh/authorized_keys]/File[/var/lib/puppet/concat/_var_root_.ssh_authorized_keys]/ensure: created info: /Stage[main]/Users::Root::Setup/Ssh::Userconfig[root]/Concat[/var/root/.ssh/authorized_keys]/File[/var/lib/puppet/concat/_var_root_.ssh_authorized_keys]: Scheduling refresh of Exec[concat_/var/root/.ssh/authorized_keys] notice: /Stage[main]/Users::Root::Setup/Ssh::Userconfig[root]/Concat[/var/root/.ssh/authorized_keys]/File[/var/lib/puppet/concat/_var_root_.ssh_authorized_keys/fragments]/ensure: created info: /Stage[main]/Users::Root::Setup/Ssh::Userconfig[root]/Concat[/var/root/.ssh/authorized_keys]/File[/var/lib/puppet/concat/_var_root_.ssh_authorized_keys/fragments]: Scheduling refresh of Exec[concat_/var/root/.ssh/authorized_keys] notice: /Stage[main]/Users::Root::Setup/Ssh::Userconfig[root]/Concat[/var/root/.ssh/authorized_keys]/File[/var/lib/puppet/concat/_var_root_.ssh_authorized_keys/fragments.concat.out]/ensure: created notice: /Stage[main]/Users::Root::Setup/Ssh::Userconfig[root]/Concat::Fragment[/var/root::]/File[/var/lib/puppet/concat/_var_root_.ssh_authorized_keys/fragments/10__var_root::]/ensure: created info: /Stage[main]/Users::Root::Setup/Ssh::Userconfig[root]/Concat::Fragment[/var/root::]/File[/var/lib/puppet/concat/_var_root_.ssh_authorized_keys/fragments/10__var_root::]: Scheduling refresh of Exec[concat_/var/root/.ssh/authorized_keys] notice: /Stage[main]/Users::Root::Setup/Ssh::Userconfig[root]/File[/var/root/.ssh/known_hosts]/content: DIFFS DISABLED - https://bugzilla.mozilla.org/show_bug.cgi?id=791102 /var/root/.ssh/known_hosts /var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/puppet-file20130617-829-rcl62l-0 notice: /Stage[main]/Users::Root::Setup/Ssh::Userconfig[root]/File[/var/root/.ssh/known_hosts]/content: content changed '{md5}998ef6f565c6f6acfa810d9dfbb8a91b' to '{md5}2bfe11a7587b5f268466f0e5c41431e4' notice: /Stage[main]/Users::Root::Setup/Ssh::Userconfig[root]/Concat[/var/root/.ssh/authorized_keys]/File[/var/lib/puppet/concat/_var_root_.ssh_authorized_keys/fragments.concat]/ensure: created notice: /Stage[main]/Users::Root::Setup/Ssh::Userconfig[root]/Concat[/var/root/.ssh/authorized_keys]/Exec[concat_/var/root/.ssh/authorized_keys]/returns: executed successfully notice: /Stage[main]/Users::Root::Setup/Ssh::Userconfig[root]/Concat[/var/root/.ssh/authorized_keys]/Exec[concat_/var/root/.ssh/authorized_keys]: Triggered 'refresh' from 3 events notice: /Stage[main]/Users::Root::Setup/Ssh::Userconfig[root]/Concat[/var/root/.ssh/authorized_keys]/File[/var/root/.ssh/authorized_keys]/content: DIFFS DISABLED - https://bugzilla.mozilla.org/show_bug.cgi?id=791102 /var/root/.ssh/authorized_keys /var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/puppet-file20130617-829-5ozdmg-0 info: FileBucket got a duplicate file {md5}9b0bf1ef7e9e1dd8161f42d58970d53e info: /Stage[main]/Users::Root::Setup/Ssh::Userconfig[root]/Concat[/var/root/.ssh/authorized_keys]/File[/var/root/.ssh/authorized_keys]: Filebucketed /var/root/.ssh/authorized_keys to puppet with sum 9b0bf1ef7e9e1dd8161f42d58970d53e notice: /Stage[main]/Users::Root::Setup/Ssh::Userconfig[root]/Concat[/var/root/.ssh/authorized_keys]/File[/var/root/.ssh/authorized_keys]/content: content changed '{md5}9b0bf1ef7e9e1dd8161f42d58970d53e' to '{md5}02af804ecda99078c95b2cc2bc25d09e' notice: /Stage[main]/Users::Root::Setup/Python::User_pip_conf[root]/File[/var/root/.pip/pip.conf]/content: DIFFS DISABLED - https://bugzilla.mozilla.org/show_bug.cgi?id=791102 /var/root/.pip/pip.conf /var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/puppet-file20130617-829-1t5de69-0 notice: /Stage[main]/Users::Root::Setup/Python::User_pip_conf[root]/File[/var/root/.pip/pip.conf]/content: content changed '{md5}ef3a4af6a10c771e701334f1e1ad2156' to '{md5}153c86c79a6a867dee2cc5941bd5bb04' notice: /Stage[main]/Puppet::Config/File[/etc/puppet/puppet.conf]/content: DIFFS DISABLED - https://bugzilla.mozilla.org/show_bug.cgi?id=791102 /etc/puppet/puppet.conf /var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/puppet-file20130617-829-1whaz98-0 notice: /Stage[main]/Puppet::Config/File[/etc/puppet/puppet.conf]/content: content changed '{md5}394daa41c4117ae6c2b029318029e52f' to '{md5}2dfd9a0a4f0b118e0aa64810ee72dc78' notice: Finished catalog run in 87.02 seconds So: * /etc/profile stuff for PS1 (bug 875893) * python symlinks (bug 760093) * puppet install * remove spotlight index (iirc the semaphore file for this changed) * known_hosts updates * authorized_keys for everyone - bug 838183 * changes to the list of masters and how puppet is run - run-puppet.sh, puppetmasters.txt, pip.conf, etc. Nothing I see that's likely to affect slave behavior.
coop - can you put this through its paces, or find someone who can? I'd love to get these rolled out ASAP, given the funny business from comment 0.
Flags: needinfo?(coop)
It's running (and passing) tests: https://secure.pub.build.mozilla.org/buildapi/recent/talos-mtnlion-r5-002
Flags: needinfo?(coop)
We'll call that sufficient to start rolling out, per irc. I'll do the set in comment 1 to start. If those turn out to be badly broken, then we'll have a reduced pool for a while, but i can reimage them back into shape reasonably quickly. After a day or so of success, I'll hit the rest.
OK, the set in comment 1 are upgraded and enabled in slavealloc.
Here's the procedure: - disable hosts in slavealloc, about 10 at a time - login to each host, and look at /builds/slave/talos-slave/twistd.log to see if it's still running. you'll see: A. "Command Complete" from >1h ago B. "..is ready" or "message from master" C. "Server shutting down" D. no such file E. a running command For A or B, reboot the host and check back; it should end up without Buildbot running. For C or D, you're good to go. For E, you'll have to wait. The host will likely reboot when it's done, and come up without Buildbot running (since it's disabled). - run: curl -L http://hg.mozilla.org/users/dmitchell_mozilla.com/puppet320/raw-file/21009387d468/modules/puppet/files/puppetize.sh > puppetize.sh PUPPET_SERVER=releng-puppet2.srv.releng.scl3.mozilla.com sh puppetize.sh (enter the password) puppet agent --test --server=releng-puppet2.srv.releng.scl3.mozilla.com puppet --version (should be 3.2.2)
- enable the host in slavealloc and reboot. If you're cautious, verify that buildbot starts after the reboot.
ALl of the mountain lion hosts that are online have been upgraded to 3.2.2. There is one host that's out for RMA (54) and a block of hosts (90-95) that haven't been installed yet because they are waiting on an additional disk. The hosts that have NOT been upgraded are: talos-mtnlion-r5-054 talos-mtnlion-r5-090 talos-mtnlion-r5-091 talos-mtnlion-r5-092 talos-mtnlion-r5-093 talos-mtnlion-r5-094 talos-mtnlion-r5-095 At this point, we have not updated the imaging task sequence to install the new version of puppet, so anything that gets imaged/reimaged will pick up the old client. This bug will remain open to track the modification of the task sequence.
Assignee: dustin → arich
Replaced /Deploy/Files/puppetize.sh and set PUPPET_SERVER on install.test.releng.scl3.mozilla.com. Moved aside package folder of /Deploy/Packages/Puppet with old facter and puppet versions and created a new folder with facter-1.7.1.pkg and puppet-3.2.2.pkg. Install verified that the correct versions of both packages were installed and a run of "puppet agent --test" completed successfully. This will take care of the machines that are not yet installed or that will be reinstalled in the future.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Component: Server Operations: RelEng → RelOps
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.