777742 - duplicate definition: File[/tools] on master-puppet1 for buildbot-master36

Status: RESOLVED INVALID

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[[e-mail] (:bhearsum)]

Could be an issue with multiple master instances on the same machine:
Jul 26 08:38:09 master-puppet1 puppetmasterd[2788]: Duplicate definition: File[/tools] is already defined in file /etc/puppet/modules/buildmaster/manifests/init.pp at line 101; cannot redefine at /etc/puppet/modules/selfserve-agent/manifests/init.pp:39 on node buildbot-master36.srv.releng.scl3.mozilla.com

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

This blocks puppet deploying changes to bm36, eg:

Aug 15 20:54:18 buildbot-master36 puppet-agent[19319]: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate definition: File[/tools] is already defined in file /etc/puppet/modules/buildmaster/manifests/init.pp at line 101; cannot redefine at /etc/puppet/modules/selfserve-agent/manifests/init.pp:39 on node buildbot-master36.srv.releng.scl3.mozilla.com
Aug 15 20:54:18 buildbot-master36 puppet-agent[19319]: Using cached catalog
Aug 15 20:54:18 buildbot-master36 puppet-agent[19319]: Could not retrieve catalog; skipping run

This seems to be from buildmaster and selfserve-agent both including python::virtualenv, but having to independently ensure /tools exists first. Perhaps we can create /tools in the virtualenv module instead ?

Comment 2

[[e-mail] (:bhearsum)]

This is pretty serious IMO.

Comment 3

[[e-mail] (:bhearsum)]

We also have:
err: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate definition: Python::Package_dir_file[pip-0.8.2.tar.gz] is already defined in file /etc/puppet/modules/python/manifests/virtualenv.pp at line 146; cannot redefine at /etc/puppet/modules/python/manifests/virtualenv.pp:146 on node buildbot-master36.srv.releng.scl3.mozilla.com

if you comment out /tools issues.

And then:
err: Could not retrieve catalog from remote server: Error 400 on SERVER: Duplicate definition: Python::Package[buildbot==0.8.4-pre-moz1] is already defined in file /etc/puppet/modules/python/manifests/virtualenv.pp at line 178; cannot redefine at /etc/puppet/modules/python/manifests/virtualenv.pp:178 on node buildbot-master36.srv.releng.scl3.mozilla.com

if you comment that out.

And then dependency errors because of commenting too many things out. ಠ_ಠ

Eventually I was able to make it sync and found the following things:
notice: //Buildmaster::Queue/File[/builds/buildbot/queue/passwords.py]/content: current_value {md5}8ff565143670ef260e15a505b58b682c, should be {md5}b69289f0b88ee1ee423927722703eea0 (noop)
--- /builds/buildbot/selfserve-agent/selfserve-agent.ini	2012-05-15 18:48:21.000000000 -0700
+++ /tmp/puppet-file.17170.0	2012-10-16 07:33:51.000000000 -0700
@@ -12,7 +12,7 @@
 [masters]
 masters-url = http://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-masters.json
 buildbot = /builds/buildbot/selfserve-agent/bin/buildbot
-sendchange-master = bm36-build_scheduler
+sendchange-master = bm10-build_scheduler
 
 [branches]
 url = http://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-branches.json
notice: //Selfserve-agent/File[/builds/buildbot/selfserve-agent/selfserve-agent.ini]/content: current_value {md5}7e31b320b6a745496f3cdb78dcd3fbda, should be {md5}c617ed74d377cfee6e1831dec9ea8355 (noop)
info: //Selfserve-agent/File[/builds/buildbot/selfserve-agent/selfserve-agent.ini]: Scheduling refresh of Service[selfserve-agent]
notice: //Selfserve-agent/Service[selfserve-agent]/ensure: current_value stopped, should be running (noop)
notice: //Selfserve-agent/Service[selfserve-agent]: Would have triggered 'refresh' from 1 events
--- /etc/yum.repos.d/packages-mozilla.repo	2012-05-03 08:05:35.000000000 -0700
+++ /tmp/puppet-file.17170.0	2012-10-16 07:33:51.000000000 -0700
@@ -1,31 +1,31 @@
 [mozilla]
 name=Mozilla Package Repo - $basearch
-baseurl=http://mrepo.mozilla.org/mrepo/$releasever-$basearch/RPMS.mozilla
+baseurl=https://mrepo.mozilla.org/mrepo/$releasever-$basearch/RPMS.mozilla
 enabled=1
 gpgcheck=0
 
 [base]
 name=Mozilla Package Repo - $basearch
-baseurl=http://mrepo.mozilla.org/mrepo/$releasever-$basearch/RPMS.centos-base
+baseurl=https://mrepo.mozilla.org/mrepo/$releasever-$basearch/RPMS.centos-base
 enabled=1
 gpgcheck=0
 
 [updates]
 name=Mozilla Package Repo - $basearch
-baseurl=http://mrepo.mozilla.org/mrepo/$releasever-$basearch/RPMS.centos-updates
+baseurl=https://mrepo.mozilla.org/mrepo/$releasever-$basearch/RPMS.centos-updates
 enabled=1
 gpgcheck=0
 
 [epel]
 name=Mozilla Package Repo - $basearch
-baseurl=http://mrepo.mozilla.org/mrepo/$releasever-$basearch/RPMS.epel
+baseurl=https://mrepo.mozilla.org/mrepo/$releasever-$basearch/RPMS.epel
 enabled=1
 gpgcheck=0
 
 
 [addons]
 name=Mozilla Package Repo - $basearch
-baseurl=http://mrepo.mozilla.org/mrepo/$releasever-$basearch/RPMS.centos-addons
+baseurl=https://mrepo.mozilla.org/mrepo/$releasever-$basearch/RPMS.centos-addons
 enabled=1
 gpgcheck=0
 
notice: //Packages::Mozilla-repo/File[/etc/yum.repos.d/packages-mozilla.repo]/content: current_value {md5}57ab9949e28cfdd567932f132d6d9867, should be {md5}bab72bf200f48d17668d8276471ead06 (noop)
notice: //Buildmaster/Package[python26]/ensure: current_value 2.6.5-6.el5, should be 2.6.8-1.el5 (noop)
notice: Finished catalog run in 5.10 seconds

As well as changes to /builds/buildbot/queue/passwords.py and /root/.my.cnf which I can't paste here due to secrets. The /root/.my.cnf is particularly concerning, as it changes database passwords, which makes me wonder if there's some cronjobs not working because of it.

Comment 5

[Chris Cooper [:coop] (he/him)]

Master is gone now, and we've moved to PuppetAgain.