781338 - rename and re-kick new bugzilla DB servers in scl3

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Sheeri Cabral [:sheeri]]

Please rename:
addons3.db.scl3.mozilla.com
addons4.db.scl3.mozilla.com
addons5.db.scl3.mozilla.com
addons6.db.scl3.mozilla.com

to:

bugzilla1.db.scl3.mozilla.com
bugzilla2.db.scl3.mozilla.com
bugzilla3.db.scl3.mozilla.com
bugzilla4.db.scl3.mozilla.com

As their names suggest, they should be in the 10.22.70 subnet.

Please also re-kickstart them. I'll add the right MySQL configs when that's done. Thanx!

Comment 1

[Sheeri Cabral [:sheeri]]

these servers are in nagios so please do make sure to take them out first.

Comment 2

[Michael Burns [:mburns]]

Attachment: x86_64 cobbler rhel 6.2 install

This is the error screenshot on the failed cobbler install

Comment 3

[Michael Burns [:mburns]]

Kickstarted and puppetized.

bugzilla1.db.scl3.mozilla.com has address 10.22.70.33
bugzilla2.db.scl3.mozilla.com has address 10.22.70.34
bugzilla3.db.scl3.mozilla.com has address 10.22.70.35
bugzilla4.db.scl3.mozilla.com has address 10.22.70.36

Comment 4

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/79e57b09177b

Comment 5

[Shyam Mani [:fox2mike]]

Burns,

Seems like none of these servers have been puppetized. Can we please fix that?

Comment 6

[Michael Burns [:mburns]]

[[e-mail]3 ~]# ls /home/ | wc -l
65
[[e-mail]3 ~]# ls /home/ | wc -l
65
[[e-mail]3 ~]# ls /home/ | wc -l
65
[[e-mail]3 ~]# ls /etc/cron.d
0hourly  hp-health  puppetcheck  raid-check  sysstat

My mistake. These look to be *actualy for realz no joke* puppetized now.

Comment 7

[Sheeri Cabral [:sheeri]]

Fixed!

Comment 8

[Sheeri Cabral [:sheeri]]

Verified! Thanx for jumping on this so quickly.

Comment 9

[Shyam Mani [:fox2mike]]

Sheeri, just to make sure : these have the 3.x fusionIO stuff on them? Thanks!

Comment 10

[Sheeri Cabral [:sheeri]]

Just to recap this AM's discusion on IRC:

These machines don't have a separate partition so I don't believe they have fusionIO stuff on them at all.

You're going to order SSD's, but that's another ticket.

Comment 11

[Shyam Mani [:fox2mike]]

Sorry Burnsie, I'm going to have to reopen, since we'll have to rekick these once the SSDs are in them.

Comment 12

[Michael Burns [:mburns]]

No problem. I'll get these kicked when the SSDs get installed.

Comment 13

[Shyam Mani [:fox2mike]]

(In reply to Michael Burns [:mburns] from comment #12)
> No problem. I'll get these kicked when the SSDs get installed.

Thanks dude. Van is replacing the drives now, feel free to re-kick when that's done.

Comment 14

[Sheeri Cabral [:sheeri]]

Looks like Van is done, so mburns whenever the pager stops ringing, feel free to have at it!

Comment 15

[Rick Bryce [:rbryce]]

burnsie is traveling to MozCamp.  I'm taking these over for him, but I won't get started until later this afternoon.

Comment 16

[Sheeri Cabral [:sheeri]]

Thanx!

Comment 17

[Rick Bryce [:rbryce]]

these have been rekicked and puppetized.  Each server had this error after the puppetze run. 

---

notice: /Stage[main]/Mysql2::Server/Service[mysql]: Dependency Package[Percona-Server-server-51] has failures: true
notice: /Stage[main]/Mysql2::Server/Service[mysql]: Dependency File[/etc/my.cnf] has failures: true
warning: /Stage[main]/Mysql2::Server/Service[mysql]: Skipping because of failed dependencies
notice: /Stage[main]/Mysql2::Server/Mysql2::Variable[slow_query_log_file]/Exec[mysql2::variable::slow_query_log_file]: Dependency Package[Percona-Server-server-51] has failures: true
notice: /Stage[main]/Mysql2::Server/Mysql2::Variable[slow_query_log_file]/Exec[mysql2::variable::slow_query_log_file]: Dependency File[/etc/my.cnf] has failures: true
warning: /Stage[main]/Mysql2::Server/Mysql2::Variable[slow_query_log_file]/Exec[mysql2::variable::slow_query_log_file]: Skipping because of failed dependencies
notice: /Stage[main]/Mysql2::Server/Mysql2::Variable[slow_query_log]/Exec[mysql2::variable::slow_query_log]: Dependency Package[Percona-Server-server-51] has failures: true
notice: /Stage[main]/Mysql2::Server/Mysql2::Variable[slow_query_log]/Exec[mysql2::variable::slow_query_log]: Dependency File[/etc/my.cnf] has failures: true
warning: /Stage[main]/Mysql2::Server/Mysql2::Variable[slow_query_log]/Exec[mysql2::variable::slow_query_log]: Skipping because of failed dependencies
notice: /Stage[main]/Mysql2::Server/Mysql2::Variable[log_output]/Exec[mysql2::variable::log_output]: Dependency Package[Percona-Server-server-51] has failures: true
notice: /Stage[main]/Mysql2::Server/Mysql2::Variable[log_output]/Exec[mysql2::variable::log_output]: Dependency File[/etc/my.cnf] has failures: true
warning: /Stage[main]/Mysql2::Server/Mysql2::Variable[log_output]/Exec[mysql2::variable::log_output]: Skipping because of failed dependencies
err: /Stage[main]/Mysql2::Server/File[/var/log/mysql]: Could not evaluate: Could not find user mysql

Comment 18

[Sheeri Cabral [:sheeri]]

Thanx! I fixed the problems and here's the latest run:

[[e-mail]3 ~]# /usr/local/sbin/puppetctl run
info: Retrieving plugin
info: Loading facts in /var/lib/puppet/lib/facter/customFacts.rb
info: Loading facts in /var/lib/puppet/lib/facter/root_home.rb
info: Loading facts in /var/lib/puppet/lib/facter/puppetmaster.rb
info: Loading facts in /var/lib/puppet/lib/facter/hp_array_utils.rb
info: Loading facts in /var/lib/puppet/lib/facter/vmware_version.rb
info: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb
info: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb
info: Loading facts in /var/lib/puppet/lib/facter/partitions.rb
info: Loading facts in /var/lib/puppet/lib/facter/datacenter.rb
info: Loading facts in /var/lib/puppet/lib/facter/available_package_updates.rb
info: Loading facts in /var/lib/puppet/lib/facter/iptables.rb
info: Loading facts in /var/lib/puppet/lib/facter/raidcontroller.rb
info: Caching catalog for bugzilla1.db.scl3.mozilla.com
info: Applying configuration version '46820'
notice: Finished catalog run in 25.58 seconds
[[e-mail]3 ~]#