1188702 - Re-image hgweb1.dmz.scl3

Status: RESOLVED FIXED

(Infrastructure & Operations :: MOC: Service Requests, task)

Description

[Gregory Szorc [:gps]]

hgweb1.dmz.scl3.mozilla.com has been out of the load balancer and disabled from Puppet and Nagios for a few weeks while I run "experiments." I'm done with my work and am ready to return this host to the pool. However, I made some rather crazy changes and I don't feel comfortable returning this host to the pool as is. Furthermore, I'd like an opportunity to test how our fresh server "bootstrap" process works. This will give me an opportunity to write documentation/automation around our disaster recovery workflows as well as make it easier to stand up new servers in the future.

Please re-image hgweb1.dmz.scl3.mozilla.com at your earliest convenience. Please wipe all partitions so the machine is starting from scratch.

Please note that /dev/sda3 has a custom amount of inodes (~8M) and instructions at https://mana.mozilla.org/wiki/display/SYSADMIN/Provisioning+extra+inodes+on+the+root+filesystem will need to be followed.

Please *do not* return the machine to Nagios or the ZLB. Reactivating Puppet is fine.

Comment 1

[Gregory Szorc [:gps]]

I should have said ~80M inodes, not ~8M.

Comment 2

[Gregory Szorc [:gps]]

Bug 1181260 comment #15 is also likely relevant and indicates a Puppet change that needs reverting.

Comment 3

[Ryan C [:ryanc] (UTC-4)]

Will have this back to you in a few hours.

Comment 4

[Ryan C [:ryanc] (UTC-4)]

Greg,

Looking at the kickstarts that we have shows that almost all of them, besides one have the extra inodes function commented out and says specifically:

# This section does not work

I've also tried proceeding with some of these with no luck. This would need some diagnosis and could take some time.


[[e-mail]3 profiles]$ grep -nr "inode" ./*
./extrainodes.cfg:50:## We need more inodes than the default
./extrainodes.cfg:51:#/bin/sed -i 's/inode_ratio = 16384/inode_ratio = 2048/g' /etc/mke2fs.conf
./github-sync4.dmz.scl3.cfg:50:## We need more inodes than the default
./github-sync4.dmz.scl3.cfg:51:#/bin/sed -i 's/inode_ratio = 16384/inode_ratio = 2048/g' /etc/mke2fs.conf
./hgweb1.dmz.scl3.mozilla.com.cfg:50:## We need more inodes than the default
./hgweb1.dmz.scl3.mozilla.com.cfg:51:#/bin/sed -i 's/inode_ratio = 16384/inode_ratio = 2048/g' /etc/mke2fs.conf
./hgweb.cfg:35:#part / --fstype ext4 --bytes-per-inode=4096 --onpart=sda3
./hgweb.cfg:51:## We need more inodes than the default
./hgweb.cfg:52:#/bin/sed -i 's/inode_ratio = 16384/inode_ratio = 2048/g' /etc/mke2fs.conf
./RHEL-6.2-x86_64-Vanilla-github.cfg:47:sed -i 's/inode_ratio = 16384/inode_ratio = 4096/g' /etc/mke2fs.conf


Setting aside the directory that is used as storage for hg would probably be the quickest/easiest solution for getting additional inodes.

Comment 5

[Gregory Szorc [:gps]]

We may want to give fubar the next ~12 hours to provide historical context. But, I wouldn't be opposed to establishing a separate partition for the repository data. If we can't figure out how to allocate the extra inodes on /dev/sda3, let's just set aside ~8 GB for the OS partition (sda3) and leave the rest unpartitioned (for hg foo) and I'll take it from there.

Comment 6

[Ryan C [:ryanc] (UTC-4)]

Greg,

Sounds good. I'd be interested as well. Another alternative may be to use XFS (assigns inodes dynamically), which I believe can be used for installation with RHEL 6.6 and later.


You're all set --

[[e-mail]3 ~]$ sudo cat /var/www/html/kickstart/profiles/hg.cfg | grep "sda"
part /boot --ondisk=sda --fstype ext4 --size=512
part swap --ondisk=sda --fstype swap --size=2048
part / --ondisk=sda --fstype ext4 --size=8192
part /hg --ondisk=sda --fstype ext4 --size=1024 --grow

[[e-mail]3 ~]$ uname -a && cat /etc/redhat-release
Linux hgweb1.dmz.scl3.mozilla.com 2.6.32-573.1.1.el6.x86_64 #1 SMP Tue Jul 14 02:46:51 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 6.7 (Santiago)
[[e-mail]3 ~]$ df -hTa
Filesystem     Type         Size  Used Avail Use% Mounted on
/dev/sda2      ext4         7.8G  1.6G  5.8G  22% /
proc           proc            0     0     0    - /proc
sysfs          sysfs           0     0     0    - /sys
devpts         devpts          0     0     0    - /dev/pts
tmpfs          tmpfs        5.9G     0  5.9G   0% /dev/shm
/dev/sda1      ext4         488M   58M  405M  13% /boot
/dev/sda5      ext4         540G   70M  513G   1% /hg
none           binfmt_misc     0     0     0    - /proc/sys/fs/binfmt_misc

Comment 7

[Kendall Libby [:fubar] (he/him)]

I believe this is what folks used in the past to provision extra inodes: 

https://mana.mozilla.org/wiki/display/SYSADMIN/Provisioning+extra+inodes+on+the+root+filesystem

Comment 8

[Gregory Szorc [:gps]]

Thank you for attending to this.