hgweb1.dmz.scl3.mozilla.com has been out of the load balancer and disabled from Puppet and Nagios for a few weeks while I run "experiments." I'm done with my work and am ready to return this host to the pool. However, I made some rather crazy changes and I don't feel comfortable returning this host to the pool as is. Furthermore, I'd like an opportunity to test how our fresh server "bootstrap" process works. This will give me an opportunity to write documentation/automation around our disaster recovery workflows as well as make it easier to stand up new servers in the future. Please re-image hgweb1.dmz.scl3.mozilla.com at your earliest convenience. Please wipe all partitions so the machine is starting from scratch. Please note that /dev/sda3 has a custom amount of inodes (~8M) and instructions at https://mana.mozilla.org/wiki/display/SYSADMIN/Provisioning+extra+inodes+on+the+root+filesystem will need to be followed. Please *do not* return the machine to Nagios or the ZLB. Reactivating Puppet is fine.
I should have said ~80M inodes, not ~8M.
Bug 1181260 comment #15 is also likely relevant and indicates a Puppet change that needs reverting.
Will have this back to you in a few hours.
Assignee: nobody → rchilds
Status: NEW → ASSIGNED
Greg, Looking at the kickstarts that we have shows that almost all of them, besides one have the extra inodes function commented out and says specifically: # This section does not work I've also tried proceeding with some of these with no luck. This would need some diagnosis and could take some time. [firstname.lastname@example.org profiles]$ grep -nr "inode" ./* ./extrainodes.cfg:50:## We need more inodes than the default ./extrainodes.cfg:51:#/bin/sed -i 's/inode_ratio = 16384/inode_ratio = 2048/g' /etc/mke2fs.conf ./github-sync4.dmz.scl3.cfg:50:## We need more inodes than the default ./github-sync4.dmz.scl3.cfg:51:#/bin/sed -i 's/inode_ratio = 16384/inode_ratio = 2048/g' /etc/mke2fs.conf ./hgweb1.dmz.scl3.mozilla.com.cfg:50:## We need more inodes than the default ./hgweb1.dmz.scl3.mozilla.com.cfg:51:#/bin/sed -i 's/inode_ratio = 16384/inode_ratio = 2048/g' /etc/mke2fs.conf ./hgweb.cfg:35:#part / --fstype ext4 --bytes-per-inode=4096 --onpart=sda3 ./hgweb.cfg:51:## We need more inodes than the default ./hgweb.cfg:52:#/bin/sed -i 's/inode_ratio = 16384/inode_ratio = 2048/g' /etc/mke2fs.conf ./RHEL-6.2-x86_64-Vanilla-github.cfg:47:sed -i 's/inode_ratio = 16384/inode_ratio = 4096/g' /etc/mke2fs.conf Setting aside the directory that is used as storage for hg would probably be the quickest/easiest solution for getting additional inodes.
We may want to give fubar the next ~12 hours to provide historical context. But, I wouldn't be opposed to establishing a separate partition for the repository data. If we can't figure out how to allocate the extra inodes on /dev/sda3, let's just set aside ~8 GB for the OS partition (sda3) and leave the rest unpartitioned (for hg foo) and I'll take it from there.
Greg, Sounds good. I'd be interested as well. Another alternative may be to use XFS (assigns inodes dynamically), which I believe can be used for installation with RHEL 6.6 and later. You're all set -- [email@example.com ~]$ sudo cat /var/www/html/kickstart/profiles/hg.cfg | grep "sda" part /boot --ondisk=sda --fstype ext4 --size=512 part swap --ondisk=sda --fstype swap --size=2048 part / --ondisk=sda --fstype ext4 --size=8192 part /hg --ondisk=sda --fstype ext4 --size=1024 --grow [firstname.lastname@example.org ~]$ uname -a && cat /etc/redhat-release Linux hgweb1.dmz.scl3.mozilla.com 2.6.32-573.1.1.el6.x86_64 #1 SMP Tue Jul 14 02:46:51 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux Red Hat Enterprise Linux Server release 6.7 (Santiago) [email@example.com ~]$ df -hTa Filesystem Type Size Used Avail Use% Mounted on /dev/sda2 ext4 7.8G 1.6G 5.8G 22% / proc proc 0 0 0 - /proc sysfs sysfs 0 0 0 - /sys devpts devpts 0 0 0 - /dev/pts tmpfs tmpfs 5.9G 0 5.9G 0% /dev/shm /dev/sda1 ext4 488M 58M 405M 13% /boot /dev/sda5 ext4 540G 70M 513G 1% /hg none binfmt_misc 0 0 0 - /proc/sys/fs/binfmt_misc
Status: ASSIGNED → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
I believe this is what folks used in the past to provision extra inodes: https://mana.mozilla.org/wiki/display/SYSADMIN/Provisioning+extra+inodes+on+the+root+filesystem
Thank you for attending to this.
You need to log in before you can comment on or make changes to this bug.