3 years ago
3 years ago


(Reporter: gps, Assigned: ryanc)





3 years ago has been out of the load balancer and disabled from Puppet and Nagios for a few weeks while I run "experiments." I'm done with my work and am ready to return this host to the pool. However, I made some rather crazy changes and I don't feel comfortable returning this host to the pool as is. Furthermore, I'd like an opportunity to test how our fresh server "bootstrap" process works. This will give me an opportunity to write documentation/automation around our disaster recovery workflows as well as make it easier to stand up new servers in the future.

Please re-image at your earliest convenience. Please wipe all partitions so the machine is starting from scratch.

Please note that /dev/sda3 has a custom amount of inodes (~8M) and instructions at will need to be followed.

Please *do not* return the machine to Nagios or the ZLB. Reactivating Puppet is fine.

Comment 1

3 years ago
I should have said ~80M inodes, not ~8M.

Comment 2

3 years ago
Bug 1181260 comment #15 is also likely relevant and indicates a Puppet change that needs reverting.

Comment 3

3 years ago
Will have this back to you in a few hours.
Assignee: nobody → rchilds

Comment 4

3 years ago

Looking at the kickstarts that we have shows that almost all of them, besides one have the extra inodes function commented out and says specifically:

# This section does not work

I've also tried proceeding with some of these with no luck. This would need some diagnosis and could take some time.

[rchilds@admin1a.private.scl3 profiles]$ grep -nr "inode" ./*
./extrainodes.cfg:50:## We need more inodes than the default
./extrainodes.cfg:51:#/bin/sed -i 's/inode_ratio = 16384/inode_ratio = 2048/g' /etc/mke2fs.conf
./github-sync4.dmz.scl3.cfg:50:## We need more inodes than the default
./github-sync4.dmz.scl3.cfg:51:#/bin/sed -i 's/inode_ratio = 16384/inode_ratio = 2048/g' /etc/mke2fs.conf
./ We need more inodes than the default
./ -i 's/inode_ratio = 16384/inode_ratio = 2048/g' /etc/mke2fs.conf
./hgweb.cfg:35:#part / --fstype ext4 --bytes-per-inode=4096 --onpart=sda3
./hgweb.cfg:51:## We need more inodes than the default
./hgweb.cfg:52:#/bin/sed -i 's/inode_ratio = 16384/inode_ratio = 2048/g' /etc/mke2fs.conf
./RHEL-6.2-x86_64-Vanilla-github.cfg:47:sed -i 's/inode_ratio = 16384/inode_ratio = 4096/g' /etc/mke2fs.conf

Setting aside the directory that is used as storage for hg would probably be the quickest/easiest solution for getting additional inodes.

Comment 5

3 years ago
We may want to give fubar the next ~12 hours to provide historical context. But, I wouldn't be opposed to establishing a separate partition for the repository data. If we can't figure out how to allocate the extra inodes on /dev/sda3, let's just set aside ~8 GB for the OS partition (sda3) and leave the rest unpartitioned (for hg foo) and I'll take it from there.

Comment 6

3 years ago

Sounds good. I'd be interested as well. Another alternative may be to use XFS (assigns inodes dynamically), which I believe can be used for installation with RHEL 6.6 and later.

You're all set --

[rchilds@admin1a.private.scl3 ~]$ sudo cat /var/www/html/kickstart/profiles/hg.cfg | grep "sda"
part /boot --ondisk=sda --fstype ext4 --size=512
part swap --ondisk=sda --fstype swap --size=2048
part / --ondisk=sda --fstype ext4 --size=8192
part /hg --ondisk=sda --fstype ext4 --size=1024 --grow

[rchilds@hgweb1.dmz.scl3 ~]$ uname -a && cat /etc/redhat-release
Linux 2.6.32-573.1.1.el6.x86_64 #1 SMP Tue Jul 14 02:46:51 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 6.7 (Santiago)
[rchilds@hgweb1.dmz.scl3 ~]$ df -hTa
Filesystem     Type         Size  Used Avail Use% Mounted on
/dev/sda2      ext4         7.8G  1.6G  5.8G  22% /
proc           proc            0     0     0    - /proc
sysfs          sysfs           0     0     0    - /sys
devpts         devpts          0     0     0    - /dev/pts
tmpfs          tmpfs        5.9G     0  5.9G   0% /dev/shm
/dev/sda1      ext4         488M   58M  405M  13% /boot
/dev/sda5      ext4         540G   70M  513G   1% /hg
none           binfmt_misc     0     0     0    - /proc/sys/fs/binfmt_misc
Last Resolved: 3 years ago
Resolution: --- → FIXED
I believe this is what folks used in the past to provision extra inodes:

Comment 8

3 years ago
Thank you for attending to this.
You need to log in before you can comment on or make changes to this bug.