Status

RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: gps, Assigned: ryanc)

Tracking

Details

(Reporter)

Description

3 years ago
hgweb1.dmz.scl3.mozilla.com has been out of the load balancer and disabled from Puppet and Nagios for a few weeks while I run "experiments." I'm done with my work and am ready to return this host to the pool. However, I made some rather crazy changes and I don't feel comfortable returning this host to the pool as is. Furthermore, I'd like an opportunity to test how our fresh server "bootstrap" process works. This will give me an opportunity to write documentation/automation around our disaster recovery workflows as well as make it easier to stand up new servers in the future.

Please re-image hgweb1.dmz.scl3.mozilla.com at your earliest convenience. Please wipe all partitions so the machine is starting from scratch.

Please note that /dev/sda3 has a custom amount of inodes (~8M) and instructions at https://mana.mozilla.org/wiki/display/SYSADMIN/Provisioning+extra+inodes+on+the+root+filesystem will need to be followed.

Please *do not* return the machine to Nagios or the ZLB. Reactivating Puppet is fine.
(Reporter)

Comment 1

3 years ago
I should have said ~80M inodes, not ~8M.
(Reporter)

Comment 2

3 years ago
Bug 1181260 comment #15 is also likely relevant and indicates a Puppet change that needs reverting.
(Assignee)

Comment 3

3 years ago
Will have this back to you in a few hours.
Assignee: nobody → rchilds
Status: NEW → ASSIGNED
(Assignee)

Comment 4

3 years ago
Greg,

Looking at the kickstarts that we have shows that almost all of them, besides one have the extra inodes function commented out and says specifically:

# This section does not work

I've also tried proceeding with some of these with no luck. This would need some diagnosis and could take some time.


[rchilds@admin1a.private.scl3 profiles]$ grep -nr "inode" ./*
./extrainodes.cfg:50:## We need more inodes than the default
./extrainodes.cfg:51:#/bin/sed -i 's/inode_ratio = 16384/inode_ratio = 2048/g' /etc/mke2fs.conf
./github-sync4.dmz.scl3.cfg:50:## We need more inodes than the default
./github-sync4.dmz.scl3.cfg:51:#/bin/sed -i 's/inode_ratio = 16384/inode_ratio = 2048/g' /etc/mke2fs.conf
./hgweb1.dmz.scl3.mozilla.com.cfg:50:## We need more inodes than the default
./hgweb1.dmz.scl3.mozilla.com.cfg:51:#/bin/sed -i 's/inode_ratio = 16384/inode_ratio = 2048/g' /etc/mke2fs.conf
./hgweb.cfg:35:#part / --fstype ext4 --bytes-per-inode=4096 --onpart=sda3
./hgweb.cfg:51:## We need more inodes than the default
./hgweb.cfg:52:#/bin/sed -i 's/inode_ratio = 16384/inode_ratio = 2048/g' /etc/mke2fs.conf
./RHEL-6.2-x86_64-Vanilla-github.cfg:47:sed -i 's/inode_ratio = 16384/inode_ratio = 4096/g' /etc/mke2fs.conf


Setting aside the directory that is used as storage for hg would probably be the quickest/easiest solution for getting additional inodes.
(Reporter)

Comment 5

3 years ago
We may want to give fubar the next ~12 hours to provide historical context. But, I wouldn't be opposed to establishing a separate partition for the repository data. If we can't figure out how to allocate the extra inodes on /dev/sda3, let's just set aside ~8 GB for the OS partition (sda3) and leave the rest unpartitioned (for hg foo) and I'll take it from there.
(Assignee)

Comment 6

3 years ago
Greg,

Sounds good. I'd be interested as well. Another alternative may be to use XFS (assigns inodes dynamically), which I believe can be used for installation with RHEL 6.6 and later.


You're all set --

[rchilds@admin1a.private.scl3 ~]$ sudo cat /var/www/html/kickstart/profiles/hg.cfg | grep "sda"
part /boot --ondisk=sda --fstype ext4 --size=512
part swap --ondisk=sda --fstype swap --size=2048
part / --ondisk=sda --fstype ext4 --size=8192
part /hg --ondisk=sda --fstype ext4 --size=1024 --grow

[rchilds@hgweb1.dmz.scl3 ~]$ uname -a && cat /etc/redhat-release
Linux hgweb1.dmz.scl3.mozilla.com 2.6.32-573.1.1.el6.x86_64 #1 SMP Tue Jul 14 02:46:51 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
Red Hat Enterprise Linux Server release 6.7 (Santiago)
[rchilds@hgweb1.dmz.scl3 ~]$ df -hTa
Filesystem     Type         Size  Used Avail Use% Mounted on
/dev/sda2      ext4         7.8G  1.6G  5.8G  22% /
proc           proc            0     0     0    - /proc
sysfs          sysfs           0     0     0    - /sys
devpts         devpts          0     0     0    - /dev/pts
tmpfs          tmpfs        5.9G     0  5.9G   0% /dev/shm
/dev/sda1      ext4         488M   58M  405M  13% /boot
/dev/sda5      ext4         540G   70M  513G   1% /hg
none           binfmt_misc     0     0     0    - /proc/sys/fs/binfmt_misc
Status: ASSIGNED → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
I believe this is what folks used in the past to provision extra inodes: 

https://mana.mozilla.org/wiki/display/SYSADMIN/Provisioning+extra+inodes+on+the+root+filesystem
(Reporter)

Comment 8

3 years ago
Thank you for attending to this.
You need to log in before you can comment on or make changes to this bug.