1135129 - vcssync1.srv.releng.usw2.mozilla.com out of inodes

Status: RESOLVED FIXED

(Developer Services :: General, task)

Description

[Hal Wine [:hwine] use NI!]

updates of gecko-dev.git failing due to lack of space in /tmp:
09:20 < gps> 4:48:14     INFO - Running command: ['git', 'push', '/opt/vcs2vcs/build/target/beagle/.git', '+refs/heads/inbound:refs/heads/inbound'] in /opt/vcs2vcs/build/conversion/beagle/.git
09:20 < gps> 04:48:14     INFO - Copy/paste: git push /opt/vcs2vcs/build/target/beagle/.git +refs/heads/inbound:refs/heads/inbound
09:20 < gps> 04:48:14     INFO - Calling ['git', 'push', '/opt/vcs2vcs/build/target/beagle/.git', '+refs/heads/inbound:refs/heads/inbound'] with output_timeout 1800
09:20 < gps> 04:48:15     INFO -  error: unable to create temporary sha1 filename : No space left on device
09:20 < gps> 04:48:15     INFO -  fatal: failed to write object
09:20 < gps> 04:48:15     INFO -  error: unpack failed: unpack-objects abnormal exit
09:20 < gps> 04:48:15     INFO -  To /opt/vcs2vcs/build/target/beagle/.git
09:20 < gps> 04:48:15    ERROR -   ! [remote rejected] inbound -> inbound (n/a (unpacker error))

Comment 1

[Hal Wine [:hwine] use NI!]

disabled cron job and rebooted to get stable system to fix

Comment 2

[Gregory Szorc [:gps]]

We're out of inodes on /dev/xvdj

Comment 3

[Hal Wine [:hwine] use NI!]

[[e-mail] ~]$ df -hi
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/xvde1              640K     42K    599K    7% /
none                    1.9M       1    1.9M    1% /dev/shm
/dev/xvdj               6.3M    6.3M       0  100% /opt

Comment 4

[Gregory Szorc [:gps]]

I deleted the local clone of esr17, locally hacked the config, and manually ran a sync so downstream users can catch up a bit.

rail is snapshoting the EBS volume so we can clone stuff easier.

Comment 5

[Gregory Szorc [:gps]]

rail made an EBS snapshot then exposed an SSD-mounted version to the host. We're now in the process of rsyncing data from a read-only SSD snapshot to a new, increased inodes SSD volume (20M inodes vs 6M). Once that is complete, we'll pause vcs syncing, do an rsync from original to the new volume, then swing /opt over to the new SSD volume and restore service.

As a result of moving to the SSD, vcs sync should complete faster, as parts of vcs sync were I/O bound.

The rsync should complete within 4 hours. We can probably have things running off the SSD by EOD.

Normally, I'd be opposed to making changes on a Friday afternoon. But everything is self-contained on the /opt mount. This is a change I'd feel comfortable making on a Friday.

Comment 6

[Hal Wine [:hwine] use NI!]

Rsync has completed. Volumes switched to use new SSD:

[[e-mail] ~]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/xvde1            9.9G  5.2G  4.2G  56% /
none                  7.4G     0  7.4G   0% /dev/shm
/dev/xvdk              96G   72G   19G  80% /opt
[[e-mail] ~]$ df -hi
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/xvde1              640K     42K    599K    7% /
none                    1.9M       1    1.9M    1% /dev/shm
/dev/xvdk                20M    6.3M     13M   33% /opt

Will leave like this for weekend. 

To do on Monday:
 - ensure everything looks reasonable
 - re-enable esr17
 - close issue

Comment 7

[Hal Wine [:hwine] use NI!]

all done