updates of gecko-dev.git failing due to lack of space in /tmp: 09:20 < gps> 4:48:14 INFO - Running command: ['git', 'push', '/opt/vcs2vcs/build/target/beagle/.git', '+refs/heads/inbound:refs/heads/inbound'] in /opt/vcs2vcs/build/conversion/beagle/.git 09:20 < gps> 04:48:14 INFO - Copy/paste: git push /opt/vcs2vcs/build/target/beagle/.git +refs/heads/inbound:refs/heads/inbound 09:20 < gps> 04:48:14 INFO - Calling ['git', 'push', '/opt/vcs2vcs/build/target/beagle/.git', '+refs/heads/inbound:refs/heads/inbound'] with output_timeout 1800 09:20 < gps> 04:48:15 INFO - error: unable to create temporary sha1 filename : No space left on device 09:20 < gps> 04:48:15 INFO - fatal: failed to write object 09:20 < gps> 04:48:15 INFO - error: unpack failed: unpack-objects abnormal exit 09:20 < gps> 04:48:15 INFO - To /opt/vcs2vcs/build/target/beagle/.git 09:20 < gps> 04:48:15 ERROR - ! [remote rejected] inbound -> inbound (n/a (unpacker error))
disabled cron job and rebooted to get stable system to fix
We're out of inodes on /dev/xvdj
Summary: vcssync1.srv.releng.usw2.mozilla.com out of /tmp disk → vcssync1.srv.releng.usw2.mozilla.com out of inodes
[firstname.lastname@example.org ~]$ df -hi Filesystem Inodes IUsed IFree IUse% Mounted on /dev/xvde1 640K 42K 599K 7% / none 1.9M 1 1.9M 1% /dev/shm /dev/xvdj 6.3M 6.3M 0 100% /opt
I deleted the local clone of esr17, locally hacked the config, and manually ran a sync so downstream users can catch up a bit. rail is snapshoting the EBS volume so we can clone stuff easier.
rail made an EBS snapshot then exposed an SSD-mounted version to the host. We're now in the process of rsyncing data from a read-only SSD snapshot to a new, increased inodes SSD volume (20M inodes vs 6M). Once that is complete, we'll pause vcs syncing, do an rsync from original to the new volume, then swing /opt over to the new SSD volume and restore service. As a result of moving to the SSD, vcs sync should complete faster, as parts of vcs sync were I/O bound. The rsync should complete within 4 hours. We can probably have things running off the SSD by EOD. Normally, I'd be opposed to making changes on a Friday afternoon. But everything is self-contained on the /opt mount. This is a change I'd feel comfortable making on a Friday.
Rsync has completed. Volumes switched to use new SSD: [email@example.com ~]$ df -h Filesystem Size Used Avail Use% Mounted on /dev/xvde1 9.9G 5.2G 4.2G 56% / none 7.4G 0 7.4G 0% /dev/shm /dev/xvdk 96G 72G 19G 80% /opt [firstname.lastname@example.org ~]$ df -hi Filesystem Inodes IUsed IFree IUse% Mounted on /dev/xvde1 640K 42K 599K 7% / none 1.9M 1 1.9M 1% /dev/shm /dev/xvdk 20M 6.3M 13M 33% /opt Will leave like this for weekend. To do on Monday: - ensure everything looks reasonable - re-enable esr17 - close issue
Assignee: nobody → gps
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.