vcssync1.srv.releng.usw2.mozilla.com out of inodes

RESOLVED FIXED

Status

RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: hwine, Assigned: gps)

Tracking

Details

(Reporter)

Description

4 years ago
updates of gecko-dev.git failing due to lack of space in /tmp:
09:20 < gps> 4:48:14     INFO - Running command: ['git', 'push', '/opt/vcs2vcs/build/target/beagle/.git', '+refs/heads/inbound:refs/heads/inbound'] in /opt/vcs2vcs/build/conversion/beagle/.git
09:20 < gps> 04:48:14     INFO - Copy/paste: git push /opt/vcs2vcs/build/target/beagle/.git +refs/heads/inbound:refs/heads/inbound
09:20 < gps> 04:48:14     INFO - Calling ['git', 'push', '/opt/vcs2vcs/build/target/beagle/.git', '+refs/heads/inbound:refs/heads/inbound'] with output_timeout 1800
09:20 < gps> 04:48:15     INFO -  error: unable to create temporary sha1 filename : No space left on device
09:20 < gps> 04:48:15     INFO -  fatal: failed to write object
09:20 < gps> 04:48:15     INFO -  error: unpack failed: unpack-objects abnormal exit
09:20 < gps> 04:48:15     INFO -  To /opt/vcs2vcs/build/target/beagle/.git
09:20 < gps> 04:48:15    ERROR -   ! [remote rejected] inbound -> inbound (n/a (unpacker error))
(Reporter)

Comment 1

4 years ago
disabled cron job and rebooted to get stable system to fix
(Assignee)

Comment 2

4 years ago
We're out of inodes on /dev/xvdj
Summary: vcssync1.srv.releng.usw2.mozilla.com out of /tmp disk → vcssync1.srv.releng.usw2.mozilla.com out of inodes
(Reporter)

Comment 3

4 years ago
[hwine@vcssync1.srv.releng.usw2.mozilla.com ~]$ df -hi
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/xvde1              640K     42K    599K    7% /
none                    1.9M       1    1.9M    1% /dev/shm
/dev/xvdj               6.3M    6.3M       0  100% /opt
(Assignee)

Comment 4

4 years ago
I deleted the local clone of esr17, locally hacked the config, and manually ran a sync so downstream users can catch up a bit.

rail is snapshoting the EBS volume so we can clone stuff easier.
(Assignee)

Comment 5

4 years ago
rail made an EBS snapshot then exposed an SSD-mounted version to the host. We're now in the process of rsyncing data from a read-only SSD snapshot to a new, increased inodes SSD volume (20M inodes vs 6M). Once that is complete, we'll pause vcs syncing, do an rsync from original to the new volume, then swing /opt over to the new SSD volume and restore service.

As a result of moving to the SSD, vcs sync should complete faster, as parts of vcs sync were I/O bound.

The rsync should complete within 4 hours. We can probably have things running off the SSD by EOD.

Normally, I'd be opposed to making changes on a Friday afternoon. But everything is self-contained on the /opt mount. This is a change I'd feel comfortable making on a Friday.
(Reporter)

Comment 6

4 years ago
Rsync has completed. Volumes switched to use new SSD:

[hwine@vcssync1.srv.releng.usw2.mozilla.com ~]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/xvde1            9.9G  5.2G  4.2G  56% /
none                  7.4G     0  7.4G   0% /dev/shm
/dev/xvdk              96G   72G   19G  80% /opt
[hwine@vcssync1.srv.releng.usw2.mozilla.com ~]$ df -hi
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/xvde1              640K     42K    599K    7% /
none                    1.9M       1    1.9M    1% /dev/shm
/dev/xvdk                20M    6.3M     13M   33% /opt

Will leave like this for weekend. 

To do on Monday:
 - ensure everything looks reasonable
 - re-enable esr17
 - close issue
(Reporter)

Updated

4 years ago
See Also: → bug 1135266
(Reporter)

Comment 7

4 years ago
all done
Assignee: nobody → gps
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.