The default bug view has changed. See this FAQ.

manage_instance_storage.py should consider root volume size

RESOLVED FIXED

Status

Release Engineering
General Automation
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: rail, Assigned: rail)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(4 attachments)

(Assignee)

Description

3 years ago
We should use ephemeral storage for /builds/slave if the storage size is greater than the root volume size (minus ~10G we can't use).
(Assignee)

Comment 1

3 years ago
Created attachment 8442167 [details] [diff] [review]
root volume check
Attachment #8442167 - Flags: review?(catlee)

Updated

3 years ago
Attachment #8442167 - Flags: review?(catlee) → review+
(Assignee)

Comment 2

3 years ago
Comment on attachment 8442167 [details] [diff] [review]
root volume check

remote:   https://hg.mozilla.org/build/puppet/rev/a9856f387367
remote:   https://hg.mozilla.org/build/puppet/rev/468cafa821f6
Attachment #8442167 - Flags: checked-in+
(Assignee)

Comment 3

3 years ago
Created attachment 8442936 [details] [diff] [review]
dev_size-puppet.diff

Need another iteration. vgs doesn't work on non  LVM layouts, so the size is always zero on instances with 1 ephemeral device.
Attachment #8442936 - Flags: review?(catlee)

Updated

3 years ago
Attachment #8442936 - Flags: review?(catlee) → review+
(Assignee)

Comment 4

3 years ago
Comment on attachment 8442936 [details] [diff] [review]
dev_size-puppet.diff

remote:   https://hg.mozilla.org/build/puppet/rev/8fab8e1860ea
remote:   https://hg.mozilla.org/build/puppet/rev/3df278c162cc
Attachment #8442936 - Flags: checked-in+
(Assignee)

Comment 5

3 years ago
Ooh, another mystery!

[root@bld-linux64-spot-324.build.releng.usw2.mozilla.com ~]# df -h /dev/xvdb # Not mounted
Filesystem            Size  Used Avail Use% Mounted on
-                      16G  144K   16G   1% /dev
[root@bld-linux64-spot-324.build.releng.usw2.mozilla.com ~]# mount /dev/xvdb /mnt
[root@bld-linux64-spot-324.build.releng.usw2.mozilla.com ~]# df -h /dev/xvdb
Filesystem            Size  Used Avail Use% Mounted on
/dev/xvdb              74G  180M   70G   1% /mnt
(Assignee)

Comment 6

3 years ago
Created attachment 8445142 [details] [diff] [review]
df-puppet.diff

Not sure if this is the most elegant way to get the size of ephemeral storage, but it should work (TM).
Attachment #8445142 - Flags: review?(catlee)
Comment on attachment 8445142 [details] [diff] [review]
df-puppet.diff

Review of attachment 8445142 [details] [diff] [review]:
-----------------------------------------------------------------

::: modules/aws/files/manage_instance_storage.py
@@ +382,4 @@
>          output = get_output_from_cmd(cmd)
> +        if need_mount:
> +            umount(device)
> +            os.rmdir(mount_point)

put os.rmdir() into finally clause maybe?
Attachment #8445142 - Flags: review?(catlee) → review+
(Assignee)

Comment 8

3 years ago
Comment on attachment 8445142 [details] [diff] [review]
df-puppet.diff

(In reply to Chris AtLee [:catlee] from comment #7)
> Comment on attachment 8445142 [details] [diff] [review]
> df-puppet.diff
> 
> Review of attachment 8445142 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: modules/aws/files/manage_instance_storage.py
> @@ +382,4 @@
> >          output = get_output_from_cmd(cmd)
> > +        if need_mount:
> > +            umount(device)
> > +            os.rmdir(mount_point)
> 
> put os.rmdir() into finally clause maybe?

I pushed the patch after I saw r+, but before I saw this comment.
I can followup for sure. Let's just see how this one sticks.


remote:   https://hg.mozilla.org/build/puppet/rev/6aa41db61598
remote:   https://hg.mozilla.org/build/puppet/rev/9be37a0ec8a5
Attachment #8445142 - Flags: checked-in+
(Assignee)

Comment 9

3 years ago
The check works now, but I didn't like how the mount output looks like:

# mount
/dev/xvda1 on / type ext4 (rw,noatime,nodiratime,commit=60)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
none on /dev/shm type tmpfs (rw)
/dev/xvdb on /mnt/instance_storage type ext4 (rw,noatime)
/mnt/instance_storage/ccache on /builds/ccache type none (rw,bind,noatime)
/mnt/instance_storage/mock_mozilla on /builds/mock_mozilla type none (rw,bind,noatime)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
/builds/slave/ccache on /builds/ccache type none (rw,bind,noatime)
/builds/slave/mock_mozilla on /builds/mock_mozilla type none (rw,bind,noatime)

I think that we should just get rid of /etc/fstab entry updates, because it makes everything more complicated and we already run thus code on every boot.
(Assignee)

Comment 10

3 years ago
Created attachment 8453263 [details] [diff] [review]
df-puppet-1.diff

* Stop managing /etc/fstab, everything is runtime
* mount() accepts more parameters now

I tested this on 2 build slaves.

This is how one of the jacuzzi machines looks like:

[root@bld-linux64-spot-450.build.releng.usw2.mozilla.com ~]# cat /etc/fstab 
LABEL=root_dev   /         ext4   defaults,noatime,nodiratime,commit=60        1 1
none       /proc     proc    defaults        0 0
none       /sys      sysfs   defaults        0 0
none       /dev/pts  devpts  gid=5,mode=620  0 0
none       /dev/shm  tmpfs   defaults        0 0

[root@bld-linux64-spot-450.build.releng.usw2.mozilla.com ~]# mount
/dev/xvda1 on / type ext4 (rw,noatime,nodiratime,commit=60)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
none on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
/dev/xvdb on /builds/slave type ext4 (rw,noatime)
/builds/slave/ccache on /builds/ccache type none (rw,bind,noatime)
/builds/slave/mock_mozilla on /builds/mock_mozilla type none (rw,bind,noatime)


A non-jacuzzi slave looks exactly the same (because instance storage is larger than the root volume - 10G)

[root@bld-linux64-spot-449.build.releng.usw2.mozilla.com ~]# cat /etc/fstab 
LABEL=root_dev   /         ext4   defaults,noatime,nodiratime,commit=60        1 1
none       /proc     proc    defaults        0 0
none       /sys      sysfs   defaults        0 0
none       /dev/pts  devpts  gid=5,mode=620  0 0
none       /dev/shm  tmpfs   defaults        0 0
[root@bld-linux64-spot-449.build.releng.usw2.mozilla.com ~]# mount
/dev/xvda1 on / type ext4 (rw,noatime,nodiratime,commit=60)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
none on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
/dev/xvdb on /builds/slave type ext4 (rw,noatime)
/builds/slave/ccache on /builds/ccache type none (rw,bind,noatime)
/builds/slave/mock_mozilla on /builds/mock_mozilla type none (rw,bind,noatime)
Attachment #8453263 - Flags: review?(catlee)

Updated

3 years ago
Attachment #8453263 - Flags: review?(catlee) → review+
(Assignee)

Comment 11

3 years ago
Comment on attachment 8453263 [details] [diff] [review]
df-puppet-1.diff

remote:   https://hg.mozilla.org/build/puppet/rev/7075a41dacba
remote:   https://hg.mozilla.org/build/puppet/rev/4648272030c8

Transplanted to production to make it in in the nightly AMI.
Attachment #8453263 - Flags: checked-in+
(Assignee)

Comment 12

3 years ago
Yay, worked as expected! Now we can shrink EBS!
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.