1027134 - manage_instance_storage.py should consider root volume size

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Rail Aliiev [:rail]]

We should use ephemeral storage for /builds/slave if the storage size is greater than the root volume size (minus ~10G we can't use).

Comment 1

[Rail Aliiev [:rail]]

Attachment: root volume check

Comment 2

[Rail Aliiev [:rail]]

Comment on attachment 8442167 [details] [diff] [review]
root volume check

remote:   https://hg.mozilla.org/build/puppet/rev/a9856f387367
remote:   https://hg.mozilla.org/build/puppet/rev/468cafa821f6

Comment 3

[Rail Aliiev [:rail]]

Attachment: dev_size-puppet.diff

Need another iteration. vgs doesn't work on non  LVM layouts, so the size is always zero on instances with 1 ephemeral device.

Comment 4

[Rail Aliiev [:rail]]

Comment on attachment 8442936 [details] [diff] [review]
dev_size-puppet.diff

remote:   https://hg.mozilla.org/build/puppet/rev/8fab8e1860ea
remote:   https://hg.mozilla.org/build/puppet/rev/3df278c162cc

Comment 5

[Rail Aliiev [:rail]]

Ooh, another mystery!

[[e-mail] ~]# df -h /dev/xvdb # Not mounted
Filesystem            Size  Used Avail Use% Mounted on
-                      16G  144K   16G   1% /dev
[[e-mail] ~]# mount /dev/xvdb /mnt
[[e-mail] ~]# df -h /dev/xvdb
Filesystem            Size  Used Avail Use% Mounted on
/dev/xvdb              74G  180M   70G   1% /mnt

Comment 6

[Rail Aliiev [:rail]]

Attachment: df-puppet.diff

Not sure if this is the most elegant way to get the size of ephemeral storage, but it should work (TM).

Comment 7

[Chris AtLee [:catlee]]

Comment on attachment 8445142 [details] [diff] [review]
df-puppet.diff

Review of attachment 8445142 [details] [diff] [review]:
-----------------------------------------------------------------

::: modules/aws/files/manage_instance_storage.py
@@ +382,4 @@
>          output = get_output_from_cmd(cmd)
> +        if need_mount:
> +            umount(device)
> +            os.rmdir(mount_point)

put os.rmdir() into finally clause maybe?

Comment 8

[Rail Aliiev [:rail]]

Comment on attachment 8445142 [details] [diff] [review]
df-puppet.diff

(In reply to Chris AtLee [:catlee] from comment #7)
> Comment on attachment 8445142 [details] [diff] [review]
> df-puppet.diff
> 
> Review of attachment 8445142 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: modules/aws/files/manage_instance_storage.py
> @@ +382,4 @@
> >          output = get_output_from_cmd(cmd)
> > +        if need_mount:
> > +            umount(device)
> > +            os.rmdir(mount_point)
> 
> put os.rmdir() into finally clause maybe?

I pushed the patch after I saw r+, but before I saw this comment.
I can followup for sure. Let's just see how this one sticks.


remote:   https://hg.mozilla.org/build/puppet/rev/6aa41db61598
remote:   https://hg.mozilla.org/build/puppet/rev/9be37a0ec8a5

Comment 9

[Rail Aliiev [:rail]]

The check works now, but I didn't like how the mount output looks like:

# mount
/dev/xvda1 on / type ext4 (rw,noatime,nodiratime,commit=60)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
none on /dev/shm type tmpfs (rw)
/dev/xvdb on /mnt/instance_storage type ext4 (rw,noatime)
/mnt/instance_storage/ccache on /builds/ccache type none (rw,bind,noatime)
/mnt/instance_storage/mock_mozilla on /builds/mock_mozilla type none (rw,bind,noatime)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
/builds/slave/ccache on /builds/ccache type none (rw,bind,noatime)
/builds/slave/mock_mozilla on /builds/mock_mozilla type none (rw,bind,noatime)

I think that we should just get rid of /etc/fstab entry updates, because it makes everything more complicated and we already run thus code on every boot.

Comment 10

[Rail Aliiev [:rail]]

Attachment: df-puppet-1.diff

* Stop managing /etc/fstab, everything is runtime
* mount() accepts more parameters now

I tested this on 2 build slaves.

This is how one of the jacuzzi machines looks like:

[[e-mail] ~]# cat /etc/fstab 
LABEL=root_dev   /         ext4   defaults,noatime,nodiratime,commit=60        1 1
none       /proc     proc    defaults        0 0
none       /sys      sysfs   defaults        0 0
none       /dev/pts  devpts  gid=5,mode=620  0 0
none       /dev/shm  tmpfs   defaults        0 0

[[e-mail] ~]# mount
/dev/xvda1 on / type ext4 (rw,noatime,nodiratime,commit=60)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
none on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
/dev/xvdb on /builds/slave type ext4 (rw,noatime)
/builds/slave/ccache on /builds/ccache type none (rw,bind,noatime)
/builds/slave/mock_mozilla on /builds/mock_mozilla type none (rw,bind,noatime)


A non-jacuzzi slave looks exactly the same (because instance storage is larger than the root volume - 10G)

[[e-mail] ~]# cat /etc/fstab 
LABEL=root_dev   /         ext4   defaults,noatime,nodiratime,commit=60        1 1
none       /proc     proc    defaults        0 0
none       /sys      sysfs   defaults        0 0
none       /dev/pts  devpts  gid=5,mode=620  0 0
none       /dev/shm  tmpfs   defaults        0 0
[[e-mail] ~]# mount
/dev/xvda1 on / type ext4 (rw,noatime,nodiratime,commit=60)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
none on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
/dev/xvdb on /builds/slave type ext4 (rw,noatime)
/builds/slave/ccache on /builds/ccache type none (rw,bind,noatime)
/builds/slave/mock_mozilla on /builds/mock_mozilla type none (rw,bind,noatime)

Comment 11

[Rail Aliiev [:rail]]

Comment on attachment 8453263 [details] [diff] [review]
df-puppet-1.diff

remote:   https://hg.mozilla.org/build/puppet/rev/7075a41dacba
remote:   https://hg.mozilla.org/build/puppet/rev/4648272030c8

Transplanted to production to make it in in the nightly AMI.

Comment 12

[Rail Aliiev [:rail]]

Yay, worked as expected! Now we can shrink EBS!