548551 - remove scratchbox from our slaves

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P1)

Description

[[e-mail] (:bhearsum)]

Nick recently noticed that the bindmount for the Scratchbox .ssh directory was getting remounted a bunch of times. I looked into this a bit and found that every time Puppet runs, it is remounted, with Puppet saying this in the log:
Feb 25 05:47:18 mv-moz2-linux-ix-slave05 puppetd[2625]: (//Node[mv-moz2-linux-ix-slave05.build.mozilla.org]/base/centos5/Mount[/scratchbox/users/cltbld/home/cltbld/.ssh]/ensure) ensure changed 'present' to 'mounted'
Feb 25 05:47:18 mv-moz2-linux-ix-slave05 puppetd[2625]: (//Node[mv-moz2-linux-ix-slave05.build.mozilla.org]/base/centos5/Mount[/scratchbox/users/cltbld/home/cltbld/.ssh]) Refreshing self



While this doesn't seem to cause any harm, it isn't ideal and should be fixed. I think this issue stems from the fact that we have both a "File" and a "Mount" named "/scratchbox/users/cltbld/home/cltbld/.ssh", but it's hard to be sure.
http://hg.mozilla.org/build/puppet-manifests/file/279ba6cded01/os/centos.pp#l52
http://hg.mozilla.org/build/puppet-manifests/file/279ba6cded01/os/centos.pp#l79

Comment 2

[Aki Sasaki (not active)]

Maybe the File should be .../.ssh/known_hosts or something?

Comment 3

[Aki Sasaki (not active)]

Maemo bug.

Comment 4

[Aki Sasaki (not active)]

Morphing.

Comment 5

[Aki Sasaki (not active)]

We might want to doublecheck

interpreter = ['/scratchbox/moz_scratchbox', '-d', '/builds/slave/nanojit-arm']

type calls in our -arm projects; that would suggest a dependency.

Comment 6

[Chris Cooper [:coop] (he/him)]

This is pretty important cleanup work IMO. Anyone want to take it on?

Comment 7

[Mike Taylor [:bear]]

Is this the rm -rf of scratchbox on the slave drives?

Comment 8

[Aki Sasaki (not active)]

Yes, but we should do it cleanly, which involves removing bindmounts and such.
An rpm -e would do it once the rpm is fixed to uninstall cleanly.

Can we wait until Lukas comments on comment 5 ?

Comment 9

[Aki Sasaki (not active)]

(In reply to Aki Sasaki [:aki] from comment #8)
> Yes, but we should do it cleanly, which involves removing bindmounts and
> such.
> An rpm -e would do it once the rpm is fixed to uninstall cleanly.
> 
> Can we wait until Lukas comments on comment 5 ?

Oops, this was Catlee!!!
http://hg.mozilla.org/build/buildbotcustom/annotate/6d0cf1ef4ad6/misc.py#l3203
Also Coop fixed the nanojit one: http://hg.mozilla.org/build/buildbotcustom/annotate/6d0cf1ef4ad6/misc.py#l3133

Coop, Catlee, do you guys know what it will take to get rid of scratchbox here?
Is it negotiating with the spidermonkey + nanojit folk?

Comment 10

[Chris AtLee [:catlee]]

(In reply to Aki Sasaki [:aki] from comment #9)
> (In reply to Aki Sasaki [:aki] from comment #8)
> > Yes, but we should do it cleanly, which involves removing bindmounts and
> > such.
> > An rpm -e would do it once the rpm is fixed to uninstall cleanly.
> > 
> > Can we wait until Lukas comments on comment 5 ?
> 
> Oops, this was Catlee!!!
> http://hg.mozilla.org/build/buildbotcustom/annotate/6d0cf1ef4ad6/misc.
> py#l3203
> Also Coop fixed the nanojit one:
> http://hg.mozilla.org/build/buildbotcustom/annotate/6d0cf1ef4ad6/misc.
> py#l3133
> 
> Coop, Catlee, do you guys know what it will take to get rid of scratchbox
> here?
> Is it negotiating with the spidermonkey + nanojit folk?

Yup, we need to know if they can go without the qemu arm support

Comment 11

[Aki Sasaki (not active)]

Recap: we're clear!
Nanojit and SpiderMonkey don't need scratchbox.

https://bugzilla.mozilla.org/show_bug.cgi?id=638349#c14
https://bugzilla.mozilla.org/show_bug.cgi?id=638349#c15
https://bugzilla.mozilla.org/show_bug.cgi?id=609413#c33

We probably need another bug to track turning off nanojit and spidermonkey scratchbox jobs; otherwise remove away!

Comment 12

[Chris AtLee [:catlee]]

Attachment: Turn off arm nanojit jobs

We don't actually run arm spidermonkey, so this is the only thing we need.

Comment 13

[Aki Sasaki (not active)]

This landed in this morning's reconfig.

Comment 15

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to Aki Sasaki [:aki] (back nov 28) from comment #13)
> This landed in this morning's reconfig.

What left to do here?

Comment 16

[Rail Aliiev [:rail]]

/me grabs the bug

Comment 17

[Rail Aliiev [:rail]]

Attachment: remove scratchbox

Comments incoming

Comment 18

[Rail Aliiev [:rail]]

Comment on attachment 576568 [details] [diff] [review]
remove scratchbox

Review of attachment 576568 [details] [diff] [review]:
-----------------------------------------------------------------

I tested the patch on moz2-linux-slave51 several times switching clean manifests (to reinstall scratchbox) and my manifests to tests if they remove the package cleanly.

I used the following tests:
* grep scratchbox /etc/fstab, no output expected
* mount | grep scratchbox, no output expected
* /scratchbox symlink and /build/scratchbox directory shouldn't exist
* ~cltbld/.ssh should be in place
* "rpm -qa|grep scratchbox |wc -l" should retirn 0

::: modules/packages/files/scratchbox-preun.sh
@@ +14,5 @@
> +    umount /home/cltbld/.ssh
> +    sleep 2
> +done
> +rm -rf /builds/scratchbox
> +rm -rf /scratchbox

In ideal world the last 2 command should be run after removing the package. However, since it requires additional "target" and dependency and even if we run these commands before we remove the package it doesn't break anything, I decided to simplify the patch and put them here.

Comment 19

[John Ford [:jhford] CET/CEST Berlin Time]

Comment on attachment 576568 [details] [diff] [review]
remove scratchbox

Review of attachment 576568 [details] [diff] [review]:
-----------------------------------------------------------------

Cool.  looks good to me

::: modules/packages/files/scratchbox-preun.sh
@@ +14,5 @@
> +    umount /home/cltbld/.ssh
> +    sleep 2
> +done
> +rm -rf /builds/scratchbox
> +rm -rf /scratchbox

sgtm

Comment 20

[Rail Aliiev [:rail]]

To be deployed tomorrow. Thanks, jhford!

Comment 21

[Rail Aliiev [:rail]]

Comment on attachment 576568 [details] [diff] [review]
remove scratchbox

http://hg.mozilla.org/build/puppet-manifests/rev/1b7919bfec4e

Comment 22

[Rail Aliiev [:rail]]

I checked the slaves today and forced update for the following (dev/pp) slaves:

linux-ix-slave03
linux-ix-slave04
moz2-linux-slave04
moz2-linux-slave10
moz2-linux-slave51
mv-moz2-linux-ix-slave01

staging-puppet symlinked to manifests-CLEAN now.

Die, scratchbox, die!!!