822750 - Moving fuzzing uploads from pvtbuilds2.dmz.scl3 to the product delivery cluster

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

B2G uploads are now going to the "normal" product delivery upload endpoints, but winding up on the pvtbuilds share.  We should do the same with fuzzing uploads, just to keep everything consistent.  The effect will be similar (files will end up in the same place).

If possible, it'd be great if we could delete pvtbuilds2.dmz.scl3 once this is done.  But if there's a need for a system for humans to login to and move stuff around in the fuzzing directories, then we should keep it.

Comment 1

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

> If possible, it'd be great if we could delete pvtbuilds2.dmz.scl3 once this
> is done.  But if there's a need for a system for humans to login to and move
> stuff around in the fuzzing directories, then we should keep it.

We do login to look at stuff AFAIK.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

I don't think that's a big deal, we'd show you where to do that on the new host. Just need to make sure we bring over any group memberships or permissions properly.

Comment 3

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

(In reply to Nick Thomas [:nthomas] from comment #2)
> I don't think that's a big deal, we'd show you where to do that on the new
> host. Just need to make sure we bring over any group memberships or
> permissions properly.

That should work for me, and possibly Jesse too. :)

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

It's a pretty minor point, but: if the actions are uploads/downloads and basic filesystem stuff (rename, delete, etc.), then doing it on upload1 is fine.  Beyond that, I think it's better to have a dedicated host -- it'd be sad to close the trees with a shell script run amok.

Comment 5

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

(In reply to Dustin J. Mitchell [:dustin] from comment #4)
> It's a pretty minor point, but: if the actions are uploads/downloads and
> basic filesystem stuff (rename, delete, etc.), then doing it on upload1 is
> fine.  Beyond that, I think it's better to have a dedicated host -- it'd be
> sad to close the trees with a shell script run amok.

Personally I only login to zcat the gzipped logs. The logs for interesting testcases are also emailed to us as they are produced. I'm not sure what permissions the script needs to keep running as it is now.

Comment 6

[Jesse Ruderman]

fuzzing/bot.py does things like "scp" (for both uploads and downloads), "ssh mv", and "ssh ls".

I occasionally use "rsync -e ssh" to copy the top-level directories to my laptop.  The total size is multiple GB.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

For production, I think we just need to swap 
 http://mxr.mozilla.org/build/source/buildbot-configs/mozilla/production_config.py#172
to stage.mozilla.org. There are already jruderman and gkwong home directories there, but please check you can log in. Also that the permissions in /mnt/pvt_builds/fuzzing are to your liking.

For staging, we should probably swap
 http://mxr.mozilla.org/build/source/buildbot-configs/mozilla/staging_config.py#133
to dev-stage01, but would need a puppet patch to make sure the directories exist there (http://hg.mozilla.org/build/puppet-manifests/file/default/stage/stagelayout.pp)

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [buildbot-configs] Switch to stage.m.o

Both pvtbuilds2.dmz.scl3 and stage.m.o mount the same partition at the same location (/mnt/pvt_builds)

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [buildbot-configs] Switch to stage.m.o (staging + prod)

Ignore the earlier comments about dev-stage01, our fuzzing staging is a different dir on the same mount.

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 750850 [details] [diff] [review]
[buildbot-configs] Switch to stage.m.o (staging + prod)

https://hg.mozilla.org/build/buildbot-configs/rev/f9643d55f2b3

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

Initial results are good, eg w64-ix-slave09 went from failing to talk to pvtbuilds2 to having a green run. Will monitor further before closing.

Comment 14

[Armen [:armenzg]]

Live in production.

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

Ok, this is looking good. Jesse, gkw - please check that you can get what you want using stage.mozilla.org (same username and ssh key you're using now). 

Dustin, there's still some pvtbuilds2 usage
  http://mxr.mozilla.org/build/search?string=pvtbuilds2
eg b2g uploads in mozharness, log uploads in postrun.cfg.erb. Lets do that on another bug.

Comment 16

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

(In reply to Nick Thomas [:nthomas] from comment #15)
> Ok, this is looking good. Jesse, gkw - please check that you can get what
> you want using stage.mozilla.org (same username and ssh key you're using
> now). 

Nope, I'm not able to ssh to stage.mozilla.com, but I'm still able to ssh to pvtbuilds2.dmz.scl3.mozilla.com to get logs as per usual.

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

Please try stage.mozilla.org (NB .org).

Comment 18

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

My bad, stage.mozilla.org works fine. However, which directories are they located in? There's nothing in my home folder..

Comment 19

[Nick Thomas [:nthomas] (UTC+12)]

Check the context of the patch.

Comment 20

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

(In reply to Nick Thomas [:nthomas] from comment #19)
> Check the context of the patch.

This is cryptic, and I don't get it.

I don't seem to see anything relevant in /home/ffxbld either. I guess I'll ping you on IRC.

Comment 21

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Found our logs. This is working.