1056306 - Please wipe sccache1.srv.scl3.mozilla.com

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[George Miroshnykov ( :gmiroshnykov )]

Please wipe sccache1.srv.scl3.mozilla.com from bug 1011629, rename it to proxxy.srv.scl3.mozilla.com and install Ubuntu 14.04 x64.

Comment 1

[Amy Rich [:arr] [:arich]]

Jake, could you please help George out with this? afaik, he'll be working on all the puppet bits, so no need to worry about that. We just need to give him a vanilla 14.04 system.

Comment 2

[Amy Rich [:arr] [:arich]]

For clarity, this should be in the releng BU. It's currently named sccache1.srv.releng.scl3.mozilla.com and should be renamed to proxxy1.srv.releng.scl3.mozilla.com (all servers have a numeric index so we can later put them behind some sort of load balancing if need be). George, does that work for you?

Comment 3

[George Miroshnykov ( :gmiroshnykov )]

Sure, sounds good.

Comment 4

[Jake Watkins [:dividehex]]

Node def have been added.
remote:   https://hg.mozilla.org/build/puppet/rev/324399a21e2b
remote:   https://hg.mozilla.org/build/puppet/rev/3013b77e1a78

Inventory update including sreg and cname

Comment 5

[Jake Watkins [:dividehex]]

System has been kickstarted but puppetizing to toplevel::server failed because of bug1006891.  As soon as that is resolved, it should finish its puppet run without issue.

Comment 6

[Amy Rich [:arr] [:arich]]

Are we landing that patch today? If not, can we disable puppet on this host (it's sending out floods of email because puppet is failing)

Comment 7

[George Miroshnykov ( :gmiroshnykov )]

All we need for now is a bare Ubuntu, so disabling Puppet should be fine.

TBH I'm new here, so I'm not sure how it's supposed to work - maybe Puppet provisions some SSH keys or something?
I really don't know how I can access that machine as well :)

Comment 8

[Amy Rich [:arr] [:arich]]

Sorry, George, that was directed at Jake. Yes, puppet does all of the initial configuration with the toplevel::server module, including updating passwords, accounts, etc.

Comment 9

[Jake Watkins [:dividehex]]

Attachment: bug1056306.patch

patch adds gmiroshnykov ssh key to root on prozzy1.srv.releng.scl3.m.c

Comment 10

[Justin Wood (:Callek)]

Comment on attachment 8476811 [details] [diff] [review]
bug1056306.patch

code-r+

additional r? to a manager for procedural r+, since I don't know/understand the history of why the need for access, or what if anything can go wrong with the access.

Comment 11

[Jake Watkins [:dividehex]]

Comment on attachment 8476811 [details] [diff] [review]
bug1056306.patch

Already got r+ on permission from Catlee. (and coop now)  Just needed your technical r?.  And I got it. thanks!

Comment 12

[Jake Watkins [:dividehex]]

Comment on attachment 8476811 [details] [diff] [review]
bug1056306.patch

remote:   https://hg.mozilla.org/build/puppet/rev/86c37910783d
remote:   https://hg.mozilla.org/build/puppet/rev/83629716bd4d

Comment 13

[Jake Watkins [:dividehex]]

(In reply to George Miroshnykov ( :gmiroshnykov ) from comment #7)
> All we need for now is a bare Ubuntu, so disabling Puppet should be fine.
> 
> TBH I'm new here, so I'm not sure how it's supposed to work - maybe Puppet
> provisions some SSH keys or something?
> I really don't know how I can access that machine as well :)

Hi George,
This server has been wiped and kickstarted with Ubuntu 14.04.  It is running puppet as a cron task so if you feel the need to disable puppet while hacking, just delete the puppetagain cron task.  You can always re-run puppet with 'puppet agent --test' which will rebuild the periodic puppet cron task.  When deploying a server within releng, it is a typical practice to make sure the node is under puppet control in order to disseminate org wide changes.

You can view the entire releng puppet repo @ http://hg.mozilla.org/build/puppet/

It will give you a good idea of what is being customized on the server.  In this case, proxxy1.srv.releng is part of the toplevel::server class which includes many basic server configurations to fit the server within releng. (eg. ssh key, repos, base packages).  Releng puppet is also well documented if you wish to make implement your changes to be approved and production ready.

Docs can be found here: https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain

Any software that is expected to be put in production should be written into the releng puppet repo.  While Proof of concept systems really don't need this while testing and hacking.

As for access, I've added your ssh-key to the root authorized keys and according ldap, you have vpn access to the releng network.  Simply bring up your vpn connection and ssh to proxxy1.srv.releng.scl3.mozilla.com

Hope that helps. If you have questions please feel free to ping me on IRC or open a bug.
-- Jake

Comment 14

[George Miroshnykov ( :gmiroshnykov )]

Jake, thanks a lot for such a detailed response!
Just like you said, I was able to SSH into that box using my SSH key via VPN.
Thanks again!