1109050 - Setup new Ubuntu 14.10 nodes for Mozmill CI in qa.scl3.mozilla.com

Status: RESOLVED FIXED

(Infrastructure & Operations :: Virtualization, task)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Ubuntu 14.10 has been released a little while ago, and we want to upgrade our production machines to make use of 14.10. Can we please get new Ubuntu 14.04 VM templates generated? The process should be similar to bug 999435, means lets create the templates first, which I will configure, and then deploy all VMs.

Comment 1

[Chris Knowles [:cknowles]]

Just want to verify - you want 14.10 templates created, *not* 14.04 as you stated?

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Machines we would need to be created are:

production:
mm-ub-1410-32-1.qa.scl3.mozilla.com
mm-ub-1410-32-2.qa.scl3.mozilla.com
mm-ub-1410-32-3.qa.scl3.mozilla.com
mm-ub-1410-32-4.qa.scl3.mozilla.com
mm-ub-1410-64-1.qa.scl3.mozilla.com
mm-ub-1410-64-2.qa.scl3.mozilla.com
mm-ub-1410-64-3.qa.scl3.mozilla.com
mm-ub-1410-64-4.qa.scl3.mozilla.com

staging:
Machines we would need to be created are:

mm-ub-1410-32.qa.scl3.mozilla.com
mm-ub-1410-64.qa.scl3.mozilla.com

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Sorry Chris, this is indeed 14.10 and not 14.04. I missed to update this part when I cloned the bug.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Oh, and for the naming the templates might become:

Linux	Ubuntu 14.10 (desktop)	x86	Ubuntu_14.10_x86
Linux	Ubuntu 14.10 (desktop)	x86-64	Ubuntu_14.10_x86-64

Please create them from fresh and not by updating the existent 14.04 templates. Thanks!

Comment 5

[Chris Knowles [:cknowles]]

Alright, the VM's for templating have been created.

ubuntu-1410-32-template.qa.scl3.mozilla.com and ubuntu-1410-64-template.qa.scl3.mozilla.com - with standard user/password for these templates.

As per last time, I've done the following:
1) set apt-proxy to the dc proxies - per your docs
2) installed open-vm-tools-desktop for proper vm support
3) added standard IT keys to the mozauto ssh account.  
4) installed and enabled ssh server on there - so you can SSH to them as mozauto, with the password we discussed on IRC earlier.

Let me know of any concerns, and when we can templatify and deploy the real boxes.

Saw your name desires for the templates - when we go to templates, I'll change the names.  And these are fresh installs.

Comment 6

[Chris Knowles [:cknowles]]

Any word on these?  Concerns?  Looks good?  Totally-haven't-had-a-chance-to-look-yet?

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I was sick the whole last week and simply haven't had any time yet. Sorry. I will try to get to it the next days.

Comment 8

[Chris Knowles [:cknowles]]

Goodness, no rush from my side.  Be well.  Do note that due to the holidays and associated PTO, things might be spotty until January.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Chris Knowles [:cknowles] from comment #5)
> ubuntu-1410-32-template.qa.scl3.mozilla.com and
> ubuntu-1410-64-template.qa.scl3.mozilla.com - with standard user/password
> for these templates.

Chris, I have problems with connecting to these machines. Have those been turned off in the meantime?

Comment 10

[Chris Knowles [:cknowles]]

I see that ubuntu-1410-32-template.qa.scl3.mozilla.com was misnamed by me, and that certainly would have thrown a wrench in there for you.  This has been corrected.

ubuntu-1410-64-template.qa.scl3.mozilla.com seems to be working fine, can you confirm?

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Chris, all is working fine and I completed the initial setup of those machines. It means we are good in getting them converted to templates, and the requested 10 machines deployed. Thanks.

Comment 12

[Chris Knowles [:cknowles]]

Alright, I'll take them down shortly, and template-ify them, and then we can get started on the deployments.

Comment 13

[Chris Knowles [:cknowles]]

Alright, they've been templated.  Also, the 10 VMs have been created.

If you have any concerns, let me know.

Comment 14

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I checked the machines today and noticed that something is wrong with the auto-login on those boxes. Here the /var/logs/auth.log output with the appropriate failure:

Dec 22 02:34:48 localhost lightdm: PAM unable to dlopen(pam_kwallet.so): /lib/security/pam_kwallet.so: cannot open shared object file: No such file or directory
Dec 22 02:34:48 localhost lightdm: PAM adding faulty module: pam_kwallet.so
Dec 22 02:34:48 localhost lightdm: pam_unix(lightdm-greeter:session): session opened for user lightdm by (uid=0)
Dec 22 02:34:48 localhost kernel: [    4.542482] systemd-logind[672]: Failed to start unit [e-mail]: Unknown unit: [e-mail]
Dec 22 02:34:48 localhost kernel: [    4.542489] systemd-logind[672]: Failed to start user service: Unknown unit: [e-mail]
Dec 22 02:34:48 localhost kernel: [    4.545498] systemd-logind[672]: New session c1 of user lightdm.
Dec 22 02:34:48 localhost kernel: [    4.545516] systemd-logind[672]: Linked /tmp/.X11-unix/X0 to /run/user/112/X11-display.
Dec 22 02:34:48 localhost lightdm: PAM unable to dlopen(pam_kwallet.so): /lib/security/pam_kwallet.so: cannot open shared object file: No such file or directory
Dec 22 02:34:48 localhost lightdm: PAM adding faulty module: pam_kwallet.so
Dec 22 02:34:48 localhost lightdm: pam_succeed_if(lightdm:auth): requirement "user ingroup nopasswdlogin" not met by user "mozauto"
Dec 22 02:34:48 localhost dbus[410]: [system] Rejected send message, 7 matched rules; type="method_return", sender=":1.25" (uid=0 pid=1358 comm="/usr/sbin/dnsmasq --no-resolv --keep-in-foreground") interface="(unset)" member="(unset)" error name="(unset)" requested_reply="0" destination=":1.7" (uid=0 pid=652 comm="NetworkManager ")

I will check what's causing this, but we have to update all the VMs and templates afterward.

Comment 15

[Henrik Skupin [:whimboo][⌚️UTC+1]]

The problem was with:

$ cat .xsession-errors
Invalid MIT-MAGIC-COOKIE-1 keyxrdb: Can't open display ':0'
Invalid MIT-MAGIC-COOKIE-1 keyxhost:  unable to open display ":0"
Invalid MIT-MAGIC-COOKIE-1 keyInvalid MIT-MAGIC-COOKIE-1 keyxhost:  unable to open display ":0"

Deleting the .Xauthority file from the home folder of mozauto fixes the autologin problem.

Further we also have to adjust the owner of ~/.dbus to mozauto.mozauto:

> sudo chown mozauto.mozauto .dbus

I will do those two things for all the new VMs. Chris, can you fix that in the templates?

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Chris, also the /etc/environment settings for no_proxy have a problem on the 64bit machines. You have set those, and two of the host definitions have a leading / instead of a .

Comment 17

[Henrik Skupin [:whimboo][⌚️UTC+1]]

For reference here the machine names and IP addresses:

mm-ub-1410-32.qa.scl3.mozilla.com	10.22.73.179	Slave	Ubuntu_14.10_x86
mm-ub-1410-64.qa.scl3.mozilla.com	10.22.73.180	Slave	Ubuntu_14.10_x86-64

mm-ub-1410-32-1.qa.scl3.mozilla.com	10.22.73.138	Slave	Ubuntu_14.10_x86
mm-ub-1410-32-2.qa.scl3.mozilla.com	10.22.73.158	Slave	Ubuntu_14.10_x86
mm-ub-1410-32-3.qa.scl3.mozilla.com	10.22.73.177	Slave	Ubuntu_14.10_x86
mm-ub-1410-32-4.qa.scl3.mozilla.com	10.22.73.178	Slave	Ubuntu_14.10_x86
mm-ub-1410-64-1.qa.scl3.mozilla.com	10.22.73.181	Slave	Ubuntu_14.10_x86_64
mm-ub-1410-64-2.qa.scl3.mozilla.com	10.22.73.182	Slave	Ubuntu_14.10_x86_64
mm-ub-1410-64-3.qa.scl3.mozilla.com	10.22.73.183	Slave	Ubuntu_14.10_x86_64
mm-ub-1410-64-4.qa.scl3.mozilla.com	10.22.73.184	Slave	Ubuntu_14.10_x86_64

Comment 18

[Chris Knowles [:cknowles]]

Correct my mistypes on the 64 bit ones.

Intrigued that the templates boot up and logged in automagically just fine... even with the .dbus being root owned.  

I have deleted .Xauthority, and chown'd .dbus on both of them.

Please take a look and verify that things are working on those as you like...

ubuntu-1410-32-template.qa.scl3.mozilla.com
ubuntu-1410-64-template.qa.scl3.mozilla.com

Once you give the OK, I'll re-template.

Comment 19

[Henrik Skupin [:whimboo][⌚️UTC+1]]

All looks fine. You can move them back to templates. There is no need to re-create the VMs given that I have already updated them all according to the above steps.

There is still one thing I wonder, why do we have .dbus as owned by root? Did you log in with root at some point? This might also be the cause why .Xauthority is broken, and why we have this file:

-rw-------  1 root    root    1117 Dec 22 03:51 .viminfo

I deleted that file in the template. If possible we should avoid logging into those machines with root.

Comment 20

[Chris Knowles [:cknowles]]

only thing I've done is sudo various commands.  such as "vi /etc/environment" which I needed to do as root in order to edit that file. 

root login isn't enabled on these... you'd stated you wanted a "freshly installed" user experience... so I haven't enabled root, and have only ever logged in as the mozauto user.  

Do you need to look into anything else?  or are we really good to move these back to templates?

Comment 21

[Chris Knowles [:cknowles]]

Per IRC conversation, those are fixed and moved back to templates.  Sorry for the confusion on the /etc/environment file.  And still not sure what happened to cause those root owned files.