1007106 - Auto-connect Jenkins slaves to the master

Status: RESOLVED WONTFIX

(Mozilla QA Graveyard :: Infrastructure, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

All of our nodes have to be updated so that we can auto-connect our slaves to the Jenkins master on start-up. This behavior is necessary so that we can successfully restart the machine after each testrun, and let Puppet update the system during startup. We would also benefit that for sporadic reboots or power outages, all nodes would not have to be manually connected.

The best what we could do here is to setup autostart entries. Solutions for OS X, Linux, and Windows will be different. But all of them have to be implemented as a script, so it can be used by Puppet later.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Here we should be able to use the hostname of the machine, which maps 1:1 with Jenkins. Except for Vista machines which might have a slightly different name. So we might have to update the name in mozmill-ci (https://github.com/mozilla/mozmill-ci/blob/production/config/production/jenkins.patch#L591)

Comment 2

[Andreea Matei [:AndreeaMatei]]

I'll start looking into this for Ubuntu first.

Comment 3

[Andreea Matei [:AndreeaMatei]]

On linux we have startup applications, as we need this script to run after login.
I tested it with:
javaws http://master_address:8080/computer/$(hostname)/slave-agent.jnlp

For OS X I found we have Automator, but I will have to test it tomorrow. I also what to test on a staging machine.
I will focus on Windows in the end as we will start with Linux and OSX first.

Comment 4

[Andreea Matei [:AndreeaMatei]]

Attachment: connect slave staging ubuntu

This is for Ubuntu, on staging. I tested it on ub-1204-32, put this script in Startup applications and I think it worked without the "Run" window check, because after restart it was connected.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Comment on attachment 8423936 [details]
connect slave staging ubuntu

That's fine and seems to be working. But we also need the code for adding it to the autostart. This cannot be done manually.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+2]]

I'm currently investigating this for the upgrade of Jenkins to 1.580.1 via https://github.com/mozilla/mozmill-ci/issues/523. So I will take it.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Ok, so I tried to let the slave auto-connect via an installed Windows service. This can be done by connecting the slave via Java Web Start, and then selecting "File | Install as a service". By that please be aware of https://issues.jenkins-ci.org/browse/JENKINS-25582, which will force you to restart Jenkins master.

From now on the Jenkins slave node will automatically connect during startup to the master. There is no javaws window present anymore. 

If you now let some tests run I see something totally baffling. Tests are getting started correctly, BUT do not run via the gui! Means you can no longer see them running. By that my first thought was: Well now a lot of them will fail due to the focus issue for popup notifications! But I was totally wrong. Everything was working like a charm. No single focus related failure! 

I accidentally let those tests report to daily:
http://mozmill-daily.blargon7.com/#/functional/report/b67428b6e502c230ff85b6a4c42e32c4

So I do not understand that! I always thought that we need a GUI to let our tests run! But that doesn't seem to be the case. Tests can run perfectly in the background without a "visible" UI.

So all this works with Windows 7. I have not tested other Windows versions. Maybe Andreea and Andrei could help me in here? I don't have them available at home, and maybe they have them.

I will also have to test Linux and OS X, but I assume it will be similar.

So what do you think? If we would consider to take this and it keeps all working we could have a couple of benefits:

* No manual connect for the Jenkins slaves after a restart
* If the JLNP connection dies (e.g. crash), it will be automatically re-connected
* We can operate on the machine without forcing the current tests to fail
* Execution times might be faster because no graphical updates

The current downsides would be:

* The tests are run by default as the system user, and e.g. crash reports are not available after the testrun finished. We may have to archive the reports!
* We do not see what is going on - kinda helpful for investigating failures

When configuring a service you can allow to interact with the desktop for the system user. This would show a notification dialog, which let you switch to the system user desktop. As I can see this is not themed, so we might not want to use it. Or you can even specify the user under which the test should be run. I did that for our mozauto user, but sadly I do not see a Firefox instance running visually. It might also be hidden and not sure if the OS theme is applied either.

I think I have to explore this a bit more tomorrow.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Henrik Skupin (:whimboo) from comment #7)
> When configuring a service you can allow to interact with the desktop for
> the system user. This would show a notification dialog, which let you switch
> to the system user desktop. As I can see this is not themed, so we might not

I do not think we want to let the tests run under the local system user account. We would inherit all the permissions from that user, and open up the box in case of malicious web sites under test.

So I will have to check how to be able to see the tests running for our local mozauto user, even it is running on another console/display.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Further thoughts here showed that there are also a couple of issues regarding installing Jenkins as a service on slave nodes. Something we might want to do here for puppet is to use SSH for slave connections. I will continue on that bug when the current upgrade is done.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+2]]

We are moving to taskcluster. So I don't want to spend anytime on that one.