1008880 - Create module for PuppetAgain to install Java 7

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Puppet, task)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

On all of our Mozmill CI nodes we need Java JRE to be installed. We need the necessary modules for PuppetAgain to run a necessary installation during startup.

Necessary instructions can be found at:
https://mana.mozilla.org/wiki/display/websites/QA+Automation+ESX+Service

Comment 1

[Andreea Matei [:AndreeaMatei]]

I'm working on this module.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

You might wanna check existing modules and packages first.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Dustin, currently we are installing Java that way:
http://www.webupd8.org/2012/01/install-oracle-java-jdk-7-in-ubuntu-via.html

Looks like we have to cache the ppa:webupd8team/java repository? That's at least for Ubuntu. Or would it be better to simply host the installers as can be found here: http://java.com/en/download/linux_manual.jsp?locale=en

For OS X we are downloading the installer from http://java.com/en/download/mac_download.jsp

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Mirroring the PPA is probably the best solution.

And best to use debmirror, like we do for other repos:
  https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Packages

If you mirror that on your puppetmaster (at /data/repos/apt/webupd8) and get things working the way you'd like, I can rsync it over to the moco distinguished master and we can keep it up to date from there.

Similarly for OS X -- put the DMGs in /data/repos/DMGs/, let me know about them when you're happy, and I'll sync them upstream.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Thanks for the info Dustin. One more question I have is if we should really use the official Oracle version of Java or if we use Open JRE on Ubuntu. But not sure if other Linux distributions also ship this by default. Also it may be better to rely on the same software across all platforms.

Comment 6

[Andreea Matei [:AndreeaMatei]]

We decided that I'll work on the flash module since I'm also duty this week and is less blocking for the goal.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I'm currently trying the 'icedtea-netx' package on our slave nodes. This package contains the javaws command and used the 'openjdk-7-jre'. If that works I would like to keep it, and not install the Oracle version of Java.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Looks like it works nicely so far. I will give it a couple of days time for testing.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

OpenJDK works on Ubuntu, but I think we should not take it but better base on a tool which is available across all platforms. So lets use Oracle's version of Java.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+1]]

So I have uploaded the DMG and EXE file to /data/repos. But when I try to install the package it fails. Reason is that we are trying to use the proxy:

Debug: Executing '/usr/bin/curl -o /var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/d20140627-78104-1fzhkr9/java-7.0.600.dmg.dmg -C - -k -L -s --fail --url http://puppet/repos/DMGs/java-7.0.600.dmg --proxy proxy.dmz.scl3.mozilla.com:3128'
Debug: curl -o /var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/d20140627-78104-1fzhkr9/java-7.0.600.dmg.dmg -C - -k -L -s --fail --url http://puppet/repos/DMGs/java-7.0.600.dmg --proxy proxy.dmz.scl3.mozilla.com:3128 did not transfer [java-7.0.600.dmg].  Falling back to local file.
Error: Could not set 'present' on ensure: No such file or directory - http://puppet/repos/DMGs/java-7.0.600.dmg at 30:/etc/puppet/environments/hskupin/env/modules/packages/manifests/pkgdmg.pp

Looks like puppet doesn't take the no_proxy setting into account:
https://ask.puppetlabs.com/question/3270/need-to-set-no_proxy-for-windows-agent/

It may be that we temporarily need an ACL so a request to puppetmaster1/puppet is served directly by squid.

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Dustin, I assume that we cannot remove the proxy settings from the puppetmaster and block it from reaching the network, or? In such a case we would have to find out where to include the webproxy module.

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Henrik Skupin (:whimboo) from comment #11)
> Dustin, I assume that we cannot remove the proxy settings from the
> puppetmaster and block it from reaching the network, or? In such a case we
> would have to find out where to include the webproxy module.

I filed bug 1032133 for the proxy issue.

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Ok, so here the updated information regarding the Java packages on Ubuntu... The referenced URL from the Webupd8 team only contains the installer. That tool downloads and installs Java. It's a nice wrapper but mirroring that necessary repository, doesn't make that much sense given that the installer file would also have to be mirrored. I feel that's a bit too much handwork to be done here.

So I would propose that in case of Ubuntu we make use of icedtea-netx and openJDK. Possibly it should also work for Cent OS. Dustin, what do you think of that?

Comment 14

[Dustin J. Mitchell [:dustin] (he/him)]

It's really quite out of my league.  Whatever suits your needs, and works without relying on external sites, works for me.

Comment 15

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Install Java v1

WIP patch just for sharing implementation details with Andreea.

Comment 16

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Dustin, how is the uninstallation packages managed? Is there such a way in puppet?

For testing the installation I called this command to remove Java:
sudo rm -fr /Library/Internet\ Plug-Ins/JavaAppletPlugin.plugin

But now puppet agent does no longer install it, even when I run it against production first. So somewhere it keeps traces for the installation happened.

Comment 17

[Dustin J. Mitchell [:dustin] (he/him)]

OS X doesn't support uninstallation of packages.

Puppet keeps track of what it has installed with some semaphore files:
  https://github.com/puppetlabs/puppet/blob/master/lib/puppet/provider/package/pkgdmg.rb#L64

so removing that file should allow you to re-run the install.

Comment 18

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Ah, that worked. Ok, re-tested and the installation works fine now. Something we also have to do is to update the system settings on OS X, so Java is not automatically checking for updates. This will raise a notification, which can conflict with Mozmill. Further we don't want updates until we have a new version managed by puppet.

Looks like this has to be done here:
~/Library/Preferences/com.oracle.javadeployment.plist

Dustin, should this check go into the java package, or to disableservices?

Comment 19

[Henrik Skupin [:whimboo][⌚️UTC+1]]

The appropriate key in this file is:

<key>deployment.macosx.check.update</key>
<string>false</string>

Comment 20

[Dustin J. Mitchell [:dustin] (he/him)]

I think that will be best done in the package manifest, since disableservices would first need to figure out whether Java is installed or not.  Also, I think that's part of installing the package -- ensure that it stays installed :)

Comment 21

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 8448707 [details] [diff] [review]
Install Java v1

This patch works with the proxy turned off for the agent now. I tested it on OS X and Ubuntu.

Something which I wonder is that the installation happens async, so puppet exists while the Java is not installed yet. Is that always the case for installations? 

Dustin, please also sync the repo folder from our puppetmaster, so that the Java DMG is getting synced. Thanks.

Comment 22

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8448707 [details] [diff] [review]
Install Java v1

sync is complete

Comment 23

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Thanks Dustin. So I got this landed on our repo:

https://hg.mozilla.org/qa/puppet/rev/639837b0fa09 (default)
https://hg.mozilla.org/qa/puppet/rev/0994a64bb26f (production)