553337 - puppet installation of hg on talos linux/mac

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Comment 1

[Armen [:armenzg]]

Lukas is there anything involved in this bug beside getting Mercurial installed on these platforms?
* Fedora 12
* Fedora 12 x64
* 10.5.8
* 10.6.2

Alice I will need another set of rev3 machines that don't have what Lukas has installed and see if I can get it installed correctly through puppet.

Could I pull one at a time with carefulness on Monday/Tuesday from production?

From bug 548768:
> I installed hg on talos-r3-snow-001 through vnc from :
> http://mercurial.berkwood.com/binaries/Mercurial-1.5-py2.6-macosx10.6.zip
> so that I didn't have to install anything else (xcode, macports)

> In testing this on talos-staging, I've installed mercurial 1.3.1 on
> talos-r3-fed-001 with:
> yum install mercurial

Comment 2

[Armen [:armenzg]]

Per IRC conversation I will be waiting on Lukas to finish her work on talos staging.

Meanwhile, I have started researching what puppet changes I will need.

Lukas I have been looking around the Fedora and Mac slaves to see which mercurial and python they have. Here are my findings:

fed      ~/bin/python     | 2.5.4 | /usr/bin/hg       | 1.3.1
fed64    ~/bin/python     | 2.5.4 |    no hg          |  N/A
snow     ~/bin/python     | 2.5.4 | /usr/local/bin/hg | 1.3.1
leopard  /usr/bin/python  | 2.5.1 | /usr/local/bin/hg | 1.2.1

(In reply to comment #1)
> From bug 548768:
> > I installed hg on talos-r3-snow-001 through vnc from :
> > http://mercurial.berkwood.com/binaries/Mercurial-1.5-py2.6-macosx10.6.zip
> > so that I didn't have to install anything else (xcode, macports)
> 
Lukas where did Mercurial get installed on this 10.6 machine? is this on talos-r3-snow-001?

Comment 3

[Armen [:armenzg]]

(In reply to comment #2)
From IRC conversation, these are the profile changes for Snow Leopard:
talos-r3-snow-001:~ cltbld$ cat .bash_profile 
 export PATH=/Users/cltbld/bin/:$PATH
 export PYTHONPATH=/usr/local/lib/python2.5/site-packages

> snow     ~/bin/python     | 2.5.4 | /usr/local/bin/hg | 1.3.1
> leopard  /usr/bin/python  | 2.5.1 | /usr/local/bin/hg | 1.2.1
> 
To note that in snow leopard ~/bin/python points to /usr/bin/python-2.5 which is different than on leopard. I will probably have to look into making this consistent in both machines (maybe by upgrading leopard to 2.5.4 and adding another .bash_profile to it).


BTW we have to be real careful with python on snow leopard since we can be using  the wrong one (which and whereis got me confused).

talos-r3-snow-001:~ cltbld$ which python
/Users/cltbld/bin//python
talos-r3-snow-001:~ cltbld$ ls -l bin/python
lrwxr-xr-x  1 root  staff  18 Feb 18 15:11 bin/python -> /usr/bin/python2.5
talos-r3-snow-001:~ cltbld$ whereis python
/usr/bin/python
talos-r3-snow-001:~ cltbld$ /usr/bin/python --version
Python 2.6.1

Comment 4

[Armen [:armenzg]]

Attachment: [WIP] install mercurial for talos machines (darwin{9,10}&fedora{32,64}) and add .bash_profile for darwin slaves

This is work in process and it has not been tested.

Two reasons for feedback:
1) bhearsum how does it look overall? I have created "talos" subdir under "os"
2) lsblakk I know the requirements are not yet ready but I wanted to get this started

Lukas wrt to download a .bash_profile for the darwin machines, could we pass PYTHONPATH and PATH through the environment on buildbot rather than managing through puppet?

Comment 5

[Armen [:armenzg]]

> (In reply to comment #1)
> > From bug 548768:
> > > I installed hg on talos-r3-snow-001 through vnc from :
> > > http://mercurial.berkwood.com/binaries/Mercurial-1.5-py2.6-macosx10.6.zip
> > > so that I didn't have to install anything else (xcode, macports)
> > 
> Lukas where did Mercurial get installed on this 10.6 machine? is this on
> talos-r3-snow-001?

Per conversation on IRC, we are using 1.3.1.

Comment 6

[Armen [:armenzg]]

Putting on the side until I can actually test any patch.

Comment 7

[[e-mail] (:bhearsum)]

Comment on attachment 434645 [details] [diff] [review]
[WIP] install mercurial for talos machines (darwin{9,10}&fedora{32,64}) and add .bash_profile for darwin slaves 

>diff --git a/os/talos/fedora.pp b/os/talos/fedora.pp
>new file mode 100644
>--- /dev/null
>+++ b/os/talos/fedora.pp
>@@ -0,0 +1,9 @@
>+# fedora.pp
>+
>+class fedora {
>+    exec {
>+        "yum install mercurial":
>+             creates => "/usr/bin/hg",
>+             alias => "install-mercurial";
>+    }
>+}

You don't want to use an exec here. Look at the 'package' type, like we use over here: 
http://hg.mozilla.org/build/puppet-manifests/file/0a59953ce8de/os/centos.pp#l177

You probably don't even need to specify a version.

>diff --git a/os/talos/osx.pp b/os/talos/osx.pp
>new file mode 100644
>--- /dev/null
>+++ b/os/talos/osx.pp
>@@ -0,0 +1,46 @@
>+# osx.pp
>+
>+class osx {
>+    $pythonHome = "/usr/bin/python2.5"
>+    $pythonPath = "/usr/local/lib/python2.5/site-packages"
>+    $mercurialVersion = "1.3.1"
>+ 
>+    # 1) create symlink in ~/bin to python2.5
>+    # 2) create ~/bin
>+    # 3) get .bash_profile specifying PATH and PYTHONPATH
>+    # 4) get mercurial
>+    # TODO: It seems that some things are common for darwin9 and darwin10
>+    #       where should we put those files in the puppet server?
>+    file {
>+        "/Users/cltbld/bin/python":
>+            ensure => "${pythonHome}",
>+            alias  => "create_python_symlink";
>+        "/Users/cltbld/bin":
>+            ensure => "directory",
>+            alias  => "create_bin_directory";
>+            before => File["create_python_symlink"];
>+        "/Users/cltbld/.bash_profile":
>+            source => "${fileroot}/darwin/.bash_profile",
>+            alias  => "get_profile";
>+        "/tools/dist/mercurial-${mercurialVersion}.tar.gz":
>+            source => "${fileroot}/darwin/${mercurialVersion}.tar.gz",
>+            alias   => "get_mercurial";
>+    }
>+
>+    exec {
>+        "python setup.py install; rm -rf /tools/dist/mercurial-${mercurialVersion}":
>+            creates     => "${pythonPath}/mercurial",
>+            cwd         => "/tools/dist/mercurial-${mercurialVersion}",
>+            path        => "Users/cltbld/bin",
>+            alias       => "install_mercurial",
>+            environment => ["PYTHONHOME=${pythonHome}",
>+                            "PYTHONPATH=${pythonPath}"],
>+            subscribe   => File["create_python_symlink"];
>+
>+        "tar xfvz mercurial-${mercurialVersion}.tar.gz":
>+            creates   => "/tools/dist/mercurial-${mercurialVersion}",
>+            cwd       => "/tools/dist",
>+            alias     => "untar-mercurial",
>+            subscribe => File["get_mercurial"]; 
>+    }
>+}

You don't need to do 'python setup.py install'. For buildslaves, we have a bunch of things installed in /tools, and we just tar them up. Do the same here:
* Install it manually on one slave
* Tar up that directory
* Deploy that tarball with a single exec, without copying the file to the slave.

You should end up with something like:
exec {
"tar -zvxf mercurial.tar.gz":
  creates => whatever
  cwd => /Users/cltbld/bin;
}

And probably a couple of file {} checks, too

>-    include base, staging-buildslave, sandbox-network
>+node "talos-r3-fed-001.build.mozilla.org" {
>+}
>+
>+node "talos-r3-fed64-001.build.mozilla.org" {
>+}
>+
>+node "talos-r3-snow-001.build.mozilla.org" {
>+}
>+
>+node "talos-r3-leopard-001.build.mozilla.org" {
> }
> 

Unless these slaves are staying in staging permanently, don't do this, except for testing.
>

Comment 8

[Armen [:armenzg]]

We are giving higher priority to Leopard and Snow Leopard and I should be soon be able to test puppet changes.

Comment 9

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: tarball of the snowleopard mercurial egg and mercurial dir

this tarball needs to be unpacked in /usr/local/lib/python2.5/site-packages on snowleopard boxes

Comment 10

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: .bash_profile for snowleopard minis

this file needs to be placed in /Users/cltbld as .bash_profile for all snowleopard boxes

Comment 11

[Lukas Blakk [:lsblakk] use ?needinfo]

Attachment: tarball of the leopard mercurial egg and mercurial dir

this gets unpacked in /Library/Python/2.5/site-packages/ on leopard machines

Comment 12

[Armen [:armenzg]]

I still believe that we could share a same PYTHONPATH by using something like "/tools/site-packages". I don't know why we have to split the puppet changes for Leopard and Snow Leopard.

Comment 13

[Armen [:armenzg]]

Attachment: Install mercurial to Leopard and Snow Leopard; Add .bash_profile for Snow Leopard

This has not been tested yet.

Comment 14

[[e-mail] (:bhearsum)]

Comment on attachment 435151 [details] [diff] [review]
Install mercurial to Leopard and Snow Leopard; Add .bash_profile for Snow Leopard

So, this is mostly good but I'm going to ask a bit more of you here.
First of all, you can't use ${fileroot} in an exec, because it translate to puppet://, which shell tools aren't going to understand.

What you *can* do is have install_devtools download the file with a file {} check, install it with an exec {}, and then remove it afterwards. I'd love to see this happen. Also, wipe out the current contents of classes/utils.pp (which aren't used) and move this function there. It'll make the switch away from the /N share go smoother, when we do it. You might need to rename it to avoid conflicting with the devtools.pp version.

You'll probably need to define $tar somewhere, too.

Other than that, this seems good. I look forward to reviewing this after its tested!

Comment 15

[Armen [:armenzg]]

Attachment: Install mercurial to Leopard and Snow Leopard; Add .bash_profile for Snow Leopard

This has been run on staging-puppet and it is almost exactly what I want. I still have to work on the subscription/require/before combination for "install_tools".

Let me know if this is looking the way you wanted or not.

Notice that I moved os/talos/osx.pp to os/talos_osx.pp.

Comment 16

[Armen [:armenzg]]

Ben I am resuming my testing of the patch to nail it down.

Taking talos-r3-leopard-001 and talos-r3-snow-001.

Comment 17

[Armen [:armenzg]]

Attachment: /usr/local/bing/hg

It seems that I have found another dependency.

I have modified the first line to match the version of python we want:
#!/usr/bin/python2.5
Which looks to be common for Leopard and Snow Leopard.

I so wished we had gone the way of "python setup.py install"

Comment 18

[Armen [:armenzg]]

Attachment: [WIP] puppet changes for Leopard and Snow Leopard

This patch has worked all through on the Leopard machine but I am blocked on testing it for Snow Leopard (I can't mount N).

Comment 19

[Armen [:armenzg]]

Attachment: puppet changes for Leopard and Snow Leopard

This should match to what you were asking me on IRC.
I have tested that it works on Leopard. I am still waiting to test that it works for Snow Leopard.

Comment 20

[[e-mail] (:bhearsum)]

Comment on attachment 435903 [details] [diff] [review]
puppet changes for Leopard and Snow Leopard

This is pretty much good. My only nit is that the mercurial tarballs shouldn't be living in the same directory; please make subdirs for 10.5 and 10.6. We could run into issues down the road if we want them on the same version, but have to use different tarballs per OS.

Other than that, if it tests out OK, it looks good.

Comment 21

[Armen [:armenzg]]

Attachment: puppet changes for Leopard and Snow Leopard

This run properly for both Leopard and Snow Leopard.

Comment 22

[Armen [:armenzg]]

Attachment: enable puppet for talos slaves

Comment 23

[[e-mail] (:bhearsum)]

Comment on attachment 435997 [details] [diff] [review]
puppet changes for Leopard and Snow Leopard

These slaves are moving back to production at some point, so please don't add them here. r=me with that changed.

Comment 24

[Armen [:armenzg]]

Attachment: puppet changes for Leopard and Snow Leopard

As landed (combined).
http://hg.mozilla.org/build/puppet-manifests/rev/1df033a8e9b0

I have been able to update Leopard and Snow Leopards manually with the expected results.
I will check other slaves after they reboot.

Comment 25

[Armen [:armenzg]]

[root@staging-puppet puppet-files]# cvs commit -m "Bug 553337. Add Mercurial to talos darwin slaves. r=bhearsum"
RCS file: /mofo/puppet-files/talos/mac/hg,v
done
Checking in talos/mac/hg;
/mofo/puppet-files/talos/mac/hg,v  <--  hg
initial revision: 1.1
done
RCS file: /mofo/puppet-files/talos/mac/darwin10/.bash_profile,v
done
Checking in talos/mac/darwin10/.bash_profile;
/mofo/puppet-files/talos/mac/darwin10/.bash_profile,v  <--  .bash_profile
initial revision: 1.1
done
RCS file: /mofo/puppet-files/talos/mac/darwin10/mercurial-1.3.1.tar.gz,v
done
Checking in talos/mac/darwin10/mercurial-1.3.1.tar.gz;
/mofo/puppet-files/talos/mac/darwin10/mercurial-1.3.1.tar.gz,v  <--  mercurial-1.3.1.tar.gz
initial revision: 1.1
done
RCS file: /mofo/puppet-files/talos/mac/darwin9/mercurial-1.2.1.tar.gz,v
done
Checking in talos/mac/darwin9/mercurial-1.2.1.tar.gz;
/mofo/puppet-files/talos/mac/darwin9/mercurial-1.2.1.tar.gz,v  <--  mercurial-1.2.1.tar.gz
initial revision: 1.1
done

Comment 26

[Armen [:armenzg]]

Attachment: [WIP] Install Mercurial on talos fedora slaves

Might as well attach the patch for testing next week.

Comment 27

[Armen [:armenzg]]

Attachment: install Mercurial to talos Fedora slaves

This has been tested successfully for both Fedora and Fedorax64 r3 talos slaves.

Comment 28

[[e-mail] (:bhearsum)]

Comment on attachment 437353 [details] [diff] [review]
install Mercurial to talos Fedora slaves

Did you forget to 'hg add' the talos_fedora.pp file? The other bits look fine, but I don't see anything that actually installs Mercurial :).

Comment 29

[Armen [:armenzg]]

Attachment: install Mercurial to talos Fedora slaves

Comment 30

[[e-mail] (:bhearsum)]

Comment on attachment 437357 [details] [diff] [review]
install Mercurial to talos Fedora slaves

>diff --git a/os/talos_fedora.pp b/os/talos_fedora.pp
>new file mode 100644
>--- /dev/null
>+++ b/os/talos_fedora.pp
>@@ -0,0 +1,9 @@
>+# talos_fedora.pp
>+
>+class talos_fedora {
>+    package {
>+        "mercurial":
>+            ensure   => "1.3.1-3.fc12",
>+            provider => yum;
>+    }
>+}

I think this will break at some point, when the yum repository updates to a new mercurial. Can you test the following scenario to check?:
* Uninstall Mercurial from a slave
* Bump the version to something newer, like 1.5.1-1.fc12, which won't exist in the yum repositories
* Sync that slave up with the Puppet master

If it still works, this patch is OK. If it doesn't work, we'll need to use ensure => installed, or keep a copy of the RPM and use provide => rpm.

Comment 31

[Armen [:armenzg]]

Attachment: install Mercurial to talos Fedora slaves

Working on staging for the Fedora (32 & 64 bits) talos slaves.

It took me a longer to test because of /N not being mounted on Fedora slaves (I should file it) and the staging-puppet apache server not running (bug 557835).

Thanks Ben.

Comment 32

[[e-mail] (:bhearsum)]

Comment on attachment 437602 [details] [diff] [review]
install Mercurial to talos Fedora slaves

Looks good to me

Comment 33

[Armen [:armenzg]]

Comment on attachment 437602 [details] [diff] [review]
install Mercurial to talos Fedora slaves

http://hg.mozilla.org/build/puppet-manifests/rev/8dcb160501ae

This has been deployed and tested once more in two Fedora talos slaves that were idle.
I will check in the next hour and see if any Fedora rebooted slave will have Mercurial on it.

# cvs ci -m "Bug 553337. Mercurial RPMs fpr Talos Fedora slaves."
cvs commit: Examining .
cvs commit: Examining RPMS
cvs commit: Examining RPMS/i686
cvs commit: Examining RPMS/x86_64
RCS file: /mofo/puppet-files/talos/linux/RPMS/i686/mercurial-1.5.1-1.fc12.i686.rpm,v
done
Checking in RPMS/i686/mercurial-1.5.1-1.fc12.i686.rpm;
/mofo/puppet-files/talos/linux/RPMS/i686/mercurial-1.5.1-1.fc12.i686.rpm,v  <--  mercurial-1.5.1-1.fc12.i686.rpm
initial revision: 1.1
done
RCS file: /mofo/puppet-files/talos/linux/RPMS/x86_64/mercurial-1.5.1-1.fc12.x86_64.rpm,v
done
Checking in RPMS/x86_64/mercurial-1.5.1-1.fc12.x86_64.rpm;
/mofo/puppet-files/talos/linux/RPMS/x86_64/mercurial-1.5.1-1.fc12.x86_64.rpm,v  <--  mercurial-1.5.1-1.fc12.x86_64.rpm
initial revision: 1.1
done

Comment 34

[Armen [:armenzg]]

I have tested two slaves after reboot and they both had mercurial.

We now have Mercurial on darwin and fedora talos slaves.