puppet installation of hg on talos linux/mac

RESOLVED FIXED

Status

defect
P2
normal
RESOLVED FIXED
10 years ago
6 years ago

People

(Reporter: anodelman, Assigned: armenzg)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [puppet])

Attachments

(6 attachments, 10 obsolete attachments)

1.31 MB, application/empty
Details
94 bytes, text/plain
Details
890.72 KB, application/empty
Details
536 bytes, text/plain
Details
7.20 KB, patch
armenzg
: checked-in+
Details | Diff | Splinter Review
6.12 KB, patch
bhearsum
: review+
Details | Diff | Splinter Review
No description provided.
Blocks: 548768
Assignee: nobody → armenzg
Status: NEW → ASSIGNED
Priority: -- → P2
Lukas is there anything involved in this bug beside getting Mercurial installed on these platforms?
* Fedora 12
* Fedora 12 x64
* 10.5.8
* 10.6.2

Alice I will need another set of rev3 machines that don't have what Lukas has installed and see if I can get it installed correctly through puppet.

Could I pull one at a time with carefulness on Monday/Tuesday from production?

From bug 548768:
> I installed hg on talos-r3-snow-001 through vnc from :
> http://mercurial.berkwood.com/binaries/Mercurial-1.5-py2.6-macosx10.6.zip
> so that I didn't have to install anything else (xcode, macports)

> In testing this on talos-staging, I've installed mercurial 1.3.1 on
> talos-r3-fed-001 with:
> yum install mercurial
Per IRC conversation I will be waiting on Lukas to finish her work on talos staging.

Meanwhile, I have started researching what puppet changes I will need.

Lukas I have been looking around the Fedora and Mac slaves to see which mercurial and python they have. Here are my findings:

fed      ~/bin/python     | 2.5.4 | /usr/bin/hg       | 1.3.1
fed64    ~/bin/python     | 2.5.4 |    no hg          |  N/A
snow     ~/bin/python     | 2.5.4 | /usr/local/bin/hg | 1.3.1
leopard  /usr/bin/python  | 2.5.1 | /usr/local/bin/hg | 1.2.1

(In reply to comment #1)
> From bug 548768:
> > I installed hg on talos-r3-snow-001 through vnc from :
> > http://mercurial.berkwood.com/binaries/Mercurial-1.5-py2.6-macosx10.6.zip
> > so that I didn't have to install anything else (xcode, macports)
> 
Lukas where did Mercurial get installed on this 10.6 machine? is this on talos-r3-snow-001?
(In reply to comment #2)
From IRC conversation, these are the profile changes for Snow Leopard:
talos-r3-snow-001:~ cltbld$ cat .bash_profile 
 export PATH=/Users/cltbld/bin/:$PATH
 export PYTHONPATH=/usr/local/lib/python2.5/site-packages

> snow     ~/bin/python     | 2.5.4 | /usr/local/bin/hg | 1.3.1
> leopard  /usr/bin/python  | 2.5.1 | /usr/local/bin/hg | 1.2.1
> 
To note that in snow leopard ~/bin/python points to /usr/bin/python-2.5 which is different than on leopard. I will probably have to look into making this consistent in both machines (maybe by upgrading leopard to 2.5.4 and adding another .bash_profile to it).


BTW we have to be real careful with python on snow leopard since we can be using  the wrong one (which and whereis got me confused).

talos-r3-snow-001:~ cltbld$ which python
/Users/cltbld/bin//python
talos-r3-snow-001:~ cltbld$ ls -l bin/python
lrwxr-xr-x  1 root  staff  18 Feb 18 15:11 bin/python -> /usr/bin/python2.5
talos-r3-snow-001:~ cltbld$ whereis python
/usr/bin/python
talos-r3-snow-001:~ cltbld$ /usr/bin/python --version
Python 2.6.1
Whiteboard: [puppet]
This is work in process and it has not been tested.

Two reasons for feedback:
1) bhearsum how does it look overall? I have created "talos" subdir under "os"
2) lsblakk I know the requirements are not yet ready but I wanted to get this started

Lukas wrt to download a .bash_profile for the darwin machines, could we pass PYTHONPATH and PATH through the environment on buildbot rather than managing through puppet?
Attachment #434645 - Flags: feedback?(lsblakk)
Attachment #434645 - Flags: feedback?(bhearsum)
> (In reply to comment #1)
> > From bug 548768:
> > > I installed hg on talos-r3-snow-001 through vnc from :
> > > http://mercurial.berkwood.com/binaries/Mercurial-1.5-py2.6-macosx10.6.zip
> > > so that I didn't have to install anything else (xcode, macports)
> > 
> Lukas where did Mercurial get installed on this 10.6 machine? is this on
> talos-r3-snow-001?

Per conversation on IRC, we are using 1.3.1.
Putting on the side until I can actually test any patch.
Priority: P2 → P3
Comment on attachment 434645 [details] [diff] [review]
[WIP] install mercurial for talos machines (darwin{9,10}&fedora{32,64}) and add .bash_profile for darwin slaves 

>diff --git a/os/talos/fedora.pp b/os/talos/fedora.pp
>new file mode 100644
>--- /dev/null
>+++ b/os/talos/fedora.pp
>@@ -0,0 +1,9 @@
>+# fedora.pp
>+
>+class fedora {
>+    exec {
>+        "yum install mercurial":
>+             creates => "/usr/bin/hg",
>+             alias => "install-mercurial";
>+    }
>+}

You don't want to use an exec here. Look at the 'package' type, like we use over here: 
http://hg.mozilla.org/build/puppet-manifests/file/0a59953ce8de/os/centos.pp#l177

You probably don't even need to specify a version.

>diff --git a/os/talos/osx.pp b/os/talos/osx.pp
>new file mode 100644
>--- /dev/null
>+++ b/os/talos/osx.pp
>@@ -0,0 +1,46 @@
>+# osx.pp
>+
>+class osx {
>+    $pythonHome = "/usr/bin/python2.5"
>+    $pythonPath = "/usr/local/lib/python2.5/site-packages"
>+    $mercurialVersion = "1.3.1"
>+ 
>+    # 1) create symlink in ~/bin to python2.5
>+    # 2) create ~/bin
>+    # 3) get .bash_profile specifying PATH and PYTHONPATH
>+    # 4) get mercurial
>+    # TODO: It seems that some things are common for darwin9 and darwin10
>+    #       where should we put those files in the puppet server?
>+    file {
>+        "/Users/cltbld/bin/python":
>+            ensure => "${pythonHome}",
>+            alias  => "create_python_symlink";
>+        "/Users/cltbld/bin":
>+            ensure => "directory",
>+            alias  => "create_bin_directory";
>+            before => File["create_python_symlink"];
>+        "/Users/cltbld/.bash_profile":
>+            source => "${fileroot}/darwin/.bash_profile",
>+            alias  => "get_profile";
>+        "/tools/dist/mercurial-${mercurialVersion}.tar.gz":
>+            source => "${fileroot}/darwin/${mercurialVersion}.tar.gz",
>+            alias   => "get_mercurial";
>+    }
>+
>+    exec {
>+        "python setup.py install; rm -rf /tools/dist/mercurial-${mercurialVersion}":
>+            creates     => "${pythonPath}/mercurial",
>+            cwd         => "/tools/dist/mercurial-${mercurialVersion}",
>+            path        => "Users/cltbld/bin",
>+            alias       => "install_mercurial",
>+            environment => ["PYTHONHOME=${pythonHome}",
>+                            "PYTHONPATH=${pythonPath}"],
>+            subscribe   => File["create_python_symlink"];
>+
>+        "tar xfvz mercurial-${mercurialVersion}.tar.gz":
>+            creates   => "/tools/dist/mercurial-${mercurialVersion}",
>+            cwd       => "/tools/dist",
>+            alias     => "untar-mercurial",
>+            subscribe => File["get_mercurial"]; 
>+    }
>+}

You don't need to do 'python setup.py install'. For buildslaves, we have a bunch of things installed in /tools, and we just tar them up. Do the same here:
* Install it manually on one slave
* Tar up that directory
* Deploy that tarball with a single exec, without copying the file to the slave.

You should end up with something like:
exec {
"tar -zvxf mercurial.tar.gz":
  creates => whatever
  cwd => /Users/cltbld/bin;
}

And probably a couple of file {} checks, too

>-    include base, staging-buildslave, sandbox-network
>+node "talos-r3-fed-001.build.mozilla.org" {
>+}
>+
>+node "talos-r3-fed64-001.build.mozilla.org" {
>+}
>+
>+node "talos-r3-snow-001.build.mozilla.org" {
>+}
>+
>+node "talos-r3-leopard-001.build.mozilla.org" {
> }
> 

Unless these slaves are staying in staging permanently, don't do this, except for testing.
>
Attachment #434645 - Flags: feedback?(bhearsum) → feedback-
We are giving higher priority to Leopard and Snow Leopard and I should be soon be able to test puppet changes.
Priority: P3 → P2
this tarball needs to be unpacked in /usr/local/lib/python2.5/site-packages on snowleopard boxes
this file needs to be placed in /Users/cltbld as .bash_profile for all snowleopard boxes
this gets unpacked in /Library/Python/2.5/site-packages/ on leopard machines
I still believe that we could share a same PYTHONPATH by using something like "/tools/site-packages". I don't know why we have to split the puppet changes for Leopard and Snow Leopard.
This has not been tested yet.
Attachment #434645 - Attachment is obsolete: true
Attachment #435151 - Flags: feedback?(bhearsum)
Attachment #434645 - Flags: feedback?(lsblakk)
Attachment #434645 - Attachment is obsolete: false
Comment on attachment 435151 [details] [diff] [review]
Install mercurial to Leopard and Snow Leopard; Add .bash_profile for Snow Leopard

So, this is mostly good but I'm going to ask a bit more of you here.
First of all, you can't use ${fileroot} in an exec, because it translate to puppet://, which shell tools aren't going to understand.

What you *can* do is have install_devtools download the file with a file {} check, install it with an exec {}, and then remove it afterwards. I'd love to see this happen. Also, wipe out the current contents of classes/utils.pp (which aren't used) and move this function there. It'll make the switch away from the /N share go smoother, when we do it. You might need to rename it to avoid conflicting with the devtools.pp version.

You'll probably need to define $tar somewhere, too.

Other than that, this seems good. I look forward to reviewing this after its tested!
Attachment #435151 - Flags: feedback?(bhearsum)
This has been run on staging-puppet and it is almost exactly what I want. I still have to work on the subscription/require/before combination for "install_tools".

Let me know if this is looking the way you wanted or not.

Notice that I moved os/talos/osx.pp to os/talos_osx.pp.
Attachment #434645 - Attachment is obsolete: true
Attachment #435151 - Attachment is obsolete: true
Attachment #435269 - Flags: feedback?(bhearsum)
Ben I am resuming my testing of the patch to nail it down.

Taking talos-r3-leopard-001 and talos-r3-snow-001.
Posted file /usr/local/bing/hg
It seems that I have found another dependency.

I have modified the first line to match the version of python we want:
#!/usr/bin/python2.5
Which looks to be common for Leopard and Snow Leopard.

I so wished we had gone the way of "python setup.py install"
Depends on: 555790
This patch has worked all through on the Leopard machine but I am blocked on testing it for Snow Leopard (I can't mount N).
Attachment #435269 - Attachment is obsolete: true
Attachment #435269 - Flags: feedback?(bhearsum)
This should match to what you were asking me on IRC.
I have tested that it works on Leopard. I am still waiting to test that it works for Snow Leopard.
Attachment #435678 - Attachment is obsolete: true
Attachment #435903 - Flags: feedback?(bhearsum)
Comment on attachment 435903 [details] [diff] [review]
puppet changes for Leopard and Snow Leopard

This is pretty much good. My only nit is that the mercurial tarballs shouldn't be living in the same directory; please make subdirs for 10.5 and 10.6. We could run into issues down the road if we want them on the same version, but have to use different tarballs per OS.

Other than that, if it tests out OK, it looks good.
This run properly for both Leopard and Snow Leopard.
Attachment #435903 - Attachment is obsolete: true
Attachment #435997 - Flags: review?(bhearsum)
Attachment #435903 - Flags: feedback?(bhearsum)
Attachment #435998 - Flags: review?(bhearsum)
Comment on attachment 435997 [details] [diff] [review]
puppet changes for Leopard and Snow Leopard

These slaves are moving back to production at some point, so please don't add them here. r=me with that changed.
Attachment #435997 - Flags: review?(bhearsum) → review+
Attachment #435998 - Flags: review?(bhearsum) → review+
As landed (combined).
http://hg.mozilla.org/build/puppet-manifests/rev/1df033a8e9b0

I have been able to update Leopard and Snow Leopards manually with the expected results.
I will check other slaves after they reboot.
Attachment #435997 - Attachment is obsolete: true
Attachment #435998 - Attachment is obsolete: true
Attachment #436180 - Flags: checked-in+
Attachment #434934 - Attachment mime type: application/octet-stream → text/plain
[root@staging-puppet puppet-files]# cvs commit -m "Bug 553337. Add Mercurial to talos darwin slaves. r=bhearsum"
RCS file: /mofo/puppet-files/talos/mac/hg,v
done
Checking in talos/mac/hg;
/mofo/puppet-files/talos/mac/hg,v  <--  hg
initial revision: 1.1
done
RCS file: /mofo/puppet-files/talos/mac/darwin10/.bash_profile,v
done
Checking in talos/mac/darwin10/.bash_profile;
/mofo/puppet-files/talos/mac/darwin10/.bash_profile,v  <--  .bash_profile
initial revision: 1.1
done
RCS file: /mofo/puppet-files/talos/mac/darwin10/mercurial-1.3.1.tar.gz,v
done
Checking in talos/mac/darwin10/mercurial-1.3.1.tar.gz;
/mofo/puppet-files/talos/mac/darwin10/mercurial-1.3.1.tar.gz,v  <--  mercurial-1.3.1.tar.gz
initial revision: 1.1
done
RCS file: /mofo/puppet-files/talos/mac/darwin9/mercurial-1.2.1.tar.gz,v
done
Checking in talos/mac/darwin9/mercurial-1.2.1.tar.gz;
/mofo/puppet-files/talos/mac/darwin9/mercurial-1.2.1.tar.gz,v  <--  mercurial-1.2.1.tar.gz
initial revision: 1.1
done
Might as well attach the patch for testing next week.
This has been tested successfully for both Fedora and Fedorax64 r3 talos slaves.
Attachment #436527 - Attachment is obsolete: true
Attachment #437353 - Flags: review?(bhearsum)
Blocks: 557587
Comment on attachment 437353 [details] [diff] [review]
install Mercurial to talos Fedora slaves

Did you forget to 'hg add' the talos_fedora.pp file? The other bits look fine, but I don't see anything that actually installs Mercurial :).
Attachment #437353 - Flags: review?(bhearsum) → review-
Attachment #437357 - Flags: review?(bhearsum)
Comment on attachment 437357 [details] [diff] [review]
install Mercurial to talos Fedora slaves

>diff --git a/os/talos_fedora.pp b/os/talos_fedora.pp
>new file mode 100644
>--- /dev/null
>+++ b/os/talos_fedora.pp
>@@ -0,0 +1,9 @@
>+# talos_fedora.pp
>+
>+class talos_fedora {
>+    package {
>+        "mercurial":
>+            ensure   => "1.3.1-3.fc12",
>+            provider => yum;
>+    }
>+}

I think this will break at some point, when the yum repository updates to a new mercurial. Can you test the following scenario to check?:
* Uninstall Mercurial from a slave
* Bump the version to something newer, like 1.5.1-1.fc12, which won't exist in the yum repositories
* Sync that slave up with the Puppet master

If it still works, this patch is OK. If it doesn't work, we'll need to use ensure => installed, or keep a copy of the RPM and use provide => rpm.
Working on staging for the Fedora (32 & 64 bits) talos slaves.

It took me a longer to test because of /N not being mounted on Fedora slaves (I should file it) and the staging-puppet apache server not running (bug 557835).

Thanks Ben.
Attachment #437353 - Attachment is obsolete: true
Attachment #437357 - Attachment is obsolete: true
Attachment #437602 - Flags: review?(bhearsum)
Attachment #437357 - Flags: review?(bhearsum)
Blocks: 557918
Comment on attachment 437602 [details] [diff] [review]
install Mercurial to talos Fedora slaves

Looks good to me
Attachment #437602 - Flags: review?(bhearsum) → review+
Comment on attachment 437602 [details] [diff] [review]
install Mercurial to talos Fedora slaves

http://hg.mozilla.org/build/puppet-manifests/rev/8dcb160501ae

This has been deployed and tested once more in two Fedora talos slaves that were idle.
I will check in the next hour and see if any Fedora rebooted slave will have Mercurial on it.

# cvs ci -m "Bug 553337. Mercurial RPMs fpr Talos Fedora slaves."
cvs commit: Examining .
cvs commit: Examining RPMS
cvs commit: Examining RPMS/i686
cvs commit: Examining RPMS/x86_64
RCS file: /mofo/puppet-files/talos/linux/RPMS/i686/mercurial-1.5.1-1.fc12.i686.rpm,v
done
Checking in RPMS/i686/mercurial-1.5.1-1.fc12.i686.rpm;
/mofo/puppet-files/talos/linux/RPMS/i686/mercurial-1.5.1-1.fc12.i686.rpm,v  <--  mercurial-1.5.1-1.fc12.i686.rpm
initial revision: 1.1
done
RCS file: /mofo/puppet-files/talos/linux/RPMS/x86_64/mercurial-1.5.1-1.fc12.x86_64.rpm,v
done
Checking in RPMS/x86_64/mercurial-1.5.1-1.fc12.x86_64.rpm;
/mofo/puppet-files/talos/linux/RPMS/x86_64/mercurial-1.5.1-1.fc12.x86_64.rpm,v  <--  mercurial-1.5.1-1.fc12.x86_64.rpm
initial revision: 1.1
done
I have tested two slaves after reboot and they both had mercurial.

We now have Mercurial on darwin and fedora talos slaves.
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
No longer blocks: 548768
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.