1032391 - update zip and unzip on mac os x build/test machines to 6.0

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Jordan Lund (:jlund)]

currently mac os x ships with 5.52 and that does not support zip64.

see: https://bugzilla.mozilla.org/show_bug.cgi?id=1028304#c21 and https://bugzilla.mozilla.org/show_bug.cgi?id=1028304#c30 for motivation

Comment 1

[James Graham [:jgraham]]

Is there anything I can do to help here? This is one of the remaining blockers for landing bug 945222 on m-c.

Comment 2

[Jordan Lund (:jlund)]

as per email thread, I'll be either implementing this or driving it through starting tomorrow

Comment 3

[Jordan Lund (:jlund)]

due to tree outages this fell off my list.

simone, I spoke to coop and he adviced me that you have recent experience in upgrading packages on some of our platforms, possibly even something on osx. Would you be able to help me with this or point me in the right direction? If you have time. :)

Comment 4

[Chris Cooper [:coop] (he/him)]

zip and unzip are different packages with different versions. The current version of zip on our Macs is 3.0, while unzip is 5.52 as reported:

[[e-mail] unzip60]$ zip -v
Copyright (c) 1990-2008 Info-ZIP - Type 'zip "-L"' for software license.
This is Zip 3.0 (July 5th 2008), by Info-ZIP.
Currently maintained by E. Gordon.  Please send bug reports to
the authors using the web page at www.info-zip.org; see README for details.

Latest sources and executables are at ftp://ftp.info-zip.org/pub/infozip,
as of above date; see http://www.info-zip.org/ for other sites.

Compiled with gcc 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00) for Unix (Mac OS X) on May 25 2011.

Zip special compilation options:
	USE_EF_UT_TIME       (store Universal Time)
	SYMLINK_SUPPORT      (symbolic links supported)
	LARGE_FILE_SUPPORT   (can read and write large files on file system)
	ZIP64_SUPPORT        (use Zip64 to store large files in archives)
	STORE_UNIX_UIDs_GIDs (store UID/GID sizes/values using new extra field)
	UIDGID_16BIT         (old Unix 16-bit UID/GID extra field also used)
	[encryption, version 2.91 of 05 Jan 2007] (modified for Zip 3)

Encryption notice:
	The encryption code of this program is not copyrighted and is
	put in the public domain.  It was originally written in Europe
	and, to the best of our knowledge, can be freely distributed
	in both source and object forms from any country, including
	the USA under License Exception TSU of the U.S. Export
	Administration Regulations (section 740.13(e)) of 6 June 2002.

Zip environment options:
             ZIP:  [none]
          ZIPOPT:  [none]

[[e-mail] unzip60]$ unzip -v
UnZip 5.52 of 28 February 2005, by Info-ZIP.  Maintained by C. Spieler.  Send
bug reports using http://www.info-zip.org/zip-bug.html; see README for details.

Latest sources and executables are at ftp://ftp.info-zip.org/pub/infozip/ ;
see ftp://ftp.info-zip.org/pub/infozip/UnZip.html for other sites.

Compiled with gcc 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00) for Unix on May 25 2011.

UnZip special compilation options:
	COPYRIGHT_CLEAN (PKZIP 0.9x unreducing method not supported)
	SET_DIR_ATTRIB
	TIMESTAMP
	USE_EF_UT_TIME
	USE_UNSHRINK (PKZIP/Zip 1.x unshrinking method supported)
	USE_DEFLATE64 (PKZIP 4.x Deflate64(tm) supported)
	VMS_TEXT_CONV
	[decryption, version 2.9 of 05 May 2000]

UnZip and ZipInfo environment options:
           UNZIP:  [none]
        UNZIPOPT:  [none]
         ZIPINFO:  [none]
      ZIPINFOOPT:  [none]

Comment 5

[Chris Cooper [:coop] (he/him)]

I've found the newer version of unzip (6.0). There's a compiled version on bld-lion-r5-001:

[[e-mail] unzip60]$ pwd
/Users/cltbld/unzip60
[[e-mail] unzip60]$ ./unzip -v
UnZip 6.00 of 20 April 2009, by Info-ZIP.  Maintained by C. Spieler.  Send
bug reports using http://www.info-zip.org/zip-bug.html; see README for details.

Latest sources and executables are at ftp://ftp.info-zip.org/pub/infozip/ ;
see ftp://ftp.info-zip.org/pub/infozip/UnZip.html for other sites.

Compiled with gcc 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00) for Unix Mac OS X on Jul 24 2014.

UnZip special compilation options:
        COPYRIGHT_CLEAN (PKZIP 0.9x unreducing method not supported)
        SET_DIR_ATTRIB
        SYMLINKS (symbolic links supported, if RTL and file system permit)
        TIMESTAMP
        UNIXBACKUP
        USE_EF_UT_TIME
        USE_UNSHRINK (PKZIP/Zip 1.x unshrinking method supported)
        USE_DEFLATE64 (PKZIP 4.x Deflate64(tm) supported)
        VMS_TEXT_CONV
        [decryption, version 2.11 of 05 Jan 2007]

UnZip and ZipInfo environment options:
           UNZIP:  [none]
        UNZIPOPT:  [none]
         ZIPINFO:  [none]
      ZIPINFOOPT:  [none]

Comment 6

[Chris Cooper [:coop] (he/him)]

Are we actually having problems creating zip files or just unpacking them, i.e. do we also need a new version of zip? The system version of zip (3.0) seems to be the most recent: ftp://ftp.info-zip.org/pub/infozip/src/

Comment 7

[James Graham [:jgraham]]

I have only seen problems with unpacking them.

Comment 8

[Jordan Lund (:jlund)]

I think the issue is zip 3.0 supports zip64 and large files while osx default unzip (5.52) does not.

At least that's what I am gathering from places like this: http://h30499.www3.hp.com/t5/General/Can-t-get-Large-file-and-ZIP64-support-with-Zip-3-0-to-work/td-p/5177290#.U9EtEie0bDQ

This sounds like good news. If we have a good version of zip already being used and unzip 6.0 is installed on our osx machines (the one coop mentioned is a build machine so I'll have to confirm the same lives on our test machine), this might be more of mozharness patch. I can take a look at that this morning.

Comment 9

[Chris Cooper [:coop] (he/him)]

(In reply to Jordan Lund (:jlund) from comment #8)
> This sounds like good news. If we have a good version of zip already being
> used and unzip 6.0 is installed on our osx machines (the one coop mentioned
> is a build machine so I'll have to confirm the same lives on our test
> machine)

I downloaded and built it on that specific machine. The new unzip is not generally available, but feel free to re-use the one I built. ;)

Comment 10

[Jordan Lund (:jlund)]

I suppose I should have looked at the path and seen it was HOME. OK, well, I think based off of: https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Packages#Darwin I will need to make a dmg out of this. Which, and I will confirm with dustin, will mean I need to:

1) make a sh script like http://mxr.mozilla.org/build/source/puppet/modules/packages/manifests/mozilla/ccache-dmg.sh
2) run sh script locally or on a slave to make the dmg
3) add the sh script like http://mxr.mozilla.org/build/source/puppet/modules/packages/manifests/mozilla/ccache-dmg.sh (for doc purposes?)
4) upload the dmg to http://puppetagain.pub.build.mozilla.org/data/repos/DMGs/
5) make puppet package installation file like http://mxr.mozilla.org/build/source/puppet/modules/packages/manifests/upx.pp

I have the tar ball of unzip 6.0. Hopefully I get time to implement this tomorrow assuming I have a quieter buildduty day. Thank you for your patience.

Comment 11

[Jordan Lund (:jlund)]

at end of last week, dustin confirmed comment 10 is the correct process

I've cleared my queue and I am no longer on buildduty. I have scheduled to fix this tomorrow.

Comment 12

[Jordan Lund (:jlund)]

Attachment: 140731_bug_1032391_unzip_60_osx-puppet.patch

finally had time to revisit this.

dustin, this patch includes:

* the sh script I used to create the unzip dmg
* the unzip.pp patch to apply it

I have not uploaded the dmg to our repos yet. Once this gets r+, I'll do that, wait for it to sync, and then land this patch.

dustin, should I be testing this on a each version of our osx test slaves by pinning them to my environment? I tested the dmg locally and after running the dmg and everything seems fine (installed into /usr/local/bin/unzip):

jlund@Hastings163:~/tmp/dump
> unzip -v
UnZip 6.00 of 20 April 2009, by Info-ZIP.  Maintained by C. Spieler.  Send
bug reports using http://www.info-zip.org/zip-bug.html; see README for details.

Latest sources and executables are at ftp://ftp.info-zip.org/pub/infozip/ ;
see ftp://ftp.info-zip.org/pub/infozip/UnZip.html for other sites.
....

Comment 13

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8465937 [details] [diff] [review]
140731_bug_1032391_unzip_60_osx-puppet.patch

Review of attachment 8465937 [details] [diff] [review]:
-----------------------------------------------------------------

No need to pin, really -- just install the package manually or run against your environment once (without pinning) and then verify that the new binary unzips correctly.  It's unlikely (but anything's possible) that this tool needs to be built separately for each OS X version.

I assume /usr/local/bin is earlier in PATH than /usr/bin, so that the new unzip will be used by default?

::: modules/packages/manifests/unzip-dmg.sh
@@ +38,5 @@
> +# Make package
> +mkdir -p $OUT
> +PKG=$OUT/$PACKAGE_FULLNAME.pkg
> +DMG=$PACKAGE_FULLNAME.dmg
> +pkgbuild --root $ROOT --identifier org.$REALNAME.$REALNAME --install-location / $PKG

From a grep of the repo, it looks like we're using pkgbuild for some dmg scripts, and packagebuilder for others.  Can you update the docs to indicate that both are OK and how to get pkgbuild installed?

Comment 14

[James Graham [:jgraham]]

I saw some reference on irc to mozharness changes needed for this? Is that correct? It seems pretty unfortunate if every mozharness job has to special case osx just to get the right zip binary. But if it is required, let me know what I have to do, because web-platform-tests uses a different mozharness script and isn't on m-c yet.

Comment 15

[Jordan Lund (:jlund)]

(In reply to James Graham [:jgraham] from comment #14)
> I saw some reference on irc to mozharness changes needed for this? Is that
> correct? It seems pretty unfortunate if every mozharness job has to special
> case osx just to get the right zip binary. But if it is required, let me
> know what I have to do, because web-platform-tests uses a different
> mozharness script and isn't on m-c yet.

I hear ya but that appears to be the case. for all osx jobs, /usr/local/bin comes after /usr/bin/:
PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin

It looks like we set up env differently for test jobs than build jobs. Ultimately I think most test jobs get there env from here: http://mxr.mozilla.org/build/source/buildbotcustom/env.py

osx's PATH is not set so nothing custom is done. I see 4 options:

1) add a custom PATH to http://mxr.mozilla.org/build/source/buildbotcustom/env.py
** I am wary of doing a big change like this across our whole osx test pool in case usr/local/bin replaces anything from /usr/bin:/bin:/usr/sbin:/sbin:
** I am not sure if every test job actually touches this file so there is probably more work to be done

2) add a custom PATH to the env in osx mozharness jobs
** this would still have the same downside as effecting all osx test jobs
** I think there are some osx test jobs that do not use mozharness (e.g. jetpack)

3) replace existing unzip-5.52 with unzip-6.0
** I doubt we would be in favor of overwriting default installed programs in case of bustage or needing to roll back. dustin might have an input or thoughts on this.

3) add a custom unzip path for os x jobs
** I agree this, too, is cumbersome but at the same time, we often need platform specific configs
** this would isolate to just unzip and would arguably be similar work to option 1 or 2

so my plan was to go for option 3. It would look something like:
* add: 'exes': {'unzip': '/usr/local/bin'} to:
** configs/talos/mac_config.py
** configs/unittests/mac_unittest.py

* add same exes unzip item under a condition like: http://mxr.mozilla.org/build/source/mozharness/configs/marionette/gaia_ui_test_prod_config.py#7 here:
*** configs/marionette/prod_config.py
*** configs/web_platform_tests/prod_config.py (or somewhere else if it differs by branch)

* we would need special case the still active buildbot factories that use class UnpackTest for things like jetpack here:
** http://mxr.mozilla.org/build/source/buildbotcustom/steps/misc.py#274

I am completely open to suggestions from anyone if I am overlooking something or choosing the wrong option :)

Comment 16

[Jordan Lund (:jlund)]

> 3) add a custom unzip path for os x jobs
/s/3/4/

> so my plan was to go for option 3. It would look something like:
s/3/4/

Comment 17

[Jordan Lund (:jlund)]

after irc disussion we resolved that going with: 'replace existing unzip-5.52 with unzip-6.0' is actually a decent option. So just a straight up replace of '/usr/bin/unzip'.

I have taken a snow leopard machine and pinned my puppet env to it. after a puppet run, I'm getting:

[[e-mail] ~]$ unzip -v
Illegal instruction

it looks like my dmg does change /usr/bin/unzip but the pkg installs a compiled version that does not work with snow leopard? I tried manually unpacking and compiling the unzip.tar.gz on a 10.6 slave but IIUC we do not install make/gcc on our test osx machines.

there seems to be a zip with a pre-made pkg that supports 10.6+: http://www.macupdate.com/app/mac/35967/unzip

but I am not sure if that is official enough or if it's too much of a sec risk.

dustin - I keep hitting road blocks on what should be trivial, could you advise me on what I should do?

Comment 18

[Dustin J. Mitchell [:dustin] (he/him)]

DMG installs are never trivial, sorry.. it's not you, it's OS X :)

We often find packages that we need to build per-architecture, or that need to be built on a specific architecture to work.  We do have xcode for all versions of OS X that we run.  I'm not sure where you built this, but it's quite possible that building it on a Snow system would result in a package that will work all the way up through Mavericky.

Using an external package from a reliable source (e.g., upstream, or Apple) isn't a bad idea (it's a great idea, because it avoids having to build it!), but that particular source doesn't seem very reliable.  Remember when C|Net Downloads seemed reliable, then it turned out they were bundling malware?  Sigh..

Comment 19

[Jordan Lund (:jlund)]

thanks for the reply

> We do have xcode for all versions of OS X that we run.

hmm, I can't find /Developer, make, or gcc on our snow test machines.

I see the following on our build lion machines:
[[e-mail] ~]$ ls /usr/bin/gcc
/usr/bin/gcc
[[e-mail] ~]$ which make
/usr/bin/make
[[e-mail] ~]$ which gcc
/usr/bin/gcc
[[e-mail] ~]$ ls /Developer/Applications/Xcode.app

> I'm not sure where you built this, but it's
> quite possible that building it on a Snow system would result in a package
> that will work all the way up through Mavericky.

I tried to build it on one of our lion machines but snow still barfed on it

> Using an external package from a reliable source (e.g., upstream, or Apple)
> isn't a bad idea (it's a great idea, because it avoids having to build it!),
> but that particular source doesn't seem very reliable.

makes sense.

Comment 20

[Dustin J. Mitchell [:dustin] (he/him)]

Right, it's not *installed* on them, but it can be installed:
 https://github.com/mozilla/build-puppet/blob/master/manifests/moco-config.pp#L262
 https://github.com/mozilla/build-puppet/blob/master/modules/packages/manifests/xcode.pp#L35
You can do so either with puppet in a puppet environment, or just by installing the xcode dmg by hand.  Sorry, that probably wasn't at all clear and I should have mentioned it earlier.

There were plans to make a set of package-building minis in relabs, but it's not risen above the "actually gets done" threshold on anyone's priority list; notably, deploystudio installs in that VLAN weren't working correctly when I tried.

When you do get this to build correctly, please leave a note in the unzip-dmg.sh script so the next person has somewhere to start.

Comment 21

[Jordan Lund (:jlund)]

ok thanks. I was pulled away today to work on hg.m.o issues (Bug 1050109) but I was able to track down a snow leopard machine in office and successfully compile/package unzip 6.0 on it.

I'll try bubbling that up through all our osx versions to make sure it works and then wrap this up.

Comment 22

[Dustin J. Mitchell [:dustin] (he/him)]

If you do end up needing to build on 10.7 and higher, let me know - I should rebuild NRPE, too, and we can share hosts with xcode installed.

Comment 23

[Jordan Lund (:jlund)]

Attachment: 140810_bug_1032391_unzip_60_osx-puppet.patch

so I didn't need to custom build this across each osx version. compiling on 10.6 seems to work on 10.7 and 10.8 as well.

this patch differs slightly than the last. I neglected to realize that modules/packages/manifests/unzip.pp wasn't currently in view to osx so I added the include under: modules/toplevel/manifests/slave/releng/test.pp. I poked around at some puppet files but test.pp seemed to fit for what I'm trying to achieve. AFAIK this should only effect osx test slaves.

I added a note in sh script that this needs to be compiled on 10.6. I also updated the dmg wiki about using pkgbuild over packagemaker.

I uploaded the appropriate dmg in /data and then ran some unzip tests on a 10.6 and 10.7 slave (10.8 I did not test against my puppet env but I scp'd my 10.6 unzip-60 compiled version over to it and verified it).

Here are the results (10.6 results were the same):

# BEFORE INSTALLING UNZIP 6.0

[[e-mail] ~]# which unzip
/usr/bin/unzip
[[e-mail] ~]# unzip -v
UnZip 5.52 of 28 February 2005, by Info-ZIP.  Maintained by C. Spieler.  Send
bug reports using http://www.info-zip.org/zip-bug.html; see README for details.

Latest sources and executables are at ftp://ftp.info-zip.org/pub/infozip/ ;
see ftp://ftp.info-zip.org/pub/infozip/UnZip.html for other sites.

Compiled with gcc 4.2.1 Compatible Apple Clang 4.0 (tags/Apple/clang-418.0.60) for Unix on Jun 20 2012.

[[e-mail] ~]# puppet agent --test --server=releng-puppet2.srv.releng.scl3.mozilla.com --environment=jlund --pluginsync --ssldir=/var/lib/puppet/ssl
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb
Info: Loading facts in /var/lib/puppet/lib/facter/concat_ruby_interpreter.rb
Info: Loading facts in /var/lib/puppet/lib/facter/env.rb
Warning:/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/rubygems/custom_require.rb:31: importenv is deprecated after Ruby 1.8.1 (no replacement)
Info: Loading facts in /var/lib/puppet/lib/facter/existing_slave_trustlevel.rb
Info: Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rb
Info: Loading facts in /var/lib/puppet/lib/facter/ip6tables_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/iptables_persistent_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/iptables_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/needs_reboot.rb
Info: Loading facts in /var/lib/puppet/lib/facter/num_masters.rb
Info: Loading facts in /var/lib/puppet/lib/facter/pe_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb
Info: Loading facts in /var/lib/puppet/lib/facter/root_home.rb
Info: Loading facts in /var/lib/puppet/lib/facter/supermicro_ipmi_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/vmwaretools_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/winrootlp.rb
Info: Caching catalog for talos-mtnlion-r5-002.test.releng.scl3.mozilla.com
Info: /Stage[main]/Tweaks::Cleanup/Tidy[/Users/cltbld/Library/Application Support/Firefox/console.log]: File does not exist
Info: Applying configuration version 'heads/unzip'
Notice: /Stage[main]/Tweaks::Cleanup/Exec[find /tmp/* -mmin +15 -print | xargs -n1 rm -rf]/returns: executed successfully
Notice: /Stage[main]/Packages::Unzip/Packages::Pkgdmg[unzip]/Package[unzip-6.0.dmg]/ensure: created
Notice: /Stage[main]/Puppet::Atboot/File[/etc/puppet/puppetmasters.txt]/content:
--- /etc/puppet/puppetmasters.txt       2014-08-10 20:30:48.000000000 -0700
+++ /var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/puppet-file20140810-1089-l6kldd-0  2014-08-10 20:32:41.000000000 -0700
@@ -1,6 +1,6 @@
 releng-puppet1.srv.releng.scl3.mozilla.com
 releng-puppet2.srv.releng.scl3.mozilla.com
-releng-puppet1.srv.releng.use1.mozilla.com
-releng-puppet2.srv.releng.usw2.mozilla.com
 releng-puppet1.srv.releng.usw2.mozilla.com
 releng-puppet2.srv.releng.use1.mozilla.com
+releng-puppet1.srv.releng.use1.mozilla.com
+releng-puppet2.srv.releng.usw2.mozilla.com

Notice: /Stage[main]/Puppet::Atboot/File[/etc/puppet/puppetmasters.txt]/content: content changed '{md5}933d878d79217a2bef2ddd04e4598cb5' to '{md5}c3bab6826623457bdd14780fa651b933'
Notice: Finished catalog run in 54.97 seconds
[[e-mail] ~]# which unzip
/usr/bin/unzip
[[e-mail] ~]# unzip -v
UnZip 6.00 of 20 April 2009, by Info-ZIP.  Maintained by C. Spieler.  Send
bug reports using http://www.info-zip.org/zip-bug.html; see README for details.

Latest sources and executables are at ftp://ftp.info-zip.org/pub/infozip/ ;
see ftp://ftp.info-zip.org/pub/infozip/UnZip.html for other sites.

Compiled with gcc 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.9) for Unix Mac OS X on Aug  6 2014.


# THIS IS A ZIP THAT FAILED ON A RECENT CEDAR MAC OSX DEBUG JOB USING UNZIP 5.52

[[e-mail] jlund_tmp]# wget https://ftp-ssl.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/cedar-macosx64-debug/1406714586/firefox-34.0a1.en-US.mac64.tests.zip
[[e-mail] jlund_tmp]# wget http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/cedar-macosx64-debug/1406714586/firefox-34.0a1.en-US.mac64.tests.zip
--2014-08-10 20:37:10--  http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/cedar-macosx64-debug/1406714586/firefox-34.0a1.en-US.mac64.tests.zip
Resolving ftp.mozilla.org... 63.245.215.56, 63.245.215.46
Connecting to ftp.mozilla.org|63.245.215.56|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 122093994 (116M) [application/zip]
Saving to: `firefox-34.0a1.en-US.mac64.tests.zip'

100%[========================================================================================================================================>] 122,093,994 56.7M/s   in 2.1s

2014-08-10 20:37:12 (56.7 MB/s) - `firefox-34.0a1.en-US.mac64.tests.zip' saved [122093994/122093994]

[[e-mail] jlund_tmp]# ls
firefox-34.0a1.en-US.mac64.tests.zip
[[e-mail] jlund_tmp]# unzip -q -o firefox-34.0a1.en-US.mac64.tests.zip bin/* certs/* modules/* mozbase/* config/* mochitest/*


# NO ERRORS

[[e-mail] jlund_tmp]# ls
bin                                     config                                  mochitest                               mozbase
certs                                   firefox-34.0a1.en-US.mac64.tests.zip    modules

Note, there is no puppet rollback func AFAIK. This patch will have to be reverted and the machines will need to be re-imaged if this goes wrong. We spoke about making a backup of old unzip but after you and I discussed this on irc, we decided it not be needed.

As I'm paranoid, I put one of my testing slaves back into prod that used my puppet env with unzip 6.0. Since unzip was not previously tracked by puppet, 6.0 stuck and it's live in the wild: https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=talos-mtnlion-r5-002

I commented in its problem tracking bug stating it has 6.0.

Does this seem like reasonable testing? Is there something I should worry about here or can I land once (1) I get an r+ and (2) ensure that talos-mtnlion-r5-002 is behaving?

thanks in advance dustin :)

Comment 24

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8470622 [details] [diff] [review]
140810_bug_1032391_unzip_60_osx-puppet.patch

There's nothing wrong with making *one* backup copy - I only objected to having puppet do the backing-up individually on each host.

Would it hurt to have this version on all macs?  It seems odd to only put this in a leaf class of the toplevel hierarchy, since unzip is *present* on every host (just not controlled by puppet).

Comment 25

[Jordan Lund (:jlund)]

Comment on attachment 8470622 [details] [diff] [review]
140810_bug_1032391_unzip_60_osx-puppet.patch

on irc we agreed to try this on test.pp first

this is checked in and merged to prod: http://hg.mozilla.org/build/puppet/rev/9e1f976314ef

I'll kick off a few cedar jobs once there has been time for puppet to sync

Comment 26

[Dustin J. Mitchell [:dustin] (he/him)]

This failed and I backed it out.

I see what you were asking about Ubuntu systems now, and you were exactly right.  Sorry about that.

Comment 27

[Jordan Lund (:jlund)]

(In reply to Dustin J. Mitchell [:dustin] from comment #26)
> This failed and I backed it out.
> 
> I see what you were asking about Ubuntu systems now, and you were exactly
> right.  Sorry about that.

oh shux. it looks like this applied to the osx slaves and I was able to succesfully run a previous broken job on Cedar: https://tbpl.mozilla.org/php/getParsedLog.php?id=45688041&full=1&branch=cedar

So at least this does show to work. I'd imagine our slaves will keep 6.0 until they are re-imaged.

1) Was this foreman where you saw the failures? I'd like to try and catch this before you or someone else has to for me.
2) Any advice on the fix here? As in, how you'd like this organized so it is limited to just osx cleanly?
3) is there a way to test these puppet patches without loaning a slave from each platform and applying my env?

thanks!

Comment 28

[Dustin J. Mitchell [:dustin] (he/him)]

I try to watch the puppet-errors folder in the releng-shared shared mailbox.  Generally when there are more than a dozen or so unread mails there, something's gone wrong (there were 750 or so this time).

I'd suggest just putting in a clause for Ubuntu that specifies the version already installed.  Then we have that version number listed explicitly, and if/when we need to change it, we can do so in a controlled fashion.

And no, testing puppet patches realistically is just hard.  We'd like to get some permanent testing hosts up in relabs, but even there we'll lack some realism.  I usually just grab an idle slave and run `puppet agent --test --noop` on it quickly, as a way of testing.

Comment 29

[Jordan Lund (:jlund)]

Attachment: 140811_bug_1032391_unzip_60_osx-puppet-interdiff.patch

only an interdiff from last patch

this adds the Ubuntu case. This is based off of http://mxr.mozilla.org/build/source/puppet/modules/packages/manifests/wget.pp#22

There are some variants to the 6.0 version depending on the ubuntu version: http://packages.ubuntu.com/search?keywords=unzip&searchon=names

Looking at a few of our linux machines I get:

[root@talos-linux64-ix-001 ~]# dpkg -s unzip | grep Version
Version: 6.0-4ubuntu1
[[e-mail] ~]$ dpkg -s unzip | grep Version
Version: 6.0-4ubuntu1
[[e-mail] ~]$ dpkg -s unzip | grep Version
Version: 6.0-4ubuntu1

So I am going with that. Once we update to say 13.04, will this mean we have to be '6.0-8ubuntu1'?

# before this interdiff patch:
[root@talos-linux64-ix-001 ~]# puppet agent --test --noop --environment=jlund --server=releng-puppet2.srv.releng.scl3.mozilla.com
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts in /var/lib/puppet/lib/facter/pe_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/env.rb
Info: Loading facts in /var/lib/puppet/lib/facter/existing_slave_trustlevel.rb
Info: Loading facts in /var/lib/puppet/lib/facter/needs_reboot.rb
Info: Loading facts in /var/lib/puppet/lib/facter/vmwaretools_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb
Info: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb
Info: Loading facts in /var/lib/puppet/lib/facter/ip6tables_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/root_home.rb
Info: Loading facts in /var/lib/puppet/lib/facter/num_masters.rb
Info: Loading facts in /var/lib/puppet/lib/facter/iptables_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/iptables_persistent_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/supermicro_ipmi_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rb
Info: Loading facts in /var/lib/puppet/lib/facter/concat_ruby_interpreter.rb
Info: Loading facts in /var/lib/puppet/lib/facter/winrootlp.rb
Error: Could not retrieve catalog from remote server: Error 400 on SERVER: cannot install on Ubuntu at /etc/puppet/environments/jlund/modules/packages/manifests/unzip.pp:22 on no
de talos-linux64-ix-001.test.releng.scl3.mozilla.com
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

# after (no errors)
[root@talos-linux64-ix-001 ~]# puppet agent --test --noop --environment=jlund --server=releng-puppet2.srv.releng.scl3.mozilla.com
....
.... output of puppet run
....
Notice: Finished catalog run in 18.84 seconds

Comment 30

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8471155 [details] [diff] [review]
140811_bug_1032391_unzip_60_osx-puppet-interdiff.patch

Review of attachment 8471155 [details] [diff] [review]:
-----------------------------------------------------------------

Sorry for the delay!

Comment 31

[Dustin J. Mitchell [:dustin] (he/him)]

And, we definitely won't be "upgrading" to 13.04, its being already a year old, but when we do upgrade Ubuntu, we'll need to change versions accordingly.

Comment 32

[Jordan Lund (:jlund)]

Comment on attachment 8471155 [details] [diff] [review]
140811_bug_1032391_unzip_60_osx-puppet-interdiff.patch

thanks. merged and pushed to prod: https://hg.mozilla.org/build/puppet/rev/11d18b625d94

/me watches releng-shared puppet emails

Comment 33

[Jordan Lund (:jlund)]

everything seemed to go smoothly here.

I ran some tests on cedar and it looks like it fixes the issue. there are a couple situations where it failed but I think I didn't give enough time for puppet to sync with all the machines.

All the recent jobs made it passed the unzip part.

marking this bug as resolved and leaving bug 1028304 to jgraham

Comment 34

[James Graham [:jgraham]]

Hurrah! Thanks everyone.