583924 - Add yasm 1.1.0 to win32 build machines, update mac/mac64 and linux/linux64 to 1.1.0

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Justin Lebar (not reading bugmail)]

In order to fix bug 583849, we need a new yasm binary on the Windows 32-bit build machines.  The build came in just this weekend, so right now it's only in the trunk.

We need the same binary for bug 573948 on Windows and Mac 32-bit.  There are some other systems (at least Linux 32 bit) that require at least yasm 1.0.

I figure we should probably upgrade all the machines to the trunk yasm.  I've confirmed that rev 2348 does what I need.  Builds are available at

  http://www.tortall.net/projects/yasm/snapshots/r2348

The yasm devs have indicated to me that they might be willing to spin a 1.0.2 release (containing the necessary fixes) sometime soon.  I can inquire further about that if it would be preferable.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

What ETA are you asking for here ? Is this a must-have or nice-to-have for Firefox 4.0 ? I see bug 583849 is not blocking.

Comment 2

[Justin Lebar (not reading bugmail)]

jst, any idea whether we want libjpeg-turbo for FF 4?

Comment 3

[Johnny Stenback (:jst)]

Justin is working on performance numbers on 32-bit windows builds, once we have those we should decide whether landing libjpeg-turbo is a blocker for Firefox 4.

Comment 4

[atzaus]

Justin FYI
http://www.tortall.net/projects/yasm/snapshots/

Comment 5

[atzaus]

sorry for the bug spam, really just read the title...my bad

Comment 6

[Justin Lebar (not reading bugmail)]

Just got 32-bit Windows numbers. 2.7x speedup, I think.  Details in bug 573948.

Comment 7

[Johnny Stenback (:jst)]

Nick, I think Justin's results means that this is a must have for Firefox 4.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

OK. Which beta try to aim for ? How much time does it need in beta testers hands ?

Comment 9

[Johnny Stenback (:jst)]

I'd say no later than beta5. Justin, does that sound ok to you?

Comment 10

[Justin Lebar (not reading bugmail)]

(In reply to comment #9)
> I'd say no later than beta5. Justin, does that sound ok to you?

Sounds good to me.

Comment 11

[Johnny Stenback (:jst)]

Nick, does beta5 sound doable to you guys?

Comment 12

[John Ford [:jhford] CET/CEST Berlin Time]

The current version (1.0.1) was deployed to Linux in bug 567302 and Mac/Mac64 in bug 568377.

Are we still ok to build a 32bit version of yasm and use it for 64bit builds on mac?

Comment 13

[Justin Lebar (not reading bugmail)]

(In reply to comment #12)
> Are we still ok to build a 32bit version of yasm and use it for 64bit builds on
> mac?

I think so.  At least, I didn't have any problems when I did that.

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Prior to 4.0b5 sounds fine. Looking for an owner in RelEng, any takers ?

Comment 15

[Justin Lebar (not reading bugmail)]

re the change of summary to "Add yasm on win32 build machines": We need a version of yasm newer than 1.0.1 on mac32 as well.  I think 1.0.1 will work everywhere else, although my preference would be to have all the machines running the same version.

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

Ah, I hadn't reread comment #0 carefully enough. It would be really helpful if you could verify that linux and mac are OK with a development snapshot post-1.0.1. Taking a development version increases the risk of regressions, and having to push another yasm out to the slaves, so it's worth some effort up front.

FTR, bug 568377 for the initial deployment on mac/mac64, bug 567302 for linux/linux64.

Comment 17

[Justin Lebar (not reading bugmail)]

When I last spoke with Peter Johnson, one of the yasm devs, he suggested he might spin a 1.0.2 release soon which would include these and other fixes.

I just e-mailed him asking what his timeframe for the release is.  If he's going to release within the next few days, I think we should wait to pick it up.  Otherwise, I'll get started testing with trunk once I hear back.

Comment 18

[Justin Lebar (not reading bugmail)]

Peter says he plans to get 1.0.2 out this weekend.

Comment 19

[Justin Lebar (not reading bugmail)]

New version released today.  It's called 1.1.0.

Comment 20

[Justin Lebar (not reading bugmail)]

I just tested 1.1.0 on linux 32 and 64 and mac 64 and it seems to produce working builds.  The Windows box I was using is currently busy, so I can't test there right now.

Not too much has changed between the trunk version I originally tested with and the released version 1.1.0, so I think it's unlikely that we'll have problems with 1.1.0.  But I can keep waiting to test myself if we want to be more sure.

Comment 21

[Justin Lebar (not reading bugmail)]

Are we still targeting beta 5 for this?

Comment 22

[Nick Thomas [:nthomas] (UTC+12)]

Sorry, we're going to miss beta5.

Comment 23

[Nick Thomas [:nthomas] (UTC+12)]

linux32 & mac32 (10.5) compile fine, had a problem linking on linux64 that needs investigating. For windows they distribute the executable without any installer or package, so it should be a pretty simple opsi package.

Comment 24

[Justin Lebar (not reading bugmail)]

fwiw, I just re-compiled yasm 1.1 without issue on my 64-bit Ubuntu 10.04 box.

Comment 25

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Updated .spec file

Really small change. yasm still needs testing but compiles OK.

Comment 26

[Nick Thomas [:nthomas] (UTC+12)]

Linux64 built on moz2-linux64-slave07 using the stock source from yasm homepage.

$ cd ~/rpmbuild/SOURCES
$ wget http://www.tortall.net/projects/yasm/releases/yasm-1.1.0.tar.gz
$ sha1sum yasm-1.1.0.tar.gz
8b252d2a50f6d0d8fe13997183596c3cd7589db9  yasm-1.1.0.tar.gz
$ cd ../SPECS
# get patched copy yasm.spec

$ cd ...
$ unset CC CXX
$ rpmbuild -ba --target=x86_64 SPECS/yasm.spec
$ sha1sum RPMS/x86_64/yasm-1.1.0-1.x86_64.rpm
4eab6738fbe0c646d79f875d84d8f1a641699d8e  RPMS/x86_64/yasm-1.1.0-1.x86_64.rpm
$ scp -o BatchMode=no RPMS/x86_64/yasm-1.1.0-1.x86_64.rpm \
                      staging-puppet.build.mozilla.org:/tmp/

Unsetting the compiler variables resolves the problem I was having compiling, I think because we were compiling with gcc 4.3.3 but picking up headers/libs for 4.1.1. This way we use 4.1.1.

Comment 27

[Nick Thomas [:nthomas] (UTC+12)]

Linux32 built on staging-master.

$ cd ~/rpmbuild/SOURCES
$ wget http://www.tortall.net/projects/yasm/releases/yasm-1.1.0.tar.gz
$ sha1sum yasm-1.1.0.tar.gz
8b252d2a50f6d0d8fe13997183596c3cd7589db9  yasm-1.1.0.tar.gz
$ cd ../checkouts/rpm-sources
$ hg diff
$ hg pull && hg up
$ curl -sL https://bugzilla.mozilla.org/attachment.cgi?id=470685 | patch -p1

$ cd ~/rpmbuild
$ rpmbuild -ba --target=i386 checkouts/rpm-sources/yasm/centos5-i686/yasm.spec
$ sha1sum RPMS/i386/yasm-1.1.0-1.i386.rpm
d2efc21483a1ca0614032ef7ea3136b78ab7d4be  RPMS/i386/yasm-1.1.0-1.i386.rpm
$ scp -o BatchMode=no RPMS/i386/yasm-1.1.0-1.i386.rpm \
                      staging-puppet.build.mozilla.org:/tmp/

Comment 28

[Nick Thomas [:nthomas] (UTC+12)]

Mac32 built on moz2-darwin9-slave03:

$ cd ~cltbld
$ mkdir dmgbuild
$ cd dmgbuild

$ wget http://www.tortall.net/projects/yasm/releases/yasm-1.1.0.tar.gz
$ openssl dgst -sha1 yasm-1.1.0.tar.gz 
SHA1(yasm-1.1.0.tar.gz)= 8b252d2a50f6d0d8fe13997183596c3cd7589db9

$ tar xfz yasm-1.1.0.tar.gz
$ cd yasm-1.1.0
$ ./configure
$ make
$ make install DESTDIR=yasm-root

$ cd ..
$ hg clone http://hg.mozilla.org/build/puppet-manifests
$ puppet-manifests/create-dmg.sh yasm-1.1.0/yasm-root/usr yasm-1.1.0 yasm /
$ openssl dgst -sha1 yasm-1.1.0.dmg 
SHA1(yasm-1.1.0.dmg)= 331acc040f4180d15e9fb6d042073a946fa223fb
$ scp -o BatchMode=no yasm-1.1.0.dmg staging-puppet.build.mozilla.org:/tmp/

Comment 29

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Naive puppet patch

This was the obvious change to make to the puppet configs but it didn't work out for me. 

On mac I get for (--test, without --noop):
notice: //Node[build]/base/osx/Install_dmg[yasm-1.1.0.dmg]/Exec[check-for-yasm-1.1.0.dmg]/returns: executed successfully
The /var/db/.puppet_pkgdmg_installed_yasm-1.1.0.dmg file is empty and 'yasm --version' is still 1.0.1. It should be overwriting the existing 1.0.1 install. The dmg installs fine for me my own machine, using the interactive install with no prior version of yasm. On linux there's no reference at all to yasm. 

Any idea what's going on both platforms ? For linux do we need to switch to this kind of usage http://hg.mozilla.org/build/puppet-manifests/file/4f06048ffe74/packages/devtools.pp ?

Comment 30

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: OPSI package for 32bit windows

I've taken a guess at what ted will do to add yasm to MozillaBuild rather than put this in /d/yasm. It's just copying in the executable, which is 
  http://www.tortall.net/projects/yasm/releases/yasm-1.1.0-win32.exe
renamed to yasm.exe, SHA1 of 67a4d9d24187bd529b9527613533db264b1dd6b7; and updating msys's /etc/profile.d/profile-extrapaths.sh to get the new dir on the path. 

Install and uninstall worked on win32-slave03.

Comment 31

[[e-mail] (:bhearsum)]

(In reply to comment #29)
> Created attachment 470723 [details] [diff] [review]
> Naive puppet patch
> 
> This was the obvious change to make to the puppet configs but it didn't work
> out for me. 
> 
> On mac I get for (--test, without --noop):
> notice:
> //Node[build]/base/osx/Install_dmg[yasm-1.1.0.dmg]/Exec[check-for-yasm-1.1.0.dmg]/returns:
> executed successfully
> The /var/db/.puppet_pkgdmg_installed_yasm-1.1.0.dmg file is empty and 'yasm
> --version' is still 1.0.1. It should be overwriting the existing 1.0.1 install.
> The dmg installs fine for me my own machine, using the interactive install with
> no prior version of yasm. On linux there's no reference at all to yasm. 
> 
> Any idea what's going on both platforms ? For linux do we need to switch to
> this kind of usage
> http://hg.mozilla.org/build/puppet-manifests/file/4f06048ffe74/packages/devtools.pp
> ?

Hmmmm, for Mac you'll need to use a plain package { } instead of install_dmg. That function purposely skips installing the package if the file listed in 'creates' exists. Doesn't work so well in the upgrade case...(to be honest, now that we're managing everything with DMG/RPM we can just drop the install_* functions entirely).

As for Linux...try setting ensure => latest there. I think it's defaulting to ensure => installed, which would mean it's skipping it because there's already *a* version of the package installed.

Comment 32

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Working puppet patch

Thanks for the help getting this going.

Comment 33

[[e-mail] (:bhearsum)]

Comment on attachment 470738 [details] [diff] [review]
OPSI package for 32bit windows

Any idea what happens if d:\mozilla-build\yasm already exists? Could you test that case? r=me as long as it doesn't hang the slave. If it hangs, we should probably protect that mkdir behind something.

Comment 34

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: OPSI package for 32bit windows

This switches to using a Files section instead of DosInAnIcon, since the 'copy'  available in Files creates any directories in the target as needed. Since copy doesn't do renames so I had to move 
  profile-extrapaths.sh          --> new/profile-extrapaths.sh
  profile-extrapaths.sh.original --> original/profile-extrapaths.sh

Verified that install, reinstall, and uninstall work. Vulnerable to installing yasm, something else changing profile-extrapaths.sh, uninstall yasm, but I don't think that's very likely.

Comment 35

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 470685 [details] [diff] [review]
Updated .spec file

This is a tiny update to the file you had to 1.0.1.

Comment 36

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 471393 [details] [diff] [review]
OPSI package for 32bit windows

http://hg.mozilla.org/build/opsi-package-sources/rev/b1ef19cd9fb1

Landing the binary in /mofo (with -kb switch):
Checking in yasm/yasm.exe;
/mofo/opsi-binaries/yasm/yasm.exe,v  <--  yasm.exe
initial revision: 1.1
done

Comment 37

[Nick Thomas [:nthomas] (UTC+12)]

Set these production windows slaves to pick yasm 1.1.0:
* win2k3-ref-img, win32-ix-ref
* mw-ix-slave02..25, try-win32-slave01..36, win32-slave01,02,05..20,22..59 

I've rebooted the ref images to get them up to date straight away. The build slaves will pick up the update on rebooting after a job. I'll check in 24 hours for any that haven't got it yet.

And also these staging slaves: mw32-ix-slave01, win32-ix-slave01, win32-ix-slave02,  win32-slave03, win32-slave04, win32-slave21, win32-slave60
which can get it at their leisure.

Comment 38

[Nick Thomas [:nthomas] (UTC+12)]

All the win32 machines have yasm now, for try and everything else. We're good to land bug 583849 now.

Comment 39

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 470685 [details] [diff] [review]
Updated .spec file

http://hg.mozilla.org/build/rpm-sources/rev/41d962673f6d

Comment 40

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 471067 [details] [diff] [review]
Working puppet patch

http://hg.mozilla.org/build/puppet-manifests/rev/ed354d1ad902

Both production puppet masters updated, slaves will start picking this up on reboot.

Comment 41

[Nick Thomas [:nthomas] (UTC+12)]

All the linux & mac machines on try have picked up yasm 1.1.0. I'll check the non-try machines tomorrow.

Comment 42

[Nick Thomas [:nthomas] (UTC+12)]

Production machines have updated yasm now, including the new ix boxes, with the exception of a few machines that are out of service. Getting them back into service requires that puppet is working so they'll be OK when they're back.