Closed Bug 583924 Opened 10 years ago Closed 10 years ago

Add yasm 1.1.0 to win32 build machines, update mac/mac64 and linux/linux64 to 1.1.0

Categories

(Release Engineering :: General, defect, P2)

defect

Tracking

(blocking2.0 final+)

RESOLVED FIXED
Tracking Status
blocking2.0 --- final+

People

(Reporter: justin.lebar+bug, Assigned: nthomas)

References

Details

(Whiteboard: [buildslaves][opsi])

Attachments

(4 files, 1 obsolete file)

In order to fix bug 583849, we need a new yasm binary on the Windows 32-bit build machines.  The build came in just this weekend, so right now it's only in the trunk.

We need the same binary for bug 573948 on Windows and Mac 32-bit.  There are some other systems (at least Linux 32 bit) that require at least yasm 1.0.

I figure we should probably upgrade all the machines to the trunk yasm.  I've confirmed that rev 2348 does what I need.  Builds are available at

  http://www.tortall.net/projects/yasm/snapshots/r2348

The yasm devs have indicated to me that they might be willing to spin a 1.0.2 release (containing the necessary fixes) sometime soon.  I can inquire further about that if it would be preferable.
What ETA are you asking for here ? Is this a must-have or nice-to-have for Firefox 4.0 ? I see bug 583849 is not blocking.
jst, any idea whether we want libjpeg-turbo for FF 4?
Justin is working on performance numbers on 32-bit windows builds, once we have those we should decide whether landing libjpeg-turbo is a blocker for Firefox 4.
sorry for the bug spam, really just read the title...my bad
Just got 32-bit Windows numbers. 2.7x speedup, I think.  Details in bug 573948.
Nick, I think Justin's results means that this is a must have for Firefox 4.
blocking2.0: --- → final+
OK. Which beta try to aim for ? How much time does it need in beta testers hands ?
I'd say no later than beta5. Justin, does that sound ok to you?
(In reply to comment #9)
> I'd say no later than beta5. Justin, does that sound ok to you?

Sounds good to me.
Nick, does beta5 sound doable to you guys?
The current version (1.0.1) was deployed to Linux in bug 567302 and Mac/Mac64 in bug 568377.

Are we still ok to build a 32bit version of yasm and use it for 64bit builds on mac?
(In reply to comment #12)
> Are we still ok to build a 32bit version of yasm and use it for 64bit builds on
> mac?

I think so.  At least, I didn't have any problems when I did that.
Prior to 4.0b5 sounds fine. Looking for an owner in RelEng, any takers ?
Priority: -- → P3
Summary: Add/update yasm binary on build machines to trunk → Add yasm on win32 build machines
Whiteboard: [buildslaves][opsi]
re the change of summary to "Add yasm on win32 build machines": We need a version of yasm newer than 1.0.1 on mac32 as well.  I think 1.0.1 will work everywhere else, although my preference would be to have all the machines running the same version.
Ah, I hadn't reread comment #0 carefully enough. It would be really helpful if you could verify that linux and mac are OK with a development snapshot post-1.0.1. Taking a development version increases the risk of regressions, and having to push another yasm out to the slaves, so it's worth some effort up front.

FTR, bug 568377 for the initial deployment on mac/mac64, bug 567302 for linux/linux64.
Summary: Add yasm on win32 build machines → Add yasm r2348 on win32 build machines, update mac/mac64 and linux/linux64 to r2348
When I last spoke with Peter Johnson, one of the yasm devs, he suggested he might spin a 1.0.2 release soon which would include these and other fixes.

I just e-mailed him asking what his timeframe for the release is.  If he's going to release within the next few days, I think we should wait to pick it up.  Otherwise, I'll get started testing with trunk once I hear back.
Peter says he plans to get 1.0.2 out this weekend.
Summary: Add yasm r2348 on win32 build machines, update mac/mac64 and linux/linux64 to r2348 → Add yasm 1.0.2 to win32 build machines, update mac/mac64 and linux/linux64 to 1.0.2
New version released today.  It's called 1.1.0.
Summary: Add yasm 1.0.2 to win32 build machines, update mac/mac64 and linux/linux64 to 1.0.2 → Add yasm 1.1.0 to win32 build machines, update mac/mac64 and linux/linux64 to 1.1.0
I just tested 1.1.0 on linux 32 and 64 and mac 64 and it seems to produce working builds.  The Windows box I was using is currently busy, so I can't test there right now.

Not too much has changed between the trunk version I originally tested with and the released version 1.1.0, so I think it's unlikely that we'll have problems with 1.1.0.  But I can keep waiting to test myself if we want to be more sure.
Depends on: 570477
Are we still targeting beta 5 for this?
Sorry, we're going to miss beta5.
Priority: P3 → P2
linux32 & mac32 (10.5) compile fine, had a problem linking on linux64 that needs investigating. For windows they distribute the executable without any installer or package, so it should be a pretty simple opsi package.
Assignee: nobody → nrthomas
fwiw, I just re-compiled yasm 1.1 without issue on my 64-bit Ubuntu 10.04 box.
Really small change. yasm still needs testing but compiles OK.
Linux64 built on moz2-linux64-slave07 using the stock source from yasm homepage.

$ cd ~/rpmbuild/SOURCES
$ wget http://www.tortall.net/projects/yasm/releases/yasm-1.1.0.tar.gz
$ sha1sum yasm-1.1.0.tar.gz
8b252d2a50f6d0d8fe13997183596c3cd7589db9  yasm-1.1.0.tar.gz
$ cd ../SPECS
# get patched copy yasm.spec

$ cd ...
$ unset CC CXX
$ rpmbuild -ba --target=x86_64 SPECS/yasm.spec
$ sha1sum RPMS/x86_64/yasm-1.1.0-1.x86_64.rpm
4eab6738fbe0c646d79f875d84d8f1a641699d8e  RPMS/x86_64/yasm-1.1.0-1.x86_64.rpm
$ scp -o BatchMode=no RPMS/x86_64/yasm-1.1.0-1.x86_64.rpm \
                      staging-puppet.build.mozilla.org:/tmp/

Unsetting the compiler variables resolves the problem I was having compiling, I think because we were compiling with gcc 4.3.3 but picking up headers/libs for 4.1.1. This way we use 4.1.1.
Linux32 built on staging-master.

$ cd ~/rpmbuild/SOURCES
$ wget http://www.tortall.net/projects/yasm/releases/yasm-1.1.0.tar.gz
$ sha1sum yasm-1.1.0.tar.gz
8b252d2a50f6d0d8fe13997183596c3cd7589db9  yasm-1.1.0.tar.gz
$ cd ../checkouts/rpm-sources
$ hg diff
$ hg pull && hg up
$ curl -sL https://bugzilla.mozilla.org/attachment.cgi?id=470685 | patch -p1

$ cd ~/rpmbuild
$ rpmbuild -ba --target=i386 checkouts/rpm-sources/yasm/centos5-i686/yasm.spec
$ sha1sum RPMS/i386/yasm-1.1.0-1.i386.rpm
d2efc21483a1ca0614032ef7ea3136b78ab7d4be  RPMS/i386/yasm-1.1.0-1.i386.rpm
$ scp -o BatchMode=no RPMS/i386/yasm-1.1.0-1.i386.rpm \
                      staging-puppet.build.mozilla.org:/tmp/
Mac32 built on moz2-darwin9-slave03:

$ cd ~cltbld
$ mkdir dmgbuild
$ cd dmgbuild

$ wget http://www.tortall.net/projects/yasm/releases/yasm-1.1.0.tar.gz
$ openssl dgst -sha1 yasm-1.1.0.tar.gz 
SHA1(yasm-1.1.0.tar.gz)= 8b252d2a50f6d0d8fe13997183596c3cd7589db9

$ tar xfz yasm-1.1.0.tar.gz
$ cd yasm-1.1.0
$ ./configure
$ make
$ make install DESTDIR=yasm-root

$ cd ..
$ hg clone http://hg.mozilla.org/build/puppet-manifests
$ puppet-manifests/create-dmg.sh yasm-1.1.0/yasm-root/usr yasm-1.1.0 yasm /
$ openssl dgst -sha1 yasm-1.1.0.dmg 
SHA1(yasm-1.1.0.dmg)= 331acc040f4180d15e9fb6d042073a946fa223fb
$ scp -o BatchMode=no yasm-1.1.0.dmg staging-puppet.build.mozilla.org:/tmp/
This was the obvious change to make to the puppet configs but it didn't work out for me. 

On mac I get for (--test, without --noop):
notice: //Node[build]/base/osx/Install_dmg[yasm-1.1.0.dmg]/Exec[check-for-yasm-1.1.0.dmg]/returns: executed successfully
The /var/db/.puppet_pkgdmg_installed_yasm-1.1.0.dmg file is empty and 'yasm --version' is still 1.0.1. It should be overwriting the existing 1.0.1 install. The dmg installs fine for me my own machine, using the interactive install with no prior version of yasm. On linux there's no reference at all to yasm. 

Any idea what's going on both platforms ? For linux do we need to switch to this kind of usage http://hg.mozilla.org/build/puppet-manifests/file/4f06048ffe74/packages/devtools.pp ?
Attached patch OPSI package for 32bit windows (obsolete) — Splinter Review
I've taken a guess at what ted will do to add yasm to MozillaBuild rather than put this in /d/yasm. It's just copying in the executable, which is 
  http://www.tortall.net/projects/yasm/releases/yasm-1.1.0-win32.exe
renamed to yasm.exe, SHA1 of 67a4d9d24187bd529b9527613533db264b1dd6b7; and updating msys's /etc/profile.d/profile-extrapaths.sh to get the new dir on the path. 

Install and uninstall worked on win32-slave03.
Attachment #470738 - Flags: review?(bhearsum)
(In reply to comment #29)
> Created attachment 470723 [details] [diff] [review]
> Naive puppet patch
> 
> This was the obvious change to make to the puppet configs but it didn't work
> out for me. 
> 
> On mac I get for (--test, without --noop):
> notice:
> //Node[build]/base/osx/Install_dmg[yasm-1.1.0.dmg]/Exec[check-for-yasm-1.1.0.dmg]/returns:
> executed successfully
> The /var/db/.puppet_pkgdmg_installed_yasm-1.1.0.dmg file is empty and 'yasm
> --version' is still 1.0.1. It should be overwriting the existing 1.0.1 install.
> The dmg installs fine for me my own machine, using the interactive install with
> no prior version of yasm. On linux there's no reference at all to yasm. 
> 
> Any idea what's going on both platforms ? For linux do we need to switch to
> this kind of usage
> http://hg.mozilla.org/build/puppet-manifests/file/4f06048ffe74/packages/devtools.pp
> ?

Hmmmm, for Mac you'll need to use a plain package { } instead of install_dmg. That function purposely skips installing the package if the file listed in 'creates' exists. Doesn't work so well in the upgrade case...(to be honest, now that we're managing everything with DMG/RPM we can just drop the install_* functions entirely).

As for Linux...try setting ensure => latest there. I think it's defaulting to ensure => installed, which would mean it's skipping it because there's already *a* version of the package installed.
Thanks for the help getting this going.
Attachment #471067 - Flags: review?(bhearsum)
Attachment #471067 - Flags: review?(bhearsum) → review+
Comment on attachment 470738 [details] [diff] [review]
OPSI package for 32bit windows

Any idea what happens if d:\mozilla-build\yasm already exists? Could you test that case? r=me as long as it doesn't hang the slave. If it hangs, we should probably protect that mkdir behind something.
Attachment #470738 - Flags: review?(bhearsum) → review+
This switches to using a Files section instead of DosInAnIcon, since the 'copy'  available in Files creates any directories in the target as needed. Since copy doesn't do renames so I had to move 
  profile-extrapaths.sh          --> new/profile-extrapaths.sh
  profile-extrapaths.sh.original --> original/profile-extrapaths.sh

Verified that install, reinstall, and uninstall work. Vulnerable to installing yasm, something else changing profile-extrapaths.sh, uninstall yasm, but I don't think that's very likely.
Attachment #470738 - Attachment is obsolete: true
Attachment #471393 - Flags: review?(bhearsum)
Attachment #471393 - Attachment is patch: true
Attachment #471393 - Attachment mime type: application/octet-stream → text/plain
Attachment #471393 - Flags: review?(bhearsum) → review+
Comment on attachment 470685 [details] [diff] [review]
Updated .spec file

This is a tiny update to the file you had to 1.0.1.
Attachment #470685 - Flags: review?(jhford)
Comment on attachment 471393 [details] [diff] [review]
OPSI package for 32bit windows

http://hg.mozilla.org/build/opsi-package-sources/rev/b1ef19cd9fb1

Landing the binary in /mofo (with -kb switch):
Checking in yasm/yasm.exe;
/mofo/opsi-binaries/yasm/yasm.exe,v  <--  yasm.exe
initial revision: 1.1
done
Attachment #471393 - Flags: checked-in+
Set these production windows slaves to pick yasm 1.1.0:
* win2k3-ref-img, win32-ix-ref
* mw-ix-slave02..25, try-win32-slave01..36, win32-slave01,02,05..20,22..59 

I've rebooted the ref images to get them up to date straight away. The build slaves will pick up the update on rebooting after a job. I'll check in 24 hours for any that haven't got it yet.

And also these staging slaves: mw32-ix-slave01, win32-ix-slave01, win32-ix-slave02,  win32-slave03, win32-slave04, win32-slave21, win32-slave60
which can get it at their leisure.
Attachment #470685 - Flags: review?(jhford) → review+
All the win32 machines have yasm now, for try and everything else. We're good to land bug 583849 now.
Whiteboard: [buildslaves][opsi] → [buildslaves][opsi] win32 done, mac&linux pending 4.0b5 shipping
Comment on attachment 471067 [details] [diff] [review]
Working puppet patch

http://hg.mozilla.org/build/puppet-manifests/rev/ed354d1ad902

Both production puppet masters updated, slaves will start picking this up on reboot.
Attachment #471067 - Flags: checked-in+
All the linux & mac machines on try have picked up yasm 1.1.0. I'll check the non-try machines tomorrow.
Whiteboard: [buildslaves][opsi] win32 done, mac&linux pending 4.0b5 shipping → [buildslaves][opsi] only non-try mac&linux to check
Blocks: 594468
Production machines have updated yasm now, including the new ix boxes, with the exception of a few machines that are out of service. Getting them back into service requires that puppet is working so they'll be OK when they're back.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Whiteboard: [buildslaves][opsi] only non-try mac&linux to check → [buildslaves][opsi]
Blocks: 646248
Blocks: 649914
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.