594474 - Deploy MozillaBuild 1.6 to buildslaves

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect, P3)

Description

[Robert Strong (they/them - no direct email)]

The current NSIS on the Tinderbox has a bug that affects silent installers. I've added a workaround for one of the bugs so this isn't a rush.

Comment 1

[John Hopkins (:jhopkins)]

Installed NSIS 2.46 Unicode on one try server build vm.

D:\tmp> nsis-2.46-Unicode-setup.exe /S /D=d:\mozilla-build\nsis-2.46u

A build is running now:

 http://build.mozillamessaging.com/buildbot/try/builders/WINNT%205.2%20try%20build/builds/118

Also checked in a PATH env change to our buildbotcustom repo:

 https://hg.mozilla.org/users/gozer_mozillamessaging.com/buildbotcustom-thunderbird-try/

Comment 2

[John Hopkins (:jhopkins)]

Should have commented on Thunderbird clone of this bug - sorry, bear!

Comment 3

[Chris Cooper [:coop] (he/him)]

Unless I'm reading this bug wrong, bug 675970 is blocked (eventually) by this, not the other way around.

Comment 5

[Chris Cooper [:coop] (he/him)]

I've started trying this out in staging, but have run into a problem. 

The current version of MozillaBuild (1.5.1) prepends D:\mozilla-build\python (which is 2.6) to the path, while the previously installed version defaulted to D:\mozilla-build\python25. I don't think the python version change itself is bad, but the python2.6 install under D:\mozilla-build\python doesn't have the pywin32 extensions installed. Several of the releng tools (purge_builds.py, hgtool.py) rely on these extensions.

What's the best path forward here?

1) Simply pre-pend D:\mozilla-build\python25 to the path so the older python version is used.

2) Run an actual install of the 2.6 version of python included with MozillaBuild (the msi is included in the package), and then install the pywin32 extensions by hand.

3) Roll a new version of MozillaBuild that already has the pywin32 extensions installed.

#3 seems like the best plan to me. #1 is certainly the easiest, but we don't make any forward progress on the python front.

Comment 6

[Chris AtLee [:catlee]]

I like #3 too, except I'd package python2.7 with the pywin32 extensions.

Comment 7

[(not currently active) Ted Mielczarek]

We already updated Python to 2.7 in mozilla-build trunk (bug 578584). bug 496535 had been investigating adding pywin32, but I got stuck because it installed stuff to windows/system32. A while ago I talked to mhammond, who said that that stuff isn't important unless you want to implement ActiveX controls with Python or something crazy like that, so we could probably resurrect that.

Comment 8

[Chris Cooper [:coop] (he/him)]

(In reply to Ted Mielczarek [:ted, :luser] from comment #7)
> We already updated Python to 2.7 in mozilla-build trunk (bug 578584). bug
> 496535 had been investigating adding pywin32, but I got stuck because it
> installed stuff to windows/system32. A while ago I talked to mhammond, who
> said that that stuff isn't important unless you want to implement ActiveX
> controls with Python or something crazy like that, so we could probably
> resurrect that.

I think we only need the file APIs, so we're probably good there. I'll reopen bug 496535.

How long before such a thing would be ready, i.e. would it be best to massage the PATH and stick with 2.5 for the short-term? 

I'm guessing "yes" since it looks like I'm going to need to massage the path anyway, since the vim installer ends up earlier in the path than /bin/install at present:

make[7]: Leaving directory `/e/builds/moz2_slave/m-cen-w32/build/obj-firefox/build/win32/crashinjectdll'
mkdir -p ../../dist/bin
install --preserve-timestamps "/d/msvs8/VC/redist/x86/Microsoft.VC80.CRT"/Microsoft.VC80.CRT.manifest "/d/msvs8/VC/redist/x86/Microsoft.VC80.CRT"/msvcm80.dll "/d/msvs8/VC/redist/x86/Microsoft.VC80.CRT"/msvcp80.dll "/d/msvs8/VC/redist/x86/Microsoft.VC80.CRT"/msvcr80.dll ../../dist/bin
This program sets up the installation of Vim 7.2

Inspecting system...

ERROR: failed to rename vim.exe to vim.exx: No error

Comment 9

[(not currently active) Ted Mielczarek]

We fixed that in 1.5.1 or something, that shouldn't be a problem.

Comment 10

[Chris Cooper [:coop] (he/him)]

Attachment: Update slaves to MozillaBuild 1.5.1

This patch targets option 1 from comment #5.

There are 3 things going on is this patch:

1) The MozillaBuild version # is now in the window title, so we need to account for that during install.

2) I copied the version of profile-extravars.sh out of hg tip and made a slight addition to add python25 back to the front of the PATH to keep our releng tools working for now (due to pywin32).

3) I delete a file, profile-extrapaths.sh, that I've noticed lingering on some slaves from a previous MozillaBuild install. This file was making the PATH longer than it needed to be.

Hopefully this won't interfere with work for bug 602908, but at the same time, hopefully we'll have bug 496535 soon as well.

Comment 11

[[e-mail] (:bhearsum)]

(In reply to Chris Cooper [:coop] from comment #10)
> 3) I delete a file, profile-extrapaths.sh, that I've noticed lingering on
> some slaves from a previous MozillaBuild install. This file was making the
> PATH longer than it needed to be.

Yup, that should be fine. This was used to workaround a bug in earlier versions of MB.

Comment 12

[Chris AtLee [:catlee]]

Comment on attachment 573930 [details] [diff] [review]
Update slaves to MozillaBuild 1.5.1

Review of attachment 573930 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good.

Is there any reason to deploy though if it doesn't have python2.6/2.7?

Comment 13

[Chris Cooper [:coop] (he/him)]

(In reply to Chris AtLee [:catlee] from comment #12) 
> Is there any reason to deploy though if it doesn't have python2.6/2.7?

It *does* have python 2.6. Any build process that wanted/needed to use 2.6 could reference it directly, 2.5 will just be ahead of it in the path.

Comment 14

[Armen [:armenzg]]

It seems that Ehsan is hitting some msys (or nsis - not sure) bug with the current mozilla-build version in bug 704475.

Comment 15

[(no longer active)]

The old NSIS that we have on the bots is crashing when compiler our uninstaller script with the changes being made in bug 481815.  Do we have a timeline on how soon this can be deployed?

Thanks!

Comment 16

[Chris Cooper [:coop] (he/him)]

It will be sometime this week. We were waiting on the 8.0.1 chemspill, and now we're due for the next beta (9.0b3). Once builds for 9.0b3 are done, we should be safe to deploy.

Comment 17

[(no longer active)]

(In reply to Chris Cooper [:coop] from comment #16)
> It will be sometime this week. We were waiting on the 8.0.1 chemspill, and
> now we're due for the next beta (9.0b3). Once builds for 9.0b3 are done, we
> should be safe to deploy.

Do you know when 9.0b3 is targeted to go to build?

Comment 18

[Chris Cooper [:coop] (he/him)]

(In reply to Ehsan Akhgari [:ehsan] from comment #17) 
> Do you know when 9.0b3 is targeted to go to build?

Later today.

Comment 19

[Chris Cooper [:coop] (he/him)]

To followup Callek's comments from IRC, the following packages will also need to be reinstalled after the new MozillaBuild is deployed:

buildbot-startup
mercurial
python-ssl
rename-vim-install
twitsed_dumbwin32proc (if installed)
yasm

If you think there are others, please let me know.

Comment 20

[Armen [:armenzg]]

Don't do those packages come already with the newer mozilla-build?

Did your testing already show that installing your package was good enough? I'm not sure we need all of these packages but I could be wrong.

On another note, twisted_dumbwin32proc could be now be bitrotten (see bug 690232). With the newer version of buildbot/twisted supposedly we don't need to deploy such patch but I am not sure of that as I wasn't involved in that work and that's what I understand from dustin's comments in bug 690232.

Comment 21

[(not currently active) Ted Mielczarek]

(In reply to Chris Cooper [:coop] from comment #19)
> mercurial
> rename-vim-install
> yasm

I would think these would be obviated by the new MozillaBuild package.

Comment 22

[Robert Strong (they/them - no direct email)]

(In reply to Ehsan Akhgari [:ehsan] from comment #15)
> The old NSIS that we have on the bots is crashing when compiler our
> uninstaller script with the changes being made in bug 481815.  Do we have a
> timeline on how soon this can be deployed?
Ehsan, can you provide a link to a log with the crash or the output from a local build where it crashed? I just compiled the uninstaller and installer using the patches from bug 481815 and it did not crash for me. Also, it should not crash and I'd like to understand what is going on here.

Comment 23

[Justin Wood (:Callek)]

(In reply to Armen Zambrano G. [:armenzg] - Release Engineer from comment #20)
> Don't do those packages come already with the newer mozilla-build?

Many of them, no.

(In reply to Chris Cooper [:coop] from comment #19)
> To followup Callek's comments from IRC, the following packages will also
> need to be reinstalled after the new MozillaBuild is deployed:
> 
> buildbot-startup

* Needs modification of many files, we took static snapshots of some from mozillabuild 1.4 and modified them for logging/etc. We need to do these modifications against whichever mozbuild we deploy.

> mercurial

* Contrary to what ted thought, we use newer than is available in this mozbuild version. (Our version is 1.7.5)

> python-ssl

* Also pywin32 (no module yet) and py2.5 directly if needed (since installing mozbuild over again will wipe out the py2.5 in mozilla-build thats on the refimage)

> profilevars

* I think is obsolete with new mozbuild, need to verify.

> rename-vim-install

* iirc new mozbuild corrected paths by default for this, but it certainly won't hurt to redo this.

> twitsed_dumbwin32proc (if installed)

* completely obsolete with new buildbotve and twisted.

> yasm

* yes, needed.

BE WARY of http://mxr.mozilla.org/build/source/opsi-package-sources/hostkey-generator/CLIENT_DATA/hostkey.ins#8 as well, since new mozbuild does NOT use python2.5 and instead uses python, which means that if this goes to get run after the new mozbuild is installed as-is, it won't be able to find python.

Also if we don't replace python2.5 separately we need to update runslave.py (both in opsi and its canonical copy in puppet) because that points to python in the 2.5 versioned directory.

In current state as well (no py2.5) we also need to update |python-scripts-path| opsi package to point registry at the right location

But there is surely more to do if we change the python location like this as well :-)

Comment 24

[Chris Cooper [:coop] (he/him)]

MozillaBuild 1.6 is coming out soon (bug 631910), and I'd rather not deploy MozillaBuild twice. 

I'm going to file a separate bug for getting the latest version of NSIS installed in the interim, and take the dependent bugs with me over there.

Comment 25

[Robert Strong (they/them - no direct email)]

btw: I think the reason it is crashing is due to using the ansi version of the plugin bbondy was using and that upgrading NSIS isn't necessary.

Comment 26

[Justin Wood (:Callek)]

(In reply to Robert Strong [:rstrong] (do not email) from comment #25)
> btw: I think the reason it is crashing is due to using the ansi version of
> the plugin bbondy was using and that upgrading NSIS isn't necessary.

New mozillabuild in mozbuild uses both a newer NSIS than we have here, and there is a separate unicode NSIS version as well, so its quite possible it has to do with NSIS version as well.

Comment 27

[Robert Strong (they/them - no direct email)]

configure uses the latest available in MozillaBuild and it is using NSIS 2.33 which doesn't support ANSI plugins which is the plugin I reference in comment #25.

Comment 28

[Robert Strong (they/them - no direct email)]

(In reply to Chris Cooper [:coop] from comment #24)
> MozillaBuild 1.6 is coming out soon (bug 631910), and I'd rather not deploy
> MozillaBuild twice. 
> 
> I'm going to file a separate bug for getting the latest version of NSIS
> installed in the interim, and take the dependent bugs with me over there.
I don't think this is necessary as long as MozillaBuild is released sometime in the next couple of months. In summary, I think your time would better be spent on deploying MozillaBuild 1.6.

Comment 29

[Brian R. Bondy [:bbondy]]

The last build on the elm branch worked with all of the latest updates so I suspect something is wrong with a commit on the oak branch.  I think an upgrade of NSIS is not needed if it can save you time for now.

Comment 30

[Brian R. Bondy [:bbondy]]

I mean for the short term by the way, I'm not advocating in general against this upgrade when time permits.

Comment 31

[Justin Wood (:Callek)]

(In reply to Chris Cooper [:coop] from comment #19)

> If you think there are others, please let me know.

Also learned of simplejson needing to be installed again if we stay with python2.5 (Was part of the *old* buildbot 0.8.2 install)

Comment 32

[(not currently active) Ted Mielczarek]

FWIW, the "json" module in Python 2.6 is simplejson.

Comment 33

[Mike Taylor [:bear]]

one of the more handier code snippets I use to deal with the json module names:

try:
  # Python >= 2.6
  import json
except ImportError:
    # Python < 2.6
    import simplejson as json

Comment 34

[Chris AtLee [:catlee]]

(In reply to Mike Taylor [:bear] from comment #33)
> one of the more handier code snippets I use to deal with the json module
> names:
> 
> try:
>   # Python >= 2.6
>   import json
> except ImportError:
>     # Python < 2.6
>     import simplejson as json

I usually swap this around and prefer simplejson first, since it's much faster than the stock json.

</bikeshed>

Comment 35

[Rail Aliiev [:rail]]

Mozilla-build upgrades 7-zip to 4.65 which is buggy: see bug 711309 for the details. I think, this should block deploying Mozilla-Build 1.6.

Comment 36

[Chris Cooper [:coop] (he/him)]

(In reply to Rail Aliiev [:rail] from comment #35)
> Mozilla-build upgrades 7-zip to 4.65 which is buggy: see bug 711309 for the
> details. I think, this should block deploying Mozilla-Build 1.6.

We'll wait for a new version then, or tackle this as we rev the ref platform for the new builders.