Deploy MozillaBuild 1.6 to buildslaves

RESOLVED WONTFIX

Status

Release Engineering
General
P3
normal
RESOLVED WONTFIX
8 years ago
5 years ago

People

(Reporter: rstrong, Assigned: coop)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [opsi][buildslaves])

Attachments

(1 obsolete attachment)

The current NSIS on the Tinderbox has a bug that affects silent installers. I've added a workaround for one of the bugs so this isn't a rush.
(Assignee)

Updated

8 years ago
Summary: Update tinderbox to the latest MozillaBuild which includes NSIS 2.46 Unicode → Deploy latest MozillaBuild (includes NSIS 2.46 Unicode) to buildslaves
Whiteboard: [opsi]

Updated

8 years ago
Priority: -- → P3
Whiteboard: [opsi] → [opsi][buildslaves]
Installed NSIS 2.46 Unicode on one try server build vm.

D:\tmp> nsis-2.46-Unicode-setup.exe /S /D=d:\mozilla-build\nsis-2.46u

A build is running now:

 http://build.mozillamessaging.com/buildbot/try/builders/WINNT%205.2%20try%20build/builds/118

Also checked in a PATH env change to our buildbotcustom repo:

 https://hg.mozilla.org/users/gozer_mozillamessaging.com/buildbotcustom-thunderbird-try/
Assignee: nobody → jhopkins
Should have commented on Thunderbird clone of this bug - sorry, bear!
Assignee: jhopkins → nobody
Whiteboard: [opsi][buildslaves] → [opsi][buildslaves][triagefollowup]
(Assignee)

Comment 3

7 years ago
Unless I'm reading this bug wrong, bug 675970 is blocked (eventually) by this, not the other way around.
Blocks: 675970
No longer depends on: 675970
(Assignee)

Updated

7 years ago
Assignee: nobody → coop
Whiteboard: [opsi][buildslaves][triagefollowup] → [opsi][buildslaves]
(Assignee)

Updated

7 years ago
Duplicate of this bug: 690894
(Assignee)

Comment 5

7 years ago
I've started trying this out in staging, but have run into a problem. 

The current version of MozillaBuild (1.5.1) prepends D:\mozilla-build\python (which is 2.6) to the path, while the previously installed version defaulted to D:\mozilla-build\python25. I don't think the python version change itself is bad, but the python2.6 install under D:\mozilla-build\python doesn't have the pywin32 extensions installed. Several of the releng tools (purge_builds.py, hgtool.py) rely on these extensions.

What's the best path forward here?

1) Simply pre-pend D:\mozilla-build\python25 to the path so the older python version is used.

2) Run an actual install of the 2.6 version of python included with MozillaBuild (the msi is included in the package), and then install the pywin32 extensions by hand.

3) Roll a new version of MozillaBuild that already has the pywin32 extensions installed.

#3 seems like the best plan to me. #1 is certainly the easiest, but we don't make any forward progress on the python front.
Status: NEW → ASSIGNED
OS: Windows 7 → Windows Server 2003
Priority: P3 → P2
I like #3 too, except I'd package python2.7 with the pywin32 extensions.
We already updated Python to 2.7 in mozilla-build trunk (bug 578584). bug 496535 had been investigating adding pywin32, but I got stuck because it installed stuff to windows/system32. A while ago I talked to mhammond, who said that that stuff isn't important unless you want to implement ActiveX controls with Python or something crazy like that, so we could probably resurrect that.
(Assignee)

Comment 8

7 years ago
(In reply to Ted Mielczarek [:ted, :luser] from comment #7)
> We already updated Python to 2.7 in mozilla-build trunk (bug 578584). bug
> 496535 had been investigating adding pywin32, but I got stuck because it
> installed stuff to windows/system32. A while ago I talked to mhammond, who
> said that that stuff isn't important unless you want to implement ActiveX
> controls with Python or something crazy like that, so we could probably
> resurrect that.

I think we only need the file APIs, so we're probably good there. I'll reopen bug 496535.

How long before such a thing would be ready, i.e. would it be best to massage the PATH and stick with 2.5 for the short-term? 

I'm guessing "yes" since it looks like I'm going to need to massage the path anyway, since the vim installer ends up earlier in the path than /bin/install at present:

make[7]: Leaving directory `/e/builds/moz2_slave/m-cen-w32/build/obj-firefox/build/win32/crashinjectdll'
mkdir -p ../../dist/bin
install --preserve-timestamps "/d/msvs8/VC/redist/x86/Microsoft.VC80.CRT"/Microsoft.VC80.CRT.manifest "/d/msvs8/VC/redist/x86/Microsoft.VC80.CRT"/msvcm80.dll "/d/msvs8/VC/redist/x86/Microsoft.VC80.CRT"/msvcp80.dll "/d/msvs8/VC/redist/x86/Microsoft.VC80.CRT"/msvcr80.dll ../../dist/bin
This program sets up the installation of Vim 7.2

Inspecting system...

ERROR: failed to rename vim.exe to vim.exx: No error
We fixed that in 1.5.1 or something, that shouldn't be a problem.
(Assignee)

Comment 10

7 years ago
Created attachment 573930 [details] [diff] [review]
Update slaves to MozillaBuild 1.5.1

This patch targets option 1 from comment #5.

There are 3 things going on is this patch:

1) The MozillaBuild version # is now in the window title, so we need to account for that during install.

2) I copied the version of profile-extravars.sh out of hg tip and made a slight addition to add python25 back to the front of the PATH to keep our releng tools working for now (due to pywin32).

3) I delete a file, profile-extrapaths.sh, that I've noticed lingering on some slaves from a previous MozillaBuild install. This file was making the PATH longer than it needed to be.

Hopefully this won't interfere with work for bug 602908, but at the same time, hopefully we'll have bug 496535 soon as well.
Attachment #573930 - Flags: review?(catlee)
(In reply to Chris Cooper [:coop] from comment #10)
> 3) I delete a file, profile-extrapaths.sh, that I've noticed lingering on
> some slaves from a previous MozillaBuild install. This file was making the
> PATH longer than it needed to be.

Yup, that should be fine. This was used to workaround a bug in earlier versions of MB.
Comment on attachment 573930 [details] [diff] [review]
Update slaves to MozillaBuild 1.5.1

Review of attachment 573930 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good.

Is there any reason to deploy though if it doesn't have python2.6/2.7?
Attachment #573930 - Flags: review?(catlee) → review+
(Assignee)

Comment 13

7 years ago
(In reply to Chris AtLee [:catlee] from comment #12) 
> Is there any reason to deploy though if it doesn't have python2.6/2.7?

It *does* have python 2.6. Any build process that wanted/needed to use 2.6 could reference it directly, 2.5 will just be ahead of it in the path.
(Assignee)

Updated

7 years ago
Blocks: 702243
It seems that Ehsan is hitting some msys (or nsis - not sure) bug with the current mozilla-build version in bug 704475.
Blocks: 704475

Comment 15

7 years ago
The old NSIS that we have on the bots is crashing when compiler our uninstaller script with the changes being made in bug 481815.  Do we have a timeline on how soon this can be deployed?

Thanks!
Blocks: 481815, 307181
No longer blocks: 704475

Updated

7 years ago
Blocks: 704475
(Assignee)

Comment 16

7 years ago
It will be sometime this week. We were waiting on the 8.0.1 chemspill, and now we're due for the next beta (9.0b3). Once builds for 9.0b3 are done, we should be safe to deploy.

Comment 17

7 years ago
(In reply to Chris Cooper [:coop] from comment #16)
> It will be sometime this week. We were waiting on the 8.0.1 chemspill, and
> now we're due for the next beta (9.0b3). Once builds for 9.0b3 are done, we
> should be safe to deploy.

Do you know when 9.0b3 is targeted to go to build?
(Assignee)

Comment 18

7 years ago
(In reply to Ehsan Akhgari [:ehsan] from comment #17) 
> Do you know when 9.0b3 is targeted to go to build?

Later today.
(Assignee)

Comment 19

7 years ago
To followup Callek's comments from IRC, the following packages will also need to be reinstalled after the new MozillaBuild is deployed:

buildbot-startup
mercurial
python-ssl
rename-vim-install
twitsed_dumbwin32proc (if installed)
yasm

If you think there are others, please let me know.
Don't do those packages come already with the newer mozilla-build?

Did your testing already show that installing your package was good enough? I'm not sure we need all of these packages but I could be wrong.

On another note, twisted_dumbwin32proc could be now be bitrotten (see bug 690232). With the newer version of buildbot/twisted supposedly we don't need to deploy such patch but I am not sure of that as I wasn't involved in that work and that's what I understand from dustin's comments in bug 690232.
(In reply to Chris Cooper [:coop] from comment #19)
> mercurial
> rename-vim-install
> yasm

I would think these would be obviated by the new MozillaBuild package.
(In reply to Ehsan Akhgari [:ehsan] from comment #15)
> The old NSIS that we have on the bots is crashing when compiler our
> uninstaller script with the changes being made in bug 481815.  Do we have a
> timeline on how soon this can be deployed?
Ehsan, can you provide a link to a log with the crash or the output from a local build where it crashed? I just compiled the uninstaller and installer using the patches from bug 481815 and it did not crash for me. Also, it should not crash and I'd like to understand what is going on here.
(In reply to Armen Zambrano G. [:armenzg] - Release Engineer from comment #20)
> Don't do those packages come already with the newer mozilla-build?

Many of them, no.

(In reply to Chris Cooper [:coop] from comment #19)
> To followup Callek's comments from IRC, the following packages will also
> need to be reinstalled after the new MozillaBuild is deployed:
> 
> buildbot-startup

* Needs modification of many files, we took static snapshots of some from mozillabuild 1.4 and modified them for logging/etc. We need to do these modifications against whichever mozbuild we deploy.

> mercurial

* Contrary to what ted thought, we use newer than is available in this mozbuild version. (Our version is 1.7.5)

> python-ssl

* Also pywin32 (no module yet) and py2.5 directly if needed (since installing mozbuild over again will wipe out the py2.5 in mozilla-build thats on the refimage)

> profilevars

* I think is obsolete with new mozbuild, need to verify.

> rename-vim-install

* iirc new mozbuild corrected paths by default for this, but it certainly won't hurt to redo this.

> twitsed_dumbwin32proc (if installed)

* completely obsolete with new buildbotve and twisted.

> yasm

* yes, needed.

BE WARY of http://mxr.mozilla.org/build/source/opsi-package-sources/hostkey-generator/CLIENT_DATA/hostkey.ins#8 as well, since new mozbuild does NOT use python2.5 and instead uses python, which means that if this goes to get run after the new mozbuild is installed as-is, it won't be able to find python.

Also if we don't replace python2.5 separately we need to update runslave.py (both in opsi and its canonical copy in puppet) because that points to python in the 2.5 versioned directory.

In current state as well (no py2.5) we also need to update |python-scripts-path| opsi package to point registry at the right location

But there is surely more to do if we change the python location like this as well :-)
(Assignee)

Updated

7 years ago
Attachment #573930 - Attachment is obsolete: true
(Assignee)

Comment 24

7 years ago
MozillaBuild 1.6 is coming out soon (bug 631910), and I'd rather not deploy MozillaBuild twice. 

I'm going to file a separate bug for getting the latest version of NSIS installed in the interim, and take the dependent bugs with me over there.
Depends on: 631910
(Assignee)

Updated

7 years ago
No longer blocks: 307181, 481815, 592414, 675970, 702243, 704475
Status: ASSIGNED → NEW
Priority: P2 → P3
Summary: Deploy latest MozillaBuild (includes NSIS 2.46 Unicode) to buildslaves → Deploy MozillaBuild 1.6 to buildslaves
btw: I think the reason it is crashing is due to using the ansi version of the plugin bbondy was using and that upgrading NSIS isn't necessary.
(In reply to Robert Strong [:rstrong] (do not email) from comment #25)
> btw: I think the reason it is crashing is due to using the ansi version of
> the plugin bbondy was using and that upgrading NSIS isn't necessary.

New mozillabuild in mozbuild uses both a newer NSIS than we have here, and there is a separate unicode NSIS version as well, so its quite possible it has to do with NSIS version as well.
configure uses the latest available in MozillaBuild and it is using NSIS 2.33 which doesn't support ANSI plugins which is the plugin I reference in comment #25.
(Assignee)

Updated

7 years ago
Blocks: 702243
(In reply to Chris Cooper [:coop] from comment #24)
> MozillaBuild 1.6 is coming out soon (bug 631910), and I'd rather not deploy
> MozillaBuild twice. 
> 
> I'm going to file a separate bug for getting the latest version of NSIS
> installed in the interim, and take the dependent bugs with me over there.
I don't think this is necessary as long as MozillaBuild is released sometime in the next couple of months. In summary, I think your time would better be spent on deploying MozillaBuild 1.6.
The last build on the elm branch worked with all of the latest updates so I suspect something is wrong with a commit on the oak branch.  I think an upgrade of NSIS is not needed if it can save you time for now.
I mean for the short term by the way, I'm not advocating in general against this upgrade when time permits.
(In reply to Chris Cooper [:coop] from comment #19)

> If you think there are others, please let me know.

Also learned of simplejson needing to be installed again if we stay with python2.5 (Was part of the *old* buildbot 0.8.2 install)

Updated

7 years ago
Depends on: 705403
FWIW, the "json" module in Python 2.6 is simplejson.

Comment 33

7 years ago
one of the more handier code snippets I use to deal with the json module names:

try:
  # Python >= 2.6
  import json
except ImportError:
    # Python < 2.6
    import simplejson as json
(In reply to Mike Taylor [:bear] from comment #33)
> one of the more handier code snippets I use to deal with the json module
> names:
> 
> try:
>   # Python >= 2.6
>   import json
> except ImportError:
>     # Python < 2.6
>     import simplejson as json

I usually swap this around and prefer simplejson first, since it's much faster than the stock json.

</bikeshed>
(Assignee)

Updated

7 years ago
No longer blocks: 702243
Mozilla-build upgrades 7-zip to 4.65 which is buggy: see bug 711309 for the details. I think, this should block deploying Mozilla-Build 1.6.
(Assignee)

Comment 36

6 years ago
(In reply to Rail Aliiev [:rail] from comment #35)
> Mozilla-build upgrades 7-zip to 4.65 which is buggy: see bug 711309 for the
> details. I think, this should block deploying Mozilla-Build 1.6.

We'll wait for a new version then, or tackle this as we rev the ref platform for the new builders.
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → WONTFIX
Depends on: 754810
No longer depends on: 754810
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.