798629 - Update MozillaBuild to include the things the releng builder images install over the top of the default MozillaBuild release

Status: RESOLVED FIXED

(Firefox Build System :: MozillaBuild, task)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

I brought this up in IRC, but it's probably better handled in a bug.

https://wiki.mozilla.org/ReferencePlatforms says to install older versions of MozillaBuild (1.5.1 for win64), and then install parts of mozillabuild *over* it - Python-2.7.3, YASM, and gvim.

This seems redundant, and more importantly it introduces an ordering dependency for installing these using automated tools, which is a bit hard to enforce with group policy.

Can we build a version of MozillaBuild that has what a win64 builder should have (including updated Python), and just install that?

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

Related: bug 791486 - but that's a much more substantial change.

If possible, I'd like to do 1.6.2 that comes out to essentially what's installed on the current win64 boxes - python-2.7.3, yasm-1.1.0, and proper settings for python under Wow6432Node:
[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Python\Pythoncore\2.6\InstallPath]
@="C:\\mozilla-build\\python"

1.7 can remain a separate (and riskier) project..

Comment 2

[(not currently active) Ted Mielczarek]

MozillaBuild doesn't currently touch the registry, so I don't think we'd add that, FWIW. I'd be happy to ship a .reg file in MozillaBuild that could be optionally applied after installation if that'd make life easier.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

That would be cool.  We can wrap it into the MSI, but it will be nice to have that described as "run this REG file" rather than enumerating specific regkeys and values.

Comment 4

[(not currently active) Ted Mielczarek]

I spun a test build to get people a MozillaBuild with hg 2.5.1:
http://people.mozilla.com/~tmielczarek/MozillaBuildSetup1.7pre.exe

The only changes in this build over 1.6.1 are hg 2.5.1, Python 2.7.3, and bug 807061.

If we need an updated YASM that should be easy, what version are you installing on the slaves?

The registry file we talked about should be simple as well.

What's the gvim you're installing? That shouldn't be anything necessary for the build, certainly. Is that just for RelEng convenience?

Comment 5

[Armen [:armenzg]]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #4)
> I spun a test build to get people a MozillaBuild with hg 2.5.1:
> http://people.mozilla.com/~tmielczarek/MozillaBuildSetup1.7pre.exe
> 
> The only changes in this build over 1.6.1 are hg 2.5.1, Python 2.7.3, and
> bug 807061.
> 
> If we need an updated YASM that should be easy, what version are you
> installing on the slaves?
> 

This is what we do on the win6 slaves:
https://wiki.mozilla.org/ReferencePlatforms/Win64#Add_yasm.exe
 wget http://www.tortall.net/projects/yasm/releases/vsyasm-1.1.0-win64.zip
 unzip vsyasm-1.1.0-win64.zip
 mv vsyasm.exe /c/mozilla-build/msys/bin/yasm.exe

> The registry file we talked about should be simple as well.
> 
> What's the gvim you're installing? That shouldn't be anything necessary for
> the build, certainly. Is that just for RelEng convenience?

Just for convenience.

Comment 6

[(not currently active) Ted Mielczarek]

Dustin:
http://people.mozilla.com/~tmielczarek/MozillaBuildSetup1.7pre.exe

I whipped up a quick patch to put a "register-python.reg" file in the root of the MozillaBuild install directory. Can you see if that does what you need?

I'll see if I can find time to update YASM. We already ship a vim in MozillaBuild, I don't think I'm going to spend time fiddling with that.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

This is great news!  I'll take a look at the zip file on Monday.

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

The reg file looks great - thanks!

What do you think about installing pywin32 into python?  We're currently doing it separately, and it's a mess.

Sorry to keep piling things into this bug - it's been open a long time, and how this all works has been a long learning process for me :)

Comment 9

[Armen [:armenzg]]

I have removed the pywin32 requirement for our Windows unit tests (bug 845973).

talos still has it but with mozharness we can install it into a Virutalenv.
We will be switching into it eventually.

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

Armen, when that's ready, will you file a bug so we can rip out the mess that's installing it now? :)  Thanks!

Comment 11

[(not currently active) Ted Mielczarek]

pywin32 is kind of a pain. We've looked into it before (bug 496535).

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

^^ upvote for understatement of the year

Comment 13

[Armen [:armenzg]]

(In reply to Dustin J. Mitchell [:dustin] from comment #10)
> Armen, when that's ready, will you file a bug so we can rip out the mess
> that's installing it now? :)  Thanks!

bug 713055 is the one to follow for that :) It will be a while.

Comment 14

[Ed Morley [:emorley]]

Clarifying summary - please adjust further if I've mis-understood this bug! :-)

Comment 15

[Armen [:armenzg]]

We don't have a need for pywin32 these days AFAIK.

Anything left in here?

Comment 16

[(not currently active) Ted Mielczarek]

Per your comment 5 we should still update YASM.

Comment 17

[Ryan VanderMeulen [:RyanVM]]

We already did in bug 791513 (shipped in 1.8)

Comment 18

[(not currently active) Ted Mielczarek]

Sounds like we fixed all this then.