549458 - Deploy MozillaBuild to XP and Win7 slaves & enable optimized unit tests on Win7 testing machines

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P4)

Description

[John Ford [:jhford] CET/CEST Berlin Time]

+++ This bug was initially created as a clone of Bug #549456 +++

Get the unittest suites running manually on both 10.5 and 10.6.

Comment 1

[John Ford [:jhford] CET/CEST Berlin Time]

I am working on the Windows 7 machine.

Comment 2

[John Ford [:jhford] CET/CEST Berlin Time]

There is an error in shutil when trying to run mochitests.  I am going to look at this again later.

Comment 3

[John Ford [:jhford] CET/CEST Berlin Time]

I am using talos-r3-w7-001 for the windows 7 tests.  I am marking a dep on the new r2 machines as I don't have hardware for the xp tests yet.

Comment 4

[alice nodelman [:alice] [:anode]]

talos-r3-w7-001 has been set aside for this work.

Please note when you are done with this box so that it can be put back in the production talos pool.

Comment 5

[alice nodelman [:alice] [:anode]]

Taking from jhford as he doesn't have time to work on it.

I believe that we are missing here is the steps necessary for installing hg on windows boxes.  Then we can come up with a rollout plan for that installation across the talos pool.

Comment 7

[Lukas Blakk [:lsblakk] use ?needinfo]

When getting this running in staging, please watch for something along the lines of bug 558240 - if rm -rf buildtools fails the first time a talos slave does a unittest build then we will need to know this before turning it on in production.

Comment 8

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

pushing to lsblakk, who is working on this while jhford is out on vacation.

Comment 9

[Lukas Blakk [:lsblakk] use ?needinfo]

installed Mercurial 1.4 on talos-r3-xp-001 from http://mercurial.berkwood.com/binaries/Mercurial-1.4.exe

talos-r3-w7-001 had mercurial on it already - installed by jhford, I assume.  I will need him to let me know which version from where so we can prepare OPSI for this.

Comment 10

[alice nodelman [:alice] [:anode]]

Taking.

Comment 11

[alice nodelman [:alice] [:anode]]

Using:

* https://developer.mozilla.org/En/Windows_Build_Prerequisites#MozillaBuild
Mozilla-build collection of tools for python25, hg, bash, etc
* http://sourceforge.net/projects/pywin32/files/
pywin32 extensions (first needed to run http://effbot.org/zone/python-register.htm) to register python25

Using slightly updated buildbotcustom that allows for setting an env to be used to all commands run (so that bash can be found).

Comment 12

[alice nodelman [:alice] [:anode]]

Installed the requirements from comment 11 onto talos-r3-xp-001/2 talos-r3-w7-001/2.  Will leave this running over the weekend and then do a review of the oranges next week.

Comment 13

[alice nodelman [:alice] [:anode]]

Attachment: buildbot custom support for win talos uni tests (WIP)

Comment 14

[alice nodelman [:alice] [:anode]]

Attachment: buildbot-config for unit test on win talos (WIP)

Comment 15

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 442749 [details] [diff] [review]
buildbot custom support for win talos uni tests (WIP)

looks good. i'm assuming you tested this env.

Comment 16

[Armen [:armenzg]]

I have unittests running on talos-staging-pool.

Let me rephrase the tool-chain requirements from previous comments.
Alice. jhford and lsblakk have a look to see what is *incorrect*.

(In reply to comment #9)
> installed Mercurial 1.4 on talos-r3-xp-001 from
> http://mercurial.berkwood.com/binaries/Mercurial-1.4.exe
> 
> talos-r3-w7-001 had mercurial on it already - installed by jhford, I assume.  I
> will need him to let me know which version from where so we can prepare OPSI
> for this.

(In reply to comment #11)
> Using:
> 
> * https://developer.mozilla.org/En/Windows_Build_Prerequisites#MozillaBuild
> Mozilla-build collection of tools for python25, hg, bash, etc
> * http://sourceforge.net/projects/pywin32/files/
> pywin32 extensions (first needed to run
> http://effbot.org/zone/python-register.htm) to register python25
> 
> Using slightly updated buildbotcustom that allows for setting an env to be used
> to all commands run (so that bash can be found).

(From bug 553339)
(In reply to comment #6)
> I am going to test
> http://bitbucket.org/tortoisehg/thg-winbuild/downloads/mercurial-1.5.1.msi on
> both xp and 7

Comment 17

[Armen [:armenzg]]

Attachment: buildbot custom support for win talos uni tests (WIP)

A small change to make it work for platforms that don't set up an 'env_name' by replacing ['env_name'] with .get('env_name') which returns None when non-existent and acceptable for MozillaEnvironments's get() call.

Comment 18

[Armen [:armenzg]]

I am looking into unzipping mozilla-build into the slaves (This will get us hg and bash). These are the steps to follow:

* ssh into slave
* cd Desktop
* wget http://staging-stage.build.mozilla.org/pub/mozilla.org/mozilla/libraries/win32/mozilla-build-1.4.zip
* unzip mozilla-build-1.4.zip -d C:\
* rm mozilla-build-1.4.zip

Any thoughts? Any inconveniences?

I could do this on this Thursday's downtime.

Comment 19

[Armen [:armenzg]]

Last week I did not manage to have enough time to get comment 18 done and I discovered another requirement.

Let me discuss the background and the plan.

Background:
* We need bash and mercurial on the talos windows slaves to run unit tests.
* Getting bash into the machine is not trivial.
* MB (MozillaBuild) already brings bash and mercurial.
* I need to run reftests-d2d on production before the end of the quarter.
* There is no OPSI on win7 machines to deploy MB unto them.

Plan:
* Unzip unto talos slaves MB since we can't use the installer
* The zipped MB will contain the python win32 extensions
* Through ssh I will connect to all talos slaves and wget a bat file to run
* This bat file will download mozilla-build-1.4.zip (with python win32 extensions) and register_python32.reg file
* Unzip MB to C:\mozilla-build
* Log in into each individual machines to merge register_python32.reg since I need elevated privileges which I can't acquire through ssh

How mozilla-build-1.4.zip gets generated:
* Download MB's installer and install it
* Register python win32 [1]
* Download python win32 extensions and install it
* Zip the whole thing and upload it to ftp

[1] https://wiki.mozilla.org/ReferencePlatforms/Win32#Move_to_Python_2.5
[2] http://staging-stage.build.mozilla.org/pub/mozilla.org/mozilla/libraries/win32/mozilla-build-1.4.zip

Comment 20

[Armen [:armenzg]]

Attachment: register_python25.reg - register mozilla-build's python and the python win32 extensions

Comment 21

[Armen [:armenzg]]

Attachment: deploy MB and download register_win32.reg

This should get MB with python2.5 win32 deployed to the windows talos slaves.

Separately I would have to go to each slave individually and register the .reg file which I can't do through ssh.

Comment 22

[Armen [:armenzg]]

Attachment: deploy MB

Comment 23

[Armen [:armenzg]]

Updated plan after chatting with bhearsum. We are not taking the register_pythonwin32.reg since we have not found out a good reason to take it.

Plan:
* ssh to all (or by sets) Windows talos-slaves using csshX
* wget https://bugzilla.mozilla.org/attachment.cgi?id=451652 -O run.bat
* run.bat (which does the following)
** this downloads mozilla-build.zip
** unzips mozilla-build.zip
** remove mozilla-build.zip
* done

Comment 24

[Armen [:armenzg]]

Attachment: deploy MB and python win32 dlls

This adds the last bit that I was missing and that I was not able to test until I actually got a clean slave.

BTW I am *not* putting the DLLs into C:/Windows/System32 (as the installer does) but in C:\mozilla-build\python25\Scripts (which is on the PATH) since cltbld does not have the permissions to do it.

Let me know if this works for you.

I already got clean runs on the clean slave I was talking about. I will let it run overnight to make sure.

Comment 25

[[e-mail] (:bhearsum)]

Comment on attachment 452072 [details]
deploy MB and python win32 dlls

 Does it actually work to drop the DLLs into scripts? I'm a bit surprised if it does....

Comment 26

[Armen [:armenzg]]

I just had to choose something that is on the PATH. If you prefer another location let me know.

An easy way to test this is to remove the DLLs and try to import win32file from python (C:\mozilla-build\python25\python.exe) and you will get a warning saying that it can't be found. As long as we put the DLLs somewhere on the path it should work.

Comment 27

[[e-mail] (:bhearsum)]

Comment on attachment 452072 [details]
deploy MB and python win32 dlls

Script looks fine.

I'd like to see mozilla-build-1.4.zip and python25win32dlls.zip move somewhere permanent, a staging system isn't too secure for long term storage. Ping me and we can figure something out.

Comment 28

[Armen [:armenzg]]

As per discussed on IRC we have stored these two packages into the ref-platforms cvs repo.

bm-xserve05:~/tmp/ref-platforms cltbld$ cvs add python25win32dlls.zip 
cvs add: scheduling file `python25win32dlls.zip' for addition
cvs add: use 'cvs commit' to add this file permanently
bm-xserve05:~/tmp/ref-platforms cltbld$ cvs add mozilla-build-1.4.zip 
cvs add: scheduling file `mozilla-build-1.4.zip' for addition
cvs add: use 'cvs commit' to add this file permanently
bm-xserve05:~/tmp/ref-platforms cltbld$ cvs ci -m "Bug 549458. Deplo MozillaBuild1.4 plus python extensions to talos windows machines. r=bhearsum"
RCS file: /mofo/ref-platforms/mozilla-build-1.4.zip,v
done
Checking in mozilla-build-1.4.zip;
/mofo/ref-platforms/mozilla-build-1.4.zip,v  <--  mozilla-build-1.4.zip
initial revision: 1.1
done
RCS file: /mofo/ref-platforms/python25win32dlls.zip,v
done
Checking in python25win32dlls.zip;
/mofo/ref-platforms/python25win32dlls.zip,v  <--  python25win32dlls.zip
initial revision: 1.1
done

Comment 30

[Armen [:armenzg]]

Attachment: [buildbot-configs] add unit tests to talos staging for Windows and enable d2d for Win7

Hi catlee,
You will see some white space noise but what this patch does is to allow different operating systems (xp and win7) to run different sets of test suites. For instance, we want reftest-d2d for win7 but not for winXp.
We also don't want to run any unit tests for WinXp since we currently can't.

Instead of "slave_platform" I would have wanted to use the term "os_target" which is more clear to me but I am keeping consistency.

Note that I also add "fedora", "fedora64", "leopard" and "snowleopard" inside of PLATFORM_UNITTEST_VARS.

In the following patch you can tell where the list of "slave_platforms" is used.

I already had this work tested and working (just to get reftest-d2d run regardless of winXp) but I had to change the patches to allow WinXP *not* to run the same tests as Win7. This patch already gets me the list of builders that I want but I will have to run it again through the weekend.

Please have a look at it and I will ask you for a formal review once I come back from the weekend.

Comment 31

[Armen [:armenzg]]

Attachment: [baking on staging] add reftests-d2d and allow different test suites for different target operating systems

Comment 32

[Armen [:armenzg]]

Status update wrt greenness/orangeness:

GREEN:
# WINNT 6.1 mozilla-central opt test mochitests-1/5
# WINNT 6.1 mozilla-central opt test mochitests-2/5
# WINNT 6.1 mozilla-central opt test mochitests-3/5
# WINNT 6.1 mozilla-central opt test crashtest
# WINNT 6.1 mozilla-central opt test jsreftest

HALF GREEN/ORANGE:
# WINNT 6.1 mozilla-central opt test mochitests-5/5

ORANGE:
# WINNT 6.1 mozilla-central opt test mochitests-4/5
# WINNT 6.1 mozilla-central opt test mochitest-other
# WINNT 6.1 mozilla-central opt test reftest
# WINNT 6.1 mozilla-central opt test reftest-d2d
# WINNT 6.1 mozilla-central opt test xpcshell

Comment 33

[Armen [:armenzg]]

Attachment: [buildbotcustom] add reftests-d2d and allow different test suites for different target operating systems

catlee it is all ready to be reviewed.

Comment 34

[Armen [:armenzg]]

Attachment: [talos-r3 configs] enable Windows unit tests

catlee this patch adds the same stuff as in attachment 452350 [details] [diff] [review].
I also disable Win64 debug unit tests since we can't yet run windows unit tests for debug builds.

This patch should get us running in production unit tests for the win7 slaves.

Comment 35

[Nick Thomas [:nthomas] (UTC+12)]

Where are the likes of 'Rev3 WINNT 6.1 x64 mozilla-central opt test xpcshell on 2010/06/23 09:52:54' on Firefox coming from now ? http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1277311974.1277312402.22859.gz&fulltext=1

Unless we know new tests are all green (or as orange as everything else) please take care to hide them on the tinderbox waterfalls as soon as they start reporting there. There was a bit of confusion when win7x64 results started turning up.

Comment 36

[Armen [:armenzg]]

That ain't me! I am working on the Win7 and XP slaves but not the Win7x64 machines.
I have not been the most careful in other occasions but I try to be now :)

Comment 37

[Armen [:armenzg]]

It seems that they have all been hidden.

Comment 38

[Armen [:armenzg]]

All XP and Win7 slaves do now have MozillaBuild 1.4 deployed except slaves w7-036 (down) and w7-40 (I don't know where it is).

talos-master02 has been reconfigured.

http://hg.mozilla.org/build/buildbot-configs/rev/13997fd0bef9
http://hg.mozilla.org/build/buildbot-configs/rev/b8671ea2863d

http://hg.mozilla.org/build/buildbotcustom/rev/ebfeb847c847

Comment 39

[Armen [:armenzg]]

I have hidden the following test suites:
* Rev3 WINNT 6.1 mozilla-central opt test mochitest-other
* Rev3 WINNT 6.1 mozilla-central opt test mochitests-4/5	
* Rev3 WINNT 6.1 mozilla-central opt test mochitests-5/5
* Rev3 WINNT 6.1 mozilla-central opt test reftest			
* Rev3 WINNT 6.1 mozilla-central opt test reftest-d2d
* Rev3 WINNT 6.1 mozilla-central opt test xpcshell

Comment 40

[[e-mail] (:bhearsum)]

Armen, did you update talos-r3-xp-ref and talos-r3-w7-ref? That'll need to happen, and IT will need to take new images.

Comment 41

[Armen [:armenzg]]

Done as well. Thanks for reminding me.

Comment 42

[Armen [:armenzg]]

This bug is done wrt to running unit tests on Windows 7 machines.

We will fix WinXP in bug 563036.

We will fix running unit tests on debug builds on bug 562459.

Comment 43

[Armen [:armenzg]]

I am cleaning up the dependencies of some of these unit tests on the minis bugs
and verifying their summaries. My apologies for the extra bugmail.

I am adjusting the summary to identify what actually happened on this bug.

In comment 23 we decided not to take register_pythonwin32.reg as there was no good reason to do so.

In comment 38 we deployed MozillaBuild to all XP and Win7 slaves. We also landed the required patches to enable optimized unit tests for Win7.

We are enabling XP debug & optimized unit tests on bug 614955.
We are enabling Win7 debug unit tests on bug 614956.