Closed Bug 549458 Opened 10 years ago Closed 9 years ago

Deploy MozillaBuild to XP and Win7 slaves & enable optimized unit tests on Win7 testing machines

Categories

(Release Engineering :: General, defect, P4)

x86
Windows 7
defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: jhford, Assigned: armenzg)

References

Details

Attachments

(5 files, 6 obsolete files)

562 bytes, text/plain
Details
400 bytes, text/plain
bhearsum
: review+
Details
9.63 KB, patch
catlee
: review+
armenzg
: checked-in+
Details | Diff | Splinter Review
7.55 KB, patch
catlee
: review+
armenzg
: checked-in+
Details | Diff | Splinter Review
11.66 KB, patch
catlee
: review+
armenzg
: checked-in+
Details | Diff | Splinter Review
+++ This bug was initially created as a clone of Bug #549456 +++

Get the unittest suites running manually on both 10.5 and 10.6.
No longer depends on: 549456
I am working on the Windows 7 machine.
Assignee: nobody → jhford
Priority: -- → P2
There is an error in shutil when trying to run mochitests.  I am going to look at this again later.
Priority: P2 → P3
I am using talos-r3-w7-001 for the windows 7 tests.  I am marking a dep on the new r2 machines as I don't have hardware for the xp tests yet.
Depends on: 549738
talos-r3-w7-001 has been set aside for this work.

Please note when you are done with this box so that it can be put back in the production talos pool.
Taking from jhford as he doesn't have time to work on it.

I believe that we are missing here is the steps necessary for installing hg on windows boxes.  Then we can come up with a rollout plan for that installation across the talos pool.
Assignee: jhford → nobody
Duplicate of this bug: 551048
When getting this running in staging, please watch for something along the lines of bug 558240 - if rm -rf buildtools fails the first time a talos slave does a unittest build then we will need to know this before turning it on in production.
pushing to lsblakk, who is working on this while jhford is out on vacation.
Assignee: nobody → lsblakk
installed Mercurial 1.4 on talos-r3-xp-001 from http://mercurial.berkwood.com/binaries/Mercurial-1.4.exe

talos-r3-w7-001 had mercurial on it already - installed by jhford, I assume.  I will need him to let me know which version from where so we can prepare OPSI for this.
Summary: Get unittests running manually on Windows XP and 7 → Get unittests running on Windows XP and 7 in talos-staging
No longer blocks: 548768
Assignee: lsblakk → anodelman
Priority: P3 → P2
Using:

* https://developer.mozilla.org/En/Windows_Build_Prerequisites#MozillaBuild
Mozilla-build collection of tools for python25, hg, bash, etc
* http://sourceforge.net/projects/pywin32/files/
pywin32 extensions (first needed to run http://effbot.org/zone/python-register.htm) to register python25

Using slightly updated buildbotcustom that allows for setting an env to be used to all commands run (so that bash can be found).
Installed the requirements from comment 11 onto talos-r3-xp-001/2 talos-r3-w7-001/2.  Will leave this running over the weekend and then do a review of the oranges next week.
Depends on: 562459
Comment on attachment 442749 [details] [diff] [review]
buildbot custom support for win talos uni tests (WIP)

looks good. i'm assuming you tested this env.
Attachment #442749 - Flags: feedback?(lsblakk) → feedback+
Assignee: anodelman → joduinn
Assignee: joduinn → armenzg
Priority: P2 → P3
I have unittests running on talos-staging-pool.

Let me rephrase the tool-chain requirements from previous comments.
Alice. jhford and lsblakk have a look to see what is *incorrect*.

(In reply to comment #9)
> installed Mercurial 1.4 on talos-r3-xp-001 from
> http://mercurial.berkwood.com/binaries/Mercurial-1.4.exe
> 
> talos-r3-w7-001 had mercurial on it already - installed by jhford, I assume.  I
> will need him to let me know which version from where so we can prepare OPSI
> for this.

(In reply to comment #11)
> Using:
> 
> * https://developer.mozilla.org/En/Windows_Build_Prerequisites#MozillaBuild
> Mozilla-build collection of tools for python25, hg, bash, etc
> * http://sourceforge.net/projects/pywin32/files/
> pywin32 extensions (first needed to run
> http://effbot.org/zone/python-register.htm) to register python25
> 
> Using slightly updated buildbotcustom that allows for setting an env to be used
> to all commands run (so that bash can be found).

(From bug 553339)
(In reply to comment #6)
> I am going to test
> http://bitbucket.org/tortoisehg/thg-winbuild/downloads/mercurial-1.5.1.msi on
> both xp and 7
Priority: P3 → P2
A small change to make it work for platforms that don't set up an 'env_name' by replacing ['env_name'] with .get('env_name') which returns None when non-existent and acceptable for MozillaEnvironments's get() call.
Attachment #442749 - Attachment is obsolete: true
Status: NEW → ASSIGNED
Priority: P2 → P3
I am looking into unzipping mozilla-build into the slaves (This will get us hg and bash). These are the steps to follow:

* ssh into slave
* cd Desktop
* wget http://staging-stage.build.mozilla.org/pub/mozilla.org/mozilla/libraries/win32/mozilla-build-1.4.zip
* unzip mozilla-build-1.4.zip -d C:\
* rm mozilla-build-1.4.zip

Any thoughts? Any inconveniences?

I could do this on this Thursday's downtime.
Last week I did not manage to have enough time to get comment 18 done and I discovered another requirement.

Let me discuss the background and the plan.

Background:
* We need bash and mercurial on the talos windows slaves to run unit tests.
* Getting bash into the machine is not trivial.
* MB (MozillaBuild) already brings bash and mercurial.
* I need to run reftests-d2d on production before the end of the quarter.
* There is no OPSI on win7 machines to deploy MB unto them.

Plan:
* Unzip unto talos slaves MB since we can't use the installer
* The zipped MB will contain the python win32 extensions
* Through ssh I will connect to all talos slaves and wget a bat file to run
* This bat file will download mozilla-build-1.4.zip (with python win32 extensions) and register_python32.reg file
* Unzip MB to C:\mozilla-build
* Log in into each individual machines to merge register_python32.reg since I need elevated privileges which I can't acquire through ssh

How mozilla-build-1.4.zip gets generated:
* Download MB's installer and install it
* Register python win32 [1]
* Download python win32 extensions and install it
* Zip the whole thing and upload it to ftp

[1] https://wiki.mozilla.org/ReferencePlatforms/Win32#Move_to_Python_2.5
[2] http://staging-stage.build.mozilla.org/pub/mozilla.org/mozilla/libraries/win32/mozilla-build-1.4.zip
This should get MB with python2.5 win32 deployed to the windows talos slaves.

Separately I would have to go to each slave individually and register the .reg file which I can't do through ssh.
Blocks: 571571
Attached file deploy MB (obsolete) —
Attachment #451376 - Attachment is obsolete: true
Updated plan after chatting with bhearsum. We are not taking the register_pythonwin32.reg since we have not found out a good reason to take it.

Plan:
* ssh to all (or by sets) Windows talos-slaves using csshX
* wget https://bugzilla.mozilla.org/attachment.cgi?id=451652 -O run.bat
* run.bat (which does the following)
** this downloads mozilla-build.zip
** unzips mozilla-build.zip
** remove mozilla-build.zip
* done
No longer blocks: 571571
This adds the last bit that I was missing and that I was not able to test until I actually got a clean slave.

BTW I am *not* putting the DLLs into C:/Windows/System32 (as the installer does) but in C:\mozilla-build\python25\Scripts (which is on the PATH) since cltbld does not have the permissions to do it.

Let me know if this works for you.

I already got clean runs on the clean slave I was talking about. I will let it run overnight to make sure.
Attachment #451652 - Attachment is obsolete: true
Attachment #452072 - Flags: review?(bhearsum)
Comment on attachment 452072 [details]
deploy MB and python win32 dlls

 Does it actually work to drop the DLLs into scripts? I'm a bit surprised if it does....
I just had to choose something that is on the PATH. If you prefer another location let me know.

An easy way to test this is to remove the DLLs and try to import win32file from python (C:\mozilla-build\python25\python.exe) and you will get a warning saying that it can't be found. As long as we put the DLLs somewhere on the path it should work.
Comment on attachment 452072 [details]
deploy MB and python win32 dlls

Script looks fine.

I'd like to see mozilla-build-1.4.zip and python25win32dlls.zip move somewhere permanent, a staging system isn't too secure for long term storage. Ping me and we can figure something out.
Attachment #452072 - Flags: review?(bhearsum) → review+
Priority: P3 → P2
As per discussed on IRC we have stored these two packages into the ref-platforms cvs repo.

bm-xserve05:~/tmp/ref-platforms cltbld$ cvs add python25win32dlls.zip 
cvs add: scheduling file `python25win32dlls.zip' for addition
cvs add: use 'cvs commit' to add this file permanently
bm-xserve05:~/tmp/ref-platforms cltbld$ cvs add mozilla-build-1.4.zip 
cvs add: scheduling file `mozilla-build-1.4.zip' for addition
cvs add: use 'cvs commit' to add this file permanently
bm-xserve05:~/tmp/ref-platforms cltbld$ cvs ci -m "Bug 549458. Deplo MozillaBuild1.4 plus python extensions to talos windows machines. r=bhearsum"
RCS file: /mofo/ref-platforms/mozilla-build-1.4.zip,v
done
Checking in mozilla-build-1.4.zip;
/mofo/ref-platforms/mozilla-build-1.4.zip,v  <--  mozilla-build-1.4.zip
initial revision: 1.1
done
RCS file: /mofo/ref-platforms/python25win32dlls.zip,v
done
Checking in python25win32dlls.zip;
/mofo/ref-platforms/python25win32dlls.zip,v  <--  python25win32dlls.zip
initial revision: 1.1
done
Duplicate of this bug: 553339
Hi catlee,
You will see some white space noise but what this patch does is to allow different operating systems (xp and win7) to run different sets of test suites. For instance, we want reftest-d2d for win7 but not for winXp.
We also don't want to run any unit tests for WinXp since we currently can't.

Instead of "slave_platform" I would have wanted to use the term "os_target" which is more clear to me but I am keeping consistency.

Note that I also add "fedora", "fedora64", "leopard" and "snowleopard" inside of PLATFORM_UNITTEST_VARS.

In the following patch you can tell where the list of "slave_platforms" is used.

I already had this work tested and working (just to get reftest-d2d run regardless of winXp) but I had to change the patches to allow WinXP *not* to run the same tests as Win7. This patch already gets me the list of builders that I want but I will have to run it again through the weekend.

Please have a look at it and I will ask you for a formal review once I come back from the weekend.
Attachment #442751 - Attachment is obsolete: true
Attachment #452350 - Flags: feedback?(catlee)
Attachment #442751 - Flags: feedback?(lsblakk)
Blocks: 571571
Status update wrt greenness/orangeness:

GREEN:
# WINNT 6.1 mozilla-central opt test mochitests-1/5
# WINNT 6.1 mozilla-central opt test mochitests-2/5
# WINNT 6.1 mozilla-central opt test mochitests-3/5
# WINNT 6.1 mozilla-central opt test crashtest
# WINNT 6.1 mozilla-central opt test jsreftest

HALF GREEN/ORANGE:
# WINNT 6.1 mozilla-central opt test mochitests-5/5

ORANGE:
# WINNT 6.1 mozilla-central opt test mochitests-4/5
# WINNT 6.1 mozilla-central opt test mochitest-other
# WINNT 6.1 mozilla-central opt test reftest
# WINNT 6.1 mozilla-central opt test reftest-d2d
# WINNT 6.1 mozilla-central opt test xpcshell
Attachment #452350 - Attachment description: [baking on staging] add unit tests to talos staging for Windows and enable d2d for Win7 → [buildbot-configs] add unit tests to talos staging for Windows and enable d2d for Win7
Attachment #452350 - Flags: feedback?(catlee) → review?(catlee)
Attachment #452351 - Attachment is obsolete: true
Attachment #452351 - Flags: feedback?(catlee)
Attachment #452350 - Flags: review?(catlee) → review+
catlee this patch adds the same stuff as in attachment 452350 [details] [diff] [review].
I also disable Win64 debug unit tests since we can't yet run windows unit tests for debug builds.

This patch should get us running in production unit tests for the win7 slaves.
Attachment #453396 - Flags: review?(catlee)
Summary: Get unittests running on Windows XP and 7 in talos-staging → Get unit tests running on Windows XP and Windows 7 rev3 machines
Attachment #453396 - Flags: review?(catlee) → review+
Attachment #452817 - Flags: review?(catlee) → review+
Where are the likes of 'Rev3 WINNT 6.1 x64 mozilla-central opt test xpcshell on 2010/06/23 09:52:54' on Firefox coming from now ? http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1277311974.1277312402.22859.gz&fulltext=1

Unless we know new tests are all green (or as orange as everything else) please take care to hide them on the tinderbox waterfalls as soon as they start reporting there. There was a bit of confusion when win7x64 results started turning up.
That ain't me! I am working on the Win7 and XP slaves but not the Win7x64 machines.
I have not been the most careful in other occasions but I try to be now :)
It seems that they have all been hidden.
All XP and Win7 slaves do now have MozillaBuild 1.4 deployed except slaves w7-036 (down) and w7-40 (I don't know where it is).

talos-master02 has been reconfigured.

http://hg.mozilla.org/build/buildbot-configs/rev/13997fd0bef9
http://hg.mozilla.org/build/buildbot-configs/rev/b8671ea2863d

http://hg.mozilla.org/build/buildbotcustom/rev/ebfeb847c847
Attachment #452350 - Flags: checked-in+
Attachment #452817 - Flags: checked-in+
Attachment #453396 - Flags: checked-in+
No longer depends on: 562459
I have hidden the following test suites:
* Rev3 WINNT 6.1 mozilla-central opt test mochitest-other
* Rev3 WINNT 6.1 mozilla-central opt test mochitests-4/5	
* Rev3 WINNT 6.1 mozilla-central opt test mochitests-5/5
* Rev3 WINNT 6.1 mozilla-central opt test reftest			
* Rev3 WINNT 6.1 mozilla-central opt test reftest-d2d
* Rev3 WINNT 6.1 mozilla-central opt test xpcshell
No longer blocks: 571571
Armen, did you update talos-r3-xp-ref and talos-r3-w7-ref? That'll need to happen, and IT will need to take new images.
Done as well. Thanks for reminding me.
Depends on: 574417
Blocks: 574444
This bug is done wrt to running unit tests on Windows 7 machines.

We will fix WinXP in bug 563036.

We will fix running unit tests on debug builds on bug 562459.
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Priority: P2 → P4
Resolution: --- → FIXED
Whiteboard: [XP left]
Blocks: 614955
Blocks: 614956
I am cleaning up the dependencies of some of these unit tests on the minis bugs
and verifying their summaries. My apologies for the extra bugmail.

I am adjusting the summary to identify what actually happened on this bug.

In comment 23 we decided not to take register_pythonwin32.reg as there was no good reason to do so.

In comment 38 we deployed MozillaBuild to all XP and Win7 slaves. We also landed the required patches to enable optimized unit tests for Win7.

We are enabling XP debug & optimized unit tests on bug 614955.
We are enabling Win7 debug unit tests on bug 614956.
Summary: Get unit tests running on Windows XP and Windows 7 rev3 machines → Deploy MozillaBuild to XP and Win7 slaves & enable optimized unit tests on Win7 testing machines
Whiteboard: [XP left]
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.