Closed Bug 557918 Opened 10 years ago Closed 9 years ago

Enable unit tests on minis and disable them on builders

Categories

(Release Engineering :: General, defect, P4)

All
Linux
defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: armenzg, Assigned: armenzg)

References

Details

Attachments

(11 files, 7 obsolete files)

6.67 KB, patch
lsblakk
: review+
lsblakk
: checked-in+
Details | Diff | Splinter Review
8.81 KB, patch
catlee
: review+
Details | Diff | Splinter Review
2.32 KB, patch
armenzg
: review+
armenzg
: checked-in+
Details | Diff | Splinter Review
1.29 KB, patch
coop
: review+
armenzg
: checked-in+
Details | Diff | Splinter Review
2.58 KB, patch
coop
: review+
armenzg
: checked-in+
Details | Diff | Splinter Review
54.95 KB, patch
armenzg
: review+
armenzg
: checked-in+
Details | Diff | Splinter Review
10.19 KB, patch
coop
: review+
armenzg
: checked-in+
Details | Diff | Splinter Review
1.72 KB, patch
coop
: review+
armenzg
: checked-in+
Details | Diff | Splinter Review
13.55 KB, patch
coop
: review+
armenzg
: checked-in+
Details | Diff | Splinter Review
4.87 KB, patch
catlee
: review+
armenzg
: checked-in+
Details | Diff | Splinter Review
1.02 KB, patch
coop
: review+
Details | Diff | Splinter Review
I just want to know if this is all we need to enable on our side to get the unit tests running on the Fedora slaves.
Attachment #437674 - Flags: feedback?(lsblakk)
Assignee: nobody → armenzg
Priority: -- → P2
Comment on attachment 437674 [details] [diff] [review]
enable unit tests on Fedora talos slaves for mozilla-central

Yes, that's what you need - why are you moving the PLATFORM_UNITTEST_VARS?
Comment on attachment 437674 [details] [diff] [review]
enable unit tests on Fedora talos slaves for mozilla-central

Sweet!

I moved PLATFORM_UNITTEST_VARS down just because I make reference to UNITTEST_SUITES inside of it and, therefore, had to declare UNITTEST_SUITES before it. No other reason than that.
Attachment #437674 - Attachment description: [WIP] enable unit tests on Fedora talos slaves for mozilla-central → enable unit tests on Fedora talos slaves for mozilla-central
Attachment #437674 - Flags: feedback?(lsblakk) → review?(lsblakk)
The Fedora unit tests have been failing since yesterday being unable to unpack the test ball which is now a zip file rather than a tar ball.
It seems to be related to bug 549427 having been commited.
I was able to reproduce that I couldn't unzip on the actual slave.
Getting the following error (which does not show on the buildbot logs):
- error:  zipfile probably corrupt (segmentation violation)

I have clobbered all checked-out trees for the unit tests and I was then able to unzip the test ball.

Unless the unit tests go back again green we will have to postpone the enabling this.
There was a missing patch on talos-staging (bug 557854#c12) after having landed bug 549427. This seems to have been fixed.
If everything continues to be green in the morning we will still proceed with enabling unit tests for Fedora.
The only difference with the previous one is that it also enables it for staging.
Attachment #437674 - Attachment is obsolete: true
Attachment #437950 - Flags: review?(lsblakk)
Attachment #437674 - Flags: review?(lsblakk)
Comment on attachment 437950 [details] [diff] [review]
enable unit tests on Fedora talos slaves for mozilla-central


>-        'macosx': {
>-            'builds_before_reboot': 1,
>-            'opt_unittest_suites': [
>-                # Turn on chunks for mochitests
>-                ('mochitests', dict(suite='mochitest-plain', chunkByDir=4, totalChunks=5)),
>-                ('mochitest-other', ['mochitest-chrome', 'mochitest-browser-chrome',
>-                    'mochitest-ipcplugins']),
>-                ('reftest', ['reftest']),
>-                ('crashtest', ['crashtest']),
>-                ('xpcshell', ['xpcshell']),
>-                ('jsreftest', ['jsreftest']),
>-            ],
>-            'debug_unittest_suites': [
>-                # Turn on chunks for mochitests
>-                ('mochitests', dict(suite='mochitest-plain', chunkByDir=4, totalChunks=5)),
>-                ('mochitest-other', ['mochitest-chrome', 'mochitest-browser-chrome',
>-                    'mochitest-ipcplugins']),
>-                ('reftest', ['reftest']),
>-                ('crashtest', ['crashtest']),
>-                ('xpcshell', ['xpcshell']),
>-                ('jsreftest', ['jsreftest']),
>-            ],
>-        },

So you can't take this out - because macosx doesn't currently run the a11y tests, which are in the default set.  

>+            'opt_unittest_suites': UNITTEST_SUITES['opt_unittest_suites'], 
>+            'debug_unittest_suites': UNITTEST_SUITES['debug_unittest_suites'], 
>+        },

You do not need to add the UNITTEST_SUITES to each platform {linux, linux64} to enable them, just turn them on per-branch.  The only reason you put the suites into the PLATFORM_UNITTEST_VARS is if you need to override the default set.

>+        'macosx': {
>+            'builds_before_reboot': 1,
>+            'opt_unittest_suites': UNITTEST_SUITES['opt_unittest_suites'], 
>+            'debug_unittest_suites': UNITTEST_SUITES['debug_unittest_suites'], 
>+        },

As mentioned above, this won't work for macosx because it will turn on the mochitest-a11y test.

Same comments for production.
Attachment #437950 - Flags: review?(lsblakk) → review-
Attachment #437950 - Attachment is obsolete: true
Attachment #438001 - Flags: review?(lsblakk)
Attachment #438001 - Flags: review?(lsblakk) → review+
Comment on attachment 438001 [details] [diff] [review]
enable unit tests on Fedora talos slaves for mozilla-central

http://hg.mozilla.org/build/buildbot-configs/rev/0ce2427f7f29
Attachment #438001 - Flags: checked-in+
No longer blocks: 557311
Got a lot of burning when turning these on because of rm -rf buildtools failing on first build.
Depends on: 558240
I have this ready to be turned on again.
http://tinderbox.mozilla.org/showbuilds.cgi?tree=Firefox&noignore=1
You can see that there are no more reds.
I am going to wait until tomorrow to enable it (show it on Firefox).
I don't want developers to worries about Fedora oranges on their end of Friday.
I will send a notification on release-drivers@ and dev.planning ton inform everyone.
Thanks for fixing this up so quickly armen.  Huzzah!
Fedora unit tests are now running on mozilla-central and the non perma-oranges are hidden from developers. The unit tests are showing up correctly on TBPL.

To enable the rest of the branches we have to get more Rev 3 machines. I will look for the bug so we can add it to the dependency.

Meanwhile I am putting this bug on the side since there isn't more for me to be done now.
Priority: P2 → P4
No longer blocks: 548768
We now have all test suites for Fedora running without perma-oranges and have enough Rev3 machines. This means that we can now work on disabling unit tests on CentOS build machines and enabled the test suites for all project branches.

This bug will keep track of it and I am now working on this actively.
Status: NEW → ASSIGNED
Priority: P4 → P2
Summary: Enable unit tests on Fedora talos slaves → Enable unit tests on Fedora machines and disable them on CentOS build machines
packageTests is to keep on uploading the unit tests and to still send the sendchanges.
Attachment #446261 - Flags: review?(catlee)
I am not going to ask review for this one yet since I want to bake it on staging for few days.
Comment on attachment 446261 [details] [diff] [review]
disable unit tests builders for linux on mozilla-central

So this is only for staging?
That's right! We want it for production as well.
Attachment #446261 - Attachment is obsolete: true
Attachment #446500 - Flags: review?(catlee)
Attachment #446261 - Flags: review?(catlee)
Attachment #446500 - Flags: review?(catlee) → review+
Duplicate of this bug: 559701
Depends on: 567910
Let's start first with two branches to see how it impacts the load.
Attachment #446263 - Attachment is obsolete: true
Attachment #447582 - Flags: review?(ccooper)
Comment on attachment 447582 [details] [diff] [review]
enable Fedora unit tests for tracemonkey and 1.9.2

Looks good.
Attachment #447582 - Flags: review?(ccooper) → review+
Attachment #447582 - Flags: checked-in+
Comment on attachment 447582 [details] [diff] [review]
enable Fedora unit tests for tracemonkey and 1.9.2

http://hg.mozilla.org/build/buildbot-configs/rev/b5ff7da8081e
http://hg.mozilla.org/build/buildbot-configs/rev/29679aa99df8
repo_path was missing in the patch.

Master reconfigured.
Depends on: 568479
(In reply to comment #21)
> (From update of attachment 447582 [details] [diff] [review])
> http://hg.mozilla.org/build/buildbot-configs/rev/b5ff7da8081e
I did not enable Fedora x64 unit tests for 1.9.2 since we are never going to release it off it.
Status update:

oranges:
* Rev3 Fedora 12x64 tracemonkey {opt|debug} test jsreftest
* Rev3 Fedora 12 tracemonkey {opt|debug} test jsreftest
* Rev3 Fedora 12 tracemonkey debug test mochitests-1/5

pending:
* I see Fedora 12 unit test pending jobs for 1.9.2. Let's check tomorrow

I will have to check TBPL in the morning since the tinderbox is currently orange.
Comment on attachment 447582 [details] [diff] [review]
enable Fedora unit tests for tracemonkey and 1.9.2

http://hg.mozilla.org/build/buildbot-configs/rev/fa1aa0b4e5e3

I had to back out the change since the talos master became over-loaded:
http://grab.by/4C9q
Attachment #447582 - Flags: checked-in+ → checked-in-
Blocks: 568655
No longer depends on: 567910
Priority: P2 → P4
Blocks: 569623
Before we add the load of Fedora unit tests on project branches I would like to see the unit tests try jobs to be run on Rev3 machines rather than the try pool.
Blocking on bug 561437.

I also want to wait on bug 558430 so we take the buffer improvement to reduce communications with the talos master.

WRT to disabling unit tests on Centos. We will have to wait until we do the switch from unit tests on Centos for releases to Fedora (bug 568479).
Depends on: 561437, 558430
No longer depends on: 568479
Depends on: 570816
Depends on: 572469
Bug 573480 should enable Fedora unit tests for 1.9.1 and 1.9.2.
Other branches should follow if talos handles these two branches.
Depends on: 573480
We don't need to wait on the 0.8.0 upgrade.
No longer depends on: 558430
r+ from IRC.

Enable unit tests for Fedora (32/64 bit) and OSX (10.5 & 10.6).
Attachment #447582 - Attachment is obsolete: true
Attachment #457568 - Flags: review+
Summary: Enable unit tests on Fedora machines and disable them on CentOS build machines → Enable unit tests on minis and disable them on builders
Comment on attachment 457568 [details] [diff] [review]
Enable unit tests on the minis for tracemonkey

http://hg.mozilla.org/build/buildbot-configs/rev/bcc4a7172743
Attachment #457568 - Flags: checked-in+
Priority: P4 → P2
After enabling tracemonkey unit tests on the minis we can now disable it on the builders.
Attachment #458429 - Flags: review?(ccooper)
Attachment #458429 - Flags: review?(ccooper) → review+
Comment on attachment 458429 [details] [diff] [review]
Disable unit tests for tracemonkey on the builders

http://hg.mozilla.org/build/buildbot-configs/rev/378d8ff33b37
Attachment #458429 - Flags: checked-in+
This patch removes a lot of lines and adds some defaults for all branches.

It first adds a default to each platform. This allows me to set Win32 and Win64 unit tests to False.

For mozilla-central I override one of the default behaviours and that is to enable Win7 and Win7 x64 unit tests.

I have also added a new flag per branch. This allows me to disable all unit tests for a branch. Specifically I have used it for 1.9.1, 1.9.2 and addontester (which only needs talos runs).

Overall the change that is happening is that we add unit tests for Fedora 32/64-bit and OSX 10.5/10/6 for the following branches:
* places
* electrolysis
* maple
* cedar
* birch
* jaegermonkey
and disable it for 1.9.1 and 1.9.2.

I have previously mentioned that the load increase should not be much since these branches only account for 11% of all pushes according to June's load post [1]. I could enable a platform per day if you want me to. Otherwise I could enable all on Friday or Monday (I am running this patch through staging to double check that things are being triggered as expected).

[1] http://oduinn.com/images/2010/blogpost_2010_06_pie.png
Attachment #459135 - Flags: review?(ccooper)
Comment on attachment 459135 [details] [diff] [review]
Enable unit tests on minis for all remaining branches

(In reply to comment #32)
> I have previously mentioned that the load increase should not be much since
> these branches only account for 11% of all pushes according to June's load post
> [1]. I could enable a platform per day if you want me to. Otherwise I could
> enable all on Friday or Monday (I am running this patch through staging to
> double check that things are being triggered as expected).

Provided this passes in staging, I'm fine with it, and would ideally like to see it enabled earlier, i.e. tomorrow, rather than wait for next week. No sense waiting.

Any particular reason why you moved the repo_path declarations?
Attachment #459135 - Flags: review?(ccooper) → review+
Attachment #459138 - Flags: review?(ccooper) → review+
(In reply to comment #34) 
> Provided this passes in staging, I'm fine with it, and would ideally like to
> see it enabled earlier, i.e. tomorrow, rather than wait for next week. No sense
> waiting.
We are on the same page.

> Any particular reason why you moved the repo_path declarations?
Just aesthetic.
As committed.
Last minute I decided not to enable new branches on platforms on test-master02 since it is lacking a 2nd CPU. (See bug 576338#c5)

I will attach a patch for making this equal for all platforms once the 2nd CPU gets added.
Attachment #459135 - Attachment is obsolete: true
Attachment #459815 - Flags: review+
Attachment #459815 - Flags: checked-in+
Comment on attachment 459138 [details] [diff] [review]
buildbotcustom changes to allow disabling certain brancehs that don't require unit tests

http://hg.mozilla.org/build/buildbotcustom/rev/466cb22f9d06 (default)
http://hg.mozilla.org/build/buildbotcustom/rev/634344c8ad2e (buildbot-0.8.0)
Attachment #459138 - Flags: checked-in+
Comment on attachment 459815 [details] [diff] [review]
Enable unit tests on minis for all remaining branches (except Fedora and 10.5)

http://hg.mozilla.org/build/buildbot-configs/rev/456b225cc3e2

I have reconfigured the following masters:
* talos-master02
* test-master01
* test-master02 (the patch had no more effects than disabling 1.9.1 and 1.9.2 for this master to avoid tipping the master)
This patch is for when test-master02 will have a 2nd CPU.
Attachment #459820 - Flags: review?(ccooper)
Attachment #459820 - Flags: review?(ccooper) → review+
The Windows 7 oranges are down to 3 out of 11 test suites.
We should enable them. talos-master02 is in excellent CPU and memory conditions.
There is no need for a downtime for this. I can land it tomorrow morning.
Attachment #460269 - Flags: review?(ccooper)
This can land as soon as attachment 459820 [details] [diff] [review] lands (which is waiting on 2nd CPU upgrade).

If by then we also have all Win7 oranges down to zero we could disable it for Win32 as well. Otherwise, we can disable on a second run.
Attachment #460270 - Flags: review?(ccooper)
Attachment #460269 - Flags: review?(ccooper) → review+
Attachment #460270 - Flags: review?(ccooper) → review+
Attachment #459820 - Flags: checked-in+
Attachment #460269 - Flags: checked-in+
Comment on attachment 460270 [details] [diff] [review]
Disable unit tests on builders for all platforms except win32

http://hg.mozilla.org/build/buildbot-configs/rev/0964ef6eea3f

Masters reconfigured.
Attachment #460270 - Flags: checked-in+
Attachment #460882 - Flags: review?(catlee)
Attachment #460882 - Attachment is obsolete: true
Attachment #460888 - Flags: review?(catlee)
Attachment #460882 - Flags: review?(catlee)
Attachment #460888 - Flags: review?(catlee) → review+
Comment on attachment 460888 [details] [diff] [review]
[bustage] Re-enable enable_checktests

http://hg.mozilla.org/build/buildbot-configs/rev/0edc006b6681

Masters reconfigured.
Attachment #460888 - Flags: checked-in+
Status update.
* we are only waiting on bug 562489 to disable the win2k3 machines from doing *optimized* unit tests
* To disable the win3k3 *debug* unit tests on the builders we should fix bug 562459
This only disables the optimized unit tests on the builders (mozilla-central, try and all other branches). Debug unit tests would still run.

damon and rsayre have agreed to disable the test that are permanent oranges.
Attachment #462528 - Flags: review?(ccooper)
NOTE: We are currently running all tests suites on Windows 7 on the minis for all branches
Attachment #462528 - Flags: review?(ccooper) → review+
Let me summarize what is running where:
* on builders
** w2k3 opt unit tests (waiting on Win7 oranges)
** w2k3 debug unit tests (waiting on bug 562459)
** releases for 1.9.1 and 1.9.2
* on minis
** m-c, 2.0 and project branches
** releases for m-c and 2.0
** all platforms except XP (bug 563036)
** we are not running debug unit tests for XP, for Win7 and Win7-64 (bug 562459)
mochitest-4/5 and mochitest-other are now able to run green.

We can now disable the optimized unit tests on the Windows 2003 builders.
We sometimes time out on xpcshell test suite on bug 581775.
We need a downtime to deploy a twisted upgrade on all Win7 slaves.
Depends on: 581775
The Windows 7 slaves should now not get stuck running xpcshell.

There is now no blockers AFAIK to disable unit tests on Windows 2003 builders.
Passing this to coop as I will be away until Tuesday and I want to describe what is involved in here.

The only patch to land in here is:
https://bugzilla.mozilla.org/attachment.cgi?id=462528&action=edit

This patch only removes the *optimized* unit test builders for all branches except 1.9.1 and 1.9.2.

This patch should only require a reconfig on pm01, pm03 and pm02:8011 (try server).

I have verified that "packageTests" is set to True (this flag allows packaging the tests regardless if this master does not run unit tests) as it is with all the other platforms.

I have also checked that this patch does not affect the 1.9.1 and 1.9.2 branches as the other platforms don't affect it.
Assignee: armenzg → ccooper
Priority: P2 → --
Landed the Win2k3 patch yesterday. Back to Armen for remaining work/wrap-up.
Assignee: ccooper → armenzg
Priority: -- → P3
Depends on: 589607
Comment on attachment 462528 [details] [diff] [review]
Disable unit tests on W2k3 builders

Bug 589607 for the fallout of not reconfig'ing the scheduler master.
Priority: P3 → P4
Win32 debug unit tests is all that there is left - bug 562459
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.