run unit tests on packaged builds

RESOLVED FIXED

Status

Release Engineering
General
P2
normal
RESOLVED FIXED
10 years ago
4 years ago

People

(Reporter: Gavin, Assigned: catlee)

Tracking

(Depends on: 2 bugs)

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(11 attachments, 4 obsolete attachments)

37.02 KB, patch
bhearsum
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
10.95 KB, patch
bhearsum
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
927 bytes, patch
bhearsum
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
2.45 KB, patch
bhearsum
: review+
sgautherie
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
4.56 KB, patch
bhearsum
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
2.62 KB, patch
bhearsum
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
5.43 KB, patch
bhearsum
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
1.02 KB, patch
bhearsum
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
5.31 KB, patch
bhearsum
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
739 bytes, patch
bhearsum
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
765 bytes, patch
bhearsum
: review+
catlee
: checked-in+
Details | Diff | Splinter Review
We frequently forget to add new components to the installer packaging manifests (packages-static), because developers usually test things straight from the objdir. This causes problems for shipped/packaged builds (Linux tarballs, Windows zips and installers), and the problems often aren't noticed until nightly testers file bugs about brokenness (see e.g. bug 380865, bug 382482). It would be great if the unit test machines tested the packaged builds so that we could catch these problems faster.

I think this would probably involve runtests.pl hacking, to get it to run |make -C browser/installer|, unzip the resultant build, and use that. Or maybe the buildbot script can do that and just pass --app to runtests.pl. Shaver also suggested deleting any file not in packages-static before running the test, but I think that might be harder and more error-prone.
could we just run the tests on dist/firefox instead of dist/bin (after running packaging)?

But, we're going to have to deal with test files that would normally live in dist/bin but aren't in the packaging manifests... the reftest bootstrap files come to mind immediately, but I'm sure there are others.
(In reply to comment #1)
> But, we're going to have to deal with test files that would normally live in
> dist/bin but aren't in the packaging manifests... the reftest bootstrap files
> come to mind immediately, but I'm sure there are others.

We could run just mochitest from a packaged build, I don't think it has any dependencies on files in dist/bin. I think it's fairly easy to modify mozBuild.py to run the packaging target and then pass the appropriate --app flag to mochitest.
This would require some considerable changes to the buildbot setup. If the purpose of this bug is to verify that necessary components were included in the packaging phase of the build, I think there are better ways of doing it.
After discussing with gavin and ben in #developers, and understanding what's being suggested a little better, this actually would be pretty easy to do. Replace "considerable" in my previous comment with "only a few simple".

How high a priority is this?
Status: NEW → ASSIGNED
Summary: Unit test boxes should test packaged builds → Mochitest on unit test boxes should test packaged builds
(In reply to comment #4)
> How high a priority is this?

Certainly not critical, but it'd be very nice to have.
We just hit another case that would have been caught by these tests in bug 425062.
Status: ASSIGNED → NEW
Component: Testing → Release Engineering
Product: Core → mozilla.org
QA Contact: testing → release
Version: Trunk → other

Updated

10 years ago
Component: Release Engineering: Talos → Release Engineering: Future
Priority: -- → P3
Summary: Mochitest on unit test boxes should test packaged builds → unit test boxes should test packaged builds
And bug 451040. 2x for password manager!
Blocks: 452861
Found during triage. 

Once we can run unittests on pre-existing builds (bug#421611), then I'd really like to run unittests as part of the release automation. After all, afaik, we've never yet run unittests on the actual release bits that we ship!?!
Depends on: 421611
If we're going to treat this as the RelEng side of my work in bug 421611, then here's what will need to happen:
1) Build machines will need to not specify --disable-tests in their mozconfigs (--enable-tests is the default)
2) After building, but before uploading the build, they will need a build step that calls "make -C $OBJDIR package-tests"
3) The "make upload" step will upload the test package alongside the build, with the filename being the same as the build, but ending in .tests.tar.bz2
4) The unit test machine should download the build + the test package, unpack them somewhere, and then run mochitest as described in bug 421611 comment 36.
In bug#421611, Ted now has mochitest suite runable separately, and is still working on the other suites. Meanwhile, we can now:
- start running mochitest by itself on an existing build
- stop running mochitest in the build-and-unittest job.

This gains us faster end-to-end turnaround times on unittests (very crude parallelism), and the ability to run mochitest 'n' times in a row and identify any intermittent unittests. 

Moving back from Future as we are now unblocked on all this yummy goodness!!



(tweaked summary, as we pooled machines, and no longer have dedicated unittest machines)
Component: Release Engineering: Future → Release Engineering
Priority: P3 → --
Summary: unit test boxes should test packaged builds → run unit tests on packaged builds
If we switch mochitests to test the builds we currently upload we're going to stop catching leaks, aren't we? (As discussed in bug 445611.)
(In reply to comment #11)
> If we switch mochitests to test the builds we currently upload we're going to
> stop catching leaks, aren't we? (As discussed in bug 445611.)

And the real question here is: Are we OK with that?
Catlee volunteered himself for this.
Assignee: nobody → catlee
I think we absolutely do want to check for leaks with the tests, as well as eventually checking assertions... The idea I heard floated was to create builds in different flavors (normal, with memory logging, with fatal asserts), and run tests on all of them. Dunno if there's a way that will scale better, though.
Depends on: 460282, 463605
(Assignee)

Updated

9 years ago
Priority: -- → P2
(Assignee)

Updated

9 years ago
Depends on: 485182
(Assignee)

Updated

9 years ago
Depends on: 486570
Blocks: 486783
(Assignee)

Comment 15

9 years ago
I'm getting strange errors trying to get packaged tests working on windows.

The problem happens in stage-xpcshell, when it's copying over _tests/xpcshell/* to test-package-stage/xpcshell/tests

If test-package-stage/xpcshell/tests is empty, then I get this error from tar:
xpcom/unit/data/presentation.key/
      0 [main] tar 2780 open_stackdumpfile: Dumping stack trace to tar.exe.stackdump

If I re-run the same tar command afterwards, then it completes fine.  If I clean out test-package/stage/xpcshell/tests and then re-run the tar command, it crashes.
(Assignee)

Comment 16

9 years ago
Initially I couldn't reproduce this on my personal VM.  However, after reproducing the exact paths from the build machine, I'm running into this on my VM.

The full path to the objdir is:
/e/builds/moz2_slave/mozilla-central-win32-unittest/build/objdir
(In reply to comment #15)
> I'm getting strange errors trying to get packaged tests working on windows.

Ted: can you help here - any ideas/suggestions?


> The problem happens in stage-xpcshell, when it's copying over _tests/xpcshell/*
> to test-package-stage/xpcshell/tests
> 
> If test-package-stage/xpcshell/tests is empty, then I get this error from tar:
> xpcom/unit/data/presentation.key/
>       0 [main] tar 2780 open_stackdumpfile: Dumping stack trace to
> tar.exe.stackdump
> 
> If I re-run the same tar command afterwards, then it completes fine.  If I
> clean out test-package/stage/xpcshell/tests and then re-run the tar command, it
> crashes.
(Assignee)

Comment 18

9 years ago
I've found one workaround and one fix:

- Using a shorter path for the objdir (e:\builds\moz2_slave\m191-win32-ut\build\objdir) worked.

- Using a newer version of tar fixes this crash.

http://sourceforge.net/project/downloading.php?group_id=2435&filename=tar-1.19.90-MSYS-1.0.11-2-bin.tar.gz&a=30604605
(Assignee)

Updated

9 years ago
Blocks: 488732
(Assignee)

Updated

9 years ago
Blocks: 474671
(Assignee)

Comment 19

9 years ago
Created attachment 373339 [details] [diff] [review]
Add support for packaged unittests to buildbotcustom

unittest.steps is renamed to steps.unittest

I factored out some of the log parsing so it could be used by the existing unittest steps, and the new versions that operate on the packaged tests.

I moved the upload steps to happen right after the compile steps in the Nightly factory so that we could get unittests started as soon as possible without having to wait for leaktests to run.
Attachment #373339 - Flags: review?(bhearsum)
(Assignee)

Updated

8 years ago
Attachment #373339 - Attachment is obsolete: true
Attachment #373339 - Flags: review?(bhearsum)
(Assignee)

Comment 20

8 years ago
Created attachment 373684 [details] [diff] [review]
Add support for packaged unittests to buildbotcustom
Attachment #373684 - Flags: review?(bhearsum)
Comment on attachment 373684 [details] [diff] [review]
Add support for packaged unittests to buildbotcustom

>         self.addBuildSteps()
>-        if self.leakTest:
>-            self.addLeakTestSteps()
>-        if self.codesighs:
>-            self.addCodesighsSteps()
>         if self.uploadSymbols or self.uploadPackages:
>             self.addBuildSymbolsStep()
>         if self.uploadSymbols:
>             self.addUploadSymbolsStep()
>         if self.uploadPackages:
>             self.addUploadSteps()
>+        if self.leakTest:
>+            self.addLeakTestSteps()
>+        if self.codesighs:
>+            self.addCodesighsSteps()

Paranoid check: Have you looked at the leak test/codesighs steps with this change to make sure they're still working? Bustage should be obvious, but just in case...

>         if self.createSnippet:
>             self.addUpdateSteps()
>         if self.doCleanup:
>@@ -556,6 +557,13 @@
>              workdir='build/',
>              haltOnFailure=True
>             )
>+        if self.packageTests:
>+            self.addStep(ShellCommand,
>+             command=['make', 'package-tests'],
>+             env=self.env,
>+             workdir='build/%s' % self.objdir,
>+             haltOnFailure=True,
>+            )
>         self.addStep(ShellCommand,
>          command=['make', 'package'],
>          env=self.env,
>@@ -674,11 +682,16 @@
> 
> 
> class NightlyBuildFactory(MercurialBuildFactory):
>-    def __init__(self, talosMasters=None, **kwargs):
>+    def __init__(self, talosMasters=None, unittestMasters=None, **kwargs):

Sortof sucks that we can't combine these into one. Probably not worthwhile adding another parameter to config.py for it.

>@@ -1847,13 +1872,26 @@
> 
> class UnittestBuildFactory(MozillaBuildFactory):

Is there a reason we can't use NightlyBuildFactory for generating the --enable-logrefcnt builds? It seems to me that most of the changes to this class duplicate MercurialBuildFactory/NightlyBuildFactory functionality...Even if you need to subclass it to use a different POST_UPLOAD_CMD I think that's better...What do you think?

(holding off reviewing that part for now)


>+packagedUnittestSuites = ['reftest', 'crashtest', 'xpcshell', 'mochitest-plain',
>+                          'mochitest-chrome', 'mochitest-browser-chrome',
>+                          'mochitest-a11y']
>+class UnittestPackagedBuildFactory(MozillaBuildFactory):
>+    def __init__(self, platform, test_suites=None, **kwargs):
>+        env_map = {
>+                'linux': 'linux-centos-unittest',
>+                'macosx': 'mac-osx-unittest',
>+                'win32': 'win32-vc8-mozbuild-unittest',
>+                }
>+
>+        self.platform = platform.split('-')[0]
>+        assert self.platform in ('linux', 'linux64', 'win32', 'macosx')
>+        self.env = MozillaEnvironments[env_map[self.platform]]
>+
>+        MozillaBuildFactory.__init__(self, **kwargs)
>+

Does this work right if there's an env in **kwargs?


It's a pity we need so many custom steps but the rest of this file looks great. I'm very happy with the new BuildFactory.

> class TalosFactory(BuildFactory):
>     """Create working talos build factory"""
> 
>diff --git a/steps/misc.py b/steps/misc.py
>--- a/steps/misc.py
>+++ b/steps/misc.py
>@@ -380,3 +380,22 @@
>             summary = "TinderboxPrint: %s clobber" % clobberType
>             self.addCompleteLog('clobberer', summary)
> 
>+class SetBuildProperty(BuildStep):

Do we really need to do this? It seems silly to have this when you can echo it...I'm not going to push if you feel strongly about it though...




>+        #TODO: Add --symbols-path when it's supported by runxpcshelltests.py
>+        self.command = ['bash', '-c', WithProperties("""cp bin/xpcshell %(exedir)s
>+cp -R bin/components/* %(exedir)s/components/
>+cp -R bin/plugins/* %(exedir)s/plugins/
>+python -u xpcshell/runxpcshelltests.py --manifest=xpcshell/tests/all-test-dirs.list %(exedir)s/xpcshell""".replace("\n", " && "))]

I think this would be more readable without the .replace() at the end. How do you feel about making that change?

I noticed some hard tabs in steps/unittest.py - can you get rid of those?


Overall, this looks great. However, unless there's a very strong reason for putting all of the upload/post upload stuff into UnittestBuildFactory I'd like to see NightlyBuildFactory being used instead. Eventually, we can probably dump UnittestBuildFactory altogether.
Attachment #373684 - Flags: review?(bhearsum) → review-
(In reply to comment #21)
> > class UnittestBuildFactory(MozillaBuildFactory):
> 
> Is there a reason we can't use NightlyBuildFactory for generating the
> --enable-logrefcnt builds? It seems to me that most of the changes to this
> class duplicate MercurialBuildFactory/NightlyBuildFactory functionality...Even
> if you need to subclass it to use a different POST_UPLOAD_CMD I think that's
> better...What do you think?
> 

Chris and I chatted about this offline. He told me that integrating UnittestBuildFactory is going to depend on a larger clean-up of the factories to properly. Given that, I'm OK with this patch other than the comments below. Chris, can you please file a general follow-up on that so we don't forget? Maybe we can sprint on this next week...

I'm slightly worried about UnittestBuildFactory and its subclass being difficult to maintain once we stop running tests regularly with it but I'm sure we'll manage.


> (holding off reviewing that part for now)
> 

My only nit is that the 'packageTests' parameter is a little misleading. It not only packages tests, but packages the build and uploads both. How do you feel about using uploadPackages to be consistent with MercurialBuildFactory?

> 
> Does this work right if there's an env in **kwargs?

Chris said he had a patch that fixed env to be overrideable and have good defaults, I'd like to review that portion, too.
(Assignee)

Comment 23

8 years ago
Created attachment 373900 [details] [diff] [review]
Add support for packaged unittests to buildbotcustom
Attachment #373684 - Attachment is obsolete: true
Attachment #373900 - Flags: review?(bhearsum)
(Assignee)

Comment 24

8 years ago
Created attachment 373903 [details] [diff] [review]
Use packaged builds on staging
Attachment #373903 - Flags: review?(bhearsum)
Comment on attachment 373903 [details] [diff] [review]
Use packaged builds on staging

>--- a/mozilla2-staging/linux/mozilla-central/debug/mozconfig
>+++ b/mozilla2-staging/linux/mozilla-central/debug/mozconfig
>@@ -2,6 +2,7 @@
> 
> ac_add_options --disable-optimize
> ac_add_options --enable-debug
>+ac_add_options --enable-libxul

Are you sure about this one? This mozconfig is for the leak test builds...as mentioned on IRC it's probably best to not do this even in staging until we're ready to actually drive it through to production.


>diff --git a/mozilla2-staging/macosx/mozilla-central/nightly/mozconfig b/mozilla2-staging/macosx/mozilla-central/nightly/mozconfig
>--- a/mozilla2-staging/macosx/mozilla-central/nightly/mozconfig
>+++ b/mozilla2-staging/macosx/mozilla-central/nightly/mozconfig
>@@ -3,7 +3,7 @@
> ac_add_options --enable-application=browser
> ac_add_options --enable-update-channel=nightly
> ac_add_options --enable-update-packaging
>-ac_add_options --disable-tests
>+ac_add_options --enable-tests

Isn't this going to cause the tests to be packaged in the dmg's because of bug 463605?

Don't we need to update the win32 nightly mozconfigs, too?

>+    for scheduler_branch, test_builders in triggeredUnittestBuilders:
>+        scheduler_name = "%s-%s" % (name, scheduler_branch)
>+        c['schedulers'].append(NoMergeScheduler(name=scheduler_name, branch=scheduler_branch, builderNames=test_builders, treeStableTimer=0))

Let's make sure we keep an eye on this when it lands in production. I think we'll end up doing more unittest runs total with this, it might back us up a bit at peak times.
(Assignee)

Comment 26

8 years ago
Created attachment 374100 [details] [diff] [review]
Use packaged builds on staging
Attachment #373903 - Attachment is obsolete: true
Attachment #374100 - Flags: review?(bhearsum)
Attachment #373903 - Flags: review?(bhearsum)
Attachment #374100 - Flags: review?(bhearsum) → review+
Comment on attachment 373900 [details] [diff] [review]
Add support for packaged unittests to buildbotcustom

This is a big long patch, but everything in it seems to be in order...
Attachment #373900 - Flags: review?(bhearsum) → review+
(Assignee)

Updated

8 years ago
Attachment #373900 - Flags: checked‑in+
(Assignee)

Comment 28

8 years ago
Comment on attachment 373900 [details] [diff] [review]
Add support for packaged unittests to buildbotcustom

changeset:   262:c63793bf2a07
(Assignee)

Comment 29

8 years ago
Comment on attachment 374100 [details] [diff] [review]
Use packaged builds on staging

changeset:   1113:3c10c7cf5031
Attachment #374100 - Flags: checked‑in+
(Assignee)

Comment 30

8 years ago
Created attachment 375276 [details] [diff] [review]
Fix for buildbot 0.7.10p1

Just removing unused parameters that now throw exceptions under buildbot 0.7.10
Attachment #375276 - Flags: review?(bhearsum)
Attachment #375276 - Flags: review?(bhearsum) → review+
Comment on attachment 373900 [details] [diff] [review]
Add support for packaged unittests to buildbotcustom

>diff --git a/unittest/steps.py b/steps/unittest.py

>+def summarizeReftest(name, log):
>+    # Counts.
>+    passCount = 0
>+    failCount = 0

>@@ -292,45 +374,7 @@
>     def createSummary(self, log):
>-        # Counts.
>-        successfulCount = -1
>-        unexpectedCount = -1
>-        knownProblemsCount = -1

Why did you change and regress this code?
Please, restore and fix.
Blocks: 473259
(Assignee)

Comment 32

8 years ago
Created attachment 376040 [details] [diff] [review]
Fix reftest parser

This fixes a yyp error from the unittest log parser refactoring.
Attachment #376040 - Flags: review?(bhearsum)
(Assignee)

Updated

8 years ago
Attachment #376040 - Flags: review?(sgautherie.bz)
Comment on attachment 376040 [details] [diff] [review]
Fix reftest parser

I checked that this was the only unwanted change / regression,
and I confirm this patch restores the code exactly as it should be.
Attachment #376040 - Flags: review?(sgautherie.bz) → review+
Comment on attachment 373900 [details] [diff] [review]
Add support for packaged unittests to buildbotcustom

>diff --git a/unittest/steps.py b/steps/unittest.py

>+class MozillaPackagedXPCShellTests(ShellCommandReportTimeout):
>+    warnOnFailure = True
>+    warnOnWarnings = True
>+    name = "xpcshell"
>+    def __init__(self, symbols_path=None, **kwargs):

Nit: can you add a blank line before the 'def' line?
(same in the 2 other classes.)

>+    def evaluateCommand(self, cmd):
>+        superResult = self.super_class.evaluateCommand(self, cmd)
>+        if SUCCESS != superResult:
>+            return superResult

Out of curiosity,
I noticed you return 'superResult' here whereas the non-packaged class(es) return 'SUCCESS'.
Is there a reason not to synchronize both (either way)?
(same in the 2 other classes.)

>+class MozillaPackagedMochitests(ShellCommandReportTimeout):
>+    def __init__(self, variant='plain', symbols_path=None, **kwargs):
>+class MozillaPackagedReftests(ShellCommandReportTimeout):
>+    def __init__(self, crashtest=False, symbols_path=None, **kwargs):

Ftr,
I noticed you seem not to support a leak threshold here whereas the non-packaged class(es) do.
Isn't that a potential regression?
(Windows SeaMonkey mochitest-plain still needs a threshold, though SM may not use these steps yet...)
(Assignee)

Updated

8 years ago
Depends on: 491675
(Assignee)

Updated

8 years ago
Depends on: 491910
(Assignee)

Comment 35

8 years ago
Comment on attachment 375276 [details] [diff] [review]
Fix for buildbot 0.7.10p1

changeset:   271:bb0f8d07403b
Attachment #375276 - Flags: checked‑in+
(Assignee)

Comment 36

8 years ago
Created attachment 376471 [details] [diff] [review]
Minor fixes for unittests on packaged builds

This adds mochitest_leak_threshold support to the UnittestPackagedBuildFactory in case we ever need it.

It also switches from setting the 'got_revision' property to use the 'revision' property.  This is so that rebuilds work from buildbot, which does different things depending on if 'got_revision' is set.

Also, we should be copying bin/plugins from the test package into the profile directory, not %(exedir)s/plugins
Attachment #376471 - Flags: review?(bhearsum)
(Assignee)

Comment 37

8 years ago
Created attachment 376473 [details] [diff] [review]
Turn on unittests on packaged builds in production

This enables unittests on packaged builds in production.

Results will go to a separate Tinderbox page per branch, e.g., Firefox-Unittest
Attachment #376473 - Flags: review?(bhearsum)
Blocks: 425012
Comment on attachment 376040 [details] [diff] [review]
Fix reftest parser

Seems fine
Attachment #376040 - Flags: review?(bhearsum) → review+
Comment on attachment 376471 [details] [diff] [review]
Minor fixes for unittests on packaged builds

Looks fine
Attachment #376471 - Flags: review?(bhearsum) → review+
Comment on attachment 376473 [details] [diff] [review]
Turn on unittests on packaged builds in production

Please make sure the unittest trees have been created when landing this. I'm pretty sure you can do that with the sheriff password.

We also need to make sure that tbpl knows how to read information from them before we shut off the other unittest machines.
Attachment #376473 - Flags: review?(bhearsum) → review+
(Assignee)

Comment 41

8 years ago
Comment on attachment 376040 [details] [diff] [review]
Fix reftest parser

changeset:   275:cdcfe7a37b72
Attachment #376040 - Flags: checked‑in+
(Assignee)

Updated

8 years ago
Attachment #376471 - Flags: checked‑in+
(Assignee)

Comment 42

8 years ago
Comment on attachment 376471 [details] [diff] [review]
Minor fixes for unittests on packaged builds

changeset:   274:3f08d00e3291
(In reply to comment #34)
> >+    def evaluateCommand(self, cmd):
> >+        superResult = self.super_class.evaluateCommand(self, cmd)
> >+        if SUCCESS != superResult:
> >+            return superResult
> 
> Out of curiosity,
> I noticed you return 'superResult' here whereas the non-packaged class(es)
> return 'SUCCESS'.
> Is there a reason not to synchronize both (either way)?
> (same in the 2 other classes.)

catlee, what about this?

I could do the sync' (or add a comment) in bug 473259 if need be.
(In reply to comment #43)
> > return 'SUCCESS'.

WARNINGS, of course!
Status: NEW → ASSIGNED
(Assignee)

Comment 45

8 years ago
Created attachment 377492 [details] [diff] [review]
Fix symbol unpacking, and related changes
Attachment #377492 - Flags: review?(bhearsum)
(Assignee)

Comment 46

8 years ago
Created attachment 377496 [details] [diff] [review]
Turn on unittests on packaged builds in production

Updated to apply cleanly to latest version of buildbot-configs
Attachment #376473 - Attachment is obsolete: true
Attachment #377496 - Flags: review?(bhearsum)
(Assignee)

Comment 47

8 years ago
Created Firefox-Unittest, Firefox3.5-Unittest and TraceMonkey-Unittest trees on tinderbox.
(Assignee)

Updated

8 years ago
Blocks: 372581
Comment on attachment 377492 [details] [diff] [review]
Fix symbol unpacking, and related changes

Seems fine
Attachment #377492 - Flags: review?(bhearsum) → review+
Comment on attachment 377496 [details] [diff] [review]
Turn on unittests on packaged builds in production

>diff --git a/mozilla2/master.cfg b/mozilla2/master.cfg

>+    for scheduler_branch, test_builders in triggeredUnittestBuilders:
>+        scheduler_name = "%s-%s" % (name, scheduler_branch)
>+        c['schedulers'].append(NoMergeScheduler(name=scheduler_name, branch=scheduler_branch, builderNames=test_builders, treeStableTimer=0))
>+        c['status'].append(TinderboxMailNotifier(
>+            fromaddr="mozilla2.buildbot@build.mozilla.org",
>+            tree=branch['tinderbox_tree'] + "-Unittest",
>+            extraRecipients=["tinderbox-daemon@tinderbox.mozilla.org"],
>+            relayhost="mail.build.mozilla.org",
>+            builders=test_builders,
>+            logCompression="bzip2",
>+            errorparser="unittest",
>+        ))

I think there's an indentation issue here....you probably want TinderboxMailNotifier outside of the loop.

r=bhearsum with that change.
Comment on attachment 377496 [details] [diff] [review]
Turn on unittests on packaged builds in production

As it turns out, this patch is entirely correct - each triggeredUnittestBuilders item has a set of test_builders that it uses - I was getting confused between those and the test_builders variables in the previous hunk.
Attachment #377496 - Flags: review?(bhearsum) → review+
(Assignee)

Comment 51

8 years ago
Comment on attachment 377492 [details] [diff] [review]
Fix symbol unpacking, and related changes

changeset:   289:2fbab479ab9e
Attachment #377492 - Flags: checked‑in+
(Assignee)

Updated

8 years ago
No longer depends on: 463605
(Assignee)

Updated

8 years ago
No longer depends on: 460282
(Assignee)

Updated

8 years ago
Attachment #377496 - Flags: checked‑in+
(Assignee)

Comment 52

8 years ago
Comment on attachment 377496 [details] [diff] [review]
Turn on unittests on packaged builds in production

changeset:   1147:bac4d574d47d
(Assignee)

Comment 53

8 years ago
Created attachment 377719 [details] [diff] [review]
Remove list of unittest suites from factory.py

These are moving into config.py in buildbot-configs
Attachment #377719 - Flags: review?(bhearsum)
(Assignee)

Comment 54

8 years ago
Created attachment 377721 [details] [diff] [review]
Move list of unittest suites into buildbot-configs, and stop running a11y on mac 191.
Attachment #377721 - Flags: review?(bhearsum)
(Assignee)

Comment 55

8 years ago
Created attachment 377723 [details] [diff] [review]
Upload unittest builds to their own directory
Attachment #377723 - Flags: review?(bhearsum)
Attachment #377723 - Flags: review?(bhearsum) → review+
Comment on attachment 377723 [details] [diff] [review]
Upload unittest builds to their own directory

wfm
(Assignee)

Updated

8 years ago
Attachment #377723 - Flags: checked‑in+
(Assignee)

Comment 57

8 years ago
Comment on attachment 377723 [details] [diff] [review]
Upload unittest builds to their own directory

changeset:   291:ccd8f62c33ea
Attachment #377719 - Flags: review?(bhearsum) → review+
Attachment #377721 - Flags: review?(bhearsum) → review+
(Assignee)

Updated

8 years ago
Attachment #377721 - Flags: checked‑in+
(Assignee)

Comment 58

8 years ago
Comment on attachment 377721 [details] [diff] [review]
Move list of unittest suites into buildbot-configs, and stop running a11y on mac 191.

changeset:   1148:d943ec01e814
(Assignee)

Comment 59

8 years ago
Comment on attachment 377719 [details] [diff] [review]
Remove list of unittest suites from factory.py

changeset:   292:723bce148300
Attachment #377719 - Flags: checked‑in+
(Assignee)

Comment 60

8 years ago
Windows reference image should be updated with the new version of tar.exe.
(Assignee)

Updated

8 years ago
Depends on: 493341
There have been a couple build failures on moz2-win32-slave23 due to running out of disk. Turns out that the symbol archives can contain more than one build and we needed much more than 0.5GB disk space. Filed bug 493371 and now fixed on m-c and m-1.9.1; tracemonkey fixed on next merge from m-c.
Depends on: 493371
(Assignee)

Updated

8 years ago
Depends on: 493755
(Assignee)

Comment 62

8 years ago
Created attachment 378637 [details] [diff] [review]
Fix call to reboot

This was accidentally added to MercurialBuildFactory.  It should have been added to the packaged build factory.
Attachment #378637 - Flags: review?(bhearsum)
Attachment #378637 - Flags: review?(bhearsum) → review+
(Assignee)

Comment 63

8 years ago
Comment on attachment 378637 [details] [diff] [review]
Fix call to reboot

changeset:   302:334a6395eb37
Attachment #378637 - Flags: checked‑in+
Blocks: 494676
[Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9.2a1pre) Gecko/20090529 Minefield/3.6a1pre] (mozilla-central-win32-unittest/1243625438) (W2Ksp4)
(http://hg.mozilla.org/mozilla-central/rev/e44d9c0f4805)

I tried running this build tests locally:
a11y: all good :-)
browser-chrome: as if I got a bunch of known randoms but all at once :-|
chrome: test_sessionStorageFromChrome.xhtml crashes :-(

I stopped there.

***

(I see the -Unittest waterfalls have been empty for a while.)
Is this supposed to be usable yet?
Blocks: 493228
(In reply to comment #64)

> browser-chrome: as if I got a bunch of known randoms but all at once :-|
> chrome: test_sessionStorageFromChrome.xhtml crashes :-(

Arf, my bad: I had mixed up a path from a previous build :-<

> Is this supposed to be usable yet?

All tests run :-)
(Assignee)

Comment 66

8 years ago
(In reply to comment #64)
> (I see the -Unittest waterfalls have been empty for a while.)
> Is this supposed to be usable yet?

They were disabled for 3.5 code freeze so they didn't tie up machine resources.
Blocks: 496534
No longer blocks: 496534
Depends on: 496534
Depends on: 499429
Depends on: 488596
(Assignee)

Updated

8 years ago
No longer depends on: 488596
(Assignee)

Comment 67

8 years ago
I'm going to mark this one as fixed.  We have bugs filed for getting unit test runs on different kind of builds, but we are successfully running unittests on packaged unittest builds.
Status: ASSIGNED → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → FIXED
Depends on: 511929
Depends on: 524130
No longer depends on: 485182
Blocks: 548768
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.