Last Comment Bug 383136 - run unit tests on packaged builds
: run unit tests on packaged builds
Status: RESOLVED FIXED
:
Product: Release Engineering
Classification: Other
Component: Other (show other bugs)
: other
: All All
: P2 normal (vote)
: ---
Assigned To: Chris AtLee [:catlee]
:
Mentors:
Depends on: 486570 524130 421611 491675 491910 493341 493371 493755 496534 499429 511929
Blocks: 372581 425012 452861 473259 474671 486783 488732 493228 494676 548768
  Show dependency treegraph
 
Reported: 2007-06-04 08:06 PDT by :Gavin Sharp [email: gavin@gavinsharp.com]
Modified: 2013-08-12 21:54 PDT (History)
20 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
Add support for packaged unittests to buildbotcustom (30.25 KB, patch)
2009-04-17 09:41 PDT, Chris AtLee [:catlee]
no flags Details | Diff | Splinter Review
Add support for packaged unittests to buildbotcustom (32.04 KB, patch)
2009-04-20 11:05 PDT, Chris AtLee [:catlee]
bhearsum: review-
Details | Diff | Splinter Review
Add support for packaged unittests to buildbotcustom (37.02 KB, patch)
2009-04-21 12:54 PDT, Chris AtLee [:catlee]
bhearsum: review+
catlee: checked‑in+
Details | Diff | Splinter Review
Use packaged builds on staging (14.24 KB, patch)
2009-04-21 13:03 PDT, Chris AtLee [:catlee]
no flags Details | Diff | Splinter Review
Use packaged builds on staging (10.95 KB, patch)
2009-04-22 10:46 PDT, Chris AtLee [:catlee]
bhearsum: review+
catlee: checked‑in+
Details | Diff | Splinter Review
Fix for buildbot 0.7.10p1 (927 bytes, patch)
2009-04-30 18:02 PDT, Chris AtLee [:catlee]
bhearsum: review+
catlee: checked‑in+
Details | Diff | Splinter Review
Fix reftest parser (2.45 KB, patch)
2009-05-06 10:52 PDT, Chris AtLee [:catlee]
bhearsum: review+
bugzillamozillaorg_serge_20140323: review+
catlee: checked‑in+
Details | Diff | Splinter Review
Minor fixes for unittests on packaged builds (4.56 KB, patch)
2009-05-08 13:17 PDT, Chris AtLee [:catlee]
bhearsum: review+
catlee: checked‑in+
Details | Diff | Splinter Review
Turn on unittests on packaged builds in production (5.43 KB, patch)
2009-05-08 13:20 PDT, Chris AtLee [:catlee]
bhearsum: review+
Details | Diff | Splinter Review
Fix symbol unpacking, and related changes (2.62 KB, patch)
2009-05-14 12:51 PDT, Chris AtLee [:catlee]
bhearsum: review+
catlee: checked‑in+
Details | Diff | Splinter Review
Turn on unittests on packaged builds in production (5.43 KB, patch)
2009-05-14 12:57 PDT, Chris AtLee [:catlee]
bhearsum: review+
catlee: checked‑in+
Details | Diff | Splinter Review
Remove list of unittest suites from factory.py (1.02 KB, patch)
2009-05-15 11:35 PDT, Chris AtLee [:catlee]
bhearsum: review+
catlee: checked‑in+
Details | Diff | Splinter Review
Move list of unittest suites into buildbot-configs, and stop running a11y on mac 191. (5.31 KB, patch)
2009-05-15 11:40 PDT, Chris AtLee [:catlee]
bhearsum: review+
catlee: checked‑in+
Details | Diff | Splinter Review
Upload unittest builds to their own directory (739 bytes, patch)
2009-05-15 11:49 PDT, Chris AtLee [:catlee]
bhearsum: review+
catlee: checked‑in+
Details | Diff | Splinter Review
Fix call to reboot (765 bytes, patch)
2009-05-20 10:37 PDT, Chris AtLee [:catlee]
bhearsum: review+
catlee: checked‑in+
Details | Diff | Splinter Review

Description :Gavin Sharp [email: gavin@gavinsharp.com] 2007-06-04 08:06:26 PDT
We frequently forget to add new components to the installer packaging manifests (packages-static), because developers usually test things straight from the objdir. This causes problems for shipped/packaged builds (Linux tarballs, Windows zips and installers), and the problems often aren't noticed until nightly testers file bugs about brokenness (see e.g. bug 380865, bug 382482). It would be great if the unit test machines tested the packaged builds so that we could catch these problems faster.

I think this would probably involve runtests.pl hacking, to get it to run |make -C browser/installer|, unzip the resultant build, and use that. Or maybe the buildbot script can do that and just pass --app to runtests.pl. Shaver also suggested deleting any file not in packages-static before running the test, but I think that might be harder and more error-prone.
Comment 1 Benjamin Smedberg [:bsmedberg] 2007-06-04 08:10:25 PDT
could we just run the tests on dist/firefox instead of dist/bin (after running packaging)?

But, we're going to have to deal with test files that would normally live in dist/bin but aren't in the packaging manifests... the reftest bootstrap files come to mind immediately, but I'm sure there are others.
Comment 2 :Gavin Sharp [email: gavin@gavinsharp.com] 2007-06-09 20:23:49 PDT
(In reply to comment #1)
> But, we're going to have to deal with test files that would normally live in
> dist/bin but aren't in the packaging manifests... the reftest bootstrap files
> come to mind immediately, but I'm sure there are others.

We could run just mochitest from a packaged build, I don't think it has any dependencies on files in dist/bin. I think it's fairly easy to modify mozBuild.py to run the packaging target and then pass the appropriate --app flag to mochitest.
Comment 3 Rob Campbell [:rc] (:robcee) 2007-06-11 07:07:29 PDT
This would require some considerable changes to the buildbot setup. If the purpose of this bug is to verify that necessary components were included in the packaging phase of the build, I think there are better ways of doing it.
Comment 4 Rob Campbell [:rc] (:robcee) 2007-06-11 07:48:21 PDT
After discussing with gavin and ben in #developers, and understanding what's being suggested a little better, this actually would be pretty easy to do. Replace "considerable" in my previous comment with "only a few simple".

How high a priority is this?
Comment 5 :Gavin Sharp [email: gavin@gavinsharp.com] 2007-06-11 07:51:32 PDT
(In reply to comment #4)
> How high a priority is this?

Certainly not critical, but it'd be very nice to have.
Comment 6 :Gavin Sharp [email: gavin@gavinsharp.com] 2008-03-25 14:18:04 PDT
We just hit another case that would have been caught by these tests in bug 425062.
Comment 7 Justin Dolske [:Dolske] 2008-08-18 09:34:09 PDT
And bug 451040. 2x for password manager!
Comment 8 John O'Duinn [:joduinn] (please use "needinfo?" flag) 2009-01-24 01:24:33 PST
Found during triage. 

Once we can run unittests on pre-existing builds (bug#421611), then I'd really like to run unittests as part of the release automation. After all, afaik, we've never yet run unittests on the actual release bits that we ship!?!
Comment 9 Ted Mielczarek [:ted.mielczarek] 2009-01-29 09:43:21 PST
If we're going to treat this as the RelEng side of my work in bug 421611, then here's what will need to happen:
1) Build machines will need to not specify --disable-tests in their mozconfigs (--enable-tests is the default)
2) After building, but before uploading the build, they will need a build step that calls "make -C $OBJDIR package-tests"
3) The "make upload" step will upload the test package alongside the build, with the filename being the same as the build, but ending in .tests.tar.bz2
4) The unit test machine should download the build + the test package, unpack them somewhere, and then run mochitest as described in bug 421611 comment 36.
Comment 10 John O'Duinn [:joduinn] (please use "needinfo?" flag) 2009-02-20 12:00:56 PST
In bug#421611, Ted now has mochitest suite runable separately, and is still working on the other suites. Meanwhile, we can now:
- start running mochitest by itself on an existing build
- stop running mochitest in the build-and-unittest job.

This gains us faster end-to-end turnaround times on unittests (very crude parallelism), and the ability to run mochitest 'n' times in a row and identify any intermittent unittests. 

Moving back from Future as we are now unblocked on all this yummy goodness!!



(tweaked summary, as we pooled machines, and no longer have dedicated unittest machines)
Comment 11 Ben Hearsum (:bhearsum) 2009-02-23 09:12:01 PST
If we switch mochitests to test the builds we currently upload we're going to stop catching leaks, aren't we? (As discussed in bug 445611.)
Comment 12 Ben Hearsum (:bhearsum) 2009-02-23 09:14:36 PST
(In reply to comment #11)
> If we switch mochitests to test the builds we currently upload we're going to
> stop catching leaks, aren't we? (As discussed in bug 445611.)

And the real question here is: Are we OK with that?
Comment 13 Ben Hearsum (:bhearsum) 2009-02-23 13:08:05 PST
Catlee volunteered himself for this.
Comment 14 Justin Dolske [:Dolske] 2009-02-23 14:18:28 PST
I think we absolutely do want to check for leaks with the tests, as well as eventually checking assertions... The idea I heard floated was to create builds in different flavors (normal, with memory logging, with fatal asserts), and run tests on all of them. Dunno if there's a way that will scale better, though.
Comment 15 Chris AtLee [:catlee] 2009-04-09 14:31:28 PDT
I'm getting strange errors trying to get packaged tests working on windows.

The problem happens in stage-xpcshell, when it's copying over _tests/xpcshell/* to test-package-stage/xpcshell/tests

If test-package-stage/xpcshell/tests is empty, then I get this error from tar:
xpcom/unit/data/presentation.key/
      0 [main] tar 2780 open_stackdumpfile: Dumping stack trace to tar.exe.stackdump

If I re-run the same tar command afterwards, then it completes fine.  If I clean out test-package/stage/xpcshell/tests and then re-run the tar command, it crashes.
Comment 16 Chris AtLee [:catlee] 2009-04-09 14:58:40 PDT
Initially I couldn't reproduce this on my personal VM.  However, after reproducing the exact paths from the build machine, I'm running into this on my VM.

The full path to the objdir is:
/e/builds/moz2_slave/mozilla-central-win32-unittest/build/objdir
Comment 17 John O'Duinn [:joduinn] (please use "needinfo?" flag) 2009-04-14 09:42:24 PDT
(In reply to comment #15)
> I'm getting strange errors trying to get packaged tests working on windows.

Ted: can you help here - any ideas/suggestions?


> The problem happens in stage-xpcshell, when it's copying over _tests/xpcshell/*
> to test-package-stage/xpcshell/tests
> 
> If test-package-stage/xpcshell/tests is empty, then I get this error from tar:
> xpcom/unit/data/presentation.key/
>       0 [main] tar 2780 open_stackdumpfile: Dumping stack trace to
> tar.exe.stackdump
> 
> If I re-run the same tar command afterwards, then it completes fine.  If I
> clean out test-package/stage/xpcshell/tests and then re-run the tar command, it
> crashes.
Comment 18 Chris AtLee [:catlee] 2009-04-16 08:26:38 PDT
I've found one workaround and one fix:

- Using a shorter path for the objdir (e:\builds\moz2_slave\m191-win32-ut\build\objdir) worked.

- Using a newer version of tar fixes this crash.

http://sourceforge.net/project/downloading.php?group_id=2435&filename=tar-1.19.90-MSYS-1.0.11-2-bin.tar.gz&a=30604605
Comment 19 Chris AtLee [:catlee] 2009-04-17 09:41:33 PDT
Created attachment 373339 [details] [diff] [review]
Add support for packaged unittests to buildbotcustom

unittest.steps is renamed to steps.unittest

I factored out some of the log parsing so it could be used by the existing unittest steps, and the new versions that operate on the packaged tests.

I moved the upload steps to happen right after the compile steps in the Nightly factory so that we could get unittests started as soon as possible without having to wait for leaktests to run.
Comment 20 Chris AtLee [:catlee] 2009-04-20 11:05:03 PDT
Created attachment 373684 [details] [diff] [review]
Add support for packaged unittests to buildbotcustom
Comment 21 Ben Hearsum (:bhearsum) 2009-04-20 12:19:09 PDT
Comment on attachment 373684 [details] [diff] [review]
Add support for packaged unittests to buildbotcustom

>         self.addBuildSteps()
>-        if self.leakTest:
>-            self.addLeakTestSteps()
>-        if self.codesighs:
>-            self.addCodesighsSteps()
>         if self.uploadSymbols or self.uploadPackages:
>             self.addBuildSymbolsStep()
>         if self.uploadSymbols:
>             self.addUploadSymbolsStep()
>         if self.uploadPackages:
>             self.addUploadSteps()
>+        if self.leakTest:
>+            self.addLeakTestSteps()
>+        if self.codesighs:
>+            self.addCodesighsSteps()

Paranoid check: Have you looked at the leak test/codesighs steps with this change to make sure they're still working? Bustage should be obvious, but just in case...

>         if self.createSnippet:
>             self.addUpdateSteps()
>         if self.doCleanup:
>@@ -556,6 +557,13 @@
>              workdir='build/',
>              haltOnFailure=True
>             )
>+        if self.packageTests:
>+            self.addStep(ShellCommand,
>+             command=['make', 'package-tests'],
>+             env=self.env,
>+             workdir='build/%s' % self.objdir,
>+             haltOnFailure=True,
>+            )
>         self.addStep(ShellCommand,
>          command=['make', 'package'],
>          env=self.env,
>@@ -674,11 +682,16 @@
> 
> 
> class NightlyBuildFactory(MercurialBuildFactory):
>-    def __init__(self, talosMasters=None, **kwargs):
>+    def __init__(self, talosMasters=None, unittestMasters=None, **kwargs):

Sortof sucks that we can't combine these into one. Probably not worthwhile adding another parameter to config.py for it.

>@@ -1847,13 +1872,26 @@
> 
> class UnittestBuildFactory(MozillaBuildFactory):

Is there a reason we can't use NightlyBuildFactory for generating the --enable-logrefcnt builds? It seems to me that most of the changes to this class duplicate MercurialBuildFactory/NightlyBuildFactory functionality...Even if you need to subclass it to use a different POST_UPLOAD_CMD I think that's better...What do you think?

(holding off reviewing that part for now)


>+packagedUnittestSuites = ['reftest', 'crashtest', 'xpcshell', 'mochitest-plain',
>+                          'mochitest-chrome', 'mochitest-browser-chrome',
>+                          'mochitest-a11y']
>+class UnittestPackagedBuildFactory(MozillaBuildFactory):
>+    def __init__(self, platform, test_suites=None, **kwargs):
>+        env_map = {
>+                'linux': 'linux-centos-unittest',
>+                'macosx': 'mac-osx-unittest',
>+                'win32': 'win32-vc8-mozbuild-unittest',
>+                }
>+
>+        self.platform = platform.split('-')[0]
>+        assert self.platform in ('linux', 'linux64', 'win32', 'macosx')
>+        self.env = MozillaEnvironments[env_map[self.platform]]
>+
>+        MozillaBuildFactory.__init__(self, **kwargs)
>+

Does this work right if there's an env in **kwargs?


It's a pity we need so many custom steps but the rest of this file looks great. I'm very happy with the new BuildFactory.

> class TalosFactory(BuildFactory):
>     """Create working talos build factory"""
> 
>diff --git a/steps/misc.py b/steps/misc.py
>--- a/steps/misc.py
>+++ b/steps/misc.py
>@@ -380,3 +380,22 @@
>             summary = "TinderboxPrint: %s clobber" % clobberType
>             self.addCompleteLog('clobberer', summary)
> 
>+class SetBuildProperty(BuildStep):

Do we really need to do this? It seems silly to have this when you can echo it...I'm not going to push if you feel strongly about it though...




>+        #TODO: Add --symbols-path when it's supported by runxpcshelltests.py
>+        self.command = ['bash', '-c', WithProperties("""cp bin/xpcshell %(exedir)s
>+cp -R bin/components/* %(exedir)s/components/
>+cp -R bin/plugins/* %(exedir)s/plugins/
>+python -u xpcshell/runxpcshelltests.py --manifest=xpcshell/tests/all-test-dirs.list %(exedir)s/xpcshell""".replace("\n", " && "))]

I think this would be more readable without the .replace() at the end. How do you feel about making that change?

I noticed some hard tabs in steps/unittest.py - can you get rid of those?


Overall, this looks great. However, unless there's a very strong reason for putting all of the upload/post upload stuff into UnittestBuildFactory I'd like to see NightlyBuildFactory being used instead. Eventually, we can probably dump UnittestBuildFactory altogether.
Comment 22 Ben Hearsum (:bhearsum) 2009-04-21 06:21:23 PDT
(In reply to comment #21)
> > class UnittestBuildFactory(MozillaBuildFactory):
> 
> Is there a reason we can't use NightlyBuildFactory for generating the
> --enable-logrefcnt builds? It seems to me that most of the changes to this
> class duplicate MercurialBuildFactory/NightlyBuildFactory functionality...Even
> if you need to subclass it to use a different POST_UPLOAD_CMD I think that's
> better...What do you think?
> 

Chris and I chatted about this offline. He told me that integrating UnittestBuildFactory is going to depend on a larger clean-up of the factories to properly. Given that, I'm OK with this patch other than the comments below. Chris, can you please file a general follow-up on that so we don't forget? Maybe we can sprint on this next week...

I'm slightly worried about UnittestBuildFactory and its subclass being difficult to maintain once we stop running tests regularly with it but I'm sure we'll manage.


> (holding off reviewing that part for now)
> 

My only nit is that the 'packageTests' parameter is a little misleading. It not only packages tests, but packages the build and uploads both. How do you feel about using uploadPackages to be consistent with MercurialBuildFactory?

> 
> Does this work right if there's an env in **kwargs?

Chris said he had a patch that fixed env to be overrideable and have good defaults, I'd like to review that portion, too.
Comment 23 Chris AtLee [:catlee] 2009-04-21 12:54:08 PDT
Created attachment 373900 [details] [diff] [review]
Add support for packaged unittests to buildbotcustom
Comment 24 Chris AtLee [:catlee] 2009-04-21 13:03:06 PDT
Created attachment 373903 [details] [diff] [review]
Use packaged builds on staging
Comment 25 Ben Hearsum (:bhearsum) 2009-04-22 10:46:05 PDT
Comment on attachment 373903 [details] [diff] [review]
Use packaged builds on staging

>--- a/mozilla2-staging/linux/mozilla-central/debug/mozconfig
>+++ b/mozilla2-staging/linux/mozilla-central/debug/mozconfig
>@@ -2,6 +2,7 @@
> 
> ac_add_options --disable-optimize
> ac_add_options --enable-debug
>+ac_add_options --enable-libxul

Are you sure about this one? This mozconfig is for the leak test builds...as mentioned on IRC it's probably best to not do this even in staging until we're ready to actually drive it through to production.


>diff --git a/mozilla2-staging/macosx/mozilla-central/nightly/mozconfig b/mozilla2-staging/macosx/mozilla-central/nightly/mozconfig
>--- a/mozilla2-staging/macosx/mozilla-central/nightly/mozconfig
>+++ b/mozilla2-staging/macosx/mozilla-central/nightly/mozconfig
>@@ -3,7 +3,7 @@
> ac_add_options --enable-application=browser
> ac_add_options --enable-update-channel=nightly
> ac_add_options --enable-update-packaging
>-ac_add_options --disable-tests
>+ac_add_options --enable-tests

Isn't this going to cause the tests to be packaged in the dmg's because of bug 463605?

Don't we need to update the win32 nightly mozconfigs, too?

>+    for scheduler_branch, test_builders in triggeredUnittestBuilders:
>+        scheduler_name = "%s-%s" % (name, scheduler_branch)
>+        c['schedulers'].append(NoMergeScheduler(name=scheduler_name, branch=scheduler_branch, builderNames=test_builders, treeStableTimer=0))

Let's make sure we keep an eye on this when it lands in production. I think we'll end up doing more unittest runs total with this, it might back us up a bit at peak times.
Comment 26 Chris AtLee [:catlee] 2009-04-22 10:46:45 PDT
Created attachment 374100 [details] [diff] [review]
Use packaged builds on staging
Comment 27 Ben Hearsum (:bhearsum) 2009-04-22 12:49:50 PDT
Comment on attachment 373900 [details] [diff] [review]
Add support for packaged unittests to buildbotcustom

This is a big long patch, but everything in it seems to be in order...
Comment 28 Chris AtLee [:catlee] 2009-04-25 19:58:47 PDT
Comment on attachment 373900 [details] [diff] [review]
Add support for packaged unittests to buildbotcustom

changeset:   262:c63793bf2a07
Comment 29 Chris AtLee [:catlee] 2009-04-25 19:59:19 PDT
Comment on attachment 374100 [details] [diff] [review]
Use packaged builds on staging

changeset:   1113:3c10c7cf5031
Comment 30 Chris AtLee [:catlee] 2009-04-30 18:02:11 PDT
Created attachment 375276 [details] [diff] [review]
Fix for buildbot 0.7.10p1

Just removing unused parameters that now throw exceptions under buildbot 0.7.10
Comment 31 Serge Gautherie (:sgautherie) 2009-05-05 12:50:53 PDT
Comment on attachment 373900 [details] [diff] [review]
Add support for packaged unittests to buildbotcustom

>diff --git a/unittest/steps.py b/steps/unittest.py

>+def summarizeReftest(name, log):
>+    # Counts.
>+    passCount = 0
>+    failCount = 0

>@@ -292,45 +374,7 @@
>     def createSummary(self, log):
>-        # Counts.
>-        successfulCount = -1
>-        unexpectedCount = -1
>-        knownProblemsCount = -1

Why did you change and regress this code?
Please, restore and fix.
Comment 32 Chris AtLee [:catlee] 2009-05-06 10:52:56 PDT
Created attachment 376040 [details] [diff] [review]
Fix reftest parser

This fixes a yyp error from the unittest log parser refactoring.
Comment 33 Serge Gautherie (:sgautherie) 2009-05-06 16:38:07 PDT
Comment on attachment 376040 [details] [diff] [review]
Fix reftest parser

I checked that this was the only unwanted change / regression,
and I confirm this patch restores the code exactly as it should be.
Comment 34 Serge Gautherie (:sgautherie) 2009-05-06 17:00:28 PDT
Comment on attachment 373900 [details] [diff] [review]
Add support for packaged unittests to buildbotcustom

>diff --git a/unittest/steps.py b/steps/unittest.py

>+class MozillaPackagedXPCShellTests(ShellCommandReportTimeout):
>+    warnOnFailure = True
>+    warnOnWarnings = True
>+    name = "xpcshell"
>+    def __init__(self, symbols_path=None, **kwargs):

Nit: can you add a blank line before the 'def' line?
(same in the 2 other classes.)

>+    def evaluateCommand(self, cmd):
>+        superResult = self.super_class.evaluateCommand(self, cmd)
>+        if SUCCESS != superResult:
>+            return superResult

Out of curiosity,
I noticed you return 'superResult' here whereas the non-packaged class(es) return 'SUCCESS'.
Is there a reason not to synchronize both (either way)?
(same in the 2 other classes.)

>+class MozillaPackagedMochitests(ShellCommandReportTimeout):
>+    def __init__(self, variant='plain', symbols_path=None, **kwargs):
>+class MozillaPackagedReftests(ShellCommandReportTimeout):
>+    def __init__(self, crashtest=False, symbols_path=None, **kwargs):

Ftr,
I noticed you seem not to support a leak threshold here whereas the non-packaged class(es) do.
Isn't that a potential regression?
(Windows SeaMonkey mochitest-plain still needs a threshold, though SM may not use these steps yet...)
Comment 35 Chris AtLee [:catlee] 2009-05-07 12:43:08 PDT
Comment on attachment 375276 [details] [diff] [review]
Fix for buildbot 0.7.10p1

changeset:   271:bb0f8d07403b
Comment 36 Chris AtLee [:catlee] 2009-05-08 13:17:50 PDT
Created attachment 376471 [details] [diff] [review]
Minor fixes for unittests on packaged builds

This adds mochitest_leak_threshold support to the UnittestPackagedBuildFactory in case we ever need it.

It also switches from setting the 'got_revision' property to use the 'revision' property.  This is so that rebuilds work from buildbot, which does different things depending on if 'got_revision' is set.

Also, we should be copying bin/plugins from the test package into the profile directory, not %(exedir)s/plugins
Comment 37 Chris AtLee [:catlee] 2009-05-08 13:20:43 PDT
Created attachment 376473 [details] [diff] [review]
Turn on unittests on packaged builds in production

This enables unittests on packaged builds in production.

Results will go to a separate Tinderbox page per branch, e.g., Firefox-Unittest
Comment 38 Ben Hearsum (:bhearsum) 2009-05-11 05:53:26 PDT
Comment on attachment 376040 [details] [diff] [review]
Fix reftest parser

Seems fine
Comment 39 Ben Hearsum (:bhearsum) 2009-05-11 05:58:08 PDT
Comment on attachment 376471 [details] [diff] [review]
Minor fixes for unittests on packaged builds

Looks fine
Comment 40 Ben Hearsum (:bhearsum) 2009-05-11 06:38:07 PDT
Comment on attachment 376473 [details] [diff] [review]
Turn on unittests on packaged builds in production

Please make sure the unittest trees have been created when landing this. I'm pretty sure you can do that with the sheriff password.

We also need to make sure that tbpl knows how to read information from them before we shut off the other unittest machines.
Comment 41 Chris AtLee [:catlee] 2009-05-11 12:41:36 PDT
Comment on attachment 376040 [details] [diff] [review]
Fix reftest parser

changeset:   275:cdcfe7a37b72
Comment 42 Chris AtLee [:catlee] 2009-05-11 12:41:47 PDT
Comment on attachment 376471 [details] [diff] [review]
Minor fixes for unittests on packaged builds

changeset:   274:3f08d00e3291
Comment 43 Serge Gautherie (:sgautherie) 2009-05-13 08:01:09 PDT
(In reply to comment #34)
> >+    def evaluateCommand(self, cmd):
> >+        superResult = self.super_class.evaluateCommand(self, cmd)
> >+        if SUCCESS != superResult:
> >+            return superResult
> 
> Out of curiosity,
> I noticed you return 'superResult' here whereas the non-packaged class(es)
> return 'SUCCESS'.
> Is there a reason not to synchronize both (either way)?
> (same in the 2 other classes.)

catlee, what about this?

I could do the sync' (or add a comment) in bug 473259 if need be.
Comment 44 Serge Gautherie (:sgautherie) 2009-05-13 08:02:27 PDT
(In reply to comment #43)
> > return 'SUCCESS'.

WARNINGS, of course!
Comment 45 Chris AtLee [:catlee] 2009-05-14 12:51:59 PDT
Created attachment 377492 [details] [diff] [review]
Fix symbol unpacking, and related changes
Comment 46 Chris AtLee [:catlee] 2009-05-14 12:57:10 PDT
Created attachment 377496 [details] [diff] [review]
Turn on unittests on packaged builds in production

Updated to apply cleanly to latest version of buildbot-configs
Comment 47 Chris AtLee [:catlee] 2009-05-14 13:16:23 PDT
Created Firefox-Unittest, Firefox3.5-Unittest and TraceMonkey-Unittest trees on tinderbox.
Comment 48 Ben Hearsum (:bhearsum) 2009-05-14 14:26:49 PDT
Comment on attachment 377492 [details] [diff] [review]
Fix symbol unpacking, and related changes

Seems fine
Comment 49 Ben Hearsum (:bhearsum) 2009-05-14 14:30:22 PDT
Comment on attachment 377496 [details] [diff] [review]
Turn on unittests on packaged builds in production

>diff --git a/mozilla2/master.cfg b/mozilla2/master.cfg

>+    for scheduler_branch, test_builders in triggeredUnittestBuilders:
>+        scheduler_name = "%s-%s" % (name, scheduler_branch)
>+        c['schedulers'].append(NoMergeScheduler(name=scheduler_name, branch=scheduler_branch, builderNames=test_builders, treeStableTimer=0))
>+        c['status'].append(TinderboxMailNotifier(
>+            fromaddr="mozilla2.buildbot@build.mozilla.org",
>+            tree=branch['tinderbox_tree'] + "-Unittest",
>+            extraRecipients=["tinderbox-daemon@tinderbox.mozilla.org"],
>+            relayhost="mail.build.mozilla.org",
>+            builders=test_builders,
>+            logCompression="bzip2",
>+            errorparser="unittest",
>+        ))

I think there's an indentation issue here....you probably want TinderboxMailNotifier outside of the loop.

r=bhearsum with that change.
Comment 50 Ben Hearsum (:bhearsum) 2009-05-14 15:07:32 PDT
Comment on attachment 377496 [details] [diff] [review]
Turn on unittests on packaged builds in production

As it turns out, this patch is entirely correct - each triggeredUnittestBuilders item has a set of test_builders that it uses - I was getting confused between those and the test_builders variables in the previous hunk.
Comment 51 Chris AtLee [:catlee] 2009-05-15 05:53:33 PDT
Comment on attachment 377492 [details] [diff] [review]
Fix symbol unpacking, and related changes

changeset:   289:2fbab479ab9e
Comment 52 Chris AtLee [:catlee] 2009-05-15 11:11:38 PDT
Comment on attachment 377496 [details] [diff] [review]
Turn on unittests on packaged builds in production

changeset:   1147:bac4d574d47d
Comment 53 Chris AtLee [:catlee] 2009-05-15 11:35:43 PDT
Created attachment 377719 [details] [diff] [review]
Remove list of unittest suites from factory.py

These are moving into config.py in buildbot-configs
Comment 54 Chris AtLee [:catlee] 2009-05-15 11:40:35 PDT
Created attachment 377721 [details] [diff] [review]
Move list of unittest suites into buildbot-configs, and stop running a11y on mac 191.
Comment 55 Chris AtLee [:catlee] 2009-05-15 11:49:57 PDT
Created attachment 377723 [details] [diff] [review]
Upload unittest builds to their own directory
Comment 56 Ben Hearsum (:bhearsum) 2009-05-15 13:20:42 PDT
Comment on attachment 377723 [details] [diff] [review]
Upload unittest builds to their own directory

wfm
Comment 57 Chris AtLee [:catlee] 2009-05-15 13:24:13 PDT
Comment on attachment 377723 [details] [diff] [review]
Upload unittest builds to their own directory

changeset:   291:ccd8f62c33ea
Comment 58 Chris AtLee [:catlee] 2009-05-15 14:21:36 PDT
Comment on attachment 377721 [details] [diff] [review]
Move list of unittest suites into buildbot-configs, and stop running a11y on mac 191.

changeset:   1148:d943ec01e814
Comment 59 Chris AtLee [:catlee] 2009-05-15 14:22:25 PDT
Comment on attachment 377719 [details] [diff] [review]
Remove list of unittest suites from factory.py

changeset:   292:723bce148300
Comment 60 Chris AtLee [:catlee] 2009-05-15 18:17:56 PDT
Windows reference image should be updated with the new version of tar.exe.
Comment 61 Nick Thomas [:nthomas] 2009-05-16 05:02:53 PDT
There have been a couple build failures on moz2-win32-slave23 due to running out of disk. Turns out that the symbol archives can contain more than one build and we needed much more than 0.5GB disk space. Filed bug 493371 and now fixed on m-c and m-1.9.1; tracemonkey fixed on next merge from m-c.
Comment 62 Chris AtLee [:catlee] 2009-05-20 10:37:51 PDT
Created attachment 378637 [details] [diff] [review]
Fix call to reboot

This was accidentally added to MercurialBuildFactory.  It should have been added to the packaged build factory.
Comment 63 Chris AtLee [:catlee] 2009-05-20 12:11:37 PDT
Comment on attachment 378637 [details] [diff] [review]
Fix call to reboot

changeset:   302:334a6395eb37
Comment 64 Serge Gautherie (:sgautherie) 2009-05-29 15:58:56 PDT
[Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.9.2a1pre) Gecko/20090529 Minefield/3.6a1pre] (mozilla-central-win32-unittest/1243625438) (W2Ksp4)
(http://hg.mozilla.org/mozilla-central/rev/e44d9c0f4805)

I tried running this build tests locally:
a11y: all good :-)
browser-chrome: as if I got a bunch of known randoms but all at once :-|
chrome: test_sessionStorageFromChrome.xhtml crashes :-(

I stopped there.

***

(I see the -Unittest waterfalls have been empty for a while.)
Is this supposed to be usable yet?
Comment 65 Serge Gautherie (:sgautherie) 2009-05-31 15:27:17 PDT
(In reply to comment #64)

> browser-chrome: as if I got a bunch of known randoms but all at once :-|
> chrome: test_sessionStorageFromChrome.xhtml crashes :-(

Arf, my bad: I had mixed up a path from a previous build :-<

> Is this supposed to be usable yet?

All tests run :-)
Comment 66 Chris AtLee [:catlee] 2009-06-01 07:28:48 PDT
(In reply to comment #64)
> (I see the -Unittest waterfalls have been empty for a while.)
> Is this supposed to be usable yet?

They were disabled for 3.5 code freeze so they didn't tie up machine resources.
Comment 67 Chris AtLee [:catlee] 2009-07-29 10:15:55 PDT
I'm going to mark this one as fixed.  We have bugs filed for getting unit test runs on different kind of builds, but we are successfully running unittests on packaged unittest builds.

Note You need to log in before you can comment on or make changes to this bug.