Closed Bug 920569 Opened 11 years ago Closed 10 years ago

Add support for webapprt-test-chrome test jobs & enable them per push on Cedar

Categories

(Release Engineering :: General, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: emorley, Assigned: jgriffin)

References

Details

(Keywords: ateam-unittests-task, Whiteboard: [leave open])

Attachments

(5 files, 1 obsolete file)

Broken out from bug 899707.

The Firefox Webapp runtime team would like to start running the webapprt test suite per push in automation. Initially these should just be scheduled on Cedar (to green them up), but eventually they will need to be enabled on all trunk trees.

Platforms: Windows, Linux, OS X
Builds types: Opt + debug

The test suites are currently run using:
|mach webapprt-test-chrome| and |mach webapprt-test-content|
Marco, should both the chrome and content suites be run inside the same job run (ie similar to how mochitest-other currently runs several sub-suites) or as completely separate jobs/symbols on TBPL?
Flags: needinfo?(mcastelluccio)
I think we should start with webapprt-test-chrome as webapprt-test-content will likely change. When we'll want to enable content tests, I think they will be separate.
Flags: needinfo?(mcastelluccio)
Do these run as part of the build, or as a separate test job? If the latter, then we can't use mach and will need to know the underlying commands that mach is executing. We'll also have to make sure all the tests are packaged up in the tests archive.
(In reply to Chris AtLee [:catlee] from comment #3)
> Do these run as part of the build, or as a separate test job? If the latter,
> then we can't use mach and will need to know the underlying commands that
> mach is executing. We'll also have to make sure all the tests are packaged
> up in the tests archive.

Yes, they run as part of the build.
The webapprt chrome tests are packaged in obj-*/_tests/testing/mochitest/webapprtChrome.
Summary: Add support for webapprt test jobs & enable them per push on Cedar → Add support for webapprt-test-chrome test jobs & enable them per push on Cedar
Catlee, this doesn't actually have to be run by mach on the builders. You can run these using the standard packaged tests using slight tweaks to the existing mochitest command lines. We need to ensure that the webapprt-stub application is bundled with our builds (which I believe it should be, since it's in the dist/bin directory of our builds. Here are the command lines required (you'd still need to supply xre-path, certificate-path, and utility-path but they are the standard values that other mochitests use so I didn't include them here.

webapprt-content tests: (important ones here are --webapprt-content, and --app)
python runtests.py --webapprt-content --app=../../../dist/bin/webapprt-stub --autorun --close-when-done --console-level=INFO
Currently on my m-c build this runs 6 tests all of which pass.

webapprt-chrome tests: (important ones here are --webapprt-chrome, --app, --browser-arg and --testing-modules-dir)
python runtests.py --webapprt-chrome --app=../../../dist/bin/webapprt-stub --autorun --close-when-done --console-level=INFO --browser-arg=-test-mode --extra-profile-file=/home/ctalbert/projects/m-c/obj-dbg-linux/dist/plugins --testing-modules-dir=/home/ctalbert/projects/m-c/obj-dbg-linux/_tests/modules/
Currently on my m-c build: this runs 220 tests which pass and 1 which fails

So, we should be able to tweak the existing mozharness scripts for this support and we don't need to run them on builders.
(In reply to Clint Talbert ( :ctalbert ) from comment #5)
> webapprt-content tests: (important ones here are --webapprt-content, and
> --app)

Note that this bug is just about the webapprt-chrome tests, which is the first test suite that we'd like to start running per-commit.


> webapprt-chrome tests: (important ones here are --webapprt-chrome, --app,
> --browser-arg and --testing-modules-dir)
...
> Currently on my m-c build: this runs 220 tests which pass and 1 which fails

I filed the failure as bug 976300.
See Also: → 762744
Depends on: 976300
Depends on: 985015
Depends on: 986163
We're back to green!  catlee: What's the next step?
Flags: needinfo?(catlee)
Per comment 7, what's the next step? Can we help in some way to make this happen?
Flags: needinfo?(emorley)
Flags: needinfo?(ctalbert)
(In reply to Marco Castelluccio [:marco] from comment #8)
> Per comment 7, what's the next step? Can we help in some way to make this
> happen?

The next step will need to come from releng (I'm A-Team).
Flags: needinfo?(emorley)
We could really use some help writing the patches. Does ateam have anybody who can help?
Flags: needinfo?(catlee) → needinfo?(jgriffin)
We're going to need three patches:  one to in-tree configs, one to mozharness, and one to buildbot-configs.  I'll start the ball rolling, but won't be able to finish before I PTO.
Flags: needinfo?(jgriffin)
Whiteboard: [leave open]
Flags: needinfo?(ctalbert)
This adds a webapprt section to the in-tree configs, which includes all of the common options for chrome/content webapprt tests.  The exceptional options will be specified in buildbot-configs.
Attachment #8412763 - Flags: review?(jmaher)
Assignee: nobody → jgriffin
Comment on attachment 8412763 [details] [diff] [review]
Add webapprt to in-tree configs,

Review of attachment 8412763 [details] [diff] [review]:
-----------------------------------------------------------------

the fun begins
Attachment #8412763 - Flags: review?(jmaher) → review+
I'm going to land this on ash to make sure it's generally harmless.
Attachment #8412801 - Flags: review?(armenzg)
Schedule the webapprt-chrome tests on all desktop platforms on cedar.  Shouldn't land until after the mozharness patch does.
Attachment #8412857 - Flags: review?(aki)
Comment on attachment 8412857 [details] [diff] [review]
Schedule webapprt-chrome on cedar,

lgtm, you're a pro at this!
Attachment #8412857 - Flags: review?(aki) → review+
Comment on attachment 8412801 [details] [diff] [review]
Mozharness support for webapprt tests,

Review of attachment 8412801 [details] [diff] [review]:
-----------------------------------------------------------------

The patch looks good to me, however, I would like to get jlund's blessings since it is a non-standard new-{mochitest,reftest,known_category} type of patch.
Attachment #8412801 - Flags: review?(armenzg) → review?(jlund)
New try run on ash: https://tbpl.mozilla.org/?tree=Ash&rev=31322f05bd0b (looks good so far)
ash looks good; I think this can be landed on mozharness after r+
Comment on attachment 8412801 [details] [diff] [review]
Mozharness support for webapprt tests,

lgtm! awesome to see more suites being added. :)

I noticed that by supplying --webapprt-content, --browser-arg will be extended with '-test-mode' directly inside mochitest's runtests.py: http://mxr.mozilla.org/mozilla-central/source/testing/mochitest/runtests.py#1250

is there a reason why we explicitly pass --browser-arg=-test-mode for webapprt-chrome? Doesn't really matter j/w.

also, we can't land this patch, even on default, until https://hg.mozilla.org/integration/mozilla-inbound/rev/d4684d3c0340 merges with m-c (as of now it hasn't) otherwise everything will burn like it did in ash: https://tbpl.mozilla.org/?tree=Ash&rev=779568f0fd70

which makes me wonder if we have implemented self.tree_config correctly. As in, even if we are trying to run a mochitest or reftest suite, by just having 'self._run_category_suites('webapprt')' in desktop_unittest.py, we will fail  due to http://mxr.mozilla.org/build/source/mozharness/scripts/desktop_unittest.py#251

maybe the answer is to not bother looking up abs_base_cmd unless we are actually running that suite: eg: http://mxr.mozilla.org/build/source/mozharness/scripts/desktop_unittest.py#381 after http://mxr.mozilla.org/build/source/mozharness/scripts/desktop_unittest.py#387

IIUC we will need a TBPL patch for the new suite.
Attachment #8412801 - Flags: review?(jlund) → review+
Assignee: jgriffin → armenzg
(In reply to Jordan Lund (:jlund) from comment #22)
> Comment on attachment 8412801 [details] [diff] [review]
> which makes me wonder if we have implemented self.tree_config correctly. As
> in, even if we are trying to run a mochitest or reftest suite, by just
> having 'self._run_category_suites('webapprt')' in desktop_unittest.py, we
> will fail  due to
> http://mxr.mozilla.org/build/source/mozharness/scripts/desktop_unittest.
> py#251
> 
> maybe the answer is to not bother looking up abs_base_cmd unless we are
> actually running that suite: eg:
> http://mxr.mozilla.org/build/source/mozharness/scripts/desktop_unittest.
> py#381 after
> http://mxr.mozilla.org/build/source/mozharness/scripts/desktop_unittest.
> py#387
> 
> IIUC we will need a TBPL patch for the new suite.

Yes, this was very surprising when this failed on ash, even though we weren't running the tests yet there.  I think we should make the change you suggest.  We'll need to do this before running this on trunk branches, so we don't have to check in the in-tree configs for this in all upstream branches.
Return this bug to jgriffin to follow up on comment 25.
Let me know if you would prefer me to take care of it next week.
Assignee: armenzg → jgriffin
This includes the change suggested in bug 920569 comment #25
Attachment #8422762 - Flags: review?(jlund)
Attachment #8412801 - Attachment is obsolete: true
Comment on attachment 8422762 [details] [diff] [review]
Mozharness support for webapprt tests,

lgtm! :)
Attachment #8422762 - Flags: review?(jlund) → review+
Comment on attachment 8422762 [details] [diff] [review]
Mozharness support for webapprt tests,

https://hg.mozilla.org/build/mozharness/rev/5f1962b1e8a4
Attachment #8422762 - Flags: checked-in+
Comment on attachment 8422762 [details] [diff] [review]
Mozharness support for webapprt tests,

in production
Something here went live today
And promptly started failing on some machines with:

========= Started '/tools/buildbot/bin/python scripts/scripts/desktop_unittest.py ...' failed (results: 2, elapsed: 0 secs) (at 2014-05-23 07:23:24.672020) =========
/tools/buildbot/bin/python scripts/scripts/desktop_unittest.py --webapprt-suite chrome --blob-upload-branch cedar --download-symbols ondemand
…
Required config file not set! (use --config-file option)
program finished with exit code 255
elapsedTime=0.288343
========= Finished '/tools/buildbot/bin/python scripts/scripts/desktop_unittest.py ...' failed (results: 2, elapsed: 0 secs) (at 2014-05-23 07:23:24.980298) =========


And other machines with:

07:44:11  WARNING -  TEST-UNEXPECTED-FAIL | (browser-test.js) | Found an unexpected webapprt:mochitest
…
07:44:57  WARNING -  TEST-UNEXPECTED-FAIL | chrome://mochitests/content/webapprtChrome/webapprt/test/chrome/browser_alarm.js | Test timed out
07:44:57  WARNING -  TEST-UNEXPECTED-FAIL | chrome://mochitests/content/webapprtChrome/webapprt/test/chrome/browser_alarm.js | Cleanup function threw an exception at chrome://mochitests/content/webapprtChrome/webapprt/test/chrome/browser_alarm.js:35 - TypeError: mutObserverAlarmSet is null
etc.
(In reply to Myk Melez [:myk] [@mykmelez] from comment #33)
> 07:44:11  WARNING -  TEST-UNEXPECTED-FAIL | (browser-test.js) | Found an
> unexpected webapprt:mochitest
> …
> 07:44:57  WARNING -  TEST-UNEXPECTED-FAIL |
> chrome://mochitests/content/webapprtChrome/webapprt/test/chrome/
> browser_alarm.js | Test timed out
> 07:44:57  WARNING -  TEST-UNEXPECTED-FAIL |
> chrome://mochitests/content/webapprtChrome/webapprt/test/chrome/
> browser_alarm.js | Cleanup function threw an exception at
> chrome://mochitests/content/webapprtChrome/webapprt/test/chrome/
> browser_alarm.js:35 - TypeError: mutObserverAlarmSet is null
> etc.

I've seen the mutObserver error happening intermittently, but bug 1007402 should fix this.

The failure in the debugger test is due to bug 859372 (filed bug 1014141 to fix the regression).
I'll look at the 'config file not set' problem.
This fixes the 'config file not found' errors.  It looks like hg's fuzzing failed on the earlier patch, due to all the similar chunks of code in this file.
Attachment #8429692 - Flags: review?(jlund)
Comment on attachment 8429692 [details] [diff] [review]
Fix 'config file not found' errors,

lgtm :)
Attachment #8429692 - Flags: review?(jlund) → review+
Blocks: 1017254
Merged to production and deployed.
The most recent builds are pretty busted in general, but this test suite is still showing a config file issue, "Required config file not set! (use --config-file option)":

https://tbpl.mozilla.org/php/getParsedLog.php?id=40662469&tree=Cedar&full=1
(In reply to Myk Melez [:myk] [@mykmelez] from comment #40)
> The most recent builds are pretty busted in general, but this test suite is
> still showing a config file issue, "Required config file not set! (use
> --config-file option)":
> 
> https://tbpl.mozilla.org/php/getParsedLog.php?id=40662469&tree=Cedar&full=1

That log is from yesterday, and the change that fixes that was just merged to production this morning.  I've just kicked off a new build on cedar, so we can see the results soon.
Linux and OSX builds are going on cedar; there's a problem with Windows builds that's likely the fault of the mozharness script; I'll take a look.
This fixes the 'Error: Path C:\slave\test\build\application\firefox\webapprt-stubexe doesn't exist.' error on Windowds.
Attachment #8431912 - Flags: review?(armenzg)
Comment on attachment 8431912 [details] [diff] [review]
Add missing . in EXE_SUFFIX,

Review of attachment 8431912 [details] [diff] [review]:
-----------------------------------------------------------------

Fun!
Attachment #8431912 - Flags: review?(armenzg) → review+
These are running on cedar now.   They will show up as U until bug 1017254 is fixed, but otherwise there aren't any infrastructure issues remaining.
In production with reconfig on 2014-06-03 00:53 PT
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.