run unittests on N8x0 device with fennec using buildbot

RESOLVED FIXED

Status

Release Engineering
General
P2
normal
RESOLVED FIXED
9 years ago
4 years ago

People

(Reporter: jmaher, Assigned: aki)

Tracking

(Depends on: 1 bug, {mobile})

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(3 attachments, 1 obsolete attachment)

(Reporter)

Description

9 years ago
Created attachment 346079 [details]
Initial python script to run mochitests to avoid OOM, timeout, hangs

We need to run mochitests on fennec in the same fashion that we do for firefox.  The issue with fennec is we need to run on a mobile device.  Currently we support the Nokia 800/810 device, but we will support windows mobile devices in the near future.  In addition to mobile devices, we can run on desktop versions as well if we find it necessary to supplement.

Here are some of the issues with running on the mobile device (N8xx):
1) getting the device, physically locating it, managing it
2) connecting to the device (N810 has USB cable, N8xx has wifi)
3) setting up device with SSH access and other required tools
4) adding a memory card for additional storage, swap space
5) cross compiling a build of fennec in scratchbox 
6) packaging up fennec (make package in $objdir/mobile)
7) packaging up test files (xulrunner/_profile, xulrunner/_tests)
8) copying packaged files to device (and not running out of room)
9) installing packages on device
10) running test drivers on device
11) uploading/copying results off device
12) dealing with 20+ hour runs for mochitests (maybe only run on nightly builds)
13) troubleshooting hangs, status, device issues remotely

I have attached a run_tests_chunked.py file which is required to avoid a long list of OOM, Timeout, and hung test errors.  This resolves the problem by running mochitests on each directory (not as a single test with many subdirs) and going a step further and splitting larger directories up into smaller chunks.  Lastly, it times out after 60 minutes if you are hung and will terminate the test.

I can provide some additional tools, scripts to aide in this process.  My understanding is there is an existing framework developed to manage this process which might make my scripts not useful or incompatible.
Attachment #346079 - Attachment mime type: text/x-python → text/plain

Comment 1

9 years ago
Some helpful how-to pages on building and moving the tests over:
* Building: https://wiki.mozilla.org/Mobile/Build/Fennec
* Putting the tests on the device: https://wiki.mozilla.org/Mobile/Fennec_Mochitest

And my scratchbox mozconfig file for building a test-enabled fennec:
mk_add_options MOZ_BUILD_PROJECTS="xulrunner mobile"
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/../mobile-dbg

# clint additions
ac_add_options --enable-debug
ac_add_options --disable-optimize
# ac_add_options --disable-dbus
ac_add_options --disable-crashreporter

ac_add_options --enable-tests
ac_add_options --enable-mochitest

# XULRunner options
ac_add_app_options xulrunner --enable-application=xulrunner
ac_add_app_options xulrunner --disable-javaxpcom

# mobile options
ac_add_app_options mobile --enable-application=mobile
ac_add_app_options mobile --with-libxul-sdk=../xulrunner/dist
(Assignee)

Updated

9 years ago
Assignee: nobody → aki
From talking with Stuart, the mochitest suite is just the first testsuite they are looking for. They actually want all the suites that are run in a typical UnitTest run for a desktop Firefox build:
  TUnit, reftest, crashtest, mochitest, chrome, browser

Updating this bug to match this.
Keywords: mobile
Priority: -- → P2
Summary: run mochitests on N8x0 device with fennec using buildbot → run unittests on N8x0 device with fennec using buildbot

Comment 3

9 years ago
(In reply to comment #2)
> From talking with Stuart, the mochitest suite is just the first testsuite they
> are looking for. They actually want all the suites that are run in a typical
> UnitTest run for a desktop Firefox build:
>   TUnit, reftest, crashtest, mochitest, chrome, browser

I totally agree with this.  However, from my understanding to date, the browser tests aren't going to work on Fennec because we've changed the underlying architecture.  For example, there is no global gBrowser object in Fennec, and so I'd expect many of the tests to fail.  Just trying to start it out using the command line:  python runtests.py --appname=../../../../mobile/dist/bin/fennec --browser-chrome  and I don't even get the browser-chrome test to come up, I just get a blank fennec window.

So, I think the list we really should focus on is: TUnit, reftest, crashtest, mochitest, chrome

I think that if we are going to get browser tests to work on Fennec, we're going to need to re-implement some of the tests and quite possibly refactor some of the test framework when it comes to browser-chrome.
(Assignee)

Updated

9 years ago
Depends on: 463156
Depends on: 463657
We can not build on these n8x0 devices; instead we can only run tests on what was build previously elsewhere. Also, given how slow these devices are, we really need to be able to run separate unittest suites concurrently on different devices. 

Both of these requirements mean we need to be able to run unittests on pre-existing build. Adding dependency and moving to Future per discussions with Aki and Ted.
Assignee: aki → nobody
Component: Release Engineering → Release Engineering: Future
Depends on: 421611
No longer depends on: 421611

Comment 5

9 years ago
can we get an eta from ted as to when the unittest without build stuff is coming?  would be nice to have releng tee'd up once this is ready.
Depends on: 421611
I'm aiming to have it done in Q1, but that's kind of a SWAG. Joel is also proceeding pretty well without me with his own set of scripts. My work is a lot of taking what he's already doing and integrating it into the build system so that it's easy for everyone to use.
(Reporter)

Comment 7

9 years ago
My original steps to this bug outline how what is necessary to get mochitests running on maemo.  We already build in a scratchbox environment, package up the bits, and copy to the device.  This process is the same for the tests.  They come from the same place you build the fennec binary (the scratchbox environment).  

Right now each test suite can run on its own (mochitest, reftest, crashtest, chrome, xpcshell) independent of each other.  One big problem is how to get mochitests to run in <1 day!  That is not in scope for any bugs or discussed as a blocking issue here.

As Ted mentioned above, he is taking pieces of the work I have done (as well as other TODO items) and rolling them into the build tree.  The end result is we will still have to copy the test bits from the build tree to the device, unpack them and run them.  This is the same process I originally outlined above.

What is the ideal environment for the unittest to run on mobile?
(Reporter)

Comment 8

9 years ago
My question in comment #7 is not too clear.  My confusion stems from wanting to know what we need to do in order to get the unittests running on tinderbox/buildbot.  It makes sense that we would only want to modify the tinderbox scripts once to accommodate fennec.  The problem is we don't have a clear understanding of what we need to do as a final goal.

Here are a list of issues we will need to overcome:
* packaging up the tests (works great, ted's patch(s) makes it more uniform)
* getting the tests to a device 
* running the tests reliably (using a script to chunk up the run)
* reporting the results
** results are in chunked logs, not a single log
** results are on a remote device
** results have a long list of failures (http://people.mozilla.com/~jmaher/mochitest.htm)
* need a method for removing tests which are not relevant on maemo (tests that require a cursor)
* need a method for making the tests run faster (mochitests take 22+ hours now) such as running in parallel
* dealing with communication and run time issues on other platforms.

These are some of the big issues that I imagine we will need to figure out.  So my question in comment #7 should be phrased more like "what framework is required for unittests to run in tinderbox?"  

I suspect the ideal environment is exactly what firefox has.  In reality we need to figure out how we would like to see this work.  If it requires a lot of patches to the test harnesses we need to outline those and get bugs filed.  

Since this is not well defined, I don't see us getting any unittests running in Q1 2009.  So is it worth it to figure out an interim solution so we can get daily results for the 1.0 release?
(Assignee)

Updated

9 years ago
Assignee: nobody → aki
(Assignee)

Comment 9

9 years ago
Joel: if you can get your new steps to me and/or in this bug, I will attempt to automate them in buildbot.
Status: NEW → ASSIGNED
Component: Release Engineering: Future → Release Engineering
Should be pretty similar to what catlee is doing in bug 383136.
(Reporter)

Comment 11

9 years ago
after the rework of the test harnesses to support running on arbitrary builds, this has become a little easier.  I have methods for running:
Mochitest - https://wiki.mozilla.org/Mobile/Fennec_Mochitest
Mochitest-chrome - https://wiki.mozilla.org/Mobile/Fennec_Chrome
reftest/crashtest - https://wiki.mozilla.org/Mobile/Fennec_Reftests
XPCShell - https://wiki.mozilla.org/Mobile/Fennec_Xpcshell

step by step:
1) build in scratchbox
2) make package <- in $(objdir)
3) make package-test PYTHON=python2.5 <- in $(xr_objdir)
4) copy over .tar files to device storage card (/media/mmc1/release)
5) copy over http://hg.mozilla.org/qa/maemkit/ to /media/mmc1/maemkit
6) in the maemkit dir, run: "python maemkit-chunked.py --testtype=mochitest"
7) wait ~10 hours (non debug build) and view /media/mmc1/maemkit/log_mochitest.txt
8) repeat for --testtype=[chrome|reftest|crashtest|xpcshell]

If there are issues, most likely it is related to the maemkit.cfg file which has the paths and config options.  

Those are the basic steps.  If you get this far, lets sort out the way to split this across multiple devices (this works in maemkit.cfg with the total-clients and client-number options).
(Assignee)

Comment 12

9 years ago
Created attachment 376148 [details] [diff] [review]
add 'make package-tests' and upload xulrunner*.tar.bz2

Package/upload maemo tests.
Patch 1 of 2.
Attachment #376148 - Flags: review?(bhearsum)
(Assignee)

Comment 13

9 years ago
Created attachment 376151 [details] [diff] [review]
update maemo mozconfig to enable tests

|make package-tests| breaks without these set =)
Attachment #376151 - Flags: review?(bhearsum)
Comment on attachment 376148 [details] [diff] [review]
add 'make package-tests' and upload xulrunner*.tar.bz2

Seems fine...out of curiosity, why do you need PYTHON=python2.5?
Attachment #376148 - Flags: review?(bhearsum) → review+
Comment on attachment 376151 [details] [diff] [review]
update maemo mozconfig to enable tests

afaict --enable-mochitest is implied with --enable-tests...is that incorrect for mobile?
(Assignee)

Comment 16

9 years ago
re: both, I'm just following jmaher's directions =)
I'm guessing python2.5 because the default python in scratchbox is python2.3.
I can try without one or both on staging if you want.
(Reporter)

Comment 17

9 years ago
we need PYTHON=python2.5 for the 'make package-tests' step where we package up the xpcshell tests.  When ted rewrote the script to work on arbitrary builds in bug 421611, he made a requirement for python2.5.  The scratchbox environment (cross compile to maemo) has python 2.3? which would not copy the xpcshell tests to the test package.
(In reply to comment #17)
> we need PYTHON=python2.5 for the 'make package-tests' step where we package up
> the xpcshell tests.  When ted rewrote the script to work on arbitrary builds in
> bug 421611, he made a requirement for python2.5.  The scratchbox environment
> (cross compile to maemo) has python 2.3? which would not copy the xpcshell
> tests to the test package.

OK, thanks for the explanation.
(In reply to comment #15)
> (From update of attachment 376151 [details] [diff] [review])
> afaict --enable-mochitest is implied with --enable-tests...is that incorrect
> for mobile?

Ted just told me that --enable-mochitest doesn't exist anymore. Please remove it - --enable-tests should do the job here.
The PYTHON bit shouldn't be required anymore, I landed a patch that explicitly looks for the `python2.5` binary before `python`:
http://mxr.mozilla.org/mozilla-central/source/configure.in#814
(this hasn't landed on 1.9.1, however)
(In reply to comment #20)
> The PYTHON bit shouldn't be required anymore, I landed a patch that explicitly
> looks for the `python2.5` binary before `python`:
> http://mxr.mozilla.org/mozilla-central/source/configure.in#814
> (this hasn't landed on 1.9.1, however)

...umm... when can this land on 191? We're working on moving mobile builds to using 1.9.1 as we get closer to release date, so will need this there also, yes?
Didn't really seem necessary, but if you want it feel free to request approval on the patch in bug 427750.
(Assignee)

Comment 23

9 years ago
Created attachment 376440 [details] [diff] [review]
remove --enable-mochitest

As for PYTHON=python2.5, I think I'll leave it there in the meantime, so it works on mozilla-1.9.1 once the build goes green =)
Unless the 1.9.1 review/landing process is relatively quick.
Attachment #376151 - Attachment is obsolete: true
Attachment #376440 - Flags: review?(bhearsum)
Attachment #376151 - Flags: review?(bhearsum)
Attachment #376440 - Flags: review?(bhearsum) → review+
(Assignee)

Updated

9 years ago
Attachment #376440 - Flags: checked‑in+
(Assignee)

Comment 24

9 years ago
Comment on attachment 376440 [details] [diff] [review]
remove --enable-mochitest

rev 1136:a76c6dfdacbd
Should get picked up automatically.
(Assignee)

Comment 25

9 years ago
Comment on attachment 376148 [details] [diff] [review]
add 'make package-tests' and upload xulrunner*.tar.bz2

rev 273:c90e65ea30d9
p-m reconfig'ed, kicked off test build.
Attachment #376148 - Flags: checked‑in+
(Assignee)

Updated

9 years ago
Depends on: 471711, 471579
(Assignee)

Comment 26

9 years ago
Current status:

o Running with jmaher's patches from bug 495164.
o Mochitest keeps crashing on production, so I'm splitting it into mochitest1 and mochitest2.
o Hoping jmaher's fixes stabilize the number of tests; I've created new unittest parsers to deal with additional knownFailCount since the functions in buildbotcustom.steps.unittest aren't easily overridable/customizable.
o mochi, chrome, and xpcshell look like they'll be ready soon. crashtest and reftest are still 100% T-FAIL.
(Assignee)

Comment 27

9 years ago
(crashtest and reftest tracked in bug 471579 )
(Assignee)

Comment 28

8 years ago
These are running and reporting to the Mobile tinderbox tree.

We should continue to adjust the threshold between orange and green, and bring the known mobile fail count to 0, but as far as running and reporting we're done.
Status: ASSIGNED → RESOLVED
Last Resolved: 9 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.