1158276 - Autophone - adapt to the split tests implemented in bug 917999

Status: RESOLVED FIXED

(Testing Graveyard :: Autophone, defect)

Comment 1

[Chris Manchester (limited bugmail, email directly)]

Bug 917999 is getting close. We already have what we should need to build in compatibility with the split test zip -- a json manifest (test_packages.json) is already being uploaded with each build. The file contains a mapping from test harness names to a list of file names in the build directory that the harness needs to run. I'm not sure which tests Autophone needs to run, but I'm planning to initially split out cppunittest, reftest, xpcshell, and mochitest.

One approach would be to download an unpack all the files that appear in the manifest (this will have the same end result as downloading an unpacking the monolithic testszip today).

Comment 2

[Bob Clary [:bc] (inactive)]

Attachment: bug-1158276-split-tests-configs.patch

Chris, do I understand correctly that for reftests I need to download common mozbase reftest and for mochitests I need to download common mozbase mochitest ?

Comment 3

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8621049 [details] [diff] [review]
bug-1158276-split-tests-configs.patch

Review of attachment 8621049 [details] [diff] [review]:
-----------------------------------------------------------------

Not exactly, sorry if I was unclear. The idea is to download test_packages.json, and look for your suite category in the keys (for instance, "reftest", or "mochitest"), and the corresponding value will be a list of packages to download.

For the initial implementation, a suite category will either have everything it needs in the "common" zip, or need the "common" zip and one other. http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-dfeaa184de1c/try-linux/ is an example of the uploads that result when this is turned on. (It's possible, but I don't think likely, some of this will change before landing.)

Comment 4

[Bob Clary [:bc] (inactive)]

Attachment: bug-1158276-split-tests-download.patch

This uses the test package names introduced in the test configs to determine which test zip files to download.

We store the test_packages.json in the build cache directory as well as the individual test zip files so we can determine later if they have been downloaded.

While testing the override-build-dir option and the use of the already downloaded test_packages.json to determine if the requested test packages had already been downloaded, I found an existing bug in buildserver.py:BuildCacheHandler that would not properly deal with an exception from the build_cache.get which would leave the server.cache_lock unreleased. 

I'm also taking advantage of this bug to fix a pyflakes warning I introduced earlier in PerfTest.publish_results.

Comment 5

[Bob Clary [:bc] (inactive)]

fyi, an example test_packages.json:

http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-inbound-android-api-9/1433956183/test_packages.json

Comment 6

[Bob Clary [:bc] (inactive)]

(In reply to Chris Manchester [:chmanchester] from comment #3)

Ok. A single category is sufficient. Thanks.

gbrown: consider the common and mozbase entries removed from the configs. I'll leave the ability to provide a list of category names though.

Comment 7

[Geoff Brown [:gbrown]]

Comment on attachment 8621049 [details] [diff] [review]
bug-1158276-split-tests-configs.patch

Review of attachment 8621049 [details] [diff] [review]:
-----------------------------------------------------------------

s/common mozbase//

Comment 8

[Geoff Brown [:gbrown]]

Comment on attachment 8621113 [details] [diff] [review]
bug-1158276-split-tests-download.patch

Review of attachment 8621113 [details] [diff] [review]:
-----------------------------------------------------------------

::: builds.py
@@ +581,5 @@
>              if enable_unittests and not os.path.exists(tests_path):
> +                raise BuildCacheException(
> +                    'Override Build Directory %s does not contain a tests directory.' %
> +                    self.override_build_dir)
> +            test_packages_json_path = os.path.join(self.override_build_dir,

it looks like test_packages_json_path could be set inside the following "if test_packages_name"...

@@ +689,4 @@
>                  # XXX: assumes fixed buildurl-> fennec_ids.txt mapping
>                  fennec_ids_url = urlparse.urljoin(buildurl, 'fennec_ids.txt')
>                  fennec_ids_path = os.path.join(cache_build_dir, 'fennec_ids.txt')
> +                if not os.path.exists(fennec_ids_path):

This reminds me, weren't you going to try removing use of fennec_ids.txt?

Comment 9

[Bob Clary [:bc] (inactive)]

I remember something about it not being necessary anymore but are you sure it isn't used by robocop any more?

Comment 10

[Geoff Brown [:gbrown]]

Oh right, fennec_ids.txt is still used by robocop.

Comment 11

[Bob Clary [:bc] (inactive)]

(In reply to Geoff Brown [:gbrown] from comment #8)
> Comment on attachment 8621113 [details] [diff] [review]
> bug-1158276-split-tests-download.patch
> 

> it looks like test_packages_json_path could be set inside the following "if
> test_packages_name"...
> 

ok

Comment 12

[Bob Clary [:bc] (inactive)]

https://github.com/mozilla/autophone/commit/41f5079ecf731e55821b34b85ae5430e5a6460d5
https://github.com/mozilla/autophone/commit/ca872113c03dd3b11609c8f3af95ed2bfc6011de

Thanks!