1268571 - Autophone - support debug builds

Status: RESOLVED FIXED

(Testing Graveyard :: Autophone, defect)

Description

[Bob Clary [:bc] (inactive)]

While we have some token coverage for supporting debug builds, in practice we only support opt.

In addition we have a global option to specifiy the build types and do not allow different build types for different tests. I think we can use a test config file and a buildtypes option to specify either one or both of opt, debug.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8756354 [details] [diff] [review]
bug-1268571-enable-debug-builds-v1.patch

Review of attachment 8756354 [details] [diff] [review]:
-----------------------------------------------------------------

some days a patch just makes sense- looking forward to this new support for debug builds!

::: autophonetreeherder.py
@@ +196,4 @@
>              tj.add_machine(machine)
>              tj.add_build_info('android', t.phone.platform, t.phone.architecture)
>              tj.add_machine_info('android',t.phone.platform, t.phone.architecture)
> +            tj.add_option_collection({build_type_from_url(build_url): True})

this is pretty tricky:)

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8756356 [details] [diff] [review]
bug-1268571-opt-debug-production-v1.patch

Review of attachment 8756356 [details] [diff] [review]:
-----------------------------------------------------------------

do we have concerns with load?  Also perf tests on debug builds is not very useful.

Comment 3

[Bob Clary [:bc] (inactive)]

(In reply to Joel Maher (:jmaher) from comment #4)
> Comment on attachment 8756354 [details] [diff] [review]
> bug-1268571-enable-debug-builds-v1.patch
> 
> Review of attachment 8756354 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> some days a patch just makes sense- looking forward to this new support for
> debug builds!
> 
> ::: autophonetreeherder.py
> @@ +196,4 @@
> >              tj.add_machine(machine)
> >              tj.add_build_info('android', t.phone.platform, t.phone.architecture)
> >              tj.add_machine_info('android',t.phone.platform, t.phone.architecture)
> > +            tj.add_option_collection({build_type_from_url(build_url): True})
> 
> this is pretty tricky:)

I'll have to do some major rework to deal with builds from the TC index.

(In reply to Joel Maher (:jmaher) from comment #5)
> Comment on attachment 8756356 [details] [diff] [review]
> bug-1268571-opt-debug-production-v1.patch
> 
> Review of attachment 8756356 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> do we have concerns with load?  Also perf tests on debug builds is not very
> useful.

Well, a bit of concern about load but since we are only enabling unit tests on mozilla-central for now and the debug builds appear to be about the same run time as opt I think we are ok at least initially. If we need to get tricky we can add more configs and narrow down the repos/buildtypes/tests.

Re perf tests: that was the motivation behind being able to restrict build types. With the updated configs, the perf tests will *only* run opt builds. You can see that in the example run on staging. phonedash no longer even supports the idea of plotting debug builds separately. I have considered the idea that we might want to enable debug builds on some of the perf tests in order to diagnose crashes but in that case I'd want to use a test/config that didn't report the actual performance measurements. Perhaps a special configuration for try would be sufficient.

Comment 4

[Bob Clary [:bc] (inactive)]

https://github.com/mozilla/autophone/commit/5e4a6904779e16d3b21fadf6133a0f25c9eb2503
https://github.com/mozilla/autophone/commit/d2f8bdb03a118bb7f169bc0c4b2eb20f06e1944e

Comment 5

[Bob Clary [:bc] (inactive)]

Attachment: bug-1268571-enable-debug-builds-v1.patch

We use the presence of debug in the build url to determine debug build status. We'll have to revisit this when we do the work to download builds from TC.

I've added buildtypes = opt to the performance related tests but left the value out of the unit tests. Therefore the perf tests will only run opt builds regardless of if autophone specifies only opt or both opt and debug. The unit tests will however follow the autophone settings and run exactly the buildtypes specified by autophone.

Comment 6

[Bob Clary [:bc] (inactive)]

Attachment: bug-1268571-opt-debug-production-v1.patch

This turns on opt and debug for production. As mentioned, the perf tests will only run opt but the unit tests will run both. Hopefully this will help developer's debug failures.

Comment 7

[Bob Clary [:bc] (inactive)]

Example run on staging:

https://treeherder.allizom.org/#/jobs?filter-searchStr=autophone&exclusion_profile=false&repo=mozilla-inbound&filter-tier=1&filter-tier=2&filter-tier=3&fromchange=a7d6b2c16a3f&tochange=a8b3e46732c7

Comment 8

[Bob Clary [:bc] (inactive)]

deployed 2016-05-25 23:30