1037059 - Make phonedash optional and generate local reports

Status: RESOLVED FIXED

(Testing Graveyard :: Autophone, defect)

Description

[Mark Côté [:mcote]]

Technically Autophone will run without a working phonedash instance, but I believe it'll report some errors in the logs.  Regardless, for Autophone to be truly useful without phonedash, it will have to be able to output results locally in some format.

The basic output should be text; a graph doesn't make sense unless we want to allow data to be accumulated locally over several runs.  Not sure if that's needed; perhaps best as a second step after seeing how useful text-only output is.  Another idea using graphs would be to somehow compare it against production results; maybe we could have a system that pulled data from a phonedash instance (defaulting to phonedash.m.o) and combined it with local results from one or more runs into a single local report.

Comment 1

[Geoff Brown [:gbrown]]

I initially liked this idea, but after setting up my own autophone development environment, I'm less inclined to make any changes here.

Phonedash is easy to set up and well documented. It took me maybe 10 minutes to get from googling "mozilla phonedash" to a working instance. It took longer to figure out autophone, but once autophone was working, I loved seeing my test results in a graph, just like phonedash.mozilla.org. In terms of "developer ergonomics" and thinking specifically of an autophone newbie trying to run autophone on their own computer, I think it's best to set up and use phonedash to see test results.

Also, there is simple text output of results in the existing logging (autophone.log):

2014-09-10 18:55:23,871|DEBUG|gbrown|15617|20140910171654|remote-blank|RESULTS: Cached: True Start Time: 187.0 Throbber Start: 1336.0 Throbber Stop: 3734.0 Total Throbber Time: 2398.0 Rejected: False

That's mixed in with other log messages of course, not in a dedicated report...but all the data is there and it is easily discoverable.

This ain't broke; let's wontfix it.

Comment 2

[Bob Clary [:bc] (inactive)]

Attachment: autophone-results-nexus-one-2.csv

What do you think of this as an output option? Each device would generate a csv file that would be appended to for each run until it is deleted?

Comment 3

[Mark Côté [:mcote]]

Comment on attachment 8498203 [details]
autophone-results-nexus-one-2.csv

Seems like a decent idea to me.  Easy both to continually capture results over multiple runs and to reset the data.

Comment 4

[James Willcox (:snorp) ([e-mail]) (he/him)]

Comment on attachment 8498203 [details]
autophone-results-nexus-one-2.csv

Yup, better than nothing I think.

Comment 5

[Bob Clary [:bc] (inactive)]

snorp, is there a different format you would like? This is for you so, if you would like something different please let me know.

csv was easiest to accumulate results via appending to a file and I just used the raw data submitted to the phonedash server. I could do a different format or support additional ones and could also munge the data to do the throbberstart-starttime, throbberstop-stoptime calculations for you.

Comment 6

[James Willcox (:snorp) ([e-mail]) (he/him)]

(In reply to Bob Clary [:bc:] from comment #5)
> snorp, is there a different format you would like? This is for you so, if
> you would like something different please let me know.
> 
> csv was easiest to accumulate results via appending to a file and I just
> used the raw data submitted to the phonedash server. I could do a different
> format or support additional ones and could also munge the data to do the
> throbberstart-starttime, throbberstop-stoptime calculations for you.

Munging stuff to get throbber start/stop would be helpful. CSV is fine too, I think. Should be easy to throw it into google docs or whatever.

If I could have anything I want, though, it would be the ability to throw a build up somewhere for autophone to run and see how it compares to nightly. This is the "did my patch improve/regress anything" use case. With your CSV thing we can do that, but it requires us to manually compare values.

Comment 7

[Bob Clary [:bc] (inactive)]

I think bug 1037064 - Allow use of custom local and remote builds will help with the comparison. I'll make sure to have the use case you mentioned covered there. Setting up a local phonedash (which is pretty easy) would allow you to visually compare the runs. If you didn't set up phonedash and just dumped the results to csv, it might be easy enough to load into a spreadsheet graph. I'll add the difference calculations and we can continue to tweak as we knock down these developer oriented bugs.

Comment 8

[Bob Clary [:bc] (inactive)]

Attachment: bug-1037059-v1.patch

Comment 9

[Mark Côté [:mcote]]

Comment on attachment 8498858 [details] [diff] [review]
bug-1037059-v1.patch

Review of attachment 8498858 [details] [diff] [review]:
-----------------------------------------------------------------

::: tests/perftest.py
@@ +58,5 @@
>              self.stderrp_attempts = 1
> +        try:
> +            self._resultfile = None
> +            self._resulturl = self.cfg.get('settings', 'resulturl')
> +            if self._resulturl == 'None':

Should probably make this case insensitive.

@@ +198,5 @@
>  
> +    def dump_results(self, starttime=0, tstrt=0, tstop=0,
> +                     testname='', cache_enabled=True,
> +                     rejected=False):
> +        self._resultfile.write(('%s' + ',%s'*15 + '\n') % (

There's also the csv module which is slightly cleaner.

Comment 10

[Bob Clary [:bc] (inactive)]

re case: ok. yeah, i looked at the csv module and only saw the full file stuff and missed the writerow stuff. I'll take another look at it. want a new review if I do?

Comment 11

[Mark Côté [:mcote]]

Nah it's pretty straightforward.

Comment 12

[Bob Clary [:bc] (inactive)]

https://github.com/mozilla/autophone/commit/c1347d92c08c16d36f6c2a52da623d1beb14b915