507544 - Regression detection emails should attach a .png of the performance graph

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect, P4)

Description

[Chris AtLee [:catlee]]

It would be great if the regression detection script could include something equivalent to a screenshot of the relevant graph from graphs.m.o.

Comment 1

[Chris Cooper [:coop] (he/him)]

Mass move of bugs from Release Engineering:Future -> Release Engineering. See
http://coop.deadsquid.com/2010/02/kiss-the-future-goodbye/ for more details.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Code is at http://hg.mozilla.org/graphs/file/1.0/server/analysis

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

beltzner, is this something that would still be useful ? Or are the links to the graph server sufficient ?

Comment 4

[Mike Beltzner [:beltzner, not reading bugmail]]

It would definitely be useful, if we can get it right, as graph server loads take a long time and aren't really visible on mobile devices.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

OK. What size image would be appropriate for viewing on mobile ? Is panning around OK if I chose one that optimizes for clarity of the data ?

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Still planning to hit this before heading out. Time will tell how delusional I am.

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

I've installed libpng-devel, freetype-devel, gcc-g++, and libstdc++-devel on cruncher, without upgrading any existing packages and at matching versions to non-devel or existing gcc installed packages.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

To create a virtualenv for Matplotlib (from scratch)
  pip install numpy
  easy_install matplotlib
Both of those are compiles so they take a while and spew lots of alarming messages. I would have used pip for matplotlib but it insisted on pulling v0.9.1 instead of v1.0.0, and the earlier version tries to import deprecated module in numpy. Upgrading pip to v0.8.2 in the venv didn't help.

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

catlee, could you suggest how I might limit the analysis to a regression for debugging purposes ? Perhaps find a regression then use --test to limit to that platform & test combo, and add a -end-time argument that ends up in the SQL in source.getTestData ? 

I'm working from a copy of your prod system, with the top-level {regression_machine}_emails set to me, and the branch level ones commented out; bz api commented out too.

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to comment #6)
> Still planning to hit this before heading out. Time will tell how delusional I
> am.

Alas, I was delusional. Got as far as setting up a test bed; need to figure out how to limit the checks to a single test in a time window that's known to regress, and turn off caching to allow iteration during development (looks like most of that is already there). Then look at the data 
 http://hg.mozilla.org/graphs/file/4e34349e884b/server/analysis/analyze_talos.py#l424
gets to massage it into something plotable, then drag out the Matlab skills to use with matplotlib. Didn't look yet at attaching a file to the email.

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Virtualenv w/ numpy & matplotlib is at cruncher:~nthomas/virtualenvs/regress-plot-matplotlib (ie workon regress-plot-matplotlib).
Code in cruncher:~nthomas/graphs/server/analysis/.

Comment 12

[Chris Cooper [:coop] (he/him)]

I don't see the point in mailing around images. 

If you think there would be some benefit in developing a version of graphs.mozilla.org that was tailored better for mobile devices, I think that's the way to go here.