615765 - better tools for showing overall crashiness trends on development builds from mozilla-central

Status: RESOLVED WORKSFORME

(Socorro :: General, task)

Description

[chris hofmann]

to ship higher quality betas and effiently create stable branch points we need some better tools that tell us when mozilla-central builds are in good shape, and when they are not.

We can't use the standard crash per 100 user metric, but we can rely on crash counts and relatively stable updates rates for development build users.

I've put together some charts that help tell us about crash volume distribution of builds spanning over the last 10 days for any given day with a couple of different ways to express relative crashiness of recent builds and help us to see if we are trending up or down.

This first one shows crash counts for a given day, and which collection of bulds are producing the most crashes 

http://people.mozilla.com/~chofmann/crash-stats/20101129/floating-build-crash-rate.png

and the second one shows the same numbers expressed as a pct. of the total crashes.

http://people.mozilla.com/~chofmann/crash-stats/20101129/floating-build-crash-rate-pct.png

we need to start producing charts like this and integrate these into places like  http://crash-stats.mozilla.com/products/Firefox/versions/4.0b8pre/builds

data retrieval is pretty simple.

  foreach day (list_of_several_days_since_4.0b8pre_started_being_built) 
   print total_crash_count 
   print crash_count_from_blds_from_last_day last2days last3days...   
   print pct_of_crashes_in_builds_from_last_day last2days last3days
  end

then plot this table of numbers in two graphs.


I'll attach a .cvs of the table I used to generate the charts above as an example.

Comment 1

[chris hofmann]

Attachment: mozilla-central crashes for 4.0b8pre nov10-29

Comment 2

[chris hofmann]

Attachment: graph of same data from above .cvs

Comment 3

[chris hofmann]

for example the graph above shows that oct 13-21 would have been a bad point to create a branch or ship a beta.  by splitting the various recent builds and getting good visibiity of what's happening with latest builds we can pick up the actually state of the latest builds two or three days to a week earlier than just looking at the overall crash count for something like the total collection of 4.0b8pre builds

Comment 4

[Lonnen :lonnen]

we're moving rates and similar overviews to mission control instead