better tools for showing overall crashiness trends on development builds from mozilla-central

RESOLVED WORKSFORME

Status

RESOLVED WORKSFORME
8 years ago
a year ago

People

(Reporter: chofmann, Unassigned)

Tracking

Trunk
x86
Mac OS X

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments)

(Reporter)

Description

8 years ago
to ship higher quality betas and effiently create stable branch points we need some better tools that tell us when mozilla-central builds are in good shape, and when they are not.

We can't use the standard crash per 100 user metric, but we can rely on crash counts and relatively stable updates rates for development build users.

I've put together some charts that help tell us about crash volume distribution of builds spanning over the last 10 days for any given day with a couple of different ways to express relative crashiness of recent builds and help us to see if we are trending up or down.

This first one shows crash counts for a given day, and which collection of bulds are producing the most crashes 

http://people.mozilla.com/~chofmann/crash-stats/20101129/floating-build-crash-rate.png

and the second one shows the same numbers expressed as a pct. of the total crashes.

http://people.mozilla.com/~chofmann/crash-stats/20101129/floating-build-crash-rate-pct.png

we need to start producing charts like this and integrate these into places like  http://crash-stats.mozilla.com/products/Firefox/versions/4.0b8pre/builds

data retrieval is pretty simple.

  foreach day (list_of_several_days_since_4.0b8pre_started_being_built) 
   print total_crash_count 
   print crash_count_from_blds_from_last_day last2days last3days...   
   print pct_of_crashes_in_builds_from_last_day last2days last3days
  end

then plot this table of numbers in two graphs.


I'll attach a .cvs of the table I used to generate the charts above as an example.
(Reporter)

Comment 1

8 years ago
Created attachment 494254 [details]
mozilla-central crashes for 4.0b8pre nov10-29
(Reporter)

Comment 2

8 years ago
Created attachment 494255 [details]
graph of same data from above .cvs
(Reporter)

Comment 3

8 years ago
for example the graph above shows that oct 13-21 would have been a bad point to create a branch or ship a beta.  by splitting the various recent builds and getting good visibiity of what's happening with latest builds we can pick up the actually state of the latest builds two or three days to a week earlier than just looking at the overall crash count for something like the total collection of 4.0b8pre builds
(Assignee)

Updated

7 years ago
Component: Socorro → General
Product: Webtools → Socorro

Comment 4

a year ago
we're moving rates and similar overviews to mission control instead
Status: NEW → RESOLVED
Last Resolved: a year ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.