569423 - Sisyphus - Crash Automation - consolidate databases

Status: RESOLVED FIXED

(Testing Graveyard :: Sisyphus, defect)

Description

[Bob Clary [:bc] (inactive)]

Attachment: patch v1

The current separation of the crashtest, unittest and history databases prevents couchdb's list functions from combining results such as assertions, valgrind messages and crashes.

This patch consolidates the separate databases into a single database. The database name is no longer hard-coded but does default to sisyphus if not explicitly specified on the command line.

Most of the existing views are eliminated. Those specific to specific worker types are segrated into design documents specific to the worker type, e.g. _design/unittest, _design/crashtest and _design/builds. The _design/default design document now only contains views for workers and the combined logs.

The UI related views for results are consolidated in the _design/bughunter design document which is maintained as a couchapp. These views are "packed" by combining the different result document types into a single view which is then selected through the document type. This eliminates the need for separate views and provides more flexibility when combining result documents into lists.

A number of bug fixes and enhancements have crept into the patch during testing:

* replace the existing unpacked spider extension if the source xpi has changed.
* use the hg repo default rather than tip when building
* capture urls/locations for assertions and valgrind messages and maintain a list of urls/locations in the history documents for assertions and valgrind messages.
* make sure to update the history bug lists whenever a new url is found or the date range should be updated.
* exclude HEAP SUMMARY from the reported valgrind messages.
* change the head|tail locations to more descriptive values of startup and shutdown.
* store the "extra" data from a crash as fields and the crash reporter document as an attachment on the result_crash documents.
* remove ac_cv_* directives from the 1.9.2 linux mozconfigs as they now cause build errors.
* increase Spider's Firefox version compatibility and handle the 1.9.1 branch's invocation of the script loader in javascript strict mode.
* remove unused *hash fields from the signature documents.
* new script crashurlloader.py to upload specific urls to be tested.
* include the steps Spider performed during crash testing in the result_header_crashtest document.
* improve url/location detection in unittest result parsing.
* When restarting the script, call logMessage with the retry so that temporary database errors do not cause the script to terminate.

Comment 1

[Bob Clary [:bc] (inactive)]

Attachment: follow on patch 1, v1

missed the workers view.

Comment 2

[cmtalbert]

Comment on attachment 448588 [details] [diff] [review]
patch v1

This looks good.  Note that with Jesse's changes to include a TEST-START for mochitest and reftest, some of that logic for tracking tests will be made easier.  We can do that simplification in a separate bug though.  (Jesse's changes are on bug 570730)

Comment 3

[Bob Clary [:bc] (inactive)]

http://hg.mozilla.org/automation/sisyphus/rev/921bf5340f87

Comment 4

[Bob Clary [:bc] (inactive)]

http://hg.mozilla.org/automation/sisyphus/rev/dea512d2d366 add missing view.