558045 - [TopFails] Figure out how to split up automation.py and automation.processLeakLog() failures

Status: RESOLVED WONTFIX

(Testing :: General, defect)

Description

[(not currently active) Ted Mielczarek]

dbaron notes:
http://groups.google.com/group/mozilla.dev.planning/msg/c73c3b4c7699ef7f

 * splitting up the items listed as automation.py (which is how it
  lists a test timeout) and automationutils.processLeakLog() (which
  is how it leaks a failure due to leaks) at least by what test
  the failure was on (mochitest-plain 2/5, mochitest-chrome, etc.)

  In the case of timeouts, though, we can generally categorize the
  timeout even more precisely by the last test that started
  running.

The current #1 and #2 entries in topfails are automation.py and automation.processLeakLog(). We should be able to split these up a bit more.

For automation.py, they're generally timeouts or crashes, so we should attribute them to the last test being run.

For processLeakLog, we should at least attribute it to the test suite being run, like "mochitest 3/5".

Comment 1

[Jeff Hammel]

This is at least mostly done, just to note.

Comment 2

[Jesse Ruderman]

We should fix this with changes to automation.py, so it can help with starring in addition to topfail statistics.  Bug 570723 will add crash signatures (but not hang samples), and bug 570730 will add which-test-was-running.