757062 - Bughunter - handle oom and oof file errors better.

Status: RESOLVED FIXED

(Testing Graveyard :: Sisyphus, defect)

Description

[Bob Clary [:bc] (inactive)]

Bughunter's crashworkers currently can get into a bad state if the worker runs out of memory.

This can occur due to a test url which consumes all or most of the memory on the machine which then results in cascading failures in Bughunter's scripts which fail due to out of memory conditions. These cascading failures can also cause Bughunter to leave open file handles which accumulate until there are no more handles available.

To fix this, catch out of memory errors as early as possible during the Popen call to test the url as well as to encapulate the entire call to runTest to catch any cascading failures before they affect the entire worker. Attempt to recover by reloading the program and if not possible to recover, then exit.

Another area where out of memory errors can affect the worker is the during the processing of ASSERTION stacks via fix_stack_using_bpsyms.py. Encapsulate the calls to fixSimples in a similar fashion and in the event of an out of memory condition simply copy the raw input to output rather than failing.

Comment 1

[Bob Clary [:bc] (inactive)]

Attachment: patch

Comment 2

[Bob Clary [:bc] (inactive)]

http://hg.mozilla.org/automation/sisyphus/rev/db003902d13a