988045 - Static and/or dynamic analysis to find SM error handling bugs

Status: NEW

(Core :: JavaScript Engine, enhancement, P5)

Description

[Jason Orendorff [:jorendorff]]

I can start with finding places where we're missing a JS_CHECK_RECURSION, since that should be evident from the call graph.

What I'm really after is stupid bugs where we don't check for an error at all, or we check but then control flows back out of the `if (!)` block without doing actual error handling.

Harder to catch: a case where we do check, and we *do* return false or null, but we leave bad state on the JS heap. Hmm.

Comment 1

[Jason Orendorff [:jorendorff]]

Competing idea: jseward mentioned you could write a valgrind tool to detect when a pointer flows from an allocation function to a read or write, without first controlling a conditional branch. Dynamic, not static, but you wouldn't actually have to trigger OOM in order to test for it. Not sure that'd be worth the substantial time investment.

Our error handling rules are not simple enough. Not simple enough for computers and evidently not simple enough for people.

Comment 2

[Nicolas B. Pierron [:nbp]]

Steve was recently asking for ideas on what we could potentially do with the call graph that is extracted when doing the rooting analysis.
This could be one of such idea.