542276 - OOM calling walk_tree from process_cp_pre_genericize

Status: RESOLVED FIXED

(Developer Infrastructure :: Source Code Analysis, defect)

Description

[Ehren Metcalfe]

Attachment: patch

Calling walk_tree from process_cp_pre_genericize will introduce an out of memory error when certain strings are encountered. The actual problem function is lazy_tree_string in treehydra.c. AFAIK this only results from compiling functions with inline assembly. stack.js is currently broken for example.

This is due to an inconsistency in the TREE_STRING_LENGTH macro which doesn't count the null byte with certain (all?) empty strings, even though it's counted in every other case. The result: a negative number gets passed to xmalloc (instant OOM).

I can post a reduced testcase but this affects many files in mozilla-central.

Comment 1

[(dormant account)]

Comment on attachment 423563 [details] [diff] [review]
patch

thats probably ok, can you get bsmedberg or dmandelin to review and land?

Comment 2

[(dormant account)]

Comment on attachment 423563 [details] [diff] [review]
patch

>diff -r d039748e7775 treehydra.c
>--- a/treehydra.c	Thu Jan 14 19:06:56 2010 -0800
>+++ b/treehydra.c	Mon Jan 25 12:42:57 2010 -0500
>@@ -206,17 +206,22 @@ void lazy_tree_string (struct Dehydra *t
> 
>   // now reflect .str, account for unicode magic (bug 526970)
>   tree str_type = TREE_TYPE (str);
>   if (str_type && TYPE_PRECISION (TREE_TYPE (str_type)) == TYPE_PRECISION (char_type_node)) {
>     wchar_bytes = 1;
>   } else {
>     wchar_bytes = TYPE_PRECISION (wchar_type_node) / BITS_PER_UNIT;
>   }
>-  num_chars = (TREE_STRING_LENGTH (str) / wchar_bytes) - 1; // skip trailing null
>+  num_chars = (TREE_STRING_LENGTH (str) / wchar_bytes);
>+  // TREE_STRING_LENGTH is 0 for certain empty strings

Eww. Nice find.

Comment 3

[Ehren Metcalfe]

http://hg.mozilla.org/rewriting-and-analysis/dehydra/rev/04027671bd96