OOM calling walk_tree from process_cp_pre_genericize

RESOLVED FIXED

Status

()

Core
Rewriting and Analysis
RESOLVED FIXED
8 years ago
7 years ago

People

(Reporter: Ehren Metcalfe, Unassigned)

Tracking

Trunk
x86
Linux
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

1.02 KB, patch
(dormant account)
: review+
Details | Diff | Splinter Review
(Reporter)

Description

8 years ago
Created attachment 423563 [details] [diff] [review]
patch

Calling walk_tree from process_cp_pre_genericize will introduce an out of memory error when certain strings are encountered. The actual problem function is lazy_tree_string in treehydra.c. AFAIK this only results from compiling functions with inline assembly. stack.js is currently broken for example.

This is due to an inconsistency in the TREE_STRING_LENGTH macro which doesn't count the null byte with certain (all?) empty strings, even though it's counted in every other case. The result: a negative number gets passed to xmalloc (instant OOM).

I can post a reduced testcase but this affects many files in mozilla-central.
Attachment #423563 - Flags: review?(tglek)

Comment 1

8 years ago
Comment on attachment 423563 [details] [diff] [review]
patch

thats probably ok, can you get bsmedberg or dmandelin to review and land?
(Reporter)

Updated

8 years ago
Attachment #423563 - Flags: review?(tglek) → review?(benjamin)
Attachment #423563 - Flags: review?(benjamin) → review?(tglek)

Comment 2

7 years ago
Comment on attachment 423563 [details] [diff] [review]
patch

>diff -r d039748e7775 treehydra.c
>--- a/treehydra.c	Thu Jan 14 19:06:56 2010 -0800
>+++ b/treehydra.c	Mon Jan 25 12:42:57 2010 -0500
>@@ -206,17 +206,22 @@ void lazy_tree_string (struct Dehydra *t
> 
>   // now reflect .str, account for unicode magic (bug 526970)
>   tree str_type = TREE_TYPE (str);
>   if (str_type && TYPE_PRECISION (TREE_TYPE (str_type)) == TYPE_PRECISION (char_type_node)) {
>     wchar_bytes = 1;
>   } else {
>     wchar_bytes = TYPE_PRECISION (wchar_type_node) / BITS_PER_UNIT;
>   }
>-  num_chars = (TREE_STRING_LENGTH (str) / wchar_bytes) - 1; // skip trailing null
>+  num_chars = (TREE_STRING_LENGTH (str) / wchar_bytes);
>+  // TREE_STRING_LENGTH is 0 for certain empty strings

Eww. Nice find.
Attachment #423563 - Flags: review?(tglek) → review+
(Reporter)

Comment 3

7 years ago
http://hg.mozilla.org/rewriting-and-analysis/dehydra/rev/04027671bd96
(Reporter)

Updated

7 years ago
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.