OOM calling walk_tree from process_cp_pre_genericize

RESOLVED FIXED

Status

RESOLVED FIXED
9 years ago
a year ago

People

(Reporter: ehren.m, Unassigned)

Tracking

Trunk
x86
Linux

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Reporter)

Description

9 years ago
Created attachment 423563 [details] [diff] [review]
patch

Calling walk_tree from process_cp_pre_genericize will introduce an out of memory error when certain strings are encountered. The actual problem function is lazy_tree_string in treehydra.c. AFAIK this only results from compiling functions with inline assembly. stack.js is currently broken for example.

This is due to an inconsistency in the TREE_STRING_LENGTH macro which doesn't count the null byte with certain (all?) empty strings, even though it's counted in every other case. The result: a negative number gets passed to xmalloc (instant OOM).

I can post a reduced testcase but this affects many files in mozilla-central.
Attachment #423563 - Flags: review?(tglek)

Comment 1

9 years ago
Comment on attachment 423563 [details] [diff] [review]
patch

thats probably ok, can you get bsmedberg or dmandelin to review and land?
(Reporter)

Updated

9 years ago
Attachment #423563 - Flags: review?(tglek) → review?(benjamin)

Updated

9 years ago
Attachment #423563 - Flags: review?(benjamin) → review?(tglek)

Comment 2

9 years ago
Comment on attachment 423563 [details] [diff] [review]
patch

>diff -r d039748e7775 treehydra.c
>--- a/treehydra.c	Thu Jan 14 19:06:56 2010 -0800
>+++ b/treehydra.c	Mon Jan 25 12:42:57 2010 -0500
>@@ -206,17 +206,22 @@ void lazy_tree_string (struct Dehydra *t
> 
>   // now reflect .str, account for unicode magic (bug 526970)
>   tree str_type = TREE_TYPE (str);
>   if (str_type && TYPE_PRECISION (TREE_TYPE (str_type)) == TYPE_PRECISION (char_type_node)) {
>     wchar_bytes = 1;
>   } else {
>     wchar_bytes = TYPE_PRECISION (wchar_type_node) / BITS_PER_UNIT;
>   }
>-  num_chars = (TREE_STRING_LENGTH (str) / wchar_bytes) - 1; // skip trailing null
>+  num_chars = (TREE_STRING_LENGTH (str) / wchar_bytes);
>+  // TREE_STRING_LENGTH is 0 for certain empty strings

Eww. Nice find.
Attachment #423563 - Flags: review?(tglek) → review+
(Reporter)

Updated

9 years ago
Status: NEW → RESOLVED
Last Resolved: 9 years ago
Resolution: --- → FIXED

Updated

a year ago
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.