797190 - Slim: Investigate memory usage of gaia homescreen app

Status: RESOLVED WORKSFORME

(Core :: General, defect)

Description

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

We need to account for every byte of memory it's using.

Currently, the USS for the prelaunch process before it's loaded the homescreen content is about 8MB.  That's the baseline "process overhead" that we need to get more resolution into.  I've noticed in the past that after the homescreen content is loaded, its memory usage goes up to 25MB or so.  That's too damn high!  Need to measure.

Comment 1

[Gabriele Svelto [:gsvelto]]

Attachment: Memory report of the Homescreen app

This is a memory report taken after having launched the Homescreen app and scrolled over all the available pages

Comment 2

[Gabriele Svelto [:gsvelto]]

Attachment: Memory report of the Homescreen app after minimization

This is the same report as the previous one but taken after having forced a minimization

Comment 3

[Nicholas Nethercote [inactive]]

Here are huge strings from the atoms compartment (307,200 B (01.56%)):

16,384 B ── string(length=3360, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...") [2]
12,288 B ── string(length=4604, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
12,288 B ── string(length=4848, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
12,288 B ── string(length=4104, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=3356, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=3480, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=3644, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=3116, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=2208, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=2524, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=2604, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=3432, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=2836, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=4036, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=2520, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=2452, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=2588, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=3096, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=3268, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=3220, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=2188, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=2880, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=3732, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=3704, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=2956, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=2748, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=3120, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=3224, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=2480, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=2168, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=4040, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=3112, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=2412, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=3176, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=3084, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")

And from the app://homescreen.gaiamobile.org/index.html compartment (1,150,976 B (05.85%)):

49,152 B  ─ string(length=7094, "data:image//png;base64,iVBORw0KG...") [3]
32,768 B  ─ string(length=11230, "<span  style=" top: 13px; left:...")
32,768 B  ─ string(length=9998, "<span  style=" top: 13px; left:...")
32,768 B  ─ string(length=11762, "<span  style=" top: 13px; left:...")
32,768 B  ─ string(length=10494, "<span  style=" top: 13px; left:...")
32,768 B  ─ string(length=10982, "<span  style=" top: 13px; left:...")
32,768 B  ─ string(length=9098, "<span  style=" top: 13px; left:...")
32,768 B  ─ string(length=8410, "<span  style=" top: 13px; left:...")
32,768 B  ─ string(length=9302, "<span  style=" top: 13px; left:...")
32,768 B  ─ string(length=10654, "<span  style=" top: 13px; left:...")
32,768 B  ─ string(length=9986, "<span  style=" top: 13px; left:...")
32,768 B  ─ string(length=8990, "<span  style=" top: 13px; left:...")
32,768 B  ─ string(length=9018, "<span  style=" top: 13px; left:...")
32,768 B  ─ string(length=9414, "<span  style=" top: 13px; left:...")
20,480 B  ─ string(length=8754, "data:image//png;base64,iVBORw0KG...")
16,384 B  ─ string(length=3431, "{"MIMEType":"image//jpeg","data"...") [2]
16,384 B  ─ string(length=3116, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...") [2]
16,384 B  ─ string(length=3360, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...") [2]
16,384 B  ─ string(length=2480, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...") [2]
12,288 B  ─ string(length=4175, "{"MIMEType":"image//jpeg","data"...")
12,288 B  ─ string(length=4919, "{"MIMEType":"image//jpeg","data"...")
12,288 B  ─ string(length=4604, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
12,288 B  ─ string(length=4848, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
12,288 B  ─ string(length=4104, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
12,288 B  ─ string(length=4675, "{"MIMEType":"image//jpeg","data"...")
12,288 B  ─ string(length=4111, "{"MIMEType":"image//jpeg","data"...")
12,288 B  ─ string(length=4107, "{"MIMEType":"image//jpeg","data"...")
 8,192 B ── string(length=2259, "{"MIMEType":"image//jpeg","data"...")
 8,192 B ── string(length=3187, "{"MIMEType":"image//jpeg","data"...")
 8,192 B ── string(length=3803, "{"MIMEType":"image//jpeg","data"...")
 8,192 B ── string(length=3339, "{"MIMEType":"image//jpeg","data"...")
 8,192 B ── string(length=2676, "{"MIMEType":"image//jpeg","data"...")
 8,192 B ── string(length=2484, "{"MIMEType":"image//jpeg","data"...")
 8,192 B ── string(length=3186, "{"MIMEType":"image//jpeg","data"...")
 8,192 B ── string(length=3774, "{"MIMEType":"image//jpeg","data"...")
 8,192 B ── string(length=3027, "{"MIMEType":"image//jpeg","data"...")
 8,192 B ── string(length=3167, "{"MIMEType":"image//jpeg","data"...") 
 8,192 B ── string(length=3295, "{"MIMEType":"image//jpeg","data"...")
 8,192 B ── string(length=3291, "{"MIMEType":"image//jpeg","data"...")
 8,192 B ── string(length=3549, "{"MIMEType":"image//jpeg","data"...")
 8,192 B ── string(length=2594, "{"MIMEType":"image//jpeg","data"...")
 8,192 B ── string(length=2591, "{"MIMEType":"image//jpeg","data"...")
 8,192 B ── string(length=3183, "{"MIMEType":"image//jpeg","data"...")
 8,192 B ── string(length=2240, "{"MIMEType":"image//jpeg","data"...")
 8,192 B ── string(length=3356, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=3480, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=3644, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=2208, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=2524, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=2604, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=3432, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 
 8,192 B ── string(length=2604, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=3432, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=2836, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=4036, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=2520, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=2452, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=2588, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=3096, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=3268, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=3220, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=2188, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=2880, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=3732, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=3704, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=2956, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=2748, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=3120, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=3224, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=2168, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=4040, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=3112, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=2412, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=3176, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=3084, "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB...")
 8,192 B ── string(length=3155, "{"MIMEType":"image//jpeg","data"...")
 8,192 B ── string(length=3503, "{"MIMEType":"image//jpeg","data"...")
 8,192 B ── string(length=3715, "{"MIMEType":"image//jpeg","data"...")
 8,192 B ── string(length=3247, "{"MIMEType":"image//jpeg","data"...")
 8,192 B ── string(length=2820, "{"MIMEType":"image//jpeg","data"...")
 8,192 B ── string(length=2551, "{"MIMEType":"image//jpeg","data"...")
 8,192 B ── string(length=3192, "{"MIMEType":"image//jpeg","data"...")
 8,192 B ── string(length=2550, "{"MIMEType":"image//jpeg","data"...")
 8,192 B ── string(length=2279, "{"MIMEType":"image//jpeg","data"...")
 8,192 B ── string(length=2951, "{"MIMEType":"image//jpeg","data"...")
 8,192 B ── string(length=2659, "{"MIMEType":"image//jpeg","data"...")
 8,192 B ── string(length=2907, "{"MIMEType":"image//jpeg","data"...")
 8,192 B ── string(length=3425, "{"MIMEType":"image//jpeg","data"...")
 8,192 B ── string(length=2523, "{"MIMEType":"image//jpeg","data"...")


There are only four unique strings (or minor variants thereof):

 "data:image//png;base64,iVBORw0KG..."
 "{"MIMEType":"image//jpeg","data"..."    # the quoting on this is not quite right, BTW
 "//9j//4AAQSkZJRgABAQAAAQABAAD//2wB..."
 "<span  style=" top: 13px; left:..."

Looks like lots of repetition.  Can we remove some of these?

Comment 4

[Gabriele Svelto [:gsvelto]]

(In reply to Nicholas Nethercote [:njn] from comment #3)
> Looks like lots of repetition.  Can we remove some of these?

I was thinking about having a look at the dump itself and seeing what the differences between those strings really are, and how many real duplicate strings are there.

In addition I'm looking into jemalloc as the dump shows a significant amount of committed but unused heap space; if it's not due to pure fragmentation it might be interesting to try to recover some of that space, possibly following a low-mem event as per bug 800166.

Comment 5

[Justin Lebar (not reading bugmail)]

> I was thinking about having a look at the dump itself and seeing what the differences between those 
> strings really are, and how many real duplicate strings are there.

If you do a GC dump from the tools directory, you'll get all the /reachable/ JS strings (i.e., the ones that wouldn't be collected by the GC).

It's also pretty easy to write code to dump out all of the strings period.  It might be useful to add that to our GC/CC log dumping code.  Something like this works as a hack:

>diff --git a/js/src/jsfriendapi.cpp b/js/src/jsfriendapi.cpp
>index e8c59e4..12758eb 100644
>--- a/js/src/jsfriendapi.cpp
>+++ b/js/src/jsfriendapi.cpp
>@@ -646,9 +646,49 @@ DumpHeapVisitChild(JSTracer *trc, void **thingp, JSGCTraceKind kind)
>     DumpHeapPushIfNew(dtrc, thingp, kind);
> }
> 
>+struct DumpStringCallbackData
>+{
>+    FILE* outFile;
>+    char* compartmentName;
>+};
>+
>+static void
>+DumpStringCallback(JSRuntime *rt, void *_data, void *cell, JSGCTraceKind kind, size_t size)
>+{
>+    JS_ASSERT(kind == JSTRACE_STRING);
>+    DumpStringCallbackData* data = reinterpret_cast<DumpStringCallbackData*>(_data);
>+    JSString *str = (JSString *)cell;
>+    char buffer[1024 * 1024];
>+    if (str->isLinear()) {
>+        PutEscapedString(buffer, sizeof(buffer), &str->asLinear(), 0);
>+        fprintf(data->outFile, "string(len=%d, compartment=%s) \"%s\"\n",
>+                (int)str->length(), data->compartmentName, buffer);
>+    } else {
>+        fprintf(data->outFile, "rope(len=%d)\n", (int)str->length());
>+    }
>+}
>+
>+static void
>+DumpStrings(JSRuntime *rt, FILE *fp)
>+{
>+    for (CompartmentsIter c(rt); !c.done(); c.next()) {
>+        char name[256];
>+        (*rt->compartmentNameCallback)(rt, c, name, sizeof(name));
>+        DumpStringCallbackData data;
>+        data.outFile = fp;
>+        data.compartmentName = name;
>+        IterateCells(rt, c, gc::FINALIZE_STRING, &data, DumpStringCallback);
>+        IterateCells(rt, c, gc::FINALIZE_SHORT_STRING, &data, DumpStringCallback);
>+        IterateCells(rt, c, gc::FINALIZE_EXTERNAL_STRING, &data, DumpStringCallback);
>+    }
>+}
>+
> void
> js::DumpHeapComplete(JSRuntime *rt, FILE *fp)
> {
>+    DumpStrings(rt, fp);
>+    return;
>+
>     JSDumpHeapTracer dtrc(fp);
>     JS_TracerInit(&dtrc, rt, DumpHeapPushIfNew);
>     if (!dtrc.visited.init(10000))

Comment 6

[Gabriele Svelto [:gsvelto]]

Attachment: Memory report of the Homescreen app

Updated report after applying the patches for bug 798491

Comment 7

[Gabriele Svelto [:gsvelto]]

Attachment: Memory report of the Homescreen app after minimization

Updated report after applying the patches for bug 798491

Comment 8

[Gabriele Svelto [:gsvelto]]

After having applied the patches in attachment 675314 [details] [diff] [review] and attachment 675315 [details] [diff] [review] the profile of the homescreen app has changed quite a bit. Huge strings have almost disappeared though I still want to try to dump them out to see how much redundancy do we have in strings.

heap-committed-unused has grown a lot, I want to investigate if this is because of very high fragmentation or just because jemalloc is not returning the pages to the OS aggressively enough.

Comment 9

[Nicholas Nethercote [inactive]]

Why does the patch from bug 798491 affect huge strings?

Comment 10

[Justin Lebar (not reading bugmail)]

(In reply to Nicholas Nethercote [:njn] from comment #9)
> Why does the patch from bug 798491 affect huge strings?

I presume we GC the one big compartment more often than we GC'ed all the little compartments.

Comment 11

[Gabriele Svelto [:gsvelto]]

Attachment: Memory report of the Homescreen app

Updated memory report after the patch for bug 798491 has landed

Comment 12

[Gabriele Svelto [:gsvelto]]

Attachment: Memory report of the Homescreen app after minimization

Updated memory report after the patch for bug 798491 has landed

Comment 13

[Gabriele Svelto [:gsvelto]]

I've looked another time into the memory report and there's good news, heap-committed dropped by almost another meg down to ~10MiB, the application is still showing around ~2MiB of dirty pages which means that after I'll be finished with bug 805855 the overall amount of explicit allocations will be slightly higher than 8MiB with a resident size of ~25Mib.

Comment 14

[Gabriele Svelto [:gsvelto]]

Attachment: Memory report of the Homescreen app

I've attached a new memory report now that all patches related to minimizing unused dirty pages have landed. It's mostly good news but with a non-trivial regression:

- First of all I haven't posted a minimized version of the report because the homescreen is now being minimized every time it's sent to the background; the difference between a minimized and a non-minimized memory report is thus very small

- RSS dropped from 24.4 MiB to 23.3 MiB, the drop is less than I had anticipated but the overall RSS is below what I was expecting so that's good

- heap-committed dropped by almost a meg from 10.25 to 9.36 MiB, this change is mostly due to unused dirty pages being close to zero and partly because of a 300 KiB drop in heap-allocated

- heap-committed-unused also dropped by half a meg though it's still worth over 30% of heap-committed, since that's pure fragmentation it's pretty bad but there's not much we can do about it

- The bad news is that there seem to be a regression in the JS heap which went up by half a meg. To make matters worse the growth is mostly within unused-gc-things so pure fragmentation again with most of the change being within the app://homescreen.gaiamobile.org/index.html#root compartment. I'm not sure if we can do something about this; is there some knob in the JS GC heuristics we could use to reduce the waste?

Looking at these results I get the feeling that it might be possible to eke out another meg if we fix the regression in the JS heap and reduce jemalloc fragmentation further (by tweaking the size classes for example). I'm not sure if it would be worth it though, considering the added risks.

Comment 15

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Memory usage can always be reduced further, but I think this bug has served its purpose.