996991 - [Email] OOM Crash on displaying and/or scrolling a long text/html email that triggers newsletter mode

Status: VERIFIED DUPLICATE of bug 984460

(Firefox OS Graveyard :: Gaia::E-Mail, defect, P1)

Description

[Tony Chung [:tchung]]

Attachment: logcat

When loading a large email body, then attempt to scroll when loaded, the app will OOM and crash.

logcat attached, screencast here:  http://youtu.be/BuKluLnmmDs

logcat:
04-15 23:36:48.739: E/OomLogger(137): [Kill]: select 890 ((Preallocated a), adj 10, size 4580, to kill
04-15 23:36:48.739: E/OomLogger(137): [Kill]: send sigkill to 890 ((Preallocated a), adj 10, size 4580
04-15 23:36:48.739: E/OomLogger(137): [Kill]: select 137 (b2g), adj 0, size 11354, to kill
04-15 23:36:48.739: E/OomLogger(137): [Kill]: select 760 (Homescreen), adj 8, size 4263, to kill
04-15 23:36:48.739: E/OomLogger(137): [Kill]: send sigkill to 760 (Homescreen), adj 8, size 4263

repro:
1) install 1.4 nightly on buri
Gaia      441c4bcd8ac4f8c01a9bc5a2f8d64eaa87844803
Gecko     https://hg.mozilla.org/releases/mozilla-aurora/rev/55cae0a17861
BuildID   20140415160204
Version   30.0a2
ro.build.version.incremental=eng.tclxa.20131223.163538
ro.build.date=Mon Dec 23 16:36:04 CST 2013

2) launch email app
3) open a really long email 
4) when the throbber stops loading, attempt to scroll the body content
5) Verify OOM and crash

Expected:
- scroll the long body of the email 

Actual:
- OOM

Comment 1

[Andrew Sutherland [:asuth] (he/him)]

So, the OomLoggerr isn't claiming it tried to kill the e-mail app as far as I can tell.  From the logcat, we can see e-mail has pid 835 and that pid is not explicitly killed.  From http://dxr.mozilla.org/mozilla-central/source/hal/gonk/GonkHal.cpp it appears that the OomLogger is based on actually reading stuff out of the kernel ring buffer, which suggests this is reliable information tracking kernel activities and not simply a case of the e-mail app managing to grow faster than a user-level process can track and then dying silently.

It is pretty clear, however, that e-mail is crashing.  So I think we should adjust our operating theory that b2g is crashing in this case.  It could still be memory-usage-related, for example failing an alloc and then aborting the process.  But we should ideally have a crash report in this case, maybe?  Hopefully?

The main contextual question is what type of message this is: text/plain segmented by varying quote levels, continuous text/plain, non-newsletter text/html, or newsletter text/html.  The best answer is accomplished by providing the message as described at https://wiki.mozilla.org/Gaia/Email/ProvidingEmailsForDebugging.  Since from the top of the message it looks like you might have been reusing a personal-ish message, I would suggest just sending the message privately to [e-mail] and [e-mail] OR reproducing with a different message you're comfortable attaching to the bug.  Going forward we will augment our logging to explicitly specify what mode we are using and ballpark figures on the body size.


There's a lot of potential mitigations for this but it also very much depends on what the actual problem is, and I think we're a bit away from determining that currently.

Comment 2

[Andrew Sutherland [:asuth] (he/him)]

Meant to update the subject to clarify we're not sure this is scrolling.  If it's possible for the system to render/paint with our JS on the stack AND/OR if the message contains multiple body part segments, it's possible we are dying while synchronously trying to cram data into the body and we're just frozen for the several seconds before the actual process apoptosis happens.

Comment 3

[Tony Chung [:tchung]]

(In reply to Andrew Sutherland (:asuth) from comment #2)
> Meant to update the subject to clarify we're not sure this is scrolling.  If
> it's possible for the system to render/paint with our JS on the stack AND/OR
> if the message contains multiple body part segments, it's possible we are
> dying while synchronously trying to cram data into the body and we're just
> frozen for the several seconds before the actual process apoptosis happens.

i sent the sample email that i was reproducing this bug to Andrew and James directly.  if anyone else wants it, just ni? me.

Comment 4

[Jason Smith [:jsmith]]

Adding qawanted to check if this happens on the past release.

Comment 5

[Andrew Sutherland [:asuth] (he/him)]

Short story, Tony's e-mail does trigger newsletter mode.  The only explicit sizes that jump out are tables capped at 400px.  But the divs around it all seem to want to size larger.  So Our newsletter heuristic may be doing itself a dis-service on this one.

Here's the key details of our doom via the app manager:

clipper div (styles we set):
  width: 274px;
  height: 33951.6px;

iframe (styles we set):
  width: 638px;
  transform: scale(0.429467);
  height: 79034px;

document.body.scrollWidth === 1120
document.body.scrollHeight === 79002
document.body.clientWidth === 622
document.body.clientHeight === 79002


b2g-procrank with the message displayed was:
$ adb shell b2g-procrank
APPLICATION        PID      Vss      Rss      Pss      Uss  cmdline
b2g                136   70484K   60484K   53885K   51264K  /system/b2g/b2g
E-Mail            1282  111660K   59016K   47380K   42672K  /system/b2g/plugin-container
Homescreen        1220   27188K   27188K   15787K   11348K  /system/b2g/plugin-container
(Preallocated a   1314   18868K   18864K    9226K    5856K  /system/b2g/plugin-container
(Nuwa)             341    9728K    9728K    5513K    4076K  /system/b2g/plugin-container
                                           ------   ------  ------
                                          141074K  124088K  TOTAL


A quick attempt to attach gdb did not work; I think I need to do more legwork to get that happy.

Comment 6

[Pi Wei Cheng [:piwei] (inactive)]

Please send the email that could reproduce the issue to my email ([e-mail]). Thanks.

Comment 7

[Tony Chung [:tchung]]

(In reply to Pi Wei Cheng from comment #6)
> Please send the email that could reproduce the issue to my email
> ([e-mail]). Thanks.

sent in raw text format.

Comment 8

[Pi Wei Cheng [:piwei] (inactive)]

Thanks Tony for emailing.

The OOM and crash issue does NOT reproduce on Buri 1.3. However, scrolling the long email isn't possible either; the email app simply doesn't react to up/down scrolling (side-scrolling works sluggishly). All the buttons (Back, Delete, Flag, Move, and Reply) within the email reading view do work.

Tested on:
Device: Buri 1.3 MOZ
BuildID: 20140418024004
Gaia: 4aae5c8e9ac9a3d7ea7e83172dda6e11169cd677
Gecko: 0fd9f6fe8832
Version: 28.0
Firmware Version: v1.2-device.cfg

Comment 9

[Pi Wei Cheng [:piwei] (inactive)]

Regression window found in Nightly Central. Unfortunately, we do not have builds available to provide a deeper window.

Nightly regression window:

Last Working Environmental Variables:
Device: Buri MOZ
BuildID: 20140108040200
Gaia: b7a7191f761933fd4878227488c75d09f5ba890c
Gecko: cf2d1bd796ea
Version: 29.0a1
Firmware Version: v1.2-device.cfg

First Broken Environmental Variables:
Device: Buri MOZ
BuildID: 20140109040203
Gaia: 47206ac66b084c6f6c4503a3b10d0e0760df2b6f
Gecko: 9409405e0739
Version: 29.0a1
Firmware Version: v1.2-device.cfg

Last Working Gaia / First Broken Gecko: Issue Does NOT reproduce
Gaia: b7a7191f761933fd4878227488c75d09f5ba890c
Gecko: 9409405e0739

Last Working Gecko / First Broken Gaia: Issue DOES reproduce
Gaia: 47206ac66b084c6f6c4503a3b10d0e0760df2b6f
Gecko: cf2d1bd796ea

Gaia Pushlog:
https://github.com/mozilla-b2g/gaia/compare/b7a7191f761933fd4878227488c75d09f5ba890c...47206ac66b084c6f6c4503a3b10d0e0760df2b6f

Comment 10

[Jason Smith [:jsmith]]

I don't think that's the right range. There's nothing that jumps out in the range at all for what the cause is. Let's recheck.

Comment 11

[Andrew Sutherland [:asuth] (he/him)]

I'll attach gdb and take a look.

Comment 12

[Pi Wei Cheng [:piwei] (inactive)]

This issue appears to be caused by the APZ option, and it occurs as early as APZ was first introduced. The option has to be enabled in order to reproduce the bug.

Nightly regression window:

Last Working Environmental Variables (APZ is not available as an option):
Device: Buri
BuildID: 20131025040252
Gaia: 15c1089aea0490b9b73bc915b1495b3faa1b023c
Gecko: 5a9ac6fed6ff
Version: 27.0a1
Base Image: V1.2-device.cfg

First Broken Environmental Variables (APZ is an option, and APZ must be enabled to repro):
Device: Buri
BuildID: 20131025100746
Gaia: afbf45f26a73b7cd5e0a831bea48087331975286
Gecko: 2f2a45f04e7c
Version: 27.0a1
Base Image: V1.2-device.cfg

Last Working Gaia / First Broken Gecko: APZ option is NOT available
Gaia: 15c1089aea0490b9b73bc915b1495b3faa1b023c
Gecko: 2f2a45f04e7c

Last Working Gecko / First Broken Gaia: APZ option IS available, and bug reproduces
Gaia: afbf45f26a73b7cd5e0a831bea48087331975286
Gecko: 5a9ac6fed6ff

Gaia Pushlog:
https://github.com/mozilla-b2g/gaia/compare/15c1089aea0490b9b73bc915b1495b3faa1b023c...afbf45f26a73b7cd5e0a831bea48087331975286

Comment 13

[Andrew Sutherland [:asuth] (he/him)]

Attachment: email-unusual-oom-deaths-dmesg.log

Okay, so the situation is always an OOM death.  The reason Tony's comment 0 log misled me is because OomLogger's regexes don't actually cover all of the possible OOM cases.  I will locate and/or file a new bug on this, but here's a demsg with a bunch of deaths in them to start with.

Comment 14

[Andrew Sutherland [:asuth] (he/him)]

I've chimed in on https://bugzilla.mozilla.org/show_bug.cgi?id=939372#c49 to try and get the ball rolling on improving the OomLogger.

Since it's now clear this is an OOM thing and email's hack of using transform to get its pinch/zoom mode is arguably not a sane thing to be doing which such a big iframe, the onus is on email to not do that.

There are a few things we can directly do around newsletter mode to prevent this type of problem:

A) If the body exceeds a certain byte size, just forget all about newsletter mode including doing the speculative layout to get the e-mail's "natural" size.

B) Do not activate newsletter mode if our speculative layout resulted in scrollWidth * scrollHeight being big.  This handles the case where the document is byte-wise small but coordinate-space big,


Other things we could do but are likely not appropriate for a blocker since the prior issues will mitigate this problem and class of problems and anything else is a usability enhancement:

C) Try and avoid the non-repeated version of this mail body triggering newsletter mode in the first place since it seems like it potentially could have been avoided.

D) Changing how newsletter mode works, assuming there is any way to get the graphics layer to do zooming without OOM-ing.  I did some exploratory work related to this for bug 982888 in bug 982888 comment 16 before learning about mozscrollgrab (which still does not help address the zoom problem, but could allow us to clip in various ways that might save the graphics layer).  There's possibly some crazy things where manual rendering to canvas could be used, but that is super ridiculously out of scope even for the more serious problem in bug 982888 that we've basically punted on.

Comment 15

[Timothy Nikkel (:tnikkel)]

Does bug 984460 fix this? Just merged to central today.

Comment 16

[Jason Smith [:jsmith]]

Leaving qawanted to retest per comment 15 on trunk.

Comment 17

[Andrew Sutherland [:asuth] (he/him)]

(In reply to Timothy Nikkel (:tn) from comment #15)
> Does bug 984460 fix this? Just merged to central today.

Yes, this fixed the problem!  Hooray hooray hooray!

Comment 18

[Tony Chung [:tchung]]

Verified on 1.4 branch that bug 984460 fixed it.