1253091 - Increase in OOM crashes on the beta channel since roughly mid-January of 2016

Status: NEW

(Firefox :: General, defect)

Description

[Robert Kaiser]

[Tracking Requested - why for this release]:

Looking at the graphs in https://arewestableyet.com/graph/?fxbeta-bcat there is a noticeable increase of the OOM crash category somewhere between early January and later February of 2016. Unfortunately, there is no clear step up at any day.

When I compare a week of the then-current 44 beta in January to a week of 45 beta in February, this becomes clear also in some numbers from more direct sources than my graphs:

https://crash-stats.mozilla.com/search/?product=Firefox&process_type=browser&process_type=content&version=44.0b&signature=~OOM&date=%3E%3D2016-01-10&date=%3C2016-01-17&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#crash-reports
--> 19710 OOM crashes

https://crash-stats.mozilla.com/search/?product=Firefox&process_type=browser&process_type=content&version=45.0b&signature=~OOM&date=%3E%3D2016-02-17&date=%3C2016-02-24&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#crash-reports
--> 25237 OOM crashes

I also tried to get crash rates for those, looking up the ADI from https://crash-stats.mozilla.com/crashes-per-day/?p=Firefox&v=44.0b&v=45.0b&hang_type=any&os=Windows&os=Mac+OS+X&os=Linux&date_start=2016-01-01&date_end=2016-03-02&submit=Generate

44.0b, 01/10-16: 14385697 sum-ADI, avg rate: 0.137 OOM crashes / 100 ADI
45.0b, 02/17-24: 13771369 sum-ADI, avg rate: 0.183 OOM crashes / 100 ADI

That's a 34% increase of OOM crash rates!

Looking at a comparison of the composition of the OOM crashes in both windows gives that percentage comparison:
https://crash-analysis.mozilla.com/rkaiser/datil/searchcompare/?common=product%3DFirefox%26process_type%3Dbrowser%26process_type%3Dcontent%26dom_ipc_enabled%3D__null__%26signature%3D%7EOOM&p1=version%3D44.0b%26date%3D%3E%253D2016-01-10%26date%3D%3C2016-01-17&p2=version%3D45.0b%26date%3D%3E%253D2016-02-17%26date%3D%3C2016-02-24&limit=30

It looks like the js::AutoEnterOOMUnsafeRegion::crash signature of bug 1233481 is the main one that increased (which we finally split up into multiple signatures now), but a number of OOM|large signatures like bug 1132499 and some unfiled ones increased as well.

We may not be able to actually fix this in 45, but given that it regressed somewhere in this area, I'm requesting tracking for that train for now.
There may not a completely clear cause here but it seems clear to me that we need get a hold of the general issue of OOMs.

Comment 1

[Robert Kaiser]

(In reply to Robert Kaiser (:[e-mail]) from comment #0)
> 44.0b, 01/10-16: 14385697 sum-ADI, avg rate: 0.137 OOM crashes / 100 ADI
> 45.0b, 02/17-24: 13771369 sum-ADI, avg rate: 0.183 OOM crashes / 100 ADI

Actually, my text is slightly wrong wrt end dates, the second should be 17-23 (the query says >=17 and <24, that's where the typo comes from). The data itself, including crash rates, is fine.

Comment 2

[Sylvestre Ledru [:Sylvestre]]

Not tracking as it wasn't really actionable unfortunately...