792339 - OUT_OF_MEMORY_KILLED telemetry breakdown

Status: RESOLVED FIXED

(Mozilla Metrics :: Frontend Reports, defect)

Description

[Gian-Carlo Pascutto [:gcp]]

We added a telemetry probe to see if Firefox on Android is being killed due to the device running out of memory (OUT_OF_MEMORY_KILLED).

The data I see in Telemetry seems to indicate 25% hits, which is very high.

Is it possible to correlate this data against architecture (ARMv6 vs ARMv7) and operating system version (Android 2.2, 2.3, 3.1, 3.2, 4.0, 4.1)? We would like to have some idea what's going on, and see if any of these is significantly higher than the others.

https://metrics.mozilla.com/ has some comboboxes to allow this filtering, but the values for Operating system version and architecture are bogus/unusable for Firefox on Android.

Comment 1

[Lawrence Mandel [:lmandel] (use needinfo)]

Annie, this is a request for analysis from the mobile team. Can you please review and prioritize?

Comment 2

[Gian-Carlo Pascutto [:gcp]]

By the way, to clarify, there are two things here from my side:

1) This specific breakdown would be very helpful to us at this point.
2) The filtering options on the site *seem* to allow this to be done but it actually doesn't:
- When selecting Android as Operating System, the values in Operating System Version seem to make no sense.
- Architecture just lists ARM and doesn't allow a distinction between ARMv6 and ARMv7. (The data to distinguish those was only recently added to Telemetry)

So it's both a request for the analysis and to consider fixing the metrics site so I don't have to bother you in the future.

Comment 3

[Lawrence Mandel [:lmandel] (use needinfo)]

I don't know if Telemetry reports ARMv6 vs ARMv7. The fix may require client and/or server side changes.

cc Nathan for comment on the client side piece.

Comment 4

[Gian-Carlo Pascutto [:gcp]]

ARMv6 vs ARMv7 support was added to the client side in bug 779687, it should sit along the other hardware/platform info.

Comment 5

[Nathan Froyd [:froydnj]]

gcp would know more than I would; he's the one who added the ARMv7 bits!

Comment 6

[Annie Elliott]

I'll get this triaged and into JIRA this week. Resources not likely available until Friday; beginning of next week would be much more realistic.

Comment 7

[Gian-Carlo Pascutto [:gcp]]

If a further breakdown in device memory range is possible, that would also help us. i.e. OUT_OF_MEMORY_KILLED percentages vs devices with 256M RAM, 384M RAM, 512M RAM...

Comment 8

[Annie Elliott]

https://metrics.mozilla.com/projects/browse/BIV-63

Comment 9

[Annie Elliott]

https://metrics.mozilla.com/projects/browse/BIV-63

Comment 10

[Xavier Stevens [:xstevens]]

lmandel, Is there a field in telemetry that has the Android version? The one that comes through in the os version field is a Linux kernel version.

Comment 11

[Xavier Stevens [:xstevens]]

I'll work on a version of this that will convert Linux kernel versions on Android to Android versions. I also opened a bug for Telemetry on android to submit the android version rather than the kernel version for the OS version. You can follow that here: https://bugzilla.mozilla.org/show_bug.cgi?id=799228

Comment 12

[Xavier Stevens [:xstevens]]

I did an informed guess on os version for now. The only thing I found out of the ideas above that is correlated is ARMv7. It heavily correlated around (0.98). Don't mistake this for a root cause because there could be other factors that explain this (For instance: Is almost everything ARMv7 these days?)

Comment 13

[Gian-Carlo Pascutto [:gcp]]

>For instance: Is almost everything ARMv7 these days?

Not really, there are loads of cheap smartphones running ARMv6 coming online too, half a year ago they were over half the market. On the other hand our ARMv6 support is very fresh (still in beta) so I'd expect much less of those running Android.

I think you should be able to tell from the Telemetry the exact ratios: either ARMv6 or ARMv7 should be set.

If the OUT_OF_MEMORY_KILLED is more correlated to ARMv7 than it is to ARMv6, this is somewhat contrary to expectation and would point to something like bug 785960 being a factor.

The latter has landed, together with a renaming of OUT_OF_MEMORY_KILLED -> FENNEC_WAS_KILLED, and I'll try to land the Android version patches ASAP. After that, we could run again on the new data...

Comment 14

[Xavier Stevens [:xstevens]]

Gian-Carlo, Okay. You might look into ARMv7 related things then. ARMv6 is not correlated at all (0.05 for comparison) with OOM_KILLED. Let me know when you want to run this again in the future. Can I close this bug for now?

Comment 15

[Gian-Carlo Pascutto [:gcp]]

Okay, I'll probably ask a similar question again in a few days when bug 799228 and bug 785960 are out in the field. Thanks.