749729 - [meta] Reduce memory pressure during sync

Status: RESOLVED INCOMPLETE

(Firefox for Android Graveyard :: Android Sync, defect, P5)

Description

[Richard Newman [:rnewman]]

We're seeing a number of OOM crashes during syncs.

Strategies:

* Compute stage instances as needed. (Also laying groundwork for dynamic stage progression.)

* `advance` should work with an Executor queue or similar, so that we can clear the stack between stages and allow GC. And earlier stages can be released.

* Stages should clear expensive objects (Synchronizer, for example) explicitly when done to encourage GC... and so on down the stack. Synchronizer should let go of objects sooner, too.

* In some circumstances we might have to use less efficient approaches to things -- e.g., eliminating guidMap.

* There might be some brutal inefficiency hiding in the JSON parser, or the parsed JSON records themselves, or ...

Comment 1

[Allison Naaktgeboren :ally]

sync triage: p2 for now, but could become a p1 depending on what happens during beta period. Probably no eligible for blocker status

Comment 2

[Richard Newman [:rnewman]]

Putting this on Nick's radar.

Comment 3

[Richard Newman [:rnewman]]

See Bug 802723 Comment 18 for some thoughts on this:

---
There are a few other things we could do here:

* Extend our local backoff interval if we die catastrophically; at least we won't keep hammering the device.
* If we get an OOM, enter a resource-constrained state, where we sync only one collection at a time. Special case of tuning for available resources.
* Track collections that fail due to OOM, and do the others first (until they're done), then last (so we eventually do try the failing connection with a possibly clean VM).


A really, really important thing to note: Android's GC doesn't compact. That's why we can end up with OOM in JSON parsing -- we get a big blob of text, we need to allocate a large chunk to process it, but the (ample!) free memory is too fragmented to allocate.

We *might* still see a benefit from running System.gc() at various points — between each engine, for example, or after each record batch. But otherwise we just have to hope we get lucky and get a fresh VM at some point.

That's probably what Sandy described in Bug 802723 Comment 14 -- the process got restarted, so we got a fresh heap and were lucky enough to complete a sync before it fragmented.

Running engines in a different order (biggest first after failure, so it gets a clean slate) might get past a logjam.
---

Comment 4

[Nick Alexander :nalexander [he/him]]

I'm doing some investigate work on this as we speak.  I think we're creating a lot of JSONParser instances.  All instances share a 64k parse table, but each one allocates a 16k look-ahead input buffer.  Depending on the paths through our parsing code, we could really win here by sharing JSONParser instances.

Comment 5

[Richard Newman [:rnewman]]

(In reply to Nick Alexander :nalexander from comment #4)
> All instances share a 64k parse table, but
> each one allocates a 16k look-ahead input buffer.

Which is to say, each time we try to parse an incoming record — which we do a lot — we need to allocate a string block for the incoming data, a 16K buffer, all of the intermediate objects, and the output object. And these lifecycles won't all be congruent, so we'll definitely get memory fragmentation. Yeah, that'll hurt.

Good sleuthing; I'd be happy to review a patch and encourage uplift.

File a bug for that specific instance and block this?

(Also, Bug 709175…)

Comment 6

[Firefox Bug Husbandry Bot]

Re-triaging per https://bugzilla.mozilla.org/show_bug.cgi?id=1473195

Needinfo :susheel if you think this bug should be re-triaged.

Comment 7

[BMO Automation]

We have completed our launch of our new Firefox on Android. The development of the new versions use GitHub for issue tracking. If the bug report still reproduces in a current version of [Firefox on Android nightly](https://play.google.com/store/apps/details?id=org.mozilla.fenix) an issue can be reported at the [Fenix GitHub project](https://github.com/mozilla-mobile/fenix/). If you want to discuss your report please use [Mozilla's chat](https://wiki.mozilla.org/Matrix#Connect_to_Matrix) server https://chat.mozilla.org and join the [#fenix](https://chat.mozilla.org/#/room/#fenix:mozilla.org) channel.