java.lang.OutOfMemoryError: at org.json.simple.parser.Yylex.<init>(Yylex.java)

NEW
Unassigned

Status

()

Firefox for Android
Android Sync
P4
normal
6 years ago
4 months ago

People

(Reporter: Scoobidiver (away), Unassigned)

Tracking

(Depends on: 1 bug, {crash})

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [native-crash], crash signature)

(Reporter)

Description

6 years ago
There are currently two crashes in 14.0a1, each one with a different stack:
bp-7b9e3bef-867f-4f70-b036-e10272120415
Build ID	20120414030731
java.lang.OutOfMemoryError
	at org.json.simple.parser.Yylex.<init>(Yylex.java:231)
	at org.json.simple.parser.JSONParser.<init>(JSONParser.java:34)
	at org.mozilla.gecko.sync.CryptoRecord.parseUTF8AsJSONObject(CryptoRecord.java:169)
	at org.mozilla.gecko.sync.CryptoRecord.fromJSONRecord(CryptoRecord.java:134)
	at org.mozilla.gecko.sync.net.WBOCollectionRequestDelegate.handleRequestProgress(WBOCollectionRequestDelegate.java:27)
	at org.mozilla.gecko.sync.net.SyncStorageCollectionRequest$SyncCollectionResourceDelegate.handleHttpResponse(SyncStorageCollectionRequest.java:113)
	at org.mozilla.gecko.sync.net.BaseResource.execute(BaseResource.java:253)
	at org.mozilla.gecko.sync.net.BaseResource.go(BaseResource.java:275)
	at org.mozilla.gecko.sync.net.BaseResource.get(BaseResource.java:281)
	at org.mozilla.gecko.sync.net.SyncStorageRequest.get(SyncStorageRequest.java:162)
	at org.mozilla.gecko.sync.repositories.Server11RepositorySession.fetchWithParameters(Server11RepositorySession.java:262)
	at org.mozilla.gecko.sync.repositories.Server11RepositorySession.fetchSince(Server11RepositorySession.java:279)
	at org.mozilla.gecko.sync.middleware.Crypto5MiddlewareRepositorySession.fetchSince(Crypto5MiddlewareRepositorySession.java:147)
	at org.mozilla.gecko.sync.synchronizer.RecordsChannel.flow(RecordsChannel.java:139)
	at org.mozilla.gecko.sync.synchronizer.RecordsChannel.onBeginSucceeded(RecordsChannel.java:239)
	at org.mozilla.gecko.sync.repositories.delegates.DeferredRepositorySessionBeginDelegate$1.run(DeferredRepositorySessionBeginDelegate.java:57)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1088)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:581)
	at java.lang.Thread.run(Thread.java:1019)

bp-ad34ea0d-dfb5-41fd-ae6e-79c722120422
Build ID	20120420030653
java.lang.OutOfMemoryError
	at org.json.simple.parser.Yylex.<init>(Yylex.java:231)
	at org.json.simple.parser.JSONParser.<init>(JSONParser.java:34)
	at org.mozilla.gecko.sync.repositories.android.RepoUtils.getJSONArrayFromCursor(RepoUtils.java:115)
	at org.mozilla.gecko.sync.repositories.android.AndroidBrowserHistoryDataExtender.visitsForGUID(AndroidBrowserHistoryDataExtender.java:154)
	at org.mozilla.gecko.sync.repositories.android.AndroidBrowserHistoryRepositorySession.addVisitsToRecord(AndroidBrowserHistoryRepositorySession.java:97)
	at org.mozilla.gecko.sync.repositories.android.AndroidBrowserHistoryRepositorySession.transformRecord(AndroidBrowserHistoryRepositorySession.java:75)
	at org.mozilla.gecko.sync.repositories.android.AndroidBrowserRepositorySession$FetchingRunnable.fetchFromCursor(AndroidBrowserRepositorySession.java:230)
	at org.mozilla.gecko.sync.repositories.android.AndroidBrowserRepositorySession$FetchSinceRunnable.run(AndroidBrowserRepositorySession.java:326)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1076)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:569)
	at java.lang.Thread.run(Thread.java:856)


More reports at:
https://crash-stats.mozilla.com/report/list?signature=java.lang.OutOfMemoryError%3A+at+org.json.simple.parser.Yylex.%3Cinit%3E%28Yylex.java%29

Comment 1

6 years ago
Richard, can you take a look?
(In reply to Tony Chung [:tchung] from comment #1)
> Richard, can you take a look?

Nothing harmful; we just use a lot of memory. (See also Bug 740334.)

We'll be exploring improvements for this in the future, but nothing before beta.
Component: General → Android Sync
Product: Fennec Native → Mozilla Services
QA Contact: general → android-sync
Version: Trunk → unspecified
Depends on: 749729
Some recent Sync OOM crashes in Yylex.java:

1136174b-9537-41ec-a56f-214652120610
2091f0e5-5ff5-4f69-8e91-78c0f2120613
status-firefox16: --- → affected
status-firefox14: --- → affected
status-firefox15: --- → affected
status-firefox16: affected → ---
Yup, json parsing is expensive. We might parse 256KB of JSON at a time, which will use half a meg for the string alone, then whatever the tokenizer and lexer use. Switching to a better json lib might help, but we are always going to be susceptible to OOMs.
Severity: critical → normal
Priority: -- → P2
This is also happening on upload:

https://crash-stats.mozilla.com/report/index/bp-34ea047c-e4f4-4d6c-a571-4d5dc2120726

which means the JSON response body from the server is big enough to cause us to run out of memory.

Hypothesis is that the failure payloads are big -- str(exception) with a DB exception. And 100 of them for a batch upload. It only takes that to include a query string, and we're parsing a meg of JSON.

In general, though, we're under memory pressure by the time we're deep enough in the stack to be parsing HTTP responses. Work in the next three releases will somewhat address this.
Server-side, we have Bug 773073 open to avoid sending (potentially large, potentially sensitive) stringified exception objects back to the client.
Depends on: 773073
(Reporter)

Updated

5 years ago
status-firefox16: --- → affected
status-firefox17: --- → affected
status-firefox18: --- → affected
(Reporter)

Comment 7

5 years ago
With combined signatures, it's #26 top crasher in 17.0.

There's another stack trace:
java.lang.OutOfMemoryError
	at org.json.simple.parser.Yylex.<init>(Yylex.java:231)
	at org.json.simple.parser.JSONParser.<init>(JSONParser.java:34)
	at org.mozilla.gecko.sync.repositories.android.RepoUtils.getJSONArrayFromCursor(RepoUtils.java:115)
	at org.mozilla.gecko.sync.repositories.android.AndroidBrowserBookmarksRepositorySession.bookmarkFromMirrorCursor(AndroidBrowserBookmarksRepositorySession.java:1070)
	at org.mozilla.gecko.sync.repositories.android.AndroidBrowserBookmarksRepositorySession.retrieveRecord(AndroidBrowserBookmarksRepositorySession.java:453)
	at org.mozilla.gecko.sync.repositories.android.AndroidBrowserBookmarksRepositorySession.retrieveDuringStore(AndroidBrowserBookmarksRepositorySession.java:395)
	at org.mozilla.gecko.sync.repositories.android.AndroidBrowserRepositorySession.retrieveByGUIDDuringStore(AndroidBrowserRepositorySession.java:611)
	at org.mozilla.gecko.sync.repositories.android.AndroidBrowserRepositorySession.findExistingRecord(AndroidBrowserRepositorySession.java:664)
	at org.mozilla.gecko.sync.repositories.android.AndroidBrowserRepositorySession$1.run(AndroidBrowserRepositorySession.java:488)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1088)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:581)
	at java.lang.Thread.run(Thread.java:1019)
See discussion in Bug 749729.

In particular:

A really, really important thing to note: Android's GC doesn't compact. That's why we can end up with OOM in JSON parsing -- we get a big blob of text, we need to allocate a large chunk to process it, but the (ample!) free memory is too fragmented to allocate.


This is not a bug we can fix directly; that is, we do not have a problem that is directly caused by Yylex in JSON parsing (though switching to a more efficient JSON parser might help). We need to address overall memory usage, and more aggressively GC (amongst possible other things) to try to reduce memory fragmentation.
(Assignee)

Updated

5 years ago
Component: Android Sync → Android Sync
Product: Mozilla Services → Android Background Services

Updated

2 years ago
Crash Signature: [@ java.lang.OutOfMemoryError: at org.json.simple.parser.Yylex.<init>(Yylex.java)] → [@ java.lang.OutOfMemoryError: at org.json.simple.parser.Yylex.<init>(Yylex.java)] [@ java.lang.OutOfMemoryError: at org.json.simple.parser.Yylex.<T>]
status-firefox14: affected → ---
status-firefox15: affected → ---
status-firefox16: affected → ---
status-firefox17: affected → ---
status-firefox18: affected → ---

Updated

6 months ago
Priority: P2 → P4

Updated

4 months ago
Product: Android Background Services → Firefox for Android
You need to log in before you can comment on or make changes to this bug.