966233 - Tune "max internal blob size" on B2G

Status: RESOLVED WONTFIX

(Core :: Storage: IndexedDB, defect, P3)

Description

[Jan Varga [:janv]]

We are adding the ability to store structured clones externally (outside of the database) in bug 964561. The boundary will be set to 100 KB on Android/B2G. However, I haven't done any perf measurements to find optimal value. The new feature could help especially when we store large array buffers in IDB on B2G. When a structured clone is being stored externally, it requires less memory allocations and storing it in a standalone file should be faster than storing in the database.

Comment 1

[Jan Varga [:janv]]

Gregor, once bug 964561 lands, could someone from B2G guys help with finding the optimal value ?

Comment 2

[Gregor Wagner [:gwagner]]

(In reply to Jan Varga [:janv] from comment #1)
> Gregor, once bug 964561 lands, could someone from B2G guys help with finding
> the optimal value ?

That's a great feature!
I guess someone from Mikes or Nicks team could help with some performance numbers once we are done with our 1.3 blockers.

Comment 3

[Nicholas Nethercote [inactive]]

erahm might be able to take a look at this. Jan, can you explain what would be some good test case, and how to evaluate them? Thanks.

Comment 4

[Jan Varga [:janv]]

(In reply to Nicholas Nethercote [:njn] from comment #3)
> erahm might be able to take a look at this. Jan, can you explain what would
> be some good test case, and how to evaluate them? Thanks.

Well, what I did so far is that I forced to store all structured clones externally and ran all IndexedDB mochitests. It's quite the opposite of what we need to do here, but I wanted to verify that even when we store everything externally, performance is ok. So it took only one more sec on a high end laptop (15 secs vs 14 secs).
I would suggest to do something similar, that is, to run a B2G test suite on real mobile device which involves storing data in IDB or just entire B2G test suite and compare total time of runs with different max internal blob size setting.
The pref is: dom.indexedDB.maxInternalBlobSize and it's in bytes

According to [1], the optimal number is around 100KB.

[1] https://www.sqlite.org/intern-v-extern-blob.html

Comment 5

[Eric Rahm [:erahm]]

Do I need to wait for bug 964561 before testing this?

Comment 6

[Jan Varga [:janv]]

(In reply to Eric Rahm [:erahm] from comment #5)
> Do I need to wait for bug 964561 before testing this?

Well, you don't if you have time right now. Patches for bug 965084 and bug 965982 already landed on mozilla-inbound today, so you just need to apply the patches attached directly to bug 964561.
Actually, it would be better if you tested this before landing the feature.

Comment 7

[Eric Rahm [:erahm]]

Happy to check it out, just have to figure out how to run the indexedDB tests on my device.

Comment 8

[Eric Rahm [:erahm]]

I've attempted to get b2g mochitests running for indexedDB but have not been able to do so from my dev machine (OS X 10.9). I've tried using the default gecko under B2G as well as gecko-inbound which had the patches we needed. I can get as far as launching the tests, a white screen is loaded (which seems to indicate at least things are starting) and then things just stall for 450 seconds until the test suite kills itself due to a timeout.

I've used the instructions from here:
https://developer.mozilla.org/en-US/Firefox_OS/Platform/Automated_testing/Mochitests#Running_mochitests_on_B2G_device_and_emulator_builds

Asking around in the #b2g channel has not lead to any helpful information. If anyone with experience running the tests on a B2G device could help me out I'd be happy to continue, but until then I'm blocked.

Comment 9

[Ben Turner (not reading bugmail, use the needinfo flag!)]

Hm, you need to make sure you're not running the dom/indexedDB/ipc tests (or any other indexedDB tests that spawn remote iframes) while in a child process. I'm not entirely clear how you're running the tests (or how mochitests are set up to work in multiprocess b2g for that matter), but we don't have any support for nested child processes.

Comment 10

[Eric Rahm [:erahm]]

I saw 2 of the tests were blacklisted in the manifest, but at this point I don't think I even get as far as running a test.

Here are my steps (taken from the mochitest wiki page):
1) Install marionette on dev machine (this was already installed from when I ran some endurance tests)
2) Setup phone and dev machine on same wifi network
3) Run the following commands
    adb forward tcp:2828 tcp:2828
    cd $GECKO_OBJ_DIR/_tests/testing/mochitest
    python runtestsb2g.py --b2gpath /Volumes/firefoxos/B2G/ --xre-path $XPCSHELL_DIR --console-level INFO --httpd-path ./ --test-path dom/indexedDB/

This causes my device to go to a white screen (AFAICT that is a good thing), then I get a fair amount of output to my console (none of which indicates a test is actually run), then nothing for 450 seconds, then the tests shutdown due to a timeout (and the device is left stuck at the blank screen).

Comment 11

[Jan Varga [:janv]]

instead of:
"--test-path dom/indexedDB/"

try:
"--test-path dom/indexedDB/test"

that will avoid running dom/indexedDB/ipc

Comment 12

[Eric Rahm [:erahm]]

That fails as well, I should have noted I tried running other tests as well.

Comment 13

[Ben Turner (not reading bugmail, use the needinfo flag!)]

jgriffin, any ideas here? See comment 10.

Comment 14

[Jonathan Griffin (:jgriffin)]

logcat output would be really helpful here.  cc'ing dclarke and ahal who recently spent some time getting this to work on device.

Comment 15

[[e-mail] [:onecyrenus]]

Just an fyi I have one small patch that needs to get applied.  Basically a one liner to disable powersaving mode on the wifi. 

After 30 seconds of no activity the wifi shuts down.. 
 
Just edit your b2g_start_script for now and add the line, bug / patch coming soon. 

+if (onDevice) {
+  let manager = navigator.mozWifiManager;
+  manager.setPowerSavingMode(false);   <----- add this line
+  let con = manager.connection;
+  if(wifiSettings) {
+    if (manager.enabled) {
+      manager.associate(wifiSettings);
+    } else {
+      manager.onenabled = function () {
+        manager.associate(wifiSettings);
+      };
+    }
+  }

Comment 16

[[e-mail] [:onecyrenus]]

erahm: 
   just an fyi. I ran into problems running the indexedDB mochitest test suite on device so probably shouldn't expect them to pass.

Comment 17

[Eric Rahm [:erahm]]

Thanks! I'll check that out when I get a working build on my device.

Comment 18

[Eric Rahm [:erahm]]

Attachment: Output from mochitest

Attempted again with a patched b2g start script, but still seeing a timeout. I've attached the output form mochitest.

Comment 19

[Jonathan Griffin (:jgriffin)]

This error looks relevant, although I don't know why you're getting it:

************************************************************
* Call to xpconnect wrapped JSObject produced this error:  *
[Exception... "Component returned failure code: 0x80004005 (NS_ERROR_FAILURE) [nsIMessageSender.sendAsyncMessage]"  nsresult: "0x80004005 (NS_ERROR_FAILURE)"  location: "JS frame :: chrome://specialpowers/content/SpecialPowersObserver.js :: SpecialPowersObserver.prototype._sendAsyncMessage :: line 96"  data: no]
************************************************************

Comment 20

[[e-mail] [:onecyrenus]]

I don't show your device as getting on the wifi ? 
adb shell netcfg

did you use --wifi to specify the wifi connection ?

Comment 21

[Eric Rahm [:erahm]]

It's connected and on the same wireless network as my host, do I still need --wifi?

erahm-25043:mochitest ericrahm$ adb shell netcfg
lo       UP                                   127.0.0.1/8   0x00000049 00:00:00:00:00:00
[snip]
wlan0    UP                               192.168.10.14/24  0x00001043 00:03:7f:05:c0:ca

erahm-25043:mochitest ericrahm$ ifconfig en0
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	ether 60:03:08:a7:25:10 
	inet6 fe80::6203:8ff:fea7:2510%en0 prefixlen 64 scopeid 0x4 
	inet 192.168.10.9 netmask 0xffffff00 broadcast 192.168.10.255
	nd6 options=1<PERFORMNUD>
	media: autoselect
	status: active

Comment 22

[[e-mail] [:onecyrenus]]

On a whim synced my environment, and noticed that I too was experiencing this same problem.

Disabled NUWA, by editing gecko/b2g/confvars.sh, rebuilt, and this problem went away. 

So I think this is a recent issue, hoping that this unblocks erahm.

Comment 23

[Jason Smith [:jsmith]]

(In reply to [e-mail]  [:onecyrenus] from comment #22)
> On a whim synced my environment, and noticed that I too was experiencing
> this same problem.
> 
> Disabled NUWA, by editing gecko/b2g/confvars.sh, rebuilt, and this problem
> went away. 
> 
> So I think this is a recent issue, hoping that this unblocks erahm.

Can we get a bug on file for this problem & cc the nuwa folks on it for them to investigate it?

Comment 24

[Ben Kelly [:bkelly, not reviewing]]

B2G is wontfix at this point.