1004405 - [tarako] JavascriptException: TypeError: storage is null when running b2gpopulate

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Vendcom, defect, P1)

Description

[Dave Hunt [:davehunt] [he/him] ⌚BST]

On April 26th the b2gperf tests for Gallery, Video and Music apps started failing on Tarako due to a JavaScript exception raised when attempting to remove resources ahead of populating test-specific resources.

Gecko
Good: d91121fbca90
Bad: 204c7e12ea8b
Pushlog: http://hg.mozilla.org/releases/mozilla-b2g28_v1_3t/pushloghtml?fromchange=d91121fbca90&tochange=204c7e12ea8b

Gaia
Good: 293b056683b506ec1b4787d0fbf6033cc30fa953
Bad: 4dd06db93607f55694953fe60751eddb4bbb5afc
Pushlog: https://github.com/mozilla-b2g/gaia/compare/293b056683b506ec1b4787d0fbf6033cc30fa953...4dd06db93607f55694953fe60751eddb4bbb5afc

There aren't many changes here, and none of them jump at me.

The full traceback for one such exception follows:

2014-04-26 01:50:51,357 B2GPerfRunner INFO | Running B2GPerfLaunchGalleryTest
Traceback (most recent call last):
  File "/var/jenkins/workspace/b2g.tarako.mozilla-b2g28_v1_3t.perf/.env/local/lib/python2.7/site-packages/b2gperf/b2gperf.py", line 230, in measure_app_perf
    test.run()
  File "/var/jenkins/workspace/b2g.tarako.mozilla-b2g28_v1_3t.perf/.env/local/lib/python2.7/site-packages/b2gperf/b2gperf.py", line 342, in run
    self.setup()
  File "/var/jenkins/workspace/b2g.tarako.mozilla-b2g28_v1_3t.perf/.env/local/lib/python2.7/site-packages/b2gperf/b2gperf.py", line 389, in setup
    B2GPerfTest.setup(self)
  File "/var/jenkins/workspace/b2g.tarako.mozilla-b2g28_v1_3t.perf/.env/local/lib/python2.7/site-packages/b2gperf/b2gperf.py", line 316, in setup
    self.populate_files()
  File "/var/jenkins/workspace/b2g.tarako.mozilla-b2g28_v1_3t.perf/.env/local/lib/python2.7/site-packages/b2gperf/b2gperf.py", line 412, in populate_files
    self.b2gpopulate.populate_pictures(700)
  File "/var/jenkins/workspace/b2g.tarako.mozilla-b2g28_v1_3t.perf/.env/local/lib/python2.7/site-packages/b2gpopulate/b2gpopulate.py", line 313, in populate_pictures
    self.populate_files('picture', source, count, destination)
  File "/var/jenkins/workspace/b2g.tarako.mozilla-b2g28_v1_3t.perf/.env/local/lib/python2.7/site-packages/b2gpopulate/b2gpopulate.py", line 320, in populate_files
    self.remove_media(file_type)
  File "/var/jenkins/workspace/b2g.tarako.mozilla-b2g28_v1_3t.perf/.env/local/lib/python2.7/site-packages/b2gpopulate/b2gpopulate.py", line 331, in remove_media
    files = getattr(self.data_layer, '%s_files' % file_type) or []
  File "/var/jenkins/workspace/b2g.tarako.mozilla-b2g28_v1_3t.perf/.env/local/lib/python2.7/site-packages/gaiatest/gaia_test.py", line 340, in picture_files
    'return GaiaDataLayer.getAllPictures();')
  File "/var/jenkins/workspace/b2g.tarako.mozilla-b2g28_v1_3t.perf/.env/local/lib/python2.7/site-packages/marionette/marionette.py", line 1162, in execute_async_script
    filename=os.path.basename(frame[0]))
  File "/var/jenkins/workspace/b2g.tarako.mozilla-b2g28_v1_3t.perf/.env/local/lib/python2.7/site-packages/marionette/marionette.py", line 613, in _send_message
    self._handle_error(response)
  File "/var/jenkins/workspace/b2g.tarako.mozilla-b2g28_v1_3t.perf/.env/local/lib/python2.7/site-packages/marionette/marionette.py", line 648, in _handle_error
    raise JavascriptException(message=message, status=status, stacktrace=stacktrace)
JavascriptException: JavascriptException: TypeError: storage is null
stacktrace:
	execute_async_script @gaia_test.py, line 340
	inline javascript, line 841
	src: "    var req = storage.enumerate();"

Comment 1

[Dave Hunt [:davehunt] [he/him] ⌚BST]

I've narrowed the regression range using tinderbox builds:

Gecko
Good: 612a5616a1b6
Bad: 6e8c66aa685b
Pushlog: http://hg.mozilla.org/releases/mozilla-b2g28_v1_3t/pushloghtml?fromchange=612a5616a1b6&tochange=6e8c66aa685b

There's only one change here:
http://hg.mozilla.org/releases/mozilla-b2g28_v1_3t/rev/6e8c66aa685b

I'm guessing this is related to our partner.

Jason: can you suggest where to go from here?

Comment 2

[Dave Hunt [:davehunt] [he/him] ⌚BST]

Hey Fabrice, is this something you can assist with?

Comment 3

[Jason Smith [:jsmith]]

I'll wait for Fabrice to confirm, but if this is a partner issue, then we'll want to contact Spreadtrum about this.

Comment 4

[Robert Wood [:rwood]]

This same issue is blocking the gaia-ui endurance tests on Tarako.

Comment 5

[[:fabrice] Fabrice Desré]

A change landed on spreadtrum side to enable internal storage. That changes the partition layout, so you should check what it looks like on your device. I haven't yet updated my base image so I can't tell you...

Comment 6

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Fabrice Desré [:fabrice] from comment #5)
> A change landed on spreadtrum side to enable internal storage. That changes
> the partition layout, so you should check what it looks like on your device.
> I haven't yet updated my base image so I can't tell you...

The line that causes this can be found here: https://github.com/mozilla-b2g/gaia/blob/01fe45d91fdb4ae1e1e83a5c8b5fbb4efd3211f8/tests/atoms/gaia_data_layer.js#L431 I wouldn't expect enabling internal storage should cause this to return null. The devices running these have not had a new base build updated recently either, is this perhaps what's needed?

Comment 7

[[:fabrice] Fabrice Desré]

Can we log which storage type it's trying to open? The change from http://hg.mozilla.org/releases/mozilla-b2g28_v1_3t/rev/6e8c66aa685b has the "support internal sdcard" change though (it's 448208b894033ea5c71d2cc9d87c88cf5caa046c in device/sprd), so I strongly suspect this to be the issue.

Can you reflash with a new base image?

Comment 8

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

FYI, There's a new PAC file that came out : 2014-04-29

root@android:/sdcard # df
Filesystem             Size   Used   Free   Blksize
/dev                    49M    68K    49M   4096
/mnt/asec               49M     0K    49M   4096
/mnt/obb                49M     0K    49M   4096
/system                110M   100M     9M   4096
/data                   87M     5M    81M   4096
/cache                   7M   864K     6M   4096
/productinfo             2M   784K     1M   4096
/runtimenv               3M     1M     2M   4096
/backupfixnv             2M   836K     1M   4096
/fixnv                   2M   836K     1M   4096
/mnt/internal-sdcard    10M     3M     6M   4096
/mnt/sdcard              1G     1G   493M   32768
/mnt/secure/asec         1G     1G   493M   32768

Comment 9

[Jason Smith [:jsmith]]

Rob - Were the gaia ui endurance tests ever running on Tarako? Just wondering if this is a regression or not.

Comment 10

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Attachment: logcat.txt

I found out that I tried to take a picture and got this error message of "Not enough space on memory card"

root@android:/ # df
Filesystem             Size   Used   Free   Blksize
/dev                    49M    68K    49M   4096
/mnt/asec               49M     0K    49M   4096
/mnt/obb                49M     0K    49M   4096
/system                110M   100M     9M   4096
/data                   87M     5M    81M   4096
/cache                   7M   872K     6M   4096
/productinfo             2M   784K     1M   4096
/runtimenv               3M     1M     2M   4096
/backupfixnv             2M   836K     1M   4096
/fixnv                   2M   836K     1M   4096
/mnt/internal-sdcard    10M     3M     6M   4096
/mnt/sdcard              1G     1G   493M   32768
/mnt/secure/asec         1G     1G   493M   32768

logcat shows something that seems odd to me : 05-01 17:27:57.332: E/slog(100): check_available_volume: statfs return err! [2]

Comment 11

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

james, I think this might be an issue.

The other reason why I think it might be an issue is:
I placed in a 2 gig SDcard and it's only showing 1 gig:
/mnt/sdcard              1G     1G   493M   32768
/mnt/secure/asec         1G     1G   493M   32768

I placed in a 4 gig and it's showing 3 gig:
/mnt/sdcard              3G   139M     3G   32768
/mnt/secure/asec         3G   139M     3G   32768

This might be a separate bug?

Comment 12

[James Zhang (Spreadtrum)]

(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #11)
> james, I think this might be an issue.
> 
> The other reason why I think it might be an issue is:
> I placed in a 2 gig SDcard and it's only showing 1 gig:
> /mnt/sdcard              1G     1G   493M   32768
> /mnt/secure/asec         1G     1G   493M   32768
> 
> I placed in a 4 gig and it's showing 3 gig:
> /mnt/sdcard              3G   139M     3G   32768
> /mnt/secure/asec         3G   139M     3G   32768
> 
> This might be a separate bug?

Loop Lianxiang.

Comment 13

[James Zhang (Spreadtrum)]

Lianxiang said it's "adb shell df" issue, please use "adb shell busybox df".

Comment 14

[Dave Hunt [:davehunt] [he/him] ⌚BST]

(In reply to Fabrice Desré [:fabrice] from comment #7)
> Can we log which storage type it's trying to open? The change from
> http://hg.mozilla.org/releases/mozilla-b2g28_v1_3t/rev/6e8c66aa685b has the
> "support internal sdcard" change though (it's
> 448208b894033ea5c71d2cc9d87c88cf5caa046c in device/sprd), so I strongly
> suspect this to be the issue.
> 
> Can you reflash with a new base image?

We get the failure with storage types: 'pictures', 'videos', and 'music'. Where can I get a new base image? Does it require Windows to reflash? If so, this might have to wait until I'm in the office next week, unless I can get it to work in a VM.

Comment 15

[Robert Wood [:rwood]]

(In reply to Jason Smith [:jsmith] from comment #9)
> Rob - Were the gaia ui endurance tests ever running on Tarako? Just
> wondering if this is a regression or not.

Yes, the endurance tests were up and running on Tarako successfully before this issue started.

Comment 16

[[:fabrice] Fabrice Desré]

(In reply to Dave Hunt (:davehunt) from comment #14)
> (In reply to Fabrice Desré [:fabrice] from comment #7)
> > Can we log which storage type it's trying to open? The change from
> > http://hg.mozilla.org/releases/mozilla-b2g28_v1_3t/rev/6e8c66aa685b has the
> > "support internal sdcard" change though (it's
> > 448208b894033ea5c71d2cc9d87c88cf5caa046c in device/sprd), so I strongly
> > suspect this to be the issue.
> > 
> > Can you reflash with a new base image?
> 
> We get the failure with storage types: 'pictures', 'videos', and 'music'.
> Where can I get a new base image? Does it require Windows to reflash? If so,
> this might have to wait until I'm in the office next week, unless I can get
> it to work in a VM.

I don't know it that works in a VM, but yes I would reflash.

Comment 17

[Joe Cheng [:jcheng] (please needinfo)]

triage: 
the 10MB of internal storage is a product decision so it is not something to be reverted
so i would not really call this a regression
this seem like not part of the build and we should change our tests to cope wit this change. 
1.3T+ so we can get tests running again
thanks

Comment 18

[Jason Smith [:jsmith]]

(In reply to Joe Cheng [:jcheng] from comment #17)
> triage: 
> the 10MB of internal storage is a product decision so it is not something to
> be reverted
> so i would not really call this a regression
> this seem like not part of the build and we should change our tests to cope
> wit this change. 
> 1.3T+ so we can get tests running again
> thanks

That doesn't make sense to me. You shouldn't forcibly cause an automation suite to go down on a landing even if it was expected change - you should make the adjustments before the landing is complete. This is in fact a regression, as this regressed an existing automation suite. The original change needs to be backed out here to get the suite back online & a solution needs to be put in place that solves the product problem without causing a suite regression, so I'm removing the NPOTB here.

Comment 19

[Robert Wood [:rwood]]

:davehunt, I am trying to reflash my tarako with the newest base image to see if that fixes the issue, but so far I can't get the flashing tool to recognize my tarako (but will keep trying). I will email you the link to the latest base images in case you want to try also at some point.

Comment 20

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

bug 1003045 has also been fixed, which may make a difference.  For some reason the internal storage is used by default it seems after flashing and I think that is what you might be encountering.

location for the internal storage is : /mnt/internal-sdcard 

Can you recheck please?

Comment 21

[James Zhang (Spreadtrum)]

I think this test case can be passed now.
We change setting.json and use external-sdcard as default media stroage, not internal-sdcard.

Please see device/sprd git repo commit.

commit deafa34c483537c9b9eca5331695d08eb9a61b7c
Author: james.zhang <[e-mail]>
Date:   Fri May 2 15:34:43 2014 +0800

    Bug#307801 - [6821][FirefoxOS_v1.3][ST_India_Delhi]Airtel,6821 Fail to open USB storage.
    
    [bug number  ] 307801
    [root cause  ] need config ums in setting.json
    [changes     ] add ums configure to setting.json
    [side effects] none
    [self test   ] yes
    [reviewers   ] james.zhang
    
    Change-Id: Ic226a28a07307f186e3fa0e2cfe19927b6febb3b

commit 448208b894033ea5c71d2cc9d87c88cf5caa046c
Author: james.zhang <[e-mail]>
Date:   Fri Apr 25 15:16:59 2014 +0800

    Bug #301827 support internal sdcard
    
    [bug number  ]
    [root cause  ]
    [changes     ]
    [side effects]
    [self test   ] mozilla verify
    [reviewers   ]
    
    Change-Id: I7e402d2e5c1e4f311524de05976b97dca24c6c77

Comment 22

[James Zhang (Spreadtrum)]

(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #20)
> bug 1003045 has also been fixed, which may make a difference.  For some
> reason the internal storage is used by default it seems after flashing and I
> think that is what you might be encountering.
> 
> location for the internal storage is : /mnt/internal-sdcard 
> 
> Can you recheck please?

And include this patch landed.