1117136 - On Android, the DeviceStorage API only finds files in the internal storage, not in the external SD Card

Status: RESOLVED WONTFIX

(Core Graveyard :: Widget: Android, defect)

Description

[Mossroy]

I have this issue on a Samsung S3, with Firefox 34.0.1

Steps to reproduce :
- install from the marketplace an application that uses the DeviceStorage API and has been made available on Android. For example https://marketplace.firefox.com/app/ebook
- put an ebook file (epub extension) in the internal storage of the device : it is found by the application
- move the same files in an external SD card inside the device
- run the application again

Actual result :
the file is not found

Expected result :
the file is found, like on Firefox OS


I have the same behavior on https://marketplace.firefox.com/app/evopedia : on Firefox OS, if finds files both in internal or external storage, but only on internal storage on Firefox Android.
It uses the enumerate() function (https://developer.mozilla.org/en-US/docs/Web/API/DeviceStorage.enumerate), without any parameters.

Comment 1

[Mossroy]

I edit the name of this issue to make it clear that it only affects Firefox on Android, not Firefox OS.
Aaron, are you sure the OS should be changed to "All" instead of "Android"?

Comment 2

[Aaron Train [:aaronmt]]

Nope, misread your initial comment. Thanks.

Comment 3

[Mossroy]

OK I changed the OS back to Android.
Regarding the Component/Version/Product, I'm not sure if I should change them back to General/Firefox 34/Firefox for Android

Comment 4

[Mossroy]

Same issue with Firefox 35 on Android

Comment 5

[Mossroy]

Same issue with Firefox 36 on Android, even after the upgrade to kitkat 4.4.4

Comment 6

[Mossroy]

Same issue with Firefox 37 on Android

Comment 7

[Mossroy]

Same issue with Firefox 38.0.1 on Android.
Could I do something to help investigating this?

Comment 8

[Mossroy]

Same issue with Firefox 39 on Android

Comment 9

[Mossroy]

Same issue with Firefox 40 on Android

Comment 10

[Mossroy]

Not better with Firefox 41 on Android

Comment 11

[Mossroy]

Not better with Firefox 42 on Android

Comment 12

[Michael Henretty [:mikehenrty][:mhenretty]]

Richard, has this bug gotten any traction internally? The use case is to have an offline wikipedia app that works the same across all platforms (Android, FxOS, desktop), which the bug reporter is actively working on at the moment.

Comment 13

[Richard Newman [:rnewman]]

DeviceStorageStatics.cpp just calls into AndroidBridge::GetExternalPublicDirectory:

// Keep MOZ_WIDGET_ANDROID above XP_UNIX,
// because both are defined in Android builds.
#elif defined (MOZ_WIDGET_ANDROID)
  nsAutoString path;
  if (NS_SUCCEEDED(mozilla::AndroidBridge::GetExternalPublicDirectory(
      NS_LITERAL_STRING(DEVICESTORAGE_PICTURES), path))) {
    NS_NewLocalFile(path, /* aFollowLinks */ true,
                    getter_AddRefs(mDirs[TYPE_PICTURES]));
  }
  …


I don't know how this is supposed to work, because there are at least two storage locations on Android, or we have to present a virtualized filesystem.

I'm going to punt this back into Core, because this seems at least two steps removed from the kind of stuff that lives in this component -- a fix is likely to touch widget/android and dom/devicestorage, so this either belongs in Core::DOM or Core::Widget: Android.

needinfo to janv and snorp to figure that out.

Comment 14

[Richard Newman [:rnewman]]

(In reply to Michael Henretty [:mhenretty] from comment #12)
> Richard, has this bug gotten any traction internally? The use case is to
> have an offline wikipedia app that works the same across all platforms
> (Android, FxOS, desktop), which the bug reporter is actively working on at
> the moment.

To answer your question specifically: no, this hasn't gotten any traction; I don't believe anyone has ever looked at it. It's not marked as tracking, nobody was assigned, it's not blocking any other bugs, and I don't think anyone is really working on web app support.

Thanks for being the squeaky wheel!

Comment 15

[James Willcox (:snorp) ([e-mail]) (he/him)]

Yeah it doesn't look like things are set up in DeviceStorage to be able to handle two physical directories for the same media type (sdcard). Merging those seems tricky, too, so not sure what should be done. I think the ideal solution is to add a new storage area ('internal' maybe? Naming is hard) and use 'sdcard' for the actual external sdcard.

Comment 16

[Mossroy]

Thanks Michael for reviving this bug.
I tested with Firefox 43 on Android and the issue is still there.

Comment 17

[Michael Henretty [:mikehenrty][:mhenretty]]

(In reply to James Willcox (:snorp) ([e-mail]) from comment #15)
> I think the ideal
> solution is to add a new storage area ('internal' maybe? Naming is hard) and
> use 'sdcard' for the actual external sdcard.

Would it be possible to guide a contributor like Mossroy for fixing this? Their use case is for an offline wikipedia app that works the same across all platforms (Android, FxOS, desktop).

Comment 18

[Mark Finkle (:mfinkle) (use needinfo?)]

This bugs says the API only finds files on internal storage, but I would think the Android "external" storage is used:

DeviceStorage calls AndroidBridge::GetExternalPublicDirectory for all the folders:
http://mxr.mozilla.org/mozilla-central/source/dom/devicestorage/DeviceStorageStatics.cpp#179

AndroidBridge::GetExternalPublicDirectory seems to call Environment.getExternalStoragePublicDirectory() for everything except "apps" (uses Context.getFilesDir() which is "internal) and "sdcard" (uses Environment.getExternalStorageDirectory() which is "external"):
http://mxr.mozilla.org/mozilla-central/source/mobile/android/base/java/org/mozilla/gecko/GeckoAppShell.java#2715

I agree with others though, I don't know how we'd make traversal work across two different folders. Which storage would we rather use, "internal" or "external"?

Comment 19

[Mossroy]

Mark,
I tested several applications from the Marketplace that browse files of the device, and they all have this behavior :
- on Firefox OS, they are able to find and read files that are either in external or internal storage, which is the expected behavior
- on Firefox on Android, the same apps only find and read files in internal storage

I think the DeviceStorage API should have the same behavior on all platforms.

Try for example with https://marketplace.firefox.com/app/pdf-viewer?src=popular or https://marketplace.firefox.com/app/weboffice?src=popular

Comment 20

[Emmanuel Engelhart]

This problem impacts a lot of popular FF marketplace apps, which are or are not (because of this limitation?) available for Android.

Here a few popular one I have tested:
* https://marketplace.firefox.com/app/pdf-viewer
* https://marketplace.firefox.com/app/weboffice

Here a list many popular one which are probably impacted (but I don't have tested them):
* https://marketplace.firefox.com/app/file-manager
* https://marketplace.firefox.com/app/files
* https://marketplace.firefox.com/app/document-reader
* https://marketplace.firefox.com/app/folder-player
* https://marketplace.firefox.com/app/playfolder
* https://marketplace.firefox.com/app/epubreader
* https://marketplace.firefox.com/app/sd-explorer
* https://marketplace.firefox.com/app/crop
* https://marketplace.firefox.com/app/explorer
* https://marketplace.firefox.com/app/mp3-download
* https://marketplace.firefox.com/app/photography
* https://marketplace.firefox.com/app/recorder
* https://marketplace.firefox.com/app/file-manager
* https://marketplace.firefox.com/app/collage
* https://marketplace.firefox.com/app/cameran
* https://marketplace.firefox.com/app/picsart
* https://marketplace.firefox.com/app/downtube

From a high-level, a lot of apps in the categories "Utilities" and "Photo & Video" are probably impacted by this problem.

Comment 21

[Michael Henretty [:mikehenrty][:mhenretty]]

(In reply to Mossroy from comment #19)
> Mark,
> I tested several applications from the Marketplace that browse files of the
> device, and they all have this behavior :
> - on Firefox OS, they are able to find and read files that are either in
> external or internal storage, which is the expected behavior
> - on Firefox on Android, the same apps only find and read files in internal
> storage

Andrew, I think you have the most expertise on the B2G side of things. Can you confirm that the deviceStorage API is able to read simultaneously from both the internal and external sources, and if so how does this work from the API perspective?

Comment 22

[James Willcox (:snorp) ([e-mail]) (he/him)]

(In reply to Michael Henretty [:mhenretty] from comment #17)
> (In reply to James Willcox (:snorp) ([e-mail]) from comment #15)
> > I think the ideal
> > solution is to add a new storage area ('internal' maybe? Naming is hard) and
> > use 'sdcard' for the actual external sdcard.
> 
> Would it be possible to guide a contributor like Mossroy for fixing this?
> Their use case is for an offline wikipedia app that works the same across
> all platforms (Android, FxOS, desktop).

I actually think something like ServiceWorkers would be a better choice for that. Apparently the DeviceStorage API is only available for privileged/certified applications on Android, and that's all going away soon-ish, so I would just move away from it.

Comment 23

[Mossroy]

James, there might be a misunderstanding.

We are currently porting http://www.kiwix.org in HTML5/javascript. It allows to read all the wikipedia content offline : you first download and put on your device a big ZIM file, and the app reads all the content in it (without Internet access).
For that, we need an API that reads this content : to my knowledge, only the DeviceStorage API can do that on Gecko. As the ZIM files can be very big, they most often must be stored on an external storage (sd-card).
All this works well on Firefox OS devices. And yes, it is a privileged application, but I don't think it's an issue.

The bug I mention here is that, on Android, the DeviceStorage API is not able to read files on external storage : only on internal storage. It's not the same behavior as on Firefox OS, and prevents us from really using the app on Android.

What do you mean by "that's all going away soon-ish"?

Regarding ServiceWorkers, we do use one inside the application. It "traps" the HTTP requests for contents (articles, images, CSS etc), and reads them in the ZIM file instead of the Internet.
But I don't see how a ServiceWorker could replace the DeviceStorage API to read some files on the device.

Comment 24

[Jan Varga [:janv]]

Dave might be able to shed some light on this.

Comment 25

[Dave Hylands [:dhylands]]

I haven't played with DeviceStorage under Fennec.

My understanding is that the browser requires a special permission in order to access files on the sdcard.

Comment 26

[Kevin Brosnan [Ex-Mozilla]]

We already request this permission for downloads and updating Aurora/Nightly builds.

Comment 27

[Mossroy]

I suppose this can be closed as "won't fix" as the whole API has been removed from Firefox Android

Comment 28

[Emmanuel Engelhart]

Do we have some kind of document explaining the future of this API? Why this has been removed from Android?

Comment 29

[Mossroy]

As far as I know, this API is completely deprecated, with all the other "WebAPIs" https://wiki.mozilla.org/WebAPI

It has been done in several steps : 
- first removal from desktop and android : see references from https://www.fxsitecompat.com/en-CA/docs/2016/web-app-runtime-has-been-removed-from-firefox-for-desktop-and-android/
- then Firefox OS has been completely discontinued by Mozilla, and they removed all the corresponding code from Gecko : https://groups.google.com/forum/#!msg/mozilla.dev.fxos/FoAwifahNPY/Lppm0VHVBAAJ