The default bug view has changed. See this FAQ.

Nested omni.jar for Android

RESOLVED FIXED in mozilla11

Status

()

Core
Widget: Android
RESOLVED FIXED
6 years ago
5 years ago

People

(Reporter: glandium, Assigned: glandium)

Tracking

(Depends on: 1 bug, Blocks: 1 bug)

Trunk
mozilla11
ARM
Android
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [mobilestartupshrink][ts][inbound])

Attachments

(11 attachments, 17 obsolete attachments)

3.52 KB, patch
bsmedberg
: review+
Details | Diff | Splinter Review
9.21 KB, patch
mrbkap
: review+
Details | Diff | Splinter Review
7.83 KB, patch
bsmedberg
: review+
Details | Diff | Splinter Review
1.26 KB, patch
mrbkap
: review+
Details | Diff | Splinter Review
16.17 KB, patch
glandium
: review+
Details | Diff | Splinter Review
3.75 KB, patch
glandium
: review+
Details | Diff | Splinter Review
2.63 KB, patch
(dormant account)
: review+
Details | Diff | Splinter Review
4.59 KB, patch
glandium
: review+
Details | Diff | Splinter Review
16.38 KB, patch
mwu
: review+
Details | Diff | Splinter Review
58.49 KB, patch
glandium
: review+
Details | Diff | Splinter Review
14.34 KB, patch
Details | Diff | Splinter Review
(Assignee)

Description

6 years ago
Around 200ms are wasted on a nexus s at startup because Android does something with the apk zip entries and takes time for this because we have so many entries in the apk. Getting the number of entries down should allow us to avoid wasting these 200ms. We can do that by keeping chrome, components and resources compressed under a single (uncompressed) file in the apk.
(Assignee)

Comment 1

6 years ago
(In reply to Mike Hommey [:glandium] from comment #0)
> Around 200ms are wasted on a nexus s at startup.

Note that it's even more on multi-language builds.
(Assignee)

Comment 2

6 years ago
Created attachment 569041 [details] [diff] [review]
[WIP] part 1 - Make Omnijar::GetReader return an nsIZipReader instead of nsZipArchive

It still needs to be modified such that nsJAR::GetZipArchive is not required.
(Assignee)

Comment 3

6 years ago
Created attachment 569042 [details] [diff] [review]
part 2 - Avoid code duplication for components manifest registration
Attachment #569042 - Flags: review?(benjamin)
(Assignee)

Comment 4

6 years ago
Created attachment 569043 [details] [diff] [review]
[WIP] part 3 - Get the component manager to use Omnijar::GetReader instead of Omnijar::GetPath

I'm not sure we should keep nsIZipReaders around, here. The impact should definitely be checked.
(Assignee)

Comment 5

6 years ago
Created attachment 569044 [details] [diff] [review]
[WIP] part 4 - Allow nested omni.jar and build one for Android

This part definitely needs more work, to have something proper for the chrome registry and js component loader. The other part that's broken with this patch is that ManifestProcessingContext, both in the component manager and the chrome registry, are using nsIFiles for the base path to resolve from, and thus fail with nested omni.jar. They probably need to store either an nsIFile or an nsIZipReader depending on cases.

(I just wish we had fake nsIFiles that would allow to use Zips transparently, like directories, but so many things rely on nsIFiles to actually be nsILocalFiles...)
(Assignee)

Updated

6 years ago
Blocks: 697356
Attachment #569042 - Flags: review?(benjamin) → review+
(Assignee)

Updated

5 years ago
Attachment #569041 - Attachment is obsolete: true
(Assignee)

Updated

5 years ago
Attachment #569043 - Attachment is obsolete: true
(Assignee)

Updated

5 years ago
Attachment #569044 - Attachment is obsolete: true
(Assignee)

Comment 6

5 years ago
Created attachment 572834 [details] [diff] [review]
part 1 - Add Refcounting on nsZipArchives
Attachment #572834 - Flags: review?(mwu)
(Assignee)

Comment 7

5 years ago
Created attachment 572835 [details] [diff] [review]
part 3 - Use mozilla::AutoFDClose instead of ad hoc AutoCloseFD classes
Attachment #572835 - Flags: review?(benjamin)
(Assignee)

Comment 8

5 years ago
Created attachment 572836 [details] [diff] [review]
part 4 - Add a FileLocation class to either store a pair (zip, path) or a file, and do some operations on these

I'm not settled on this one. I may switch to using nsZipHandles instead of nsZipArchives in FileLocation. The class also needs some documentation, and the API may require some tweaking.
(Assignee)

Updated

5 years ago
Attachment #572836 - Flags: feedback?(mwu)
(Assignee)

Comment 9

5 years ago
Created attachment 572838 [details] [diff] [review]
part 5 - Allow nsZipCursor to always use the given buffer
Attachment #572838 - Flags: review?(mwu)
(Assignee)

Comment 10

5 years ago
Created attachment 572840 [details] [diff] [review]
part 6 - Don't separate file and JAR modules in component manager
Attachment #572840 - Flags: review?(benjamin)
(Assignee)

Comment 11

5 years ago
(In reply to Mike Hommey [:glandium] from comment #10)
> Created attachment 572840 [details] [diff] [review] [diff] [details] [review]
> part 6 - Don't separate file and JAR modules in component manager

Note that the use of net_GetURLSpecFromActualFile goes away in a subsequent patch.
(Assignee)

Comment 12

5 years ago
Created attachment 572841 [details] [diff] [review]
part 7 - Use the URI string as the key for the various mozJSComponentLoader hashtables
Attachment #572841 - Flags: review?(mrbkap)
(Assignee)

Comment 13

5 years ago
Created attachment 572842 [details] [diff] [review]
part 8 - Properly handle jar-in-jars when importing modules
Attachment #572842 - Flags: review?(mrbkap)
(Assignee)

Comment 14

5 years ago
Created attachment 572843 [details] [diff] [review]
part 9 - Use FileLocations in the component manager

This uses the API from part 4.
(Assignee)

Updated

5 years ago
Attachment #572843 - Flags: feedback?(benjamin)
(Assignee)

Comment 15

5 years ago
Created attachment 572844 [details] [diff] [review]
part 10 - Allow nested omni.jar and build one for Android
Attachment #572844 - Flags: review?(benjamin)
(Assignee)

Comment 16

5 years ago
Created attachment 572848 [details] [diff] [review]
part 6 - Don't separate file and JAR modules in component manager
Attachment #572848 - Flags: review?(benjamin)
(Assignee)

Updated

5 years ago
Attachment #572840 - Attachment is obsolete: true
Attachment #572840 - Flags: review?(benjamin)

Comment 17

5 years ago
Comment on attachment 572838 [details] [diff] [review]
part 5 - Allow nsZipCursor to always use the given buffer

Wouldn't it be cleaner to add a nsZipCursor::Copy?

Updated

5 years ago
Attachment #572834 - Flags: review?(mwu) → review+

Comment 18

5 years ago
Comment on attachment 572838 [details] [diff] [review]
part 5 - Allow nsZipCursor to always use the given buffer

Taras seems to have more of an opinion on this one than I do.
Attachment #572838 - Flags: review?(mwu) → review?(tglek)
(Assignee)

Comment 19

5 years ago
Try result on this patch queue:
https://tbpl.mozilla.org/?noignore=1&tree=Try&rev=cc189b22e21d

Summary:
- 2 nsZipArchives are leaked
- Bootstrap extensions test seems to go in an infinite loop on windows.
(Assignee)

Comment 20

5 years ago
(In reply to Mike Hommey [:glandium] from comment #19)
> - 2 nsZipArchives are leaked

Found the culprit. Will update the relevant patch later.

Another issue: jars are likely to be mapped twice in memory with the current patch queue. Bug 504217 should help there.
(Assignee)

Comment 21

5 years ago
Created attachment 573171 [details] [diff] [review]
part 8 - Properly handle jar-in-jars when importing modules
Attachment #573171 - Flags: review?(mrbkap)
(Assignee)

Updated

5 years ago
Attachment #572842 - Attachment is obsolete: true
Attachment #572842 - Flags: review?(mrbkap)
(Assignee)

Comment 22

5 years ago
Created attachment 573172 [details] [diff] [review]
part 1 - Add Refcounting on nsZipArchives

Same patch, which removes a "mZip.forget()" that was a leftover from the original use of nsAutoPtr.
(Assignee)

Updated

5 years ago
Attachment #572834 - Attachment is obsolete: true
(Assignee)

Comment 23

5 years ago
Comment on attachment 573172 [details] [diff] [review]
part 1 - Add Refcounting on nsZipArchives

Carrying previous r+, considering the changes.
Attachment #573172 - Flags: review+
(Assignee)

Comment 24

5 years ago
(In reply to Mike Hommey [:glandium] from comment #19)
> - Bootstrap extensions test seems to go in an infinite loop on windows.

The test goes on waiting for nothing because it doesn't handle failure gracefully. The failure itself is due to the xpi file still being open (iow, a nsZipArchive still existing) when it is removed (NS_ERROR_FILE_ACCESS_DENIED on nsIFile.remove) in test_bootstrap.js line 113, function manuallyUninstall. Obviously that kind of thing only fail on windows.

Comment 25

5 years ago
Comment on attachment 572838 [details] [diff] [review]
part 5 - Allow nsZipCursor to always use the given buffer

We decided to add a nsZipCursor::Copy
Attachment #572838 - Flags: review?(tglek) → review-

Comment 26

5 years ago
Comment on attachment 572836 [details] [diff] [review]
part 4 - Add a FileLocation class to either store a pair (zip, path) or a file, and do some operations on these

So, I think FileLocation makes a lot of sense for the component manager code but not as much for nsZipHandle. That being said, this is one of the easier ways for you to generate uris and it doesn't mess with nsZipHandle too much. I suspect that having nsZipHandle directly generate the information you want is a cleaner way to go here, and might be useful for some other code along the way. Up to you though - if moving things to nsZipHandle is too complicated for now, that's fine.
Attachment #572836 - Flags: feedback?(mwu)

Updated

5 years ago
Attachment #573171 - Flags: review?(mrbkap) → review+

Updated

5 years ago
Attachment #572841 - Flags: review?(mrbkap) → review+
Attachment #572835 - Flags: review?(benjamin) → review+
Attachment #572843 - Flags: feedback?(benjamin) → feedback+
Attachment #572844 - Flags: review?(benjamin) → review+
Comment on attachment 572848 [details] [diff] [review]
part 6 - Don't separate file and JAR modules in component manager

I'm not sure that hashing on the URI representation is correct or will work correctly: we've had issues in the past with short and long path names not comparing equal correctly. What is the actual behavior of net_GetURLSpecFromActualFile WRT case differences or things like that?
(Assignee)

Comment 28

5 years ago
(In reply to Benjamin Smedberg  [:bsmedberg] from comment #27)
> Comment on attachment 572848 [details] [diff] [review] [diff] [details] [review]
> part 6 - Don't separate file and JAR modules in component manager
> 
> I'm not sure that hashing on the URI representation is correct or will work
> correctly: we've had issues in the past with short and long path names not
> comparing equal correctly. What is the actual behavior of
> net_GetURLSpecFromActualFile WRT case differences or things like that?

net_GetURLSpecFromActualFile doesn't ensure a canonical path before turning the path into an url :(
The same concern would also apply on part 7. I guess we could canonicalize paths before putting them in the uri string used for keys.
(Assignee)

Comment 29

5 years ago
Note that the nsIHashable part of nsLocalFile* doesn't handle case sensitivity on Mac, AFAICS, so we may already have the case sensitivity problem.
Comment on attachment 573172 [details] [diff] [review]
part 1 - Add Refcounting on nsZipArchives

>--- a/startupcache/StartupCache.cpp
>+++ b/startupcache/StartupCache.cpp
>+nsresult
>+GetBufferFromZipArchive(nsZipArchive *zip, bool doCRC, const char* id,
>+                        char** outbuf, PRUint32* length)
>+{
>+  if (zip) {
>+    nsZipItemPtr<char> zipItem(zip, id, doCRC);
>+    if (zipItem) {
>+      *outbuf = zipItem.Forget();
>+      *length = zipItem.Length();
>+      return NS_OK;
>+    }
>+  }
>+  return NS_ERROR_NOT_AVAILABLE;
>+}

Early returns, please.

>+  nsresult rv;
>+  rv = GetBufferFromZipArchive(mArchive, true, id, outbuf, length);

nsresult rv = GetBufferFromZipArchive(mArchive, true, id, outbuf, length);

>--- a/xpcom/build/Omnijar.cpp
>+++ b/xpcom/build/Omnijar.cpp
> Omnijar::InitOne(nsIFile *aPath, Type aType)
>+    nsRefPtr<nsZipArchive> zipReader = new nsZipArchive();
>+    if (!zipReader || NS_FAILED(zipReader->OpenArchive(file))) {

Remove the '!zipReader' check, 'new' can't fail.

>--- a/xpcom/build/Omnijar.h
>+++ b/xpcom/build/Omnijar.h
>+static inline already_AddRefed<nsZipArchive>
> GetReader(Type aType)
> {
>     NS_ABORT_IF_FALSE(IsInitialized(), "Omnijar not initialized");
>+    NS_IF_ADDREF(sReader[aType]);
>     return sReader[aType];

nsRefPtr<nsZipArchive> reader = sReader[aType];
return reader.forget();
Comment on attachment 572836 [details] [diff] [review]
part 4 - Add a FileLocation class to either store a pair (zip, path) or a file, and do some operations on these

>--- /dev/null
>+++ b/xpcom/build/FileLocation.cpp
>+namespace mozilla {
>+
>+nsresult FileLocation::GetData(char *buf, PRUint32 len)
>+{
>+    nsZipCursor cursor(item, mBaseZip, (PRUint8 *)buf, len, true);

static_cast

>+  mozilla::AutoFDClose fd;

No need for the 'mozilla::'.

>--- /dev/null
>+++ b/xpcom/build/FileLocation.h
>+  void Init(nsILocalFile *file)
>+  {
>+    mBaseFile = file;
>+    mPath = (char *)NULL;

mPath.Truncate()?

>+/* This needs to be included at the end because of the cross-references */
>+#include "nsZipArchive.h"

Uh, really? Why does this need to be included at all?
Comment on attachment 572844 [details] [diff] [review]
part 10 - Allow nested omni.jar and build one for Android

>--- a/xpcom/build/Omnijar.cpp
>+++ b/xpcom/build/Omnijar.cpp
>@@ -105,16 +106,24 @@ Omnijar::InitOne(nsIFile *aPath, Type aT
>+    nsRefPtr<nsZipHandle> handle;
>+    if (NS_SUCCEEDED(nsZipHandle::Init(zipReader, "omni.jar", getter_AddRefs(handle)))) {
>+        zipReader = new nsZipArchive();
>+        if (!zipReader...

And here
Comment on attachment 572848 [details] [diff] [review]
part 6 - Don't separate file and JAR modules in component manager

Ugh. I guess that since we're using manifests now it's unlikely that we'll have mixed-case issues; I guess we'll find out later!
Attachment #572848 - Flags: review?(benjamin) → review+
(Assignee)

Comment 34

5 years ago
Created attachment 579619 [details] [diff] [review]
part 1 - Add Refcounting on nsZipArchives.

Updated with Ms2ger comments.
(Assignee)

Comment 35

5 years ago
Comment on attachment 579619 [details] [diff] [review]
part 1 - Add Refcounting on nsZipArchives.

Carrying over r+
Attachment #579619 - Flags: review+
(Assignee)

Updated

5 years ago
Attachment #573172 - Attachment is obsolete: true
(Assignee)

Comment 36

5 years ago
Created attachment 579620 [details] [diff] [review]
part 3 - Use mozilla::AutoFDClose instead of ad hoc AutoCloseFD classes

Refreshed against current m-c
(Assignee)

Updated

5 years ago
Attachment #572835 - Attachment is obsolete: true
(Assignee)

Comment 37

5 years ago
Comment on attachment 579620 [details] [diff] [review]
part 3 - Use mozilla::AutoFDClose instead of ad hoc AutoCloseFD classes

Carrying over r+
Attachment #579620 - Flags: review+
(Assignee)

Comment 38

5 years ago
Created attachment 579621 [details] [diff] [review]
part 4 - Allow nsZipCursor to always use the given buffer
Attachment #579621 - Flags: review?(tglek)
(Assignee)

Updated

5 years ago
Attachment #572838 - Attachment is obsolete: true
(Assignee)

Comment 39

5 years ago
Created attachment 579624 [details] [diff] [review]
part 5 - Add a FileLocation class to either store a pair (zip, path) or a file, and do some operations on these
Attachment #579624 - Flags: review?(mwu)
(Assignee)

Updated

5 years ago
Attachment #572836 - Attachment is obsolete: true
(Assignee)

Updated

5 years ago
Attachment #579624 - Flags: review?(benjamin)
(Assignee)

Comment 40

5 years ago
Created attachment 579626 [details] [diff] [review]
part 9 - Use FileLocations in the component manager
Attachment #579626 - Flags: review?(benjamin)
(Assignee)

Updated

5 years ago
Attachment #572843 - Attachment is obsolete: true
(Assignee)

Comment 41

5 years ago
Created attachment 579627 [details] [diff] [review]
part 10 - Allow nested omni.jar and build one for Android

Basically the same as previous version, but uses OMNIJAR_NAME instead of hardcoding omni.jar.
(Assignee)

Updated

5 years ago
Attachment #572844 - Attachment is obsolete: true
(Assignee)

Comment 42

5 years ago
Comment on attachment 579627 [details] [diff] [review]
part 10 - Allow nested omni.jar and build one for Android

Carrying over r+
Attachment #579627 - Flags: review+
(Assignee)

Comment 43

5 years ago
Created attachment 579628 [details] [diff] [review]
extra - Use more canonical keys for component manager and JS component loader hash tables

In the case we end up having problems with short vs. long path names making uris not stable enough to be hash keys for the component manager and the JS component loader, here is a patch that makes the relevant hash tables use something more canonical. Attaching so that it can be found later if needed, since I already did the work (instead of being lost on my harddrive)

Comment 44

5 years ago
Comment on attachment 579621 [details] [diff] [review]
part 4 - Allow nsZipCursor to always use the given buffer

thanks
Attachment #579621 - Flags: review?(mozilla) → review+
Comment on attachment 579624 [details] [diff] [review]
part 5 - Add a FileLocation class to either store a pair (zip, path) or a file, and do some operations on these

Some of the single-line ifs in FileLocation::FileLocation aren't braced and I find this hard to read: please brace them all.

Please comment namespace-closing brace in FileLocation.cpp, e.g.:

} // namespace mozilla

The opening brace of "class FileLocation {" and FileLocation::Data should be on the next line

It's unusual to return an nsCString (in FileLocation::GetURIString and FileLocation::GetPath). Normally methods that return a string use: 

void ReturnAString(nsACString& result);

Since we have copy-on-write strings, perhaps this isn't a problem, but I'd prefer to be consistent with other parts of our code.

r=me with those changes
Attachment #579624 - Flags: review?(benjamin) → review+
Attachment #579626 - Flags: review?(benjamin) → review+
(Assignee)

Comment 46

5 years ago
Created attachment 579972 [details] [diff] [review]
part 5 - Add a FileLocation class to either store a pair (zip, path) or a file, and do some operations on these.

With bsmedberg's comments addressed
Attachment #579972 - Flags: review?(mwu)
(Assignee)

Updated

5 years ago
Attachment #579624 - Attachment is obsolete: true
Attachment #579624 - Flags: review?(mwu)
(Assignee)

Comment 47

5 years ago
Created attachment 579973 [details] [diff] [review]
part 9 - Use FileLocations in the component manager.

Refreshed with the new modified GetURIString signature from part 5.
(Assignee)

Updated

5 years ago
Attachment #579626 - Attachment is obsolete: true
(Assignee)

Comment 48

5 years ago
Created attachment 579974 [details] [diff] [review]
part 5 - Add a FileLocation class to either store a pair (zip, path) or a file, and do some operations on these.

Removed a bit that is meant to be in the "extra" patch.
Attachment #579974 - Flags: review?(mwu)
(Assignee)

Updated

5 years ago
Attachment #579972 - Attachment is obsolete: true
Attachment #579972 - Flags: review?(mwu)
(Assignee)

Comment 49

5 years ago
Created attachment 579975 [details] [diff] [review]
part 9 - Use FileLocations in the component manager.

Removed an unused variable
(Assignee)

Updated

5 years ago
Attachment #579973 - Attachment is obsolete: true
(Assignee)

Comment 50

5 years ago
Created attachment 579976 [details] [diff] [review]
extra - Use more canonical keys for component manager and JS component loader hash tables

Refreshed against refreshed part 9.
(Assignee)

Updated

5 years ago
Attachment #579628 - Attachment is obsolete: true
(Assignee)

Comment 51

5 years ago
Comment on attachment 579975 [details] [diff] [review]
part 9 - Use FileLocations in the component manager.

Carrying over r+.
Attachment #579975 - Flags: review+

Comment 52

5 years ago
Comment on attachment 579974 [details] [diff] [review]
part 5 - Add a FileLocation class to either store a pair (zip, path) or a file, and do some operations on these.

Review of attachment 579974 [details] [diff] [review]:
-----------------------------------------------------------------

Looks ok overall though I haven't really studied how some of the callers use some of this API. Comments are just nits.

::: xpcom/build/FileLocation.cpp
@@ +106,5 @@
> +  if (IsZip() && mBaseZip) {
> +    nsRefPtr<nsZipHandle> handler = mBaseZip->GetFD();
> +    if (handler)
> +      return handler->mFile.GetBaseFile();
> +  } else {

I think this else could be removed after moving return NULL to the "if (IsZip() && mBaseZip)" block, or maybe invert this if/else. Having the return NULL only apply to one of the if/else blocks is a bit weird.

@@ +139,5 @@
> +
> +nsresult
> +FileLocation::GetData(Data &data)
> +{
> +  if (IsZip()) {

if (!IsZip())
  return mBaseFile->OpenNSPRFileDesc(PR_RDONLY, 0444, &data.mFd);

::: xpcom/build/FileLocation.h
@@ +132,5 @@
> +
> +  /**
> +   * Returns whether the "base file" (see GetBaseFile) is an archive
> +   */
> +  bool IsZip() const {

Curly brace style inconsistent here and in the next two functions.
Attachment #579974 - Flags: review?(mwu) → review+
(Assignee)

Comment 53

5 years ago
https://hg.mozilla.org/integration/mozilla-inbound/rev/a31949628a7e
https://hg.mozilla.org/integration/mozilla-inbound/rev/8e345e9d93e9
https://hg.mozilla.org/integration/mozilla-inbound/rev/3032fd2bd96a
https://hg.mozilla.org/integration/mozilla-inbound/rev/692b6b2207b4
https://hg.mozilla.org/integration/mozilla-inbound/rev/616b0239a9fb
https://hg.mozilla.org/integration/mozilla-inbound/rev/c5a971a7dc4d
https://hg.mozilla.org/integration/mozilla-inbound/rev/00d65419d422
https://hg.mozilla.org/integration/mozilla-inbound/rev/2dd63b2aa323
https://hg.mozilla.org/integration/mozilla-inbound/rev/502c67d69baa
https://hg.mozilla.org/integration/mozilla-inbound/rev/792a9ba2aa26
Whiteboard: [mobilestartupshrink][ts] → [mobilestartupshrink][ts][inbound]
https://hg.mozilla.org/mozilla-central/rev/a31949628a7e
https://hg.mozilla.org/mozilla-central/rev/8e345e9d93e9
https://hg.mozilla.org/mozilla-central/rev/3032fd2bd96a
https://hg.mozilla.org/mozilla-central/rev/692b6b2207b4
https://hg.mozilla.org/mozilla-central/rev/616b0239a9fb
https://hg.mozilla.org/mozilla-central/rev/c5a971a7dc4d
https://hg.mozilla.org/mozilla-central/rev/00d65419d422
https://hg.mozilla.org/mozilla-central/rev/2dd63b2aa323
https://hg.mozilla.org/mozilla-central/rev/502c67d69baa
https://hg.mozilla.org/mozilla-central/rev/792a9ba2aa26
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla11
Depends on: 709250
Note: this seems to have regressed being able to access APK resources from Java using jar: URIs

Updated

5 years ago
Depends on: 712433
You need to log in before you can comment on or make changes to this bug.