Nested omni.jar for Android

RESOLVED FIXED in mozilla11

Status

()

defect
RESOLVED FIXED
8 years ago
8 years ago

People

(Reporter: glandium, Assigned: glandium)

Tracking

Trunk
mozilla11
ARM
Android
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [mobilestartupshrink][ts][inbound])

Attachments

(11 attachments, 17 obsolete attachments)

3.52 KB, patch
benjamin
: review+
Details | Diff | Splinter Review
9.21 KB, patch
mrbkap
: review+
Details | Diff | Splinter Review
7.83 KB, patch
benjamin
: review+
Details | Diff | Splinter Review
1.26 KB, patch
mrbkap
: review+
Details | Diff | Splinter Review
16.17 KB, patch
glandium
: review+
Details | Diff | Splinter Review
3.75 KB, patch
glandium
: review+
Details | Diff | Splinter Review
2.63 KB, patch
taras.mozilla
: review+
Details | Diff | Splinter Review
4.59 KB, patch
glandium
: review+
Details | Diff | Splinter Review
16.38 KB, patch
mwu
: review+
Details | Diff | Splinter Review
58.49 KB, patch
glandium
: review+
Details | Diff | Splinter Review
14.34 KB, patch
Details | Diff | Splinter Review
Assignee

Description

8 years ago
Around 200ms are wasted on a nexus s at startup because Android does something with the apk zip entries and takes time for this because we have so many entries in the apk. Getting the number of entries down should allow us to avoid wasting these 200ms. We can do that by keeping chrome, components and resources compressed under a single (uncompressed) file in the apk.
Assignee

Comment 1

8 years ago
(In reply to Mike Hommey [:glandium] from comment #0)
> Around 200ms are wasted on a nexus s at startup.

Note that it's even more on multi-language builds.
Assignee

Comment 2

8 years ago
It still needs to be modified such that nsJAR::GetZipArchive is not required.
Assignee

Comment 4

8 years ago
I'm not sure we should keep nsIZipReaders around, here. The impact should definitely be checked.
Assignee

Comment 5

8 years ago
This part definitely needs more work, to have something proper for the chrome registry and js component loader. The other part that's broken with this patch is that ManifestProcessingContext, both in the component manager and the chrome registry, are using nsIFiles for the base path to resolve from, and thus fail with nested omni.jar. They probably need to store either an nsIFile or an nsIZipReader depending on cases.

(I just wish we had fake nsIFiles that would allow to use Zips transparently, like directories, but so many things rely on nsIFiles to actually be nsILocalFiles...)
Assignee

Updated

8 years ago
Blocks: 697356

Updated

8 years ago
Attachment #569042 - Flags: review?(benjamin) → review+
Assignee

Updated

8 years ago
Attachment #569041 - Attachment is obsolete: true
Assignee

Updated

8 years ago
Attachment #569043 - Attachment is obsolete: true
Assignee

Updated

8 years ago
Attachment #569044 - Attachment is obsolete: true
Assignee

Comment 6

8 years ago
Attachment #572834 - Flags: review?(mwu)
Assignee

Comment 8

8 years ago
I'm not settled on this one. I may switch to using nsZipHandles instead of nsZipArchives in FileLocation. The class also needs some documentation, and the API may require some tweaking.
Assignee

Updated

8 years ago
Attachment #572836 - Flags: feedback?(mwu)
Assignee

Comment 11

8 years ago
(In reply to Mike Hommey [:glandium] from comment #10)
> Created attachment 572840 [details] [diff] [review] [diff] [details] [review]
> part 6 - Don't separate file and JAR modules in component manager

Note that the use of net_GetURLSpecFromActualFile goes away in a subsequent patch.
Assignee

Comment 14

8 years ago
This uses the API from part 4.
Assignee

Updated

8 years ago
Attachment #572843 - Flags: feedback?(benjamin)
Assignee

Comment 15

8 years ago
Attachment #572844 - Flags: review?(benjamin)
Assignee

Updated

8 years ago
Attachment #572840 - Attachment is obsolete: true
Attachment #572840 - Flags: review?(benjamin)
Comment on attachment 572838 [details] [diff] [review]
part 5 - Allow nsZipCursor to always use the given buffer

Wouldn't it be cleaner to add a nsZipCursor::Copy?

Updated

8 years ago
Attachment #572834 - Flags: review?(mwu) → review+
Comment on attachment 572838 [details] [diff] [review]
part 5 - Allow nsZipCursor to always use the given buffer

Taras seems to have more of an opinion on this one than I do.
Attachment #572838 - Flags: review?(mwu) → review?(tglek)
Assignee

Comment 19

8 years ago
Try result on this patch queue:
https://tbpl.mozilla.org/?noignore=1&tree=Try&rev=cc189b22e21d

Summary:
- 2 nsZipArchives are leaked
- Bootstrap extensions test seems to go in an infinite loop on windows.
Assignee

Comment 20

8 years ago
(In reply to Mike Hommey [:glandium] from comment #19)
> - 2 nsZipArchives are leaked

Found the culprit. Will update the relevant patch later.

Another issue: jars are likely to be mapped twice in memory with the current patch queue. Bug 504217 should help there.
Assignee

Updated

8 years ago
Attachment #572842 - Attachment is obsolete: true
Attachment #572842 - Flags: review?(mrbkap)
Assignee

Comment 22

8 years ago
Same patch, which removes a "mZip.forget()" that was a leftover from the original use of nsAutoPtr.
Assignee

Updated

8 years ago
Attachment #572834 - Attachment is obsolete: true
Assignee

Comment 23

8 years ago
Comment on attachment 573172 [details] [diff] [review]
part 1 - Add Refcounting on nsZipArchives

Carrying previous r+, considering the changes.
Attachment #573172 - Flags: review+
Assignee

Comment 24

8 years ago
(In reply to Mike Hommey [:glandium] from comment #19)
> - Bootstrap extensions test seems to go in an infinite loop on windows.

The test goes on waiting for nothing because it doesn't handle failure gracefully. The failure itself is due to the xpi file still being open (iow, a nsZipArchive still existing) when it is removed (NS_ERROR_FILE_ACCESS_DENIED on nsIFile.remove) in test_bootstrap.js line 113, function manuallyUninstall. Obviously that kind of thing only fail on windows.
Comment on attachment 572838 [details] [diff] [review]
part 5 - Allow nsZipCursor to always use the given buffer

We decided to add a nsZipCursor::Copy
Attachment #572838 - Flags: review?(tglek) → review-
Comment on attachment 572836 [details] [diff] [review]
part 4 - Add a FileLocation class to either store a pair (zip, path) or a file, and do some operations on these

So, I think FileLocation makes a lot of sense for the component manager code but not as much for nsZipHandle. That being said, this is one of the easier ways for you to generate uris and it doesn't mess with nsZipHandle too much. I suspect that having nsZipHandle directly generate the information you want is a cleaner way to go here, and might be useful for some other code along the way. Up to you though - if moving things to nsZipHandle is too complicated for now, that's fine.
Attachment #572836 - Flags: feedback?(mwu)
Attachment #573171 - Flags: review?(mrbkap) → review+
Attachment #572841 - Flags: review?(mrbkap) → review+

Updated

8 years ago
Attachment #572835 - Flags: review?(benjamin) → review+

Updated

8 years ago
Attachment #572843 - Flags: feedback?(benjamin) → feedback+

Updated

8 years ago
Attachment #572844 - Flags: review?(benjamin) → review+
Comment on attachment 572848 [details] [diff] [review]
part 6 - Don't separate file and JAR modules in component manager

I'm not sure that hashing on the URI representation is correct or will work correctly: we've had issues in the past with short and long path names not comparing equal correctly. What is the actual behavior of net_GetURLSpecFromActualFile WRT case differences or things like that?
Assignee

Comment 28

8 years ago
(In reply to Benjamin Smedberg  [:bsmedberg] from comment #27)
> Comment on attachment 572848 [details] [diff] [review] [diff] [details] [review]
> part 6 - Don't separate file and JAR modules in component manager
> 
> I'm not sure that hashing on the URI representation is correct or will work
> correctly: we've had issues in the past with short and long path names not
> comparing equal correctly. What is the actual behavior of
> net_GetURLSpecFromActualFile WRT case differences or things like that?

net_GetURLSpecFromActualFile doesn't ensure a canonical path before turning the path into an url :(
The same concern would also apply on part 7. I guess we could canonicalize paths before putting them in the uri string used for keys.
Assignee

Comment 29

8 years ago
Note that the nsIHashable part of nsLocalFile* doesn't handle case sensitivity on Mac, AFAICS, so we may already have the case sensitivity problem.
Comment on attachment 573172 [details] [diff] [review]
part 1 - Add Refcounting on nsZipArchives

>--- a/startupcache/StartupCache.cpp
>+++ b/startupcache/StartupCache.cpp
>+nsresult
>+GetBufferFromZipArchive(nsZipArchive *zip, bool doCRC, const char* id,
>+                        char** outbuf, PRUint32* length)
>+{
>+  if (zip) {
>+    nsZipItemPtr<char> zipItem(zip, id, doCRC);
>+    if (zipItem) {
>+      *outbuf = zipItem.Forget();
>+      *length = zipItem.Length();
>+      return NS_OK;
>+    }
>+  }
>+  return NS_ERROR_NOT_AVAILABLE;
>+}

Early returns, please.

>+  nsresult rv;
>+  rv = GetBufferFromZipArchive(mArchive, true, id, outbuf, length);

nsresult rv = GetBufferFromZipArchive(mArchive, true, id, outbuf, length);

>--- a/xpcom/build/Omnijar.cpp
>+++ b/xpcom/build/Omnijar.cpp
> Omnijar::InitOne(nsIFile *aPath, Type aType)
>+    nsRefPtr<nsZipArchive> zipReader = new nsZipArchive();
>+    if (!zipReader || NS_FAILED(zipReader->OpenArchive(file))) {

Remove the '!zipReader' check, 'new' can't fail.

>--- a/xpcom/build/Omnijar.h
>+++ b/xpcom/build/Omnijar.h
>+static inline already_AddRefed<nsZipArchive>
> GetReader(Type aType)
> {
>     NS_ABORT_IF_FALSE(IsInitialized(), "Omnijar not initialized");
>+    NS_IF_ADDREF(sReader[aType]);
>     return sReader[aType];

nsRefPtr<nsZipArchive> reader = sReader[aType];
return reader.forget();
Comment on attachment 572836 [details] [diff] [review]
part 4 - Add a FileLocation class to either store a pair (zip, path) or a file, and do some operations on these

>--- /dev/null
>+++ b/xpcom/build/FileLocation.cpp
>+namespace mozilla {
>+
>+nsresult FileLocation::GetData(char *buf, PRUint32 len)
>+{
>+    nsZipCursor cursor(item, mBaseZip, (PRUint8 *)buf, len, true);

static_cast

>+  mozilla::AutoFDClose fd;

No need for the 'mozilla::'.

>--- /dev/null
>+++ b/xpcom/build/FileLocation.h
>+  void Init(nsILocalFile *file)
>+  {
>+    mBaseFile = file;
>+    mPath = (char *)NULL;

mPath.Truncate()?

>+/* This needs to be included at the end because of the cross-references */
>+#include "nsZipArchive.h"

Uh, really? Why does this need to be included at all?
Comment on attachment 572844 [details] [diff] [review]
part 10 - Allow nested omni.jar and build one for Android

>--- a/xpcom/build/Omnijar.cpp
>+++ b/xpcom/build/Omnijar.cpp
>@@ -105,16 +106,24 @@ Omnijar::InitOne(nsIFile *aPath, Type aT
>+    nsRefPtr<nsZipHandle> handle;
>+    if (NS_SUCCEEDED(nsZipHandle::Init(zipReader, "omni.jar", getter_AddRefs(handle)))) {
>+        zipReader = new nsZipArchive();
>+        if (!zipReader...

And here
Comment on attachment 572848 [details] [diff] [review]
part 6 - Don't separate file and JAR modules in component manager

Ugh. I guess that since we're using manifests now it's unlikely that we'll have mixed-case issues; I guess we'll find out later!
Attachment #572848 - Flags: review?(benjamin) → review+
Assignee

Comment 34

8 years ago
Updated with Ms2ger comments.
Assignee

Comment 35

8 years ago
Comment on attachment 579619 [details] [diff] [review]
part 1 - Add Refcounting on nsZipArchives.

Carrying over r+
Attachment #579619 - Flags: review+
Assignee

Updated

8 years ago
Attachment #573172 - Attachment is obsolete: true
Assignee

Updated

8 years ago
Attachment #572835 - Attachment is obsolete: true
Assignee

Comment 37

8 years ago
Comment on attachment 579620 [details] [diff] [review]
part 3 - Use mozilla::AutoFDClose instead of ad hoc AutoCloseFD classes

Carrying over r+
Attachment #579620 - Flags: review+
Assignee

Updated

8 years ago
Attachment #572838 - Attachment is obsolete: true
Assignee

Updated

8 years ago
Attachment #572836 - Attachment is obsolete: true
Assignee

Updated

8 years ago
Attachment #579624 - Flags: review?(benjamin)
Assignee

Comment 40

8 years ago
Attachment #579626 - Flags: review?(benjamin)
Assignee

Updated

8 years ago
Attachment #572843 - Attachment is obsolete: true
Assignee

Comment 41

8 years ago
Basically the same as previous version, but uses OMNIJAR_NAME instead of hardcoding omni.jar.
Assignee

Updated

8 years ago
Attachment #572844 - Attachment is obsolete: true
Assignee

Comment 42

8 years ago
Comment on attachment 579627 [details] [diff] [review]
part 10 - Allow nested omni.jar and build one for Android

Carrying over r+
Attachment #579627 - Flags: review+
Assignee

Comment 43

8 years ago
In the case we end up having problems with short vs. long path names making uris not stable enough to be hash keys for the component manager and the JS component loader, here is a patch that makes the relevant hash tables use something more canonical. Attaching so that it can be found later if needed, since I already did the work (instead of being lost on my harddrive)
Comment on attachment 579621 [details] [diff] [review]
part 4 - Allow nsZipCursor to always use the given buffer

thanks
Attachment #579621 - Flags: review?(mozilla) → review+
Comment on attachment 579624 [details] [diff] [review]
part 5 - Add a FileLocation class to either store a pair (zip, path) or a file, and do some operations on these

Some of the single-line ifs in FileLocation::FileLocation aren't braced and I find this hard to read: please brace them all.

Please comment namespace-closing brace in FileLocation.cpp, e.g.:

} // namespace mozilla

The opening brace of "class FileLocation {" and FileLocation::Data should be on the next line

It's unusual to return an nsCString (in FileLocation::GetURIString and FileLocation::GetPath). Normally methods that return a string use: 

void ReturnAString(nsACString& result);

Since we have copy-on-write strings, perhaps this isn't a problem, but I'd prefer to be consistent with other parts of our code.

r=me with those changes
Attachment #579624 - Flags: review?(benjamin) → review+

Updated

8 years ago
Attachment #579626 - Flags: review?(benjamin) → review+
Assignee

Updated

8 years ago
Attachment #579624 - Attachment is obsolete: true
Attachment #579624 - Flags: review?(mwu)
Assignee

Comment 47

8 years ago
Refreshed with the new modified GetURIString signature from part 5.
Assignee

Updated

8 years ago
Attachment #579626 - Attachment is obsolete: true
Assignee

Comment 48

8 years ago
Removed a bit that is meant to be in the "extra" patch.
Attachment #579974 - Flags: review?(mwu)
Assignee

Updated

8 years ago
Attachment #579972 - Attachment is obsolete: true
Attachment #579972 - Flags: review?(mwu)
Assignee

Comment 49

8 years ago
Removed an unused variable
Assignee

Updated

8 years ago
Attachment #579973 - Attachment is obsolete: true
Assignee

Updated

8 years ago
Attachment #579628 - Attachment is obsolete: true
Assignee

Comment 51

8 years ago
Comment on attachment 579975 [details] [diff] [review]
part 9 - Use FileLocations in the component manager.

Carrying over r+.
Attachment #579975 - Flags: review+
Comment on attachment 579974 [details] [diff] [review]
part 5 - Add a FileLocation class to either store a pair (zip, path) or a file, and do some operations on these.

Review of attachment 579974 [details] [diff] [review]:
-----------------------------------------------------------------

Looks ok overall though I haven't really studied how some of the callers use some of this API. Comments are just nits.

::: xpcom/build/FileLocation.cpp
@@ +106,5 @@
> +  if (IsZip() && mBaseZip) {
> +    nsRefPtr<nsZipHandle> handler = mBaseZip->GetFD();
> +    if (handler)
> +      return handler->mFile.GetBaseFile();
> +  } else {

I think this else could be removed after moving return NULL to the "if (IsZip() && mBaseZip)" block, or maybe invert this if/else. Having the return NULL only apply to one of the if/else blocks is a bit weird.

@@ +139,5 @@
> +
> +nsresult
> +FileLocation::GetData(Data &data)
> +{
> +  if (IsZip()) {

if (!IsZip())
  return mBaseFile->OpenNSPRFileDesc(PR_RDONLY, 0444, &data.mFd);

::: xpcom/build/FileLocation.h
@@ +132,5 @@
> +
> +  /**
> +   * Returns whether the "base file" (see GetBaseFile) is an archive
> +   */
> +  bool IsZip() const {

Curly brace style inconsistent here and in the next two functions.
Attachment #579974 - Flags: review?(mwu) → review+
Note: this seems to have regressed being able to access APK resources from Java using jar: URIs

Updated

8 years ago
Depends on: 712433
You need to log in before you can comment on or make changes to this bug.