Closed Bug 695843 Opened 13 years ago Closed 12 years ago

Nested omni.jar for Android

Categories

(Core Graveyard :: Widget: Android, defect)

ARM
Android
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED
mozilla11

People

(Reporter: glandium, Assigned: glandium)

References

Details

(Whiteboard: [mobilestartupshrink][ts][inbound])

Attachments

(11 files, 17 obsolete files)

3.52 KB, patch
benjamin
: review+
Details | Diff | Splinter Review
9.21 KB, patch
mrbkap
: review+
Details | Diff | Splinter Review
7.83 KB, patch
benjamin
: review+
Details | Diff | Splinter Review
1.26 KB, patch
mrbkap
: review+
Details | Diff | Splinter Review
16.17 KB, patch
glandium
: review+
Details | Diff | Splinter Review
3.75 KB, patch
glandium
: review+
Details | Diff | Splinter Review
2.63 KB, patch
taras.mozilla
: review+
Details | Diff | Splinter Review
4.59 KB, patch
glandium
: review+
Details | Diff | Splinter Review
16.38 KB, patch
mwu
: review+
Details | Diff | Splinter Review
58.49 KB, patch
glandium
: review+
Details | Diff | Splinter Review
14.34 KB, patch
Details | Diff | Splinter Review
Around 200ms are wasted on a nexus s at startup because Android does something with the apk zip entries and takes time for this because we have so many entries in the apk. Getting the number of entries down should allow us to avoid wasting these 200ms. We can do that by keeping chrome, components and resources compressed under a single (uncompressed) file in the apk.
(In reply to Mike Hommey [:glandium] from comment #0)
> Around 200ms are wasted on a nexus s at startup.

Note that it's even more on multi-language builds.
It still needs to be modified such that nsJAR::GetZipArchive is not required.
I'm not sure we should keep nsIZipReaders around, here. The impact should definitely be checked.
This part definitely needs more work, to have something proper for the chrome registry and js component loader. The other part that's broken with this patch is that ManifestProcessingContext, both in the component manager and the chrome registry, are using nsIFiles for the base path to resolve from, and thus fail with nested omni.jar. They probably need to store either an nsIFile or an nsIZipReader depending on cases.

(I just wish we had fake nsIFiles that would allow to use Zips transparently, like directories, but so many things rely on nsIFiles to actually be nsILocalFiles...)
Blocks: 697356
Attachment #569042 - Flags: review?(benjamin) → review+
Attachment #569041 - Attachment is obsolete: true
Attachment #569043 - Attachment is obsolete: true
Attachment #569044 - Attachment is obsolete: true
Attachment #572834 - Flags: review?(mwu)
I'm not settled on this one. I may switch to using nsZipHandles instead of nsZipArchives in FileLocation. The class also needs some documentation, and the API may require some tweaking.
Attachment #572836 - Flags: feedback?(mwu)
(In reply to Mike Hommey [:glandium] from comment #10)
> Created attachment 572840 [details] [diff] [review] [diff] [details] [review]
> part 6 - Don't separate file and JAR modules in component manager

Note that the use of net_GetURLSpecFromActualFile goes away in a subsequent patch.
This uses the API from part 4.
Attachment #572843 - Flags: feedback?(benjamin)
Attachment #572844 - Flags: review?(benjamin)
Attachment #572840 - Attachment is obsolete: true
Attachment #572840 - Flags: review?(benjamin)
Comment on attachment 572838 [details] [diff] [review]
part 5 - Allow nsZipCursor to always use the given buffer

Wouldn't it be cleaner to add a nsZipCursor::Copy?
Attachment #572834 - Flags: review?(mwu) → review+
Comment on attachment 572838 [details] [diff] [review]
part 5 - Allow nsZipCursor to always use the given buffer

Taras seems to have more of an opinion on this one than I do.
Attachment #572838 - Flags: review?(mwu) → review?(tglek)
Try result on this patch queue:
https://tbpl.mozilla.org/?noignore=1&tree=Try&rev=cc189b22e21d

Summary:
- 2 nsZipArchives are leaked
- Bootstrap extensions test seems to go in an infinite loop on windows.
(In reply to Mike Hommey [:glandium] from comment #19)
> - 2 nsZipArchives are leaked

Found the culprit. Will update the relevant patch later.

Another issue: jars are likely to be mapped twice in memory with the current patch queue. Bug 504217 should help there.
Attachment #572842 - Attachment is obsolete: true
Attachment #572842 - Flags: review?(mrbkap)
Same patch, which removes a "mZip.forget()" that was a leftover from the original use of nsAutoPtr.
Attachment #572834 - Attachment is obsolete: true
Comment on attachment 573172 [details] [diff] [review]
part 1 - Add Refcounting on nsZipArchives

Carrying previous r+, considering the changes.
Attachment #573172 - Flags: review+
(In reply to Mike Hommey [:glandium] from comment #19)
> - Bootstrap extensions test seems to go in an infinite loop on windows.

The test goes on waiting for nothing because it doesn't handle failure gracefully. The failure itself is due to the xpi file still being open (iow, a nsZipArchive still existing) when it is removed (NS_ERROR_FILE_ACCESS_DENIED on nsIFile.remove) in test_bootstrap.js line 113, function manuallyUninstall. Obviously that kind of thing only fail on windows.
Comment on attachment 572838 [details] [diff] [review]
part 5 - Allow nsZipCursor to always use the given buffer

We decided to add a nsZipCursor::Copy
Attachment #572838 - Flags: review?(tglek) → review-
Comment on attachment 572836 [details] [diff] [review]
part 4 - Add a FileLocation class to either store a pair (zip, path) or a file, and do some operations on these

So, I think FileLocation makes a lot of sense for the component manager code but not as much for nsZipHandle. That being said, this is one of the easier ways for you to generate uris and it doesn't mess with nsZipHandle too much. I suspect that having nsZipHandle directly generate the information you want is a cleaner way to go here, and might be useful for some other code along the way. Up to you though - if moving things to nsZipHandle is too complicated for now, that's fine.
Attachment #572836 - Flags: feedback?(mwu)
Attachment #573171 - Flags: review?(mrbkap) → review+
Attachment #572841 - Flags: review?(mrbkap) → review+
Attachment #572835 - Flags: review?(benjamin) → review+
Attachment #572843 - Flags: feedback?(benjamin) → feedback+
Attachment #572844 - Flags: review?(benjamin) → review+
Comment on attachment 572848 [details] [diff] [review]
part 6 - Don't separate file and JAR modules in component manager

I'm not sure that hashing on the URI representation is correct or will work correctly: we've had issues in the past with short and long path names not comparing equal correctly. What is the actual behavior of net_GetURLSpecFromActualFile WRT case differences or things like that?
(In reply to Benjamin Smedberg  [:bsmedberg] from comment #27)
> Comment on attachment 572848 [details] [diff] [review] [diff] [details] [review]
> part 6 - Don't separate file and JAR modules in component manager
> 
> I'm not sure that hashing on the URI representation is correct or will work
> correctly: we've had issues in the past with short and long path names not
> comparing equal correctly. What is the actual behavior of
> net_GetURLSpecFromActualFile WRT case differences or things like that?

net_GetURLSpecFromActualFile doesn't ensure a canonical path before turning the path into an url :(
The same concern would also apply on part 7. I guess we could canonicalize paths before putting them in the uri string used for keys.
Note that the nsIHashable part of nsLocalFile* doesn't handle case sensitivity on Mac, AFAICS, so we may already have the case sensitivity problem.
Comment on attachment 573172 [details] [diff] [review]
part 1 - Add Refcounting on nsZipArchives

>--- a/startupcache/StartupCache.cpp
>+++ b/startupcache/StartupCache.cpp
>+nsresult
>+GetBufferFromZipArchive(nsZipArchive *zip, bool doCRC, const char* id,
>+                        char** outbuf, PRUint32* length)
>+{
>+  if (zip) {
>+    nsZipItemPtr<char> zipItem(zip, id, doCRC);
>+    if (zipItem) {
>+      *outbuf = zipItem.Forget();
>+      *length = zipItem.Length();
>+      return NS_OK;
>+    }
>+  }
>+  return NS_ERROR_NOT_AVAILABLE;
>+}

Early returns, please.

>+  nsresult rv;
>+  rv = GetBufferFromZipArchive(mArchive, true, id, outbuf, length);

nsresult rv = GetBufferFromZipArchive(mArchive, true, id, outbuf, length);

>--- a/xpcom/build/Omnijar.cpp
>+++ b/xpcom/build/Omnijar.cpp
> Omnijar::InitOne(nsIFile *aPath, Type aType)
>+    nsRefPtr<nsZipArchive> zipReader = new nsZipArchive();
>+    if (!zipReader || NS_FAILED(zipReader->OpenArchive(file))) {

Remove the '!zipReader' check, 'new' can't fail.

>--- a/xpcom/build/Omnijar.h
>+++ b/xpcom/build/Omnijar.h
>+static inline already_AddRefed<nsZipArchive>
> GetReader(Type aType)
> {
>     NS_ABORT_IF_FALSE(IsInitialized(), "Omnijar not initialized");
>+    NS_IF_ADDREF(sReader[aType]);
>     return sReader[aType];

nsRefPtr<nsZipArchive> reader = sReader[aType];
return reader.forget();
Comment on attachment 572836 [details] [diff] [review]
part 4 - Add a FileLocation class to either store a pair (zip, path) or a file, and do some operations on these

>--- /dev/null
>+++ b/xpcom/build/FileLocation.cpp
>+namespace mozilla {
>+
>+nsresult FileLocation::GetData(char *buf, PRUint32 len)
>+{
>+    nsZipCursor cursor(item, mBaseZip, (PRUint8 *)buf, len, true);

static_cast

>+  mozilla::AutoFDClose fd;

No need for the 'mozilla::'.

>--- /dev/null
>+++ b/xpcom/build/FileLocation.h
>+  void Init(nsILocalFile *file)
>+  {
>+    mBaseFile = file;
>+    mPath = (char *)NULL;

mPath.Truncate()?

>+/* This needs to be included at the end because of the cross-references */
>+#include "nsZipArchive.h"

Uh, really? Why does this need to be included at all?
Comment on attachment 572844 [details] [diff] [review]
part 10 - Allow nested omni.jar and build one for Android

>--- a/xpcom/build/Omnijar.cpp
>+++ b/xpcom/build/Omnijar.cpp
>@@ -105,16 +106,24 @@ Omnijar::InitOne(nsIFile *aPath, Type aT
>+    nsRefPtr<nsZipHandle> handle;
>+    if (NS_SUCCEEDED(nsZipHandle::Init(zipReader, "omni.jar", getter_AddRefs(handle)))) {
>+        zipReader = new nsZipArchive();
>+        if (!zipReader...

And here
Comment on attachment 572848 [details] [diff] [review]
part 6 - Don't separate file and JAR modules in component manager

Ugh. I guess that since we're using manifests now it's unlikely that we'll have mixed-case issues; I guess we'll find out later!
Attachment #572848 - Flags: review?(benjamin) → review+
Updated with Ms2ger comments.
Comment on attachment 579619 [details] [diff] [review]
part 1 - Add Refcounting on nsZipArchives.

Carrying over r+
Attachment #579619 - Flags: review+
Attachment #573172 - Attachment is obsolete: true
Attachment #572835 - Attachment is obsolete: true
Comment on attachment 579620 [details] [diff] [review]
part 3 - Use mozilla::AutoFDClose instead of ad hoc AutoCloseFD classes

Carrying over r+
Attachment #579620 - Flags: review+
Attachment #572838 - Attachment is obsolete: true
Attachment #572836 - Attachment is obsolete: true
Attachment #579624 - Flags: review?(benjamin)
Attachment #579626 - Flags: review?(benjamin)
Attachment #572843 - Attachment is obsolete: true
Basically the same as previous version, but uses OMNIJAR_NAME instead of hardcoding omni.jar.
Attachment #572844 - Attachment is obsolete: true
Comment on attachment 579627 [details] [diff] [review]
part 10 - Allow nested omni.jar and build one for Android

Carrying over r+
Attachment #579627 - Flags: review+
In the case we end up having problems with short vs. long path names making uris not stable enough to be hash keys for the component manager and the JS component loader, here is a patch that makes the relevant hash tables use something more canonical. Attaching so that it can be found later if needed, since I already did the work (instead of being lost on my harddrive)
Comment on attachment 579621 [details] [diff] [review]
part 4 - Allow nsZipCursor to always use the given buffer

thanks
Attachment #579621 - Flags: review?(mozilla) → review+
Comment on attachment 579624 [details] [diff] [review]
part 5 - Add a FileLocation class to either store a pair (zip, path) or a file, and do some operations on these

Some of the single-line ifs in FileLocation::FileLocation aren't braced and I find this hard to read: please brace them all.

Please comment namespace-closing brace in FileLocation.cpp, e.g.:

} // namespace mozilla

The opening brace of "class FileLocation {" and FileLocation::Data should be on the next line

It's unusual to return an nsCString (in FileLocation::GetURIString and FileLocation::GetPath). Normally methods that return a string use: 

void ReturnAString(nsACString& result);

Since we have copy-on-write strings, perhaps this isn't a problem, but I'd prefer to be consistent with other parts of our code.

r=me with those changes
Attachment #579624 - Flags: review?(benjamin) → review+
Attachment #579626 - Flags: review?(benjamin) → review+
Attachment #579624 - Attachment is obsolete: true
Attachment #579624 - Flags: review?(mwu)
Refreshed with the new modified GetURIString signature from part 5.
Attachment #579626 - Attachment is obsolete: true
Removed a bit that is meant to be in the "extra" patch.
Attachment #579974 - Flags: review?(mwu)
Attachment #579972 - Attachment is obsolete: true
Attachment #579972 - Flags: review?(mwu)
Removed an unused variable
Attachment #579973 - Attachment is obsolete: true
Attachment #579628 - Attachment is obsolete: true
Comment on attachment 579975 [details] [diff] [review]
part 9 - Use FileLocations in the component manager.

Carrying over r+.
Attachment #579975 - Flags: review+
Comment on attachment 579974 [details] [diff] [review]
part 5 - Add a FileLocation class to either store a pair (zip, path) or a file, and do some operations on these.

Review of attachment 579974 [details] [diff] [review]:
-----------------------------------------------------------------

Looks ok overall though I haven't really studied how some of the callers use some of this API. Comments are just nits.

::: xpcom/build/FileLocation.cpp
@@ +106,5 @@
> +  if (IsZip() && mBaseZip) {
> +    nsRefPtr<nsZipHandle> handler = mBaseZip->GetFD();
> +    if (handler)
> +      return handler->mFile.GetBaseFile();
> +  } else {

I think this else could be removed after moving return NULL to the "if (IsZip() && mBaseZip)" block, or maybe invert this if/else. Having the return NULL only apply to one of the if/else blocks is a bit weird.

@@ +139,5 @@
> +
> +nsresult
> +FileLocation::GetData(Data &data)
> +{
> +  if (IsZip()) {

if (!IsZip())
  return mBaseFile->OpenNSPRFileDesc(PR_RDONLY, 0444, &data.mFd);

::: xpcom/build/FileLocation.h
@@ +132,5 @@
> +
> +  /**
> +   * Returns whether the "base file" (see GetBaseFile) is an archive
> +   */
> +  bool IsZip() const {

Curly brace style inconsistent here and in the next two functions.
Attachment #579974 - Flags: review?(mwu) → review+
Note: this seems to have regressed being able to access APK resources from Java using jar: URIs
Depends on: 712433
Product: Core → Core Graveyard
You need to log in before you can comment on or make changes to this bug.