Note: There are a few cases of duplicates in user autocompletion which are being worked on.

Can not extract omni.jar by using 7-zip, And the anti-virus (avast!5) fails in the scan of omni.jar.

RESOLVED INCOMPLETE

Status

Plugins Graveyard
Avast AV
RESOLVED INCOMPLETE
7 years ago
a year ago

People

(Reporter: Alice0775 White, Unassigned)

Tracking

({dev-doc-complete})

Details

(Reporter)

Description

7 years ago
Build Identifier: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b6pre) Gecko/20100910 Firefox/4.0b6pre ID:20100910041829

I am not sure which component I should file a bug for.

I can not extract omni.jar by using 7-zip.
It says "can not extract omni.jar by using 7-zip".
I tried in Windows7 and Windows Xp, and 7-zip 4.6.5 and 9.1.6.

What can extract files in omni.jar is useful for the making of extension.

Reproducible: Always, However, it depends on a build.

Steps to Reproduce:
1. Download zip file from ( http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/ , http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32/ or http://hourly-archive.localgho.st/hourly-archive2/mozilla-central-win32/ )
2. Extract zip file.
3. Try to extract omni.jar by using 7-zip

Actual Results:
 extract fails

Expected Results:
 extract successfully


[nightly build]
success: http://hg.mozilla.org/mozilla-central/rev/8e0fce7d5b49
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b6pre) Gecko/20100909 Firefox/4.0b6pre ID:20100909041137

fail   : http://hg.mozilla.org/mozilla-central/rev/79d0beec27b5
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b6pre) Gecko/20100910 Firefox/4.0b6pre ID:20100910041829


[hourly build]
success: http://hg.mozilla.org/mozilla-central/rev/29b9772d0c3a
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b6pre) Gecko/20100910 Firefox/4.0b6pre ID:20100910141259

fail   : http://hg.mozilla.org/mozilla-central/rev/8e2027140923
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b6pre) Gecko/20100910 Firefox/4.0b6pre ID:20100910171333

fail   : http://hg.mozilla.org/mozilla-central/rev/fa6c83c1a10e
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b6pre) Gecko/20100910 Firefox/4.0b6pre ID:20100910174327

fail   : http://hg.mozilla.org/mozilla-central/rev/da5a386a0346
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b6pre) Gecko/20100910 Firefox/4.0b6pre ID:20100910185754

fail   : http://hg.mozilla.org/mozilla-central/rev/7e651ffb7a63
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b6pre) Gecko/20100910 Firefox/4.0b6pre ID:20100910192350

fail   : http://hg.mozilla.org/mozilla-central/rev/2779c55431a4
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b6pre) Gecko/20100910 Firefox/4.0b6pre ID:20100910221019

success: http://hg.mozilla.org/mozilla-central/rev/3d42767d964d
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b6pre) Gecko/20100910 Firefox/4.0b6pre ID:20100910234453
(Reporter)

Comment 1

7 years ago
And The anti-virus (avast!5) fails in the scan of omni.jar.
Summary: Can not extract omni.jar by using 7-zip → Can not extract omni.jar by using 7-zip, And the anti-virus (avast!5) fails in the scan of omni.jar.

Comment 2

7 years ago
Sorry guys, your zip programs are broken. More competent zip programs(eg Windows 7 builtin zip support) work fine with our "optimized" zip layout introduced in bug 559961.
Status: UNCONFIRMED → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → WORKSFORME
(Reporter)

Comment 3

7 years ago
(In reply to comment #2)
> Sorry guys, your zip programs are broken. More competent zip programs(eg
> Windows 7 builtin zip support) work fine with our "optimized" zip layout
> introduced in bug 559961.

What about stand alone program working on XP?

Comment 4

7 years ago
(In reply to comment #3)
> (In reply to comment #2)
> > Sorry guys, your zip programs are broken. More competent zip programs(eg
> > Windows 7 builtin zip support) work fine with our "optimized" zip layout
> > introduced in bug 559961.
> 
> What about stand alone program working on XP?

WinRAR (though it will print out some warnings).
Keywords: dev-doc-needed
John/Christian: at the very least we'll need outreach to the antivirus vendors on this.

Alice: do we know if it's just Avast! that fails?
(Reporter)

Comment 6

7 years ago
(In reply to comment #4)
> WinRAR (though it will print out some warnings).

Thanks it works.

(In reply to comment #5)
> John/Christian: at the very least we'll need outreach to the antivirus vendors
> on this.
> 
> Alice: do we know if it's just Avast! that fails?

Avast! fails in scan files which are in omni.jar.

1.right click omni.jar
2.select Scan omni.jar

You can see in result dialog, "Tested files:1"
If Avast! succeed in scan files,  "Tested files:1052" something like that.

Comment 7

7 years ago
This also effects IZArc and a slew of other FREE zip utilities.  This will impact theme and extension developers as they will need to access omni.jar, and 7Zip and IZArc are the two most commonly recommended ZIP utilities that will generate a JAR/XPI file that Firefox can open.  I find it odd that people developing free open source addons for a free open source application should be forced to use pay software to do their job.
The zip program we ship with mozilla-build (infozip) can handle this, no?  That's free software ...

Comment 9

7 years ago
I tried InfoZip for Windows (WiZ) and it threw up several errors and only extracted chrome.manifest... but maybe I wasn't using it correctly as I've never tried it before.

Comment 10

7 years ago
(In reply to comment #7)
> This also effects IZArc and a slew of other FREE zip utilities.  This will
> impact theme and extension developers as they will need to access omni.jar, and
> 7Zip and IZArc are the two most commonly recommended ZIP utilities that will
> generate a JAR/XPI file that Firefox can open.  I find it odd that people
> developing free open source addons for a free open source application should be
> forced to use pay software to do their job.

Nobody is forcing you to use payware. File bugs with relevant open source utilities, they are too strict in their interpretations of the zip format.

Comment 11

7 years ago
(In reply to comment #5)
> John/Christian: at the very least we'll need outreach to the antivirus vendors
> on this.
> 
> Alice: do we know if it's just Avast! that fails?

I'll try to contact tomorrow...
(Reporter)

Comment 12

7 years ago
FYI
Microsoft Security Essentials  version 1.0.1963.0 also fail to scan inside of omni.jar.

Latest Nightly http://hg.mozilla.org/mozilla-central/rev/10a6b2c105ae
MSE reported that Scanned file count is  1. 

Build http://hg.mozilla.org/mozilla-central/rev/8e0fce7d5b49
MSE reported that Scanned file count is  1539.
The result of the failed scan is what, precisely? That MSE marks us as infected?
(Reporter)

Comment 14

7 years ago
MSE can not scan inside file of omni.jar
Yes, but is that bad? It's a bug in Avast, but it's not clear why it matters to us.
(Reporter)

Comment 16

7 years ago
(In reply to comment #15)
> Yes, but is that bad? It's a bug in Avast, but it's not clear why it matters to
> us.

SEE http://blog.mozilla.com/tglek/2010/09/14/firefox-4-jar-jar-jar/
This blog reports "broken antivirus (it’s a security risk to be overly picky) fail."

So, If Omni.jar infected, antivirus may not be able to detect .

Comment 17

7 years ago
the problem is not Avast but 7-zip that cannot open jar files. And Avast is using 7-zip. WinRar opens them with no problem. But Avast has now an issue that doesn't come from the core software.

Comment 18

7 years ago
okay, according to an avast dev the integration of 7-zip is for nothing in this failure, it's not used to scan anything else then *.7z files
I think we need to reinvestigate this, based on the effects we're seeing.

Can someone please confirm the *effects* of the various antivirus programs not being able to scan into omni.jar? Do they mark Firefox as dangerous?
Status: RESOLVED → REOPENED
Ever confirmed: true
Resolution: WORKSFORME → ---
(In reply to comment #19)
> I think we need to reinvestigate this, based on the effects we're seeing.
> 
> Can someone please confirm the *effects* of the various antivirus programs not
> being able to scan into omni.jar? Do they mark Firefox as dangerous?

Microsoft Security Essentials doesn't flag omni.jar/Firefox as dangerous, but it also can't scan files within omni.jar, so if some naughty program is able to sneak a file into the archive, it can't be scanned directly.

Comment 21

7 years ago
no Avast 5 doesn't flag firefox at all

Comment 22

7 years ago
Flagging this as invalid. beltzner reopened this worried about AV software flagging omnijars as a virus which is not the case. 
We can revisit this bug if that turns out to be a problem.

Updated

7 years ago
Status: REOPENED → RESOLVED
Last Resolved: 7 years ago7 years ago
Resolution: --- → INVALID

Updated

7 years ago
Duplicate of this bug: 605524
Duplicate of this bug: 630037
The docs were updated for this a while ago.
Keywords: dev-doc-needed → dev-doc-complete
JFYI - Symantec Corporate 9.0 also fails to scan omni.jar

Comment 27

6 years ago
We're now tracking such bugs. This doesn't mean it's something we can fix, merely something we hope to be able to point vendors to so they can investigate. This is an automated message.
Status: RESOLVED → UNCONFIRMED
Component: Build Config → Avast AV
Ever confirmed: false
Product: Core → Plugins
QA Contact: build-config → avast-antivirus
Resolution: INVALID → ---
Version: Trunk → unspecified

Comment 28

6 years ago
Cygwin unzip (UnZip 6.00) seems to unzip it ok, but gives the following error:
warning [omni.jar]:  4035823 extra bytes at beginning or within zipfile
  (attempting to process anyway)
error [omni.jar]:  reported length of central directory is
  -4035823 bytes too long (Atari STZip zipfile?  J.H.Holm ZIPSPLIT 1.1
  zipfile?).  Compensating...

Comment 29

6 years ago
Have you seen https://bugzilla.mozilla.org/show_bug.cgi?id=605524 ?

There is no attempt to provide a valid header.

Comment 30

6 years ago
Weird, thanks @Ulli. Ok I will post it there also as an FYI. I know I would have like to have seen someone else had seen the same error.

Comment 31

6 years ago
I wonder why the problem is not simply fixed ?
Why is it so difficult to give omni.jar a usual signature, to let most zip extraction programs access the contained files without problem ?

Previous .jar files always contained a usual signature.
One of the reasons given for using this format was that tools to access it were widely available.
Why the change in attitude ?

Can anyone give a logical reason for this ?

(In the meantime, many, especially those not in a Microsoft environment, are shut out of contributing to Mozilla.)

Comment 32

6 years ago
Nobody is being shut out. I develop on Linux. I agree that omni.jar a non-standard mozilla zip that only works with specific zip tools.

This wont be fixed because the standard way of storing data in a zip does not match how we access it. I would rather save 0.5seconds for a few million users than make sure that every zip utility on the planet can cope with it.
(In reply to comment #32)
> Nobody is being shut out. I develop on Linux. I agree that omni.jar a
> non-standard mozilla zip that only works with specific zip tools.
> 
> This wont be fixed because the standard way of storing data in a zip does
> not match how we access it. I would rather save 0.5seconds for a few million
> users than make sure that every zip utility on the planet can cope with it.

I don't get it. The problem is only with the omni.jar files of the *localized* builds ("repacks") - omni.jar files from the en-US builds can be used together with 7-zip without any problems. So if there is a "non-standard mozilla way" of doing the zips, why isn't it being used for the en-US builds too?

Comment 34

6 years ago
Are you saying that there is a Linux utility available that is able to read omni.jar ?
I have the latest stable version of unzip installed on my system (produced by infozip) and it doesn't work.

If you do have a free tool that works on Linux (and other platforms), I'd like to know where to find it.
And I think that omni.jar should have a different name so it is evident that it is not a zip format (even though it is similar).  And make the availability of the tools more evident.
The problem is not necessarily a custom format (as long as the tools are available), but mascarading as a zip format.

By the way, if I repack it as a standard zip file, will Mozilla still be able to use it ?  (I'm interested in comparing the performance.)

<adside>
0.5 seconds doesn't make much difference to an individual user.
My changes to previous .jar files, which I optimally compressed, made considerably more difference than that -- with standard zip compression format.
Although it is apparent that Mozilla now loads much faster than the unchanged previous versions.
</aside>

Comment 35

6 years ago
unzip should work fine. non-optimized zips work fine

Comment 36

6 years ago
> I don't get it. The problem is only with the omni.jar files of the
> *localized* builds ("repacks") - omni.jar files from the en-US builds can be
> used together with 7-zip without any problems.

It's actually the opposite, only en-us has had the optimized layout, until 6.0. Which incidentally suggests that the perf gains are somewhere between non-existent and unnoticeable, otherwise this would have been fixed in 4.0.1 or at least 5.0.

@taras - deoptimizing omni.jar from 6.0b1/win32/en-US with optimizejars.py (both central and beta) does not yield a file that 7-zip can open, which suggests it's still not a strictly adhering zip file.  This is new in 6.0.  Please look into this.
(In reply to comment #36)
> > I don't get it. The problem is only with the omni.jar files of the
> > *localized* builds ("repacks") - omni.jar files from the en-US builds can be
> > used together with 7-zip without any problems.
> 
> It's actually the opposite, only en-us has had the optimized layout, until
> 6.0. Which incidentally suggests that the perf gains are somewhere between
> non-existent and unnoticeable, otherwise this would have been fixed in 4.0.1
> or at least 5.0.

Heh, so why was it broken on the localized builds already for a long, long time (see e.g. Bug 630037, which was duped to this one)? And why can I still unpack omni.jar from Fx 5.0 en-US with 7-zip?
But ok. If 6.0 will break it completely, then yeah, nothing we can do here...

Comment 38

6 years ago
Bug 671746 - deoptimized 6.0b1/win32/en-US omni.jar can not be opened by 7-zip, is likely not a strictly valid zip archive
Duplicate of this bug: 671746
Duplicate of this bug: 680654
Will bug 684347 help with this? (I'm not at all familiar with this part of the codebase, so perhaps not)

Comment 42

6 years ago
I doubt it.
Mozilla has created a new file format for omni.jar, based on zip but different.
(They discuss the differences, which are significant in my view.)
Most standard zip programs can't reliably read this new format.
And their code for this new format will not reliably read a standard zip file.
Yet they insist on calling it a zip format.
I wish they would simply admit that it is a new format, and publish tools to manipulate this new format.  Not rocket science.

Note that an optimally compressed zip format provides most of the loading speed improvements that they proclaim.  They have (a least up to the omni format) distributed non-optimally formatted .jar files to reduce download size, which significantly increases loading time.
Closing old bugs in the Plugins component. We aren't going to track issues in 3rd-party plugins in the Mozilla bug tracker. In addition, support for NPAPI plugins will be removed at the end of this year; for more details see the post at https://blog.mozilla.org/futurereleases/2015/10/08/npapi-plugins-in-firefox/

If there is a serious bug in Firefox, it needs to be filed in the "Core" product, "Plug-Ins" component.
Status: UNCONFIRMED → RESOLVED
Last Resolved: 7 years agoa year ago
Resolution: --- → INCOMPLETE
(Assignee)

Updated

a year ago
Product: Plugins → Plugins Graveyard
You need to log in before you can comment on or make changes to this bug.