Closed Bug 595473 Opened 14 years ago Closed 8 years ago

Can not extract omni.jar by using 7-zip, And the anti-virus (avast!5) fails in the scan of omni.jar.

Categories

(Plugins Graveyard :: Avast AV, defect)

Product:
Plugins Graveyard
Component:
Avast AV
Platform:
x86
Windows 7
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED INCOMPLETE

People

(Reporter: alice0775, Unassigned)

References

Details

(Keywords: dev-doc-complete) 

Build Identifier: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b6pre) Gecko/20100910 Firefox/4.0b6pre ID:20100910041829

I am not sure which component I should file a bug for.

I can not extract omni.jar by using 7-zip.
It says "can not extract omni.jar by using 7-zip".
I tried in Windows7 and Windows Xp, and 7-zip 4.6.5 and 9.1.6.

What can extract files in omni.jar is useful for the making of extension.

Reproducible: Always, However, it depends on a build.

Steps to Reproduce:
1. Download zip file from ( http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/ , http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/mozilla-central-win32/ or http://hourly-archive.localgho.st/hourly-archive2/mozilla-central-win32/ )
2. Extract zip file.
3. Try to extract omni.jar by using 7-zip

Actual Results:
 extract fails

Expected Results:
 extract successfully


[nightly build]
success: http://hg.mozilla.org/mozilla-central/rev/8e0fce7d5b49
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b6pre) Gecko/20100909 Firefox/4.0b6pre ID:20100909041137

fail   : http://hg.mozilla.org/mozilla-central/rev/79d0beec27b5
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b6pre) Gecko/20100910 Firefox/4.0b6pre ID:20100910041829


[hourly build]
success: http://hg.mozilla.org/mozilla-central/rev/29b9772d0c3a
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b6pre) Gecko/20100910 Firefox/4.0b6pre ID:20100910141259

fail   : http://hg.mozilla.org/mozilla-central/rev/8e2027140923
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b6pre) Gecko/20100910 Firefox/4.0b6pre ID:20100910171333

fail   : http://hg.mozilla.org/mozilla-central/rev/fa6c83c1a10e
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b6pre) Gecko/20100910 Firefox/4.0b6pre ID:20100910174327

fail   : http://hg.mozilla.org/mozilla-central/rev/da5a386a0346
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b6pre) Gecko/20100910 Firefox/4.0b6pre ID:20100910185754

fail   : http://hg.mozilla.org/mozilla-central/rev/7e651ffb7a63
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b6pre) Gecko/20100910 Firefox/4.0b6pre ID:20100910192350

fail   : http://hg.mozilla.org/mozilla-central/rev/2779c55431a4
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b6pre) Gecko/20100910 Firefox/4.0b6pre ID:20100910221019

success: http://hg.mozilla.org/mozilla-central/rev/3d42767d964d
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:2.0b6pre) Gecko/20100910 Firefox/4.0b6pre ID:20100910234453
And The anti-virus (avast!5) fails in the scan of omni.jar.
Summary: Can not extract omni.jar by using 7-zip → Can not extract omni.jar by using 7-zip, And the anti-virus (avast!5) fails in the scan of omni.jar.
Sorry guys, your zip programs are broken. More competent zip programs(eg Windows 7 builtin zip support) work fine with our "optimized" zip layout introduced in bug 559961.
Status: UNCONFIRMED → RESOLVED
Closed: 14 years ago
Resolution: --- → WORKSFORME
(In reply to comment #2)
> Sorry guys, your zip programs are broken. More competent zip programs(eg
> Windows 7 builtin zip support) work fine with our "optimized" zip layout
> introduced in bug 559961.

What about stand alone program working on XP?
(In reply to comment #3)
> (In reply to comment #2)
> > Sorry guys, your zip programs are broken. More competent zip programs(eg
> > Windows 7 builtin zip support) work fine with our "optimized" zip layout
> > introduced in bug 559961.
> 
> What about stand alone program working on XP?

WinRAR (though it will print out some warnings).
Keywords: dev-doc-needed
John/Christian: at the very least we'll need outreach to the antivirus vendors on this.

Alice: do we know if it's just Avast! that fails?
(In reply to comment #4)
> WinRAR (though it will print out some warnings).

Thanks it works.

(In reply to comment #5)
> John/Christian: at the very least we'll need outreach to the antivirus vendors
> on this.
> 
> Alice: do we know if it's just Avast! that fails?

Avast! fails in scan files which are in omni.jar.

1.right click omni.jar
2.select Scan omni.jar

You can see in result dialog, "Tested files:1"
If Avast! succeed in scan files,  "Tested files:1052" something like that.
This also effects IZArc and a slew of other FREE zip utilities.  This will impact theme and extension developers as they will need to access omni.jar, and 7Zip and IZArc are the two most commonly recommended ZIP utilities that will generate a JAR/XPI file that Firefox can open.  I find it odd that people developing free open source addons for a free open source application should be forced to use pay software to do their job.
The zip program we ship with mozilla-build (infozip) can handle this, no?  That's free software ...
I tried InfoZip for Windows (WiZ) and it threw up several errors and only extracted chrome.manifest... but maybe I wasn't using it correctly as I've never tried it before.
(In reply to comment #7)
> This also effects IZArc and a slew of other FREE zip utilities.  This will
> impact theme and extension developers as they will need to access omni.jar, and
> 7Zip and IZArc are the two most commonly recommended ZIP utilities that will
> generate a JAR/XPI file that Firefox can open.  I find it odd that people
> developing free open source addons for a free open source application should be
> forced to use pay software to do their job.

Nobody is forcing you to use payware. File bugs with relevant open source utilities, they are too strict in their interpretations of the zip format.
(In reply to comment #5)
> John/Christian: at the very least we'll need outreach to the antivirus vendors
> on this.
> 
> Alice: do we know if it's just Avast! that fails?

I'll try to contact tomorrow...
FYI
Microsoft Security Essentials  version 1.0.1963.0 also fail to scan inside of omni.jar.

Latest Nightly http://hg.mozilla.org/mozilla-central/rev/10a6b2c105ae
MSE reported that Scanned file count is  1. 

Build http://hg.mozilla.org/mozilla-central/rev/8e0fce7d5b49
MSE reported that Scanned file count is  1539.
The result of the failed scan is what, precisely? That MSE marks us as infected?
MSE can not scan inside file of omni.jar
Yes, but is that bad? It's a bug in Avast, but it's not clear why it matters to us.
(In reply to comment #15)
> Yes, but is that bad? It's a bug in Avast, but it's not clear why it matters to
> us.

SEE http://blog.mozilla.com/tglek/2010/09/14/firefox-4-jar-jar-jar/
This blog reports "broken antivirus (it’s a security risk to be overly picky) fail."

So, If Omni.jar infected, antivirus may not be able to detect .
the problem is not Avast but 7-zip that cannot open jar files. And Avast is using 7-zip. WinRar opens them with no problem. But Avast has now an issue that doesn't come from the core software.
okay, according to an avast dev the integration of 7-zip is for nothing in this failure, it's not used to scan anything else then *.7z files
I think we need to reinvestigate this, based on the effects we're seeing.

Can someone please confirm the *effects* of the various antivirus programs not being able to scan into omni.jar? Do they mark Firefox as dangerous?
Status: RESOLVED → REOPENED
Ever confirmed: true
Resolution: WORKSFORME → ---
(In reply to comment #19)
> I think we need to reinvestigate this, based on the effects we're seeing.
> 
> Can someone please confirm the *effects* of the various antivirus programs not
> being able to scan into omni.jar? Do they mark Firefox as dangerous?

Microsoft Security Essentials doesn't flag omni.jar/Firefox as dangerous, but it also can't scan files within omni.jar, so if some naughty program is able to sneak a file into the archive, it can't be scanned directly.
no Avast 5 doesn't flag firefox at all
Flagging this as invalid. beltzner reopened this worried about AV software flagging omnijars as a virus which is not the case. 
We can revisit this bug if that turns out to be a problem.
Status: REOPENED → RESOLVED
Closed: 14 years ago → 14 years ago
Resolution: --- → INVALID
The docs were updated for this a while ago.
Keywords: dev-doc-needed → dev-doc-complete
JFYI - Symantec Corporate 9.0 also fails to scan omni.jar
We're now tracking such bugs. This doesn't mean it's something we can fix, merely something we hope to be able to point vendors to so they can investigate. This is an automated message.
Status: RESOLVED → UNCONFIRMED
Component: Build Config → Avast AV
Ever confirmed: false
Product: Core → Plugins
QA Contact: build-config → avast-antivirus
Resolution: INVALID → ---
Version: Trunk → unspecified
Cygwin unzip (UnZip 6.00) seems to unzip it ok, but gives the following error:
warning [omni.jar]:  4035823 extra bytes at beginning or within zipfile
  (attempting to process anyway)
error [omni.jar]:  reported length of central directory is
  -4035823 bytes too long (Atari STZip zipfile?  J.H.Holm ZIPSPLIT 1.1
  zipfile?).  Compensating...
Have you seen https://bugzilla.mozilla.org/show_bug.cgi?id=605524 ?

There is no attempt to provide a valid header.
Weird, thanks @Ulli. Ok I will post it there also as an FYI. I know I would have like to have seen someone else had seen the same error.
I wonder why the problem is not simply fixed ?
Why is it so difficult to give omni.jar a usual signature, to let most zip extraction programs access the contained files without problem ?

Previous .jar files always contained a usual signature.
One of the reasons given for using this format was that tools to access it were widely available.
Why the change in attitude ?

Can anyone give a logical reason for this ?

(In the meantime, many, especially those not in a Microsoft environment, are shut out of contributing to Mozilla.)
Nobody is being shut out. I develop on Linux. I agree that omni.jar a non-standard mozilla zip that only works with specific zip tools.

This wont be fixed because the standard way of storing data in a zip does not match how we access it. I would rather save 0.5seconds for a few million users than make sure that every zip utility on the planet can cope with it.
(In reply to comment #32)
> Nobody is being shut out. I develop on Linux. I agree that omni.jar a
> non-standard mozilla zip that only works with specific zip tools.
> 
> This wont be fixed because the standard way of storing data in a zip does
> not match how we access it. I would rather save 0.5seconds for a few million
> users than make sure that every zip utility on the planet can cope with it.

I don't get it. The problem is only with the omni.jar files of the *localized* builds ("repacks") - omni.jar files from the en-US builds can be used together with 7-zip without any problems. So if there is a "non-standard mozilla way" of doing the zips, why isn't it being used for the en-US builds too?
Are you saying that there is a Linux utility available that is able to read omni.jar ?
I have the latest stable version of unzip installed on my system (produced by infozip) and it doesn't work.

If you do have a free tool that works on Linux (and other platforms), I'd like to know where to find it.
And I think that omni.jar should have a different name so it is evident that it is not a zip format (even though it is similar).  And make the availability of the tools more evident.
The problem is not necessarily a custom format (as long as the tools are available), but mascarading as a zip format.

By the way, if I repack it as a standard zip file, will Mozilla still be able to use it ?  (I'm interested in comparing the performance.)

<adside>
0.5 seconds doesn't make much difference to an individual user.
My changes to previous .jar files, which I optimally compressed, made considerably more difference than that -- with standard zip compression format.
Although it is apparent that Mozilla now loads much faster than the unchanged previous versions.
</aside>
unzip should work fine. non-optimized zips work fine
> I don't get it. The problem is only with the omni.jar files of the
> *localized* builds ("repacks") - omni.jar files from the en-US builds can be
> used together with 7-zip without any problems.

It's actually the opposite, only en-us has had the optimized layout, until 6.0. Which incidentally suggests that the perf gains are somewhere between non-existent and unnoticeable, otherwise this would have been fixed in 4.0.1 or at least 5.0.

@taras - deoptimizing omni.jar from 6.0b1/win32/en-US with optimizejars.py (both central and beta) does not yield a file that 7-zip can open, which suggests it's still not a strictly adhering zip file.  This is new in 6.0.  Please look into this.
(In reply to comment #36)
> > I don't get it. The problem is only with the omni.jar files of the
> > *localized* builds ("repacks") - omni.jar files from the en-US builds can be
> > used together with 7-zip without any problems.
> 
> It's actually the opposite, only en-us has had the optimized layout, until
> 6.0. Which incidentally suggests that the perf gains are somewhere between
> non-existent and unnoticeable, otherwise this would have been fixed in 4.0.1
> or at least 5.0.

Heh, so why was it broken on the localized builds already for a long, long time (see e.g. Bug 630037, which was duped to this one)? And why can I still unpack omni.jar from Fx 5.0 en-US with 7-zip?
But ok. If 6.0 will break it completely, then yeah, nothing we can do here...
Bug 671746 - deoptimized 6.0b1/win32/en-US omni.jar can not be opened by 7-zip, is likely not a strictly valid zip archive
Will bug 684347 help with this? (I'm not at all familiar with this part of the codebase, so perhaps not)
I doubt it.
Mozilla has created a new file format for omni.jar, based on zip but different.
(They discuss the differences, which are significant in my view.)
Most standard zip programs can't reliably read this new format.
And their code for this new format will not reliably read a standard zip file.
Yet they insist on calling it a zip format.
I wish they would simply admit that it is a new format, and publish tools to manipulate this new format.  Not rocket science.

Note that an optimally compressed zip format provides most of the loading speed improvements that they proclaim.  They have (a least up to the omni format) distributed non-optimally formatted .jar files to reduce download size, which significantly increases loading time.
Closing old bugs in the Plugins component. We aren't going to track issues in 3rd-party plugins in the Mozilla bug tracker. In addition, support for NPAPI plugins will be removed at the end of this year; for more details see the post at https://blog.mozilla.org/futurereleases/2015/10/08/npapi-plugins-in-firefox/

If there is a serious bug in Firefox, it needs to be filed in the "Core" product, "Plug-Ins" component.
Status: UNCONFIRMED → RESOLVED
Closed: 14 years ago → 8 years ago
Resolution: --- → INCOMPLETE
Product: Plugins → Plugins Graveyard
You need to log in before you can comment on or make changes to this bug.