Open Bug 1199754 Opened 11 years ago Updated 2 years ago

Let's stop packing Windows installer with UPX

Categories

(Firefox :: Installer, enhancement, P3)

Product:
Firefox
Component:
Installer
Version:
40 Branch
Platform:
x86
Windows
Type:
enhancement

Tracking

()

People

(Reporter: michalpurzynski1, Unassigned)

References

Details

To my surprise, the Windows installer for many versions are packed with UPX. This might be done in an effort to make the binary smaller but the real savings are 0.12% (or less) and make Firefox installer look like a malware. Full story below UPX is used to make Windows binaries smaller. It self-extracts and required no external tools, it's transparent, so you just run the binary, it unpacks itself and runs the main application. To avoid being detected by the IDS/IPS (Intrusion Detection System / Intrusion Prevention System) most of malware is packed with UPX these days. It is so common it actually became like a hallmark of malware and a reason for many defense systems to become suspicious when they see a UPX packed EXE. In case of IDS, they will raise an alert, in case of IPS they will block the installer download - both will flag it as suspicious and likely malware. Furthermore, we don't really save bandwidth anymore. Below are test results on various Firefox versions. We have 38 ESR, 40 stable, 41 beta (32 and 64 bit) and the stub installer wget "http://ftp.mozilla.org/pub/mozilla.org//firefox/releases/38.2.0esr/win32/ro/Firefox Setup 38.2.0esr.exe" wget http://ftp.mozilla.org/pub/firefox/releases/latest/win32/en-US/Firefox%20Setup%2040.0.3.exe wget http://ftp.mozilla.org/pub/firefox/releases/latest/win32/en-US/Firefox%20Setup%20Stub%2040.0.3.exe wget http://ftp.mozilla.org/pub/firefox/releases/latest-beta/win32/en-US/Firefox%20Setup%2041.0b5.exe wget http://ftp.mozilla.org/pub/firefox/releases/latest-beta/win32/en-US/Firefox%20Setup%20Stub%2041.0b5.exe wget http://ftp.mozilla.org/pub/firefox/releases/latest-beta/win64/en-US/Firefox%20Setup%2041.0b5.exe wget http://ftp.mozilla.org/pub/firefox/releases/latest-esr/win32/en-US/Firefox%20Setup%2038.2.1esr.exe Michals-MacBook-Air :: ~/tmp/upx » ls -lh total 409480 -rw-r--r-- 1 mpurzynski staff 38M Aug 6 14:42 Firefox Setup 38.2.0esr.exe -rw-r--r-- 1 mpurzynski staff 38M Aug 26 07:56 Firefox Setup 38.2.1esr.exe -rw-r--r-- 1 mpurzynski staff 40M Aug 26 05:59 Firefox Setup 40.0.3.exe -rw-r--r-- 1 mpurzynski staff 41M Aug 27 19:19 Firefox Setup 41.0b5.exe -rw-r--r-- 1 mpurzynski staff 43M Aug 27 18:58 Firefox Setup 41.0b5.exe.1 -rw-r--r-- 1 mpurzynski staff 237K Aug 26 05:59 Firefox Setup Stub 40.0.3.exe -rw-r--r-- 1 mpurzynski staff 237K Aug 27 19:19 Firefox Setup Stub 41.0b5.exe Michals-MacBook-Air :: ~/tmp/upx » openssl sha1 * SHA1(Firefox Setup 38.2.0esr.exe)= 38e599de0181ed09e3c9f5a9c4ded64af9fc57d5 SHA1(Firefox Setup 38.2.1esr.exe)= 8fd1a45294ce80105ab89c31feb3f51c6b2e3c93 SHA1(Firefox Setup 40.0.3.exe)= 8a3e7ad06ebb09008908194b9ec3fa84ec6d637b SHA1(Firefox Setup 41.0b5.exe)= b8ad5b4852489ed8939caefcf60a9be4f92b8043 SHA1(Firefox Setup 41.0b5.exe.1)= ee75106af6b7edf5d62e9dbcd750d809f7551a78 SHA1(Firefox Setup Stub 40.0.3.exe)= b8ed299ac17360992b51db2f0fa80fa1450a8b66 SHA1(Firefox Setup Stub 41.0b5.exe)= ed6c7ec39f594694d49f39205fbda2d4b8b342c2 SHA1(i)= 8b7eff1ec54ca3b8f489cd9d4b7a35b6a0c6bdea Michals-MacBook-Air :: ~/tmp/upx » upx -d * Ultimate Packer for eXecutables Copyright (C) 1996 - 2013 UPX 3.91 Markus Oberhumer, Laszlo Molnar & John Reiser Sep 30th 2013 File size Ratio Format Name -------------------- ------ ----------- ----------- 40080032 <- 40028832 99.87% win32/pe Firefox Setup 38.2.0esr.exe 39732488 <- 39681288 99.87% win32/pe Firefox Setup 38.2.1esr.exe 42063912 <- 42012712 99.88% win32/pe Firefox Setup 40.0.3.exe 42800232 <- 42749032 99.88% win32/pe Firefox Setup 41.0b5.exe 44727848 <- 44676648 99.89% win32/pe Firefox Setup 41.0b5.exe.1 293952 <- 242752 82.58% win32/pe Firefox Setup Stub 40.0.3.exe 294160 <- 242960 82.59% win32/pe Firefox Setup Stub 41.0b5.exe upx: i: NotPackedException: not packed by UPX -------------------- ------ ----------- ----------- 209992624 <- 209634224 99.83% [ 7 files ] Unpacked 7 files. The average savings on the full installer is 50KB and that's like 0.12% of the original size of the binary. Stub installer is about the same, also 50KB. That is not much bandwidth saving for our users, at the significant cost for the enterprise environment where such binaries might be flagged as malicious and stopped from downloading.
We've been doing this a long time (2001?), see bug 110919 and bug 247115. Maybe it's not necessary now for space savings, but if it's been causing problems I would expect to see bugs or SUMO threads about it. Our installs are signed with a known-good signature, don't most security systems whitelist those?
Flags: needinfo?(robert.strong.bugs)
The times have changed, and I would say, we might re-think that decision, especially that the real space savings are 50KB on tens of MB big binary and a high chance of tripping a false positive. Some security systems are better than others, I haven't seen many (if any) checking for a signature, unfortunately. Most vendors like to take a shortcuts, and while I agree it's not us who should be fixing their things, I would say in this case the potential for problem avoidance is bigger than the not-existing savings (again, 0.12%, not 12% ;).
I would like to see evidence first that this actually occurs in the wild before making changes since I would expect bug reports for this if it were. Also, 50KB for the stub installer is a significant portion of the stub installer size.
Flags: needinfo?(robert.strong.bugs)
Emerging Threats rule set, the most widely used for Suricata and second most popular for Snort (which itself is the most popular IPS out there), enabled by default alert http $EXTERNAL_NET any -> $HOME_NET any (msg:"ET MALWARE UPX encrypted file download possible malware"; flow:established,from_server; content:"|0d 0a 0d 0a|MZ"; byte_jump:4,58,relative,little; content:"PE|00 00|"; distance:-64; within:4; content:"|00|code|00|"; content:"|00 C0|text|00|"; reference:url,doc.emergingthreats.net/2001047; classtype:misc-activity; sid:2001047; rev:13;) alert http $REALLY_EXTERNAL_NET any -> $HOME_NET any (msg:"ET TROJAN UPX compressed file download possible malware"; flow:established,from_server; content:"|0d 0a 0d 0a|MZ"; byte_jump:4,58,relative,little; content:"PE|00 00|"; distance:-64; within:4; content:"UPX0"; content:"UPX1"; content:"UPX!"; reference:url,doc.emergingthreats.net/2001046; classtype:misc-activity; sid:2001046; rev:13;)
16434 <-> POLICY Ultimate Packer for Executables/UPX v0.51-v0.61 packed file download attempt (policy.rules, Low) 16435 <-> POLICY Ultimate Packer for Executables/UPX v0.62-v1.22 packed file download attempt (policy.rules, Low) 16436 <-> POLICY Ultimate Packer for Executables/UPX v2.90,v2.93-3.00 packed file download attempt (policy.rules, Low) from VRT (Snort / Sourcefire now Cisco)
and have you verified that a binary signed with a valid certificate doesn't bypass the rule?
We pack the binary signed executable? If so, the IPS is only going to look for the packing, see the above signatures looking for UPX in the file header.
I have yet to see an IDS with a rule language sophisticated enough to bypass its own rules based on external factors, like signature verification. I haven't seen many (any?) IDS that do signature verification, to be honest. Some host based solutions might do it, but they will flag UPX as well.
OK, it still would be a "good thing" to verify that it actually affects us since in all of the years we have been doing this I don't recall there being even one bug report.
I could reverse it and ask for a prof why we should keep it :) No, 50KB savings, even in bandwidth hungry users are not an excuse, since what follows is either a tens of MBs binary or, in the case of stab installer, ehm, wait, the same thing but packed in a different way. Read: if one can download 50MB he can download 50KB on top of it. If 50KB is a problem, then 50MB is not even an option. Having seem more IDS installations I could name I know how these operate, in the best case someone's either leaving the rule in place and ignoring it, even if it denies Firefox downloads (i.e. who cares - Mozilla browser is not "enterprise" enough) or just adds a rule to ignore us.
Correction of the first sentence, one should not type when eating ;) No, 50KB savings, even in bandwidth hungry countries are not an excuse, since what follows is either a tens of MB binary or, in the case of stub installer, ehm, wait, the same thing but packed in a different way.
Simply put, change for the sake of change without evidence that this improves things. I'm not saying we won't change it but I have also seen numerous cases where a change didn't actually improve things. Instead of debating this there should be evidence that it is in fact an issue in at least some cases.
(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #13) > Simply put, change for the sake of change without evidence that this > improves things. I'm not saying we won't change it but I have also seen > numerous cases where a change didn't actually improve things. Instead of > debating this there should be evidence that it is in fact an issue in at > least some cases. The evidence has been shown in the first comment, go ahead and read it, I'll wait.
I did and yet we haven't had a single bug report about this (which I have asked for) since Firefox 1.0 was released.
How about 1199754
Since comment 0 was written, the difference seems to have increased to about 90 KB: Ultimate Packer for eXecutables Copyright (C) 1996 - 2017 UPX 3.94w Markus Oberhumer, Laszlo Molnar & John Reiser May 12th 2017 File size Ratio Format Name -------------------- ------ ----------- ----------- 332280 <- 243192 73.19% win32/pe Firefox Installer.en-US.exe I'm not sure if this is due more to using a newer version of UPX or to changes in the code itself. As the installer engineering owner, I am unwilling to make things even that much worse in the absence of some solid evidence that the benefits would exceed the damage. In this case, that would mean showing that our usage of UPX is causing real problems for a considerable number of users or preventing a considerable number of potential users from being able to use Firefox. That has not been shown in this bug; all we have here are potential ways that problems could arise and no proof of actual harm in practice. And in fact there is opposing evidence in that we've received no other complaints about this that I am aware of or can find. I searched here in Bugzilla, our support site support.mozilla.org, archives of relevant mailing lists, Twitter, and I even went into the MozillaZine forums and searched those. All I found aside from this bug was bug 487825, which was an AV false positive on a specific signature, and we would not reduce our exposure to false positives of that kind by removing UPX. Other posts around UPX in those places are actually about expanding its use. I am planning on closing this bug soon unless new information arises.
FWIW: I'd wager that we are getting a pass from the various AV engines due to their whitelisting common/popular executables. Using UPX is a common indicator in the same way that malformed headers trigger spam engines: https://webmasters.stackexchange.com/a/81680 It may be more informative to survey the other installation stubs for common practices? http://www.webdevelopersnotes.com/browsers-for-windows FWIW: Chrome's setup does not pack: https://virustotal.com/en/file/dde873395078c614c09c19bb40172c91e1897716b8da3b9cd743883db5647483/analysis/1502122307/ Opera uses UPX: https://virustotal.com/en/file/a635b9bd229b6e84762c47db3388a0a12144c553ed9f9f3821c80e4dd05d1ce9/analysis/1502122477/ and is also flagged as malware by a couple AV engines (most notably cylance). Brave does not use a packer: https://virustotal.com/en/file/e4975a47946e43f0a678ee84af74952b42be1d16cea678229966a3a861130854/analysis/ I can't find a standalone Edge installer and didn't wade through the entire list linked above, but it would be interesting to submit non-whitelisted samples with and without UPX to see how AV behaves. Agreed that the end-decision lies with installer engineering.
Priority: -- → P3
Component: Build Config → Installer
Product: Core → Firefox
Cliqz browser, which based on Firefox, faced with same problem: https://www.virustotal.com/en/file/388752fac345b2fb4a9f65b03fe08902db9cdbc2099b1564d0c01c9f43b3a4c7/analysis/1513247385/ So, +1 for removing UPX from build process.
https://bugzilla.mozilla.org/show_bug.cgi?id=1246387 shows that the download size isn't a problem anymore. When are we removing the UPX?
Bug 1246387 is about using TLS, it has absolutely nothing to do with file size.
TLS connections have certain overhead and there's a certificate chain to download + OCSP checks, proving that UPX packing saving us several KBs isn't a concern.
Type: defect → enhancement
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.