Last Comment Bug 673002 - Analyze binary add-ons from
: Analyze binary add-ons from
Product: Core
Classification: Components
Component: JavaScript Engine (show other bugs)
: unspecified
: All All
-- normal (vote)
: ---
Assigned To: Bill McCloskey (:billm)
: Jason Orendorff [:jorendorff]
Depends on:
  Show dependency treegraph
Reported: 2011-07-20 17:03 PDT by Bill McCloskey (:billm)
Modified: 2011-08-01 12:59 PDT (History)
7 users (show)
See Also:
Crash Signature:
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---

script to find binary addons (2.00 KB, text/x-python)
2011-07-20 17:07 PDT, Bill McCloskey (:billm)
no flags Details
output of Python script (7.00 KB, text/plain)
2011-07-20 17:09 PDT, Bill McCloskey (:billm)
no flags Details
list of symbols imported by each addon (341.02 KB, text/plain)
2011-07-20 17:15 PDT, Bill McCloskey (:billm)
no flags Details

Description User image Bill McCloskey (:billm) 2011-07-20 17:03:04 PDT
Dave Herman gave me a big tarball with all the addons. The high-level breakdown in as follows:

  Valid: 9129, non-zip: 56, corrupted: 20, binary: 51

This means that out of 9129 addons, 51 are binary (0.56%). I looked at all the Linux and Windows binaries, and it looks like none of them use JSAPI! To make sure I didn't make a mistake, I did the same analysis on the Kikin addon, which is not on AMO and which is known to use JSAPI. It does indeed import a bunch of JSAPI entry points.

I'll post more detailed results here. Also, I have a zip file with all the binaries extracted if anyone wants to examine them.
Comment 1 User image David Mandelin [:dmandelin] 2011-07-20 17:05:27 PDT
Good to hear. It sounds like for the most part when we change JSAPI we just need to put out the word ahead of time on m.d.t.js-engine and the AMO channels.
Comment 2 User image Bill McCloskey (:billm) 2011-07-20 17:07:59 PDT
Created attachment 547298 [details]
script to find binary addons

This is the script I used to process all the addons. The first argument is a path to the directory containing all the addons. The second argument is a fresh directory where it will put all the dlls it finds.

It turns out that a bunch of them are messed up, so most of the code deals with that. For some reason some of the .xpi files actually contain HTML. Also, some of the zip files are reported as corrupted by unzip. I skipped over all these, although the ones in the latter category could probably be recovered.
Comment 3 User image Bill McCloskey (:billm) 2011-07-20 17:09:57 PDT
Created attachment 547300 [details]
output of Python script

This is the output of the analysis script. It just shows which xpi files were skipped. Note that some of them have references in their chrome manifest to files that don't exist. Mostly this seems to happen when a Mac addon claims to have a .dll or a .so, but it really only has a .dylib. Or some variation of that.
Comment 4 User image Bill McCloskey (:billm) 2011-07-20 17:15:13 PDT
Created attachment 547302 [details]
list of symbols imported by each addon

This is the result of dumpbin.exe -IMPORTS foo.dll (thanks khuey). As can be seen, mozjs.dll does not appear.

I also ran a Windows version of strings on each dll and grepped for service names, of the form "@xyz;1". A lot of addons used some sort of xpcom proxying service, which seems to have something to do with threads. One addon (cooliris) wanted the XPCOM context stack service. I don't know enough to say if this matters. I can upload the list of anyone is interested. (I left it on my Windows partition by accident.)
Comment 5 User image Dave Herman [:dherman] 2011-07-20 18:11:45 PDT
9129 seems small to me -- I thought there were somewhere between 15k and 20k addons. Also, the XPI's that contained HTML might possibly be the result of a failed wget in Wil Clouser's script that collected all the addons from AMO.

CC'ing Wil and Matt Basta, who does the AMO validator. Wil & Matt: do you think this suggests that the tarball might not be good? Could you guys help Bill and me make sure we can get an accurate snapshot of the full set of AMO addons?

Comment 6 User image Matt Basta [:basta] 2011-07-20 18:17:06 PDT
I'll run this through our stats tool, Grizwald, and get back to you tomorrow.
Comment 7 User image Bill McCloskey (:billm) 2011-07-20 19:36:22 PDT
Another odd thing is that, out of ~12,000 directories in the tarball, only about 9,000 have xpi files in them.
Comment 8 User image Matt Basta [:basta] 2011-07-20 19:55:48 PDT
Themes are packaged in JAR files. That might account for the other chunk.
Comment 9 User image Bill McCloskey (:billm) 2011-07-25 14:23:34 PDT
The failed wget seems quite possible. How can we figure that out?
Comment 10 User image Wil Clouser [:clouserw] 2011-07-26 09:20:00 PDT
There is no wget.  I ran the script on a box with the add-ons on an NFS mount.  The script is super simple, it just makes a best guess on what the latest file is and copies it:
Comment 11 User image Bill McCloskey (:billm) 2011-08-01 12:59:35 PDT
I'm going to mark this as FIXED, I guess, since the work is done.

Note You need to log in before you can comment on or make changes to this bug.