522480 - Write a script that copies the latest version of each add-on to somewhere

Status: RESOLVED FIXED

(addons.mozilla.org Graveyard :: Maintenance Scripts, defect, P2)

Description

[Wil Clouser [:clouserw]]

From bug 358591:

We need a script that copies the files of the latest version of all add-ons with viewsource=1 to a specified directory.  When the script is re-run, it should add/remove files as necessary and replace old files with new if another version is uploaded.

Comment 1

[Justin Scott [:fligtar]]

So, from the other bug, we'll just go with a private repository that includes all add-ons regardless of the viewsource setting.

Comment 2

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

Attachment: v1

Does this do what you need?

It takes 2 mins to copy, even if this is copying from NFS to somewhere else it shouldn't take too much longer, so instead of detecting what's changed, just rm -rf latest_version/* and rerun the script.

Comment 3

[Wil Clouser [:clouserw]]

Comment on attachment 408947 [details] [diff] [review]
v1

This is copying the earliest version, not the latest.  Example, addon id 10 it copied adblock-0.5.2.055-fx+fl+mz+ns.xpi when 0.5.3.043 was the latest.

Comment 4

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

Attachment: v2

Man, you're so picky ;)

Comment 5

[Wil Clouser [:clouserw]]

Comment on attachment 409135 [details] [diff] [review]
v2

Traceback (most recent call last):
  File "latest_addon_extractor.py", line 42, in <module>
    """)
  File "build/bdist.linux-i686/egg/MySQLdb/cursors.py", line 173, in execute
  File "build/bdist.linux-i686/egg/MySQLdb/connections.py", line 36, in defaulterrorhandler
_mysql_exceptions.OperationalError: (1052, "Column 'addon_id' in field list is ambiguous")

Comment 6

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

Attachment: v3

Sorry, didn't save the file before I diff'd

Comment 7

[Wil Clouser [:clouserw]]

Comment on attachment 409140 [details] [diff] [review]
v3

clouserw@khan:~/public_html/remora/bin$ rm -rf ~/temp/addons/*
clouserw@khan:~/public_html/remora/bin$ ls ~/temp/addons/
clouserw@khan:~/public_html/remora/bin$ python26 latest_addon_extractor.py /data/remora-files/ ~/temp/addons/
Traceback (most recent call last):
  File "latest_addon_extractor.py", line 52, in <module>
    os.mkdir(dir_out)
OSError: [Errno 17] File exists: '/home/clouserw/temp/addons/29'
clouserw@khan:~/public_html/remora/bin$ ls ~/temp/addons/
10  11  12  15  16  17  18  19  23  25  26  27  28  29

Comment 8

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

Attachment: v4

Checks to see if a directory exists first.

Comment 9

[Wil Clouser [:clouserw]]

Comment on attachment 409154 [details] [diff] [review]
v4

works as advertised.  Copied 1.6G of files.

Comment 10

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

woot finally ;)

r54578 = 8d717786bb596ada59ae71a0cb1a570ee2ac5141 (trunk)

Comment 11

[Stephen Donner [:stephend] Not actively reading bugmail]

Is there any way that QA can verify this, or is it mostly backend (and already proven)?  Thanks.

Comment 12

[Dave Miller [:justdave]]

This is now up and running at http://mxr.mozilla.org/addons/source/ - however, it doesn't appear to be unpacking the source, just copying the binary files over.  Makes it kind of hard to index it. :)

Comment 13

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

Attachment: v5

If either of you can review this, I'll commit it.  Dave M, I imagine you can just patch it and see if it works for you to verify that it's good.

-d

Comment 14

[Dave Miller [:justdave]]

Comment on attachment 416132 [details] [diff] [review]
v5

[root@dm-webtools04 bin]# python26 -S latest_addon_extractor.py /data/addons /data/mxr-data/amo/amo
Traceback (most recent call last):
  File "latest_addon_extractor.py", line 61, in <module>
    except zipfile.BadZipFile:
AttributeError: 'module' object has no attribute 'BadZipFile'

Comment 15

[Wil Clouser [:clouserw]]

Comment on attachment 416132 [details] [diff] [review]
v5

r- for comment 14

Comment 16

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

Attachment: v6

Comment 17

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

Dave M, feel free to patch this again and let me know.

Comment 18

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

Markign fixed, the patches are applied to this file, justdave, ping me/reopen if there's issues.

r57837 | [e-mail] | 2009-12-14 11:44:39 -0800 (Mon, 14 Dec 2009) | 2 li

[bug 522480] Some zip fixes

Comment 19

[Stephen Donner [:stephend] Not actively reading bugmail]

Sumod, how were you able to verify this?  Also, when you verify a bug as fixed, please leave a comment indicating the testing steps (when necessary).  Thanks!

Comment 20

[sumod]

There was no real direct way for me to test this. So I read through all the comments in the bug. I read the script and checked that it was doing what was expected.

Comment 21

[Dave Miller [:justdave]]

It still crashes.

Got this 9 min 11 seconds in after starting it going:

Traceback (most recent call last):
  File "latest_addon_extractor.py", line 58, in <module>
    xpi.extractall(dir_out)
  File "/usr/lib64/python2.6/zipfile.py", line 935, in extractall
    self.extract(zipinfo, path, pwd)
  File "/usr/lib64/python2.6/zipfile.py", line 923, in extract
    return self._extract_member(member, path, pwd)
  File "/usr/lib64/python2.6/zipfile.py", line 947, in _extract_member
    if member.filename[0] == '/':
IndexError: string index out of range

Comment 22

[Dave Miller [:justdave]]

It also needs to recursively unpack any .jar files it finds (it doesn't appear to be doing that currently).

Comment 23

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

recursively, or just one level deep?

Comment 24

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

svn up and try the next version, let me know asap if there are errors... I'd like to have the whole issue resolved.

Comment 25

[Dave Miller [:justdave]]

Comment on attachment 416804 [details] [diff] [review]
v6

canceling unneeded reviews on resolved bug