Write a script that copies the latest version of each add-on to somewhere

RESOLVED FIXED in 5.5

Status

defect
P2
normal
RESOLVED FIXED
10 years ago
3 years ago

People

(Reporter: clouserw, Assigned: davedash)

Tracking

unspecified
Dependency tree / graph

Details

Attachments

(1 attachment, 5 obsolete attachments)

Reporter

Description

10 years ago
From bug 358591:

We need a script that copies the files of the latest version of all add-ons with viewsource=1 to a specified directory.  When the script is re-run, it should add/remove files as necessary and replace old files with new if another version is uploaded.
Reporter

Updated

10 years ago
Blocks: amo-mxr
So, from the other bug, we'll just go with a private repository that includes all add-ons regardless of the viewsource setting.
Assignee: nobody → dd
Posted patch v1 (obsolete) — Splinter Review
Does this do what you need?

It takes 2 mins to copy, even if this is copying from NFS to somewhere else it shouldn't take too much longer, so instead of detecting what's changed, just rm -rf latest_version/* and rerun the script.
Attachment #408947 - Flags: review?(clouserw)
Reporter

Comment 3

10 years ago
Comment on attachment 408947 [details] [diff] [review]
v1

This is copying the earliest version, not the latest.  Example, addon id 10 it copied adblock-0.5.2.055-fx+fl+mz+ns.xpi when 0.5.3.043 was the latest.
Attachment #408947 - Flags: review?(clouserw) → review-
Posted patch v2 (obsolete) — Splinter Review
Man, you're so picky ;)
Attachment #408947 - Attachment is obsolete: true
Attachment #409135 - Flags: review?(clouserw)
Reporter

Updated

10 years ago
Attachment #409135 - Flags: review?(clouserw) → review-
Reporter

Comment 5

10 years ago
Comment on attachment 409135 [details] [diff] [review]
v2

Traceback (most recent call last):
  File "latest_addon_extractor.py", line 42, in <module>
    """)
  File "build/bdist.linux-i686/egg/MySQLdb/cursors.py", line 173, in execute
  File "build/bdist.linux-i686/egg/MySQLdb/connections.py", line 36, in defaulterrorhandler
_mysql_exceptions.OperationalError: (1052, "Column 'addon_id' in field list is ambiguous")
Posted patch v3 (obsolete) — Splinter Review
Sorry, didn't save the file before I diff'd
Attachment #409135 - Attachment is obsolete: true
Attachment #409140 - Flags: review?(clouserw)
Reporter

Comment 7

10 years ago
Comment on attachment 409140 [details] [diff] [review]
v3

clouserw@khan:~/public_html/remora/bin$ rm -rf ~/temp/addons/*
clouserw@khan:~/public_html/remora/bin$ ls ~/temp/addons/
clouserw@khan:~/public_html/remora/bin$ python26 latest_addon_extractor.py /data/remora-files/ ~/temp/addons/
Traceback (most recent call last):
  File "latest_addon_extractor.py", line 52, in <module>
    os.mkdir(dir_out)
OSError: [Errno 17] File exists: '/home/clouserw/temp/addons/29'
clouserw@khan:~/public_html/remora/bin$ ls ~/temp/addons/
10  11  12  15  16  17  18  19  23  25  26  27  28  29
Attachment #409140 - Flags: review?(clouserw) → review-
Posted patch v4 (obsolete) — Splinter Review
Checks to see if a directory exists first.
Attachment #409140 - Attachment is obsolete: true
Attachment #409154 - Flags: review?(clouserw)
Reporter

Updated

10 years ago
Attachment #409154 - Flags: review?(clouserw) → review+
Reporter

Comment 9

10 years ago
Comment on attachment 409154 [details] [diff] [review]
v4

works as advertised.  Copied 1.6G of files.
woot finally ;)

r54578 = 8d717786bb596ada59ae71a0cb1a570ee2ac5141 (trunk)
Status: NEW → RESOLVED
Last Resolved: 10 years ago
Resolution: --- → FIXED
Is there any way that QA can verify this, or is it mostly backend (and already proven)?  Thanks.
This is now up and running at http://mxr.mozilla.org/addons/source/ - however, it doesn't appear to be unpacking the source, just copying the binary files over.  Makes it kind of hard to index it. :)
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Posted patch v5 (obsolete) — Splinter Review
If either of you can review this, I'll commit it.  Dave M, I imagine you can just patch it and see if it works for you to verify that it's good.

-d
Attachment #409154 - Attachment is obsolete: true
Attachment #416132 - Flags: review?(clouserw)
Attachment #416132 - Flags: review?(justdave)
Comment on attachment 416132 [details] [diff] [review]
v5

[root@dm-webtools04 bin]# python26 -S latest_addon_extractor.py /data/addons /data/mxr-data/amo/amo
Traceback (most recent call last):
  File "latest_addon_extractor.py", line 61, in <module>
    except zipfile.BadZipFile:
AttributeError: 'module' object has no attribute 'BadZipFile'
Reporter

Comment 15

10 years ago
Comment on attachment 416132 [details] [diff] [review]
v5

r- for comment 14
Attachment #416132 - Flags: review?(justdave)
Attachment #416132 - Flags: review?(clouserw)
Attachment #416132 - Flags: review-
Reporter

Updated

10 years ago
Priority: -- → P2
Target Milestone: 5.3 → 5.5
Posted patch v6Splinter Review
Attachment #416132 - Attachment is obsolete: true
Attachment #416804 - Flags: review?(justdave)
Attachment #416804 - Flags: review?(clouserw)
Dave M, feel free to patch this again and let me know.
Markign fixed, the patches are applied to this file, justdave, ping me/reopen if there's issues.

r57837 | ddash@mozilla.com | 2009-12-14 11:44:39 -0800 (Mon, 14 Dec 2009) | 2 li

[bug 522480] Some zip fixes
Status: REOPENED → RESOLVED
Last Resolved: 10 years ago10 years ago
Resolution: --- → FIXED

Updated

10 years ago
Status: RESOLVED → VERIFIED
Sumod, how were you able to verify this?  Also, when you verify a bug as fixed, please leave a comment indicating the testing steps (when necessary).  Thanks!

Comment 20

10 years ago
There was no real direct way for me to test this. So I read through all the comments in the bug. I read the script and checked that it was doing what was expected.
It still crashes.

Got this 9 min 11 seconds in after starting it going:

Traceback (most recent call last):
  File "latest_addon_extractor.py", line 58, in <module>
    xpi.extractall(dir_out)
  File "/usr/lib64/python2.6/zipfile.py", line 935, in extractall
    self.extract(zipinfo, path, pwd)
  File "/usr/lib64/python2.6/zipfile.py", line 923, in extract
    return self._extract_member(member, path, pwd)
  File "/usr/lib64/python2.6/zipfile.py", line 947, in _extract_member
    if member.filename[0] == '/':
IndexError: string index out of range
Status: VERIFIED → REOPENED
Resolution: FIXED → ---
It also needs to recursively unpack any .jar files it finds (it doesn't appear to be doing that currently).
recursively, or just one level deep?
Status: REOPENED → ASSIGNED
svn up and try the next version, let me know asap if there are errors... I'd like to have the whole issue resolved.
Status: ASSIGNED → RESOLVED
Last Resolved: 10 years ago10 years ago
Resolution: --- → FIXED
Comment on attachment 416804 [details] [diff] [review]
v6

canceling unneeded reviews on resolved bug
Attachment #416804 - Flags: review?(justdave)
Attachment #416804 - Flags: review?(clouserw)
Product: addons.mozilla.org → addons.mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.