1221170 - Autophone - add prefix and keypattern arguments to S3.(ls|rm)

Status: RESOLVED FIXED

(Testing Graveyard :: Autophone, defect)

Description

[Bob Clary [:bc] (inactive)]

Boto S3 bucket.list queries are limited in terms of their ability to restrict the keys which are returned:

http://docs.pythonboto.org/en/latest/ref/s3.html#boto.s3.bucket.Bucket.list

It helps a bit to be able to restrict the key prefix. I've had this patch in a branch for a while and just want to get it checked in to get it in the tree.

maja, syd: If I remember correctly, you are using a version of this file in your projects. This changes the signatures of ls and rm to be:

-    def ls(self, keypattern='.*'):
+    def ls(self, keypattern='.*', prefix=''):

-    def rm(self, keys):
+    def rm(self, keys=[], keypattern=None, prefix=''):

I put the prefix argument at the end to help with backward compatibility.

For a comparison:

$ time python s3.py --config=developer-credentials.ini --ls=pub/mozilla.org/mobile/tinderbox-builds/mozilla-inbound-android-api-9/1446480008/.*

real	0m21.236s
user	0m1.259s
sys	0m0.066s

$ time python s3.py --config=developer-credentials.ini --ls=pub/mozilla.org/mobile/tinderbox-builds/mozilla-inbound-android-api-9/1446480008/.* --prefix=pub/mozilla.org/mobile/tinderbox-builds/mozilla-inbound-android-api-9/1446480008/

real	0m0.750s
user	0m0.098s
sys	0m0.038s

Comment 1

[Bob Clary [:bc] (inactive)]

Attachment: s3.patch

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

Comment on attachment 8682557 [details] [diff] [review]
s3.patch

Review of attachment 8682557 [details] [diff] [review]:
-----------------------------------------------------------------

good find!

Comment 3

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Thanks for the heads up, bc. I believe we'll be moving to a different solution for log uploads anyway as we migrate our jobs from pf-jenkins to the mozmill-ci Jenkins instance managed by :whimboo.

Comment 4

[Bob Clary [:bc] (inactive)]

https://github.com/mozilla/autophone/commit/de0c09a2cb08fcaf0a584114becd5a794374895d