1167271 - [tracker] Make sure zamboni uses DEFAULT_FILE_STORAGE backend for storage related operations

Status: RESOLVED WONTFIX

(Marketplace Graveyard :: General, defect, P2)

Description

[Jason Thomas [:jason]]

I've been testing zamboni + AWS S3 Storage backend using django-storages-redux and for the most part things work but I am seeing issues were we use shutil and open which makes direct file system operations without using DEFAULT_FILE_STORAGE. Can we make sure that storage operations make use of DEFAULT_FILE_STORAGE?

Here is a few places where I see this but I may have missed others:

https://github.com/mozilla/zamboni/blob/12430707e7b1810e91a777016b2583408711ea35/mkt/files/cron.py
https://github.com/mozilla/zamboni/blob/1c7232d47d78f8552f8aa8d8c32630494446ea3c/lib/video/tasks.py
https://github.com/mozilla/zamboni/blob/1c7232d47d78f8552f8aa8d8c32630494446ea3c/lib/crypto/packaged.py
https://github.com/mozilla/zamboni/blob/1c7232d47d78f8552f8aa8d8c32630494446ea3c/mkt/webapps/fakedata.py

Comment 1

[Wil Clouser [:clouserw]]

Jason says this is the only bug he found running the site on AWS.  If we can get this fixed that project can move forward.  If there isn't a slot now, let's talk with dbialer about priority and get back to Jason & co with a target date.

Comment 2

[Wil Clouser [:clouserw]]

Hi David,  This bug is blocking the Marketplace transition to AWS.  I don't think the bug would take too long, but it's not on any of our projects right now.  Can you help us prioritize it among the current projects?  Nancy Toland is PMing the transition project if you need more details or I'm happy to answer questions if I can.

Comment 3

[David Bialer [:dbialer]]

Whenever it makes sense for you that doesn't pull someone off a project.   Not a huge priority from a product point of view since we have some time, but it sounds small so we should get this done and won.    P3ish.

Comment 4

[ntoland]

Do we have a date yet on when this will get done?  I was tracking to 6/26, but I assume that did not happen.

Comment 5

[ntoland]

David - Do you have an ETA on this work?  - Nancy

Comment 6

[David Durst [:ddurst]]

I do not. We had to pre-empt it for a database blocker. Would likely not begin for two more weeks; let me know if this is burning.

Comment 7

[ntoland]

Hi David - this is blocking some work on the Marketplace Migration project.  We were expecting this done by end of June.  Jason can speak to when it starts to impact our cutover date.  - Nancy

Comment 8

[Mathieu Pillard [:mat]]

FWIW I did an initial exploration of this bug and found a lot more files to change by grepping for:
- os.path.exists
- os.listdir
- os.makedir
- open

Once false positives are removed (applying grep -v stor, grep -v storage helps reducing them) I estimate there are between 30 and 40 files left to change, and I bet there are more I didn't think about (for instance. some uses of os.path.* will be problematic. And I have not accounted for variations in importing style).

Comment 9

[Jason Thomas [:jason]]

(In reply to ntoland from comment #7)
> Hi David - this is blocking some work on the Marketplace Migration project. 
> We were expecting this done by end of June.  Jason can speak to when it
> starts to impact our cutover date.  - Nancy

This is a blocker for QA testing on AWS dev/stage env which is scheduled to begin on 8/6.

Comment 10

[Allen Short [:ashort]]

To clarify: is the plan for zamboni deployments to use _no_ local storage? The previous work assumed that local tempfile storage was available.

Comment 11

[Mathieu Pillard [:mat]]

AFAIK a local storage will be available ; but everything shared needs to go through the storage class that will use AWS S3.

The problem is we have a lot of code that expects / needs shared files. For instance, almost everything that uses settings.TMP_PATH (and basically all other *_PATH settings) needs to be shared between web servers and/or celery workers.