1110801 - Change wiki attachments, etc. to use S3

Status: RESOLVED WONTFIX

(developer.mozilla.org Graveyard :: File attachments, defect)

Description

[Luke Crouch [:groovecoder]]

To move from NetApp to S3, we need to change Demo Studio and Wiki apps' code to upload demos and wiki attachments to MDN S3 buckets instead of the NetApp in SCL3.

Comment 1

[Rob Hudson [:robhudson]]

Updating title to reflect that we are removing demos.

Also, copying some notes from Luke here if they are useful.


Staying at the high level, here's a quick analysis of how the app accesses
each file:

* Wiki Attachments - uses plain django models.FileField, which uses
django's files.storage module
* Sitemaps - uses python native os module to access file system
* Humans - uses python native os module to access file system

So, at a high level, we're looking at:

1. Refactoring attachments models from FileSystemStorage to S3 storage.
2. Refactoring the sitemaps and humans code to use Amazon S3.

I can't speak to uploading the site tarball to S3 ...

/endquote

Comment 2

[Jannis Leidel [:jezdez]]

There seems to be a bunch of other files that need porting, see bug 1076893

Comment 3

[MDN Team (:mdn-dev)]

Commits pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/916659a1a0d9cefc98bb060deb14d2cd1859ce52
Bug 1110801 - Removed leftover code from 92a50f8e3ced75d5e52f19ef7666a96d1bc6aaad in attachements decorators.

https://github.com/mozilla/kuma/commit/9197b8188fdfe1a832a27e5057bbef2f624f78ed
Bug 1110801 - Remove unused attachment file listing view in favor of improved admin UI.

https://github.com/mozilla/kuma/commit/9d6a473f24e47352b96b5b2143bf7c380f41d09c
Bug 1110801 - Remove unused embed rendering.

https://github.com/mozilla/kuma/commit/739e711e6802fb6fb990227a191763367f759d4d
Bug 1110801 - Make attachment filename a property and use os.path for path splitting.

https://github.com/mozilla/kuma/commit/0fa98a43d763dec8219603a56655eedd7c43d20b
Bug 1110801 - Updated wiki spam related fields.

https://github.com/mozilla/kuma/commit/65af2f1d3291eb43a905942f16ac1599f8854966
Bug 1110801 - Updated locales and locale fields.

https://github.com/mozilla/kuma/commit/a0ff3729535279d4c53db6531d6a50dbb919ed0d
Bug 1110801 - Stopped a deprecation warning to show up.

https://github.com/mozilla/kuma/commit/1c8f8d795c9a3b5320a111539789c922f2d3ce28
Bug 1110801 - Delete the attachment detail view in favor of the admin.

https://github.com/mozilla/kuma/commit/d99a832af14fc3d929768feef9a82db237b6a1d5
Bug 1110801 - Deleted unused code.

https://github.com/mozilla/kuma/commit/6507ae914ddb08ffc042ace4b8e281a942fc31db
Bug 1110801 - Refactor attachments.

Fix bug 1026645, 665747 - Extend attachments admin integration.
Fix bug 1077172 - Store document attachment relation in the database instead of running a regex over the content.

https://github.com/mozilla/kuma/commit/db665b109d0dbe018b346627ed33074fb4eee099
Bug 1110801 - Redirect back to edit view instead of returning 405.

https://github.com/mozilla/kuma/commit/d565447bc708675692a35f91627a87cbc1f5dbae
Bug 1110801 - Install all dependencies to make requests make secure HTTP requests on Python < 2.7.9.

https://github.com/mozilla/kuma/commit/8b0a00e3498898a834114fee368c9b43d88a04a9
Bug 1110801 - Further attachment admin tweaks.

https://github.com/mozilla/kuma/commit/fc0f71fdb8491767ad79956eb75986eb71e65cd9
Bug 1110801 - Add force option for empty trash management command.

https://github.com/mozilla/kuma/commit/93db364d50ba86f7d7ca17c7e0cfc7ee626cbd7b
Bug 1110801 - Minor cosmetic fixes.

https://github.com/mozilla/kuma/commit/b14c0b4ff24ec4a41236c5cb6ce46487cffea0d1
Bug 1110801 - Don't allow creating multiple identical document/attachment M2M relationships.

https://github.com/mozilla/kuma/commit/c7f622853197d8b96b2c3fd2e18f72d5b13e5c50
Bug 1110801 - Some more details about the deletion of attachment references in wiki content.

https://github.com/mozilla/kuma/commit/c27ce6929552c5c9da71be7051552d198481af84
Merge branch 'master' into bug1110801

https://github.com/mozilla/kuma/commit/e2e2808ce2b1e2d6e9b689dea3d0a4394e45d454
Merge pull request #3769 from mozilla/bug1110801

Attachments cleanup/refactor for AWS

Comment 4

[MDN Team (:mdn-dev)]

Commit pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/79d9431379aaeed962d072f73d89ad7d56d701a6
Bug 1110801 - Better translated labels for attachments feed.

Comment 5

[Jannis Leidel [:jezdez]]

We've made lots of progress in refactoring the attachments app but the S3 goal is on hold until MDN's future platform is decided on. Unassigning myself in the meantime.

Comment 6

[John Whitlock [:jwhitlock]]

For the initial AWS migration, we're using Amazon's Elastic File System (EFS) [1]. This allows uploading attachments from one web instance and serving it from any others, and is most similar to NetApp in SCL3.  Some filenames had to be adjusted to fit the requirements of EFS.  Other files like humans.txt and sitemaps are generated and written to EFS.

This makes it harder to server MDN from multiple regions, as AWS doesn't support EFS across regions. However, it will be easier to explore other cloud-native options after the AWS move.

Removing from AWS blockers, will re-evaluate and re-prioritize later.

[1] https://aws.amazon.com/efs/

Comment 7

[BMO Automation]

MDN Web Docs' bug reporting has now moved to GitHub. From now on, please file content bugs at https://github.com/mdn/sprints/issues/ and platform bugs at https://github.com/mdn/kuma/issues/.