1058682 - Stability problems on AMO since 2014-08 push

Status: RESOLVED FIXED

(addons.mozilla.org Graveyard :: Developer Pages, defect, P1)

Description

[Jorge Villalobos [:jorgev] (he/him)]

Since the August push, we have been receiving constant complaints from developers that AMO is failing for them. The problems we have seen reported so far:

* Uploading a new add-on or new version fails after validation.
* After a new version is reviewed, updates aren't pushed to users.
* Sporadic breakage in review tools: missing files in code viewer, errors when submitting reviews, and multiple review form submissions.

This is a critical problem that may be costing us developers and needs to be addressed ASAP.

Comment 1

[Wil Clouser [:clouserw]]

CCing people who may have specific ideas.  Developers - did we adjust any of this (looks like several bugs with the validator?).  Jason - any changes on the ops side or obvious broken bits/alerts?

Comment 2

[Kris Maglione [:kmag]]

We've also been getting a lot of 504s, both in the editor tools and developer pages, especially when trying to submit reviews. The latter case is probably a factor in comments for reviews appearing multiple times in review logs.

Comment 3

[WBT]

I've been experiencing the same issue, especially when trying to upload a new add-on or an updated version, either of which would be queued for full review.  It's not clear what was and wasn't successfully uploaded. 

> This is a critical problem that may be costing us developers and needs to be
> addressed ASAP.
I can definitely vouch for that.  After several attempts at uploading my first public add-on and getting 504s and other errors and having to go back to the beginning, I was one attempt away from completely giving up and determining that I was not able and not going to be able to contribute public Firefox add-ons because I just couldn't figure out how to get the system to work.  Fortunately, it actually worked on that last attempt, and the add-on later passed full review.

Comment 4

[Jan [:Jan\]]

 504 Gateway Time-out
when submitting addon

Comment 5

[Jeremy Orem [:oremj]]

This sounds like the same problem I was seeing in Bug 897317. You can use the XPI attached there to reproduce on dev and stage.

Comment 6

[Jeremy Orem [:oremj]]

I've reverted https://github.com/washort/django-cache-machine/commit/fecd5a281500018d42c6c7c2ff8d8bfc92545c1a#diff-d41d8cd98f00b204e9800998ecf8427e, which makes django-cache-machine compatible with redis sharding. Addon submission is still slow, but no longer 504ing, as far as I can tell.

Something, in the submit_license and version_delete views, is causing mass invalidation to occur. This resulted in 2,000+ SMEMBER calls and now ~15 giant SUNION calls. We should either keep this bug open to debug that issue or close this and open another bug to address it.

Comment 7

[Mathieu Agopian [:magopian]]

Jorge, Kris, are the issues listed here still there since Jeremy's fix?

Comment 8

[Jorge Villalobos [:jorgev] (he/him)]

I haven't seen new reports recently. We should give it a couple of days before calling this fixed, however.

Comment 9

[Kris Maglione [:kmag]]

Agree, let's give it until the end of the week before we call it fixed.

Comment 10

[backy0175]

Hi all,

I uploaded new version of a addon for android during AMO had problem.
After validation finished, AMO said new version is existing.
But It seems upload new version was finished in background. Because "Manage Status & Versions" shows new version and queued for review.
Then new version was approved for full review.

Yesterday, I noticed download button is active when I access addon page from desktop firefox.
But Version Informations said "Works with Firefox for Android 26.0 - 35.0" only, not for desktop.

Did I made any mistake or something related to AMO problem?


"Cache Tool for Android"
https://addons.mozilla.org/en-US/android/addon/cache-tool-for-android/versions/

Thank you for your help,
backy0175

Comment 11

[Jorge Villalobos [:jorgev] (he/him)]

Both versions show up as compatible only with Android. You need to select "All Platforms" when uploading a new version so that Firefox compatibility is displayed. I think you can add it here: https://addons.mozilla.org/developers/addon/cache-tool-for-android/versions/1595024

In any case, this appears unrelated to this bug.

Comment 12

[backy0175]

jorge, thank you for a reply.

I checked version information. It had been already "All Platforms".
I uploaded new version as usual. But this time, something weird.

I tried to add and remove desktop compatibility. But it didn't fix this problem.
What should I do next?

backy0175

Comment 13

[Jorge Villalobos [:jorgev] (he/him)]

(In reply to backy0175 from comment #12)
> I tried to add and remove desktop compatibility. But it didn't fix this
> problem.
> What should I do next?

Please email the amo-admins list. This discussion is offtopic for this bug.