540863 - Partial upload on AMO causes update to be created with incorrect state

Status: RESOLVED FIXED

(addons.mozilla.org Graveyard :: Developer Pages, defect, P3)

Description

[Eric Jung [:ericjung]]

Attachment: FoxyProxy Standard 2.18

I haven't been able to upload a new version of FoxyProxy standard in 3-4 day. I don't precisely remember the error I got a few days ago, but it was after clicking "Continue" on the Validation Results page.

Today I received a different error on the same page:

Oops! There seems to be a problem with this file...

An error occurred moving 59605400 1264009425.xpi.

Please correct this problem and upload your file again.

I deleted the 2.18 file/version from the https://addons.mozilla.org/en-US/developers/versions/add/2464 and tried again. Now I'm just getting the spinner/throbber indefinitely after pressing "Continue" on the Validation Results page.

XPI attached.

Comment 1

[Dave Garrett]

This maybe a dupe of bug 539372?

For the record, I had a hell of a time uploading a version of Flagfox not too long ago. Occasionally getting a never-ending throbber. Occasionally just getting blank pages instead of actually doing something. It's been really sketchy lately.

Comment 2

[Jorge Villalobos [:jorgev] (he/him)]

Wil, can you take a look?

Comment 4

[Wil Clouser [:clouserw]]

Fine!  I'll dig into this scary code.

Anyway, thanks everyone for pasting in the error messages - that helps a lot.  The problem is, we're using that same error message in 3 different places in this workflow so I'm not sure where it is breaking.

Can this problem be reproduced on preview.amo?  If so, that would speed this process up.

Comment 5

[Eric Jung [:ericjung]]

(In reply to comment #4)

> Can this problem be reproduced on preview.amo?  If so, that would speed this
> process up.

Just uploaded FoxyProxy Standard 2.18 to preview.amo successfully :(  I'll delete it and try a few more times. If it fails, I'll comment again.

Comment 6

[Wil Clouser [:clouserw]]

Thanks, and paste in the error if you get one.  I added tags so we can tell where it fails.

Comment 7

[Dave Garrett]

The error also sounds like what I was getting in bug 530511.

Comment 8

[Eric Jung [:ericjung]]

(In reply to comment #6)
> Thanks, and paste in the error if you get one.  I added tags so we can tell
> where it fails.

Didn't get any errors with 4 or 5 tries. For kicks, I tried uploading the same XPI to production AMO again. This time I can't get past the Validation Results page ("Now validating your add-on ... " with infinite spinner/throbber). See attached screenshot.

If this is related to validation, is there any way you can selectively and temporarily disable validation so I can publish this? Alternatively, can you publish it manually on my behalf?

Comment 9

[Eric Jung [:ericjung]]

Attachment: infinite throbber

Comment 10

[Wil Clouser [:clouserw]]

Please try this again

Comment 11

[Wil Clouser [:clouserw]]

I mean, try uploading on the live site again.  I think this is fixed.  I tried validating the attached add-on on the live site and it's working now.

Comment 12

[Eric Jung [:ericjung]]

(In reply to comment #11)
> I mean, try uploading on the live site again.  I think this is fixed.  I tried
> validating the attached add-on on the live site and it's working now.

Worked, thanks. Did you change anything on the site or are we just being lucky? IOW, should this bug be closed?

Comment 13

[Wil Clouser [:clouserw]]

Live site is changed and the reason things were broken was stupid. :(

Comment 15

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to comment #13)
> Live site is changed and the reason things were broken was stupid. :(

What was the fix?  I take it that it was something on IT's side?

Comment 17

[Jorge Villalobos [:jorgev] (he/him)]

We're still receiving messages from authors saying that their uploads are being created with an In Sandbox status, rather than pending review.

Comment 18

[Bee]

Hi!!!!!!!!!!!!!!!!!!

Yeah!!!! I think that something of similar is happening also to Bee Free!!!!!!!
Maybe, this could help!!!! I also noticed that in the "Manage Versions and Files" page ( https://addons.mozilla.org/en-US/developers/versions/57131 ) all the values under the "Validation" column turned into "No Test Results"!!!!!!!!!!!!!!!
I remember there were written things about all the passed tests, and the `warnings' received!!!!!!!!!!!!! (and, also, the warnings themselves were shot without real reasons against unreferenced files!!!!!!!!!)

bye!!!!!!!!!!!!!!!!!!!!
Bee!!!!!!!!!!!!

Comment 19

[Dave Garrett]

(In reply to comment #18)
> the values under the "Validation" column turned into "No Test
> Results"!!!!!!!!!!!!!!!
> I remember there were written things about all the passed tests, and the
> `warnings' received!!!!!!!!!!!!!

First of all, the exclamations were clearly excessive...

What you're describing is correct. It only saves the results for a day. This is intentional to avoid caching obsolete results after test changes. (bug 513350)

Comment 20

[Bee]

Well, OKAY!!!!!!! So the two things aren't related!!!!
(though what you said is written nowhere, so i didn't knew it!!!!!!!!!!)!!!
THANK YOU DAVE!!!!!!!!!!!!!!!!

However, there is for sure something of strange in that page; i don't know why, but i feel it!!!!!!!!!!!!!!!!! maybe, it is right the same bug reported here!!!!but im almost sure that page doesn't work as it should!!!!!!

bye!!!!!!!!!!!!
~bee!!!!!!!!!

Comment 21

[Wil Clouser [:clouserw]]

Someone please attach an add-on that is failing

Comment 22

[Wil Clouser [:clouserw]]

Alright, I can't reproduce this on dev, stage, or prod.  Can someone try it on preview.addons.mozilla.org?  If it fails, please copy the entire error message and paste it here (or screenshot).  Thanks.

Comment 23

[Michael Kraft [:morac]]

I've found it's intermittent.  See bug 544037 for screen shots.

Comment 24

[Wil Clouser [:clouserw]]

Thanks Michael.  I got the feeling that wasn't the same bug but I replied to your bug with my idea.

Can anyone reproduce this bug right now on preview?

Comment 25

[Jorge Villalobos [:jorgev] (he/him)]

Attachment: List of add-ons and versions affected by this bug

Comment 26

[Jorge Villalobos [:jorgev] (he/him)]

(Hit submit too soon)

Since the problem kept coming up from developers, I took the time today to query the DB for add-ons that potentially could be presenting this issue. I came with this list of over 100 add-on updates that had an incorrect status.

A handful of these are false positives because I didn't cover the multiple file per version case, but there are still over a hundred cases here, the majority of which are fairly recent.

Comment 27

[Wil Clouser [:clouserw]]

What query did you use to create that.  I'm looking at the first one on your list, 12, and I'm not sure what the problem is.

Comment 28

[Jorge Villalobos [:jorgev] (he/him)]

You'll have to run it in preview or some previous snapshot of the DB. I fixed the status for all of those before I posted that comment. The all had the latest file with an In Sandbox status and no editor review.

Comment 29

[Jorge Villalobos [:jorgev] (he/him)]

Attachment: SQL to get add-ons with incorrect status

And FWIW, here's the query.

Comment 30

[Jorge Villalobos [:jorgev] (he/him)]

Attachment: New list, generated 2010-04-23

Here's a new list I generated today. These are the updates that have had an incorrect status assigned to them in the past 2 weeks.

There are 26 add-ons on this list, and I already fixed their status in production.

Comment 31

[Jorge Villalobos [:jorgev] (he/him)]

Attachment: New list, generated 2010-04-30

More data.

Comment 32

[Jorge Villalobos [:jorgev] (he/him)]

Attachment: New list, generated 2010-05-07

11 add-on updates with incorrect status this week.

Comment 33

[Chuck Baker]

Yesterday (05/07/2010) I tried uploading FEBE 6.3.3 (addon_id 2109) to AMO.  At the end of the process I got a message "Oops ... cannot move febe-6.3.3.xpi [3]" (or something very similar).

An email to fligtar got the status changed correctly to "1 In Sandbox; Pending Review file".

Could this problem be related to file size?  FEBE is a relatively large addon (~874k).

Comment 34

[Wil Clouser [:clouserw]]

(In reply to comment #33)
> Yesterday (05/07/2010) I tried uploading FEBE 6.3.3 (addon_id 2109) to AMO.  At
> the end of the process I got a message "Oops ... cannot move febe-6.3.3.xpi
> [3]" (or something very similar).

Please copy the exact error next time.  Ending with a [3] is very different than ending with a [2] or [1].

Comment 35

[Chuck Baker]

Well, the "[3]" I am certain about.  It's the "Oops" part that may be paraphrased.

Comment 36

[Wil Clouser [:clouserw]]

Attachment: lame sauce

I'm grasping at straws here since I can't reproduce this.  My latest idea is that it's a race condition.  While processing an update we put all pending files in the sandbox (so the new one will be the only pending one).  If that update query takes a long time, the new file might already be in the db by the time it finishes, and in that case it will mark the latest one as a sandbox add-on too.  -> r?fwenzel for crazy idea check

Comment 37

[Fred Wenzel [:wenzel]]

Comment on attachment 445004 [details] [diff] [review]
lame sauce

That's fine, but I really don't think this is it. ACID dictates write queries are executed in order of arrival.

What does the [3] indicate? I wasn't able to find what part of the code that points at.

Comment 38

[Wil Clouser [:clouserw]]

(In reply to comment #37)
> (From update of attachment 445004 [details] [diff] [review])
> That's fine, but I really don't think this is it. ACID dictates write queries
> are executed in order of arrival.
Maybe they get there out of order?  I dunno.  Thanks for the vote of confidence. ;) r67298

> What does the [3] indicate? I wasn't able to find what part of the code that
> points at.
We have the same (crappy) generic error message in 3 places, so I tacked that onto the end so I could see what was actually failing.

Alright, I'm closing this again.  Let's see what happens.

Comment 39

[Chuck Baker]

(In reply to comment #38)
> Alright, I'm closing this again.  Let's see what happens.

Here's what happens:

Oops! There seems to be a problem with this file...

An error occurred moving febe-6.3.3.2.xpi [3].

Please correct this problem and upload your file again.

I just tried to upload a new version and got this message.  It is copied/pasted and therefore the exact message.  This time I did not try to re-upload, I just left it as is so that status can be viewed.

Comment 40

[Chuck Baker]

I just checked the "Recent Activity for FEBE" and it shows that the same version  was uploaded five times within two minutes.  The first entry has the status of "1 In Sandbox file" and the other four state "No Files".  As I mentioned before, I only attempted to upload once and just walked away after getting the above error message.

Comment 41

[Wil Clouser [:clouserw]]

The latest patch is only on preview.  It will be pushed to the live site next week.

Comment 42

[Jorge Villalobos [:jorgev] (he/him)]

Attachment: Incorrect status list 2010-06-11

It looks like this isn't fixed yet. Here's the list of add-on updates that I had to fix today. There are a number of updates there that were submitted days after the push.

Comment 43

[Jorge Villalobos [:jorgev] (he/him)]

Reopening.

Comment 44

[Michael Kraft [:morac]]

Happened to me again.  

An error occurred moving 18086500 1276749623.xpi [3].


I notice a pattern though.  It seems to depend on how fast I hit the continue button when I upload.  If I take the time to look at the warnings that show up after the upload (or I switch tabs and don't come back for a minute or two), there's a high chance that I'll get an error when I press "continue".

If I press "continue" immediately after the warnings page displays, I usually don't get an error.

There's a work around to fix this when you hit the error.  Go to the versions and files and remove all the duplicated empty versions of the version you just uploaded.  In the version that contains a file, delete the file and re-upload it.

Comment 45

[Jorge Villalobos [:jorgev] (he/him)]

Attachment: Incorrect status list 2010-06-25

Comment 46

[Jorge Villalobos [:jorgev] (he/him)]

Attachment: Incorrect status list 2010-08-09

New list.

Comment 47

[Jorge Villalobos [:jorgev] (he/him)]

While reviewing the corrected add-ons today, I noticed that every single one of them was a very simple add-on with a rather simple update. This may have some relation with comment 44, that refers to the time the validator takes to review the file and when the Continue button is clicked.

Comment 48

[Jorge Villalobos [:jorgev] (he/him)]

This might be fixed now, maybe due to some other fix. Today I checked the DB for the first time in about a month, and I only had to restore one status, which was caused by the beta detection algorithm (so not really this bug).

I'll continue checking this weekly for some time and reopen if necessary, but it looks like it's fixed.