Last Comment Bug 540863 - Partial upload on AMO causes update to be created with incorrect state
: Partial upload on AMO causes update to be created with incorrect state
Status: RESOLVED FIXED
:
Product: addons.mozilla.org Graveyard
Classification: Graveyard
Component: Developer Pages (show other bugs)
: unspecified
: All All
: P3 major
: 5.12.2
Assigned To: Wil Clouser [:clouserw]
:
Mentors:
: 527250 540872 541951 (view as bug list)
Depends on: 602911
Blocks:
  Show dependency treegraph
 
Reported: 2010-01-20 09:51 PST by Eric H. Jung
Modified: 2016-02-04 14:49 PST (History)
8 users (show)
See Also:
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
FoxyProxy Standard 2.18 (630.57 KB, application/x-xpinstall)
2010-01-20 09:51 PST, Eric H. Jung
no flags Details
infinite throbber (74.14 KB, image/png)
2010-01-21 14:29 PST, Eric H. Jung
no flags Details
List of add-ons and versions affected by this bug (4.23 KB, text/csv)
2010-04-09 17:46 PDT, Jorge Villalobos [:jorgev]
no flags Details
SQL to get add-ons with incorrect status (348 bytes, text/plain)
2010-04-09 23:23 PDT, Jorge Villalobos [:jorgev]
no flags Details
New list, generated 2010-04-23 (1.07 KB, text/csv)
2010-04-23 10:38 PDT, Jorge Villalobos [:jorgev]
no flags Details
New list, generated 2010-04-30 (606 bytes, text/csv)
2010-04-30 10:31 PDT, Jorge Villalobos [:jorgev]
no flags Details
New list, generated 2010-05-07 (498 bytes, text/csv)
2010-05-07 14:40 PDT, Jorge Villalobos [:jorgev]
no flags Details
lame sauce (1.29 KB, patch)
2010-05-12 16:06 PDT, Wil Clouser [:clouserw]
bugmail: review+
Details | Diff | Review
Incorrect status list 2010-06-11 (698 bytes, text/csv)
2010-06-11 09:23 PDT, Jorge Villalobos [:jorgev]
no flags Details
Incorrect status list 2010-06-25 (452 bytes, text/csv)
2010-06-25 12:58 PDT, Jorge Villalobos [:jorgev]
no flags Details
Incorrect status list 2010-08-09 (455 bytes, text/csv)
2010-08-09 14:43 PDT, Jorge Villalobos [:jorgev]
no flags Details

Description Eric H. Jung 2010-01-20 09:51:01 PST
Created attachment 422557 [details]
FoxyProxy Standard 2.18

I haven't been able to upload a new version of FoxyProxy standard in 3-4 day. I don't precisely remember the error I got a few days ago, but it was after clicking "Continue" on the Validation Results page.

Today I received a different error on the same page:

Oops! There seems to be a problem with this file...

An error occurred moving 59605400 1264009425.xpi.

Please correct this problem and upload your file again.

I deleted the 2.18 file/version from the https://addons.mozilla.org/en-US/developers/versions/add/2464 and tried again. Now I'm just getting the spinner/throbber indefinitely after pressing "Continue" on the Validation Results page.

XPI attached.
Comment 1 Dave Garrett 2010-01-20 10:10:36 PST
This maybe a dupe of bug 539372?

For the record, I had a hell of a time uploading a version of Flagfox not too long ago. Occasionally getting a never-ending throbber. Occasionally just getting blank pages instead of actually doing something. It's been really sketchy lately.
Comment 2 Jorge Villalobos [:jorgev] 2010-01-20 10:25:49 PST
Wil, can you take a look?
Comment 3 Wil Clouser [:clouserw] 2010-01-21 09:54:12 PST
*** Bug 540872 has been marked as a duplicate of this bug. ***
Comment 4 Wil Clouser [:clouserw] 2010-01-21 10:08:12 PST
Fine!  I'll dig into this scary code.

Anyway, thanks everyone for pasting in the error messages - that helps a lot.  The problem is, we're using that same error message in 3 different places in this workflow so I'm not sure where it is breaking.

Can this problem be reproduced on preview.amo?  If so, that would speed this process up.
Comment 5 Eric H. Jung 2010-01-21 10:19:38 PST
(In reply to comment #4)

> Can this problem be reproduced on preview.amo?  If so, that would speed this
> process up.

Just uploaded FoxyProxy Standard 2.18 to preview.amo successfully :(  I'll delete it and try a few more times. If it fails, I'll comment again.
Comment 6 Wil Clouser [:clouserw] 2010-01-21 10:20:50 PST
Thanks, and paste in the error if you get one.  I added tags so we can tell where it fails.
Comment 7 Dave Garrett 2010-01-21 10:36:40 PST
The error also sounds like what I was getting in bug 530511.
Comment 8 Eric H. Jung 2010-01-21 14:28:43 PST
(In reply to comment #6)
> Thanks, and paste in the error if you get one.  I added tags so we can tell
> where it fails.

Didn't get any errors with 4 or 5 tries. For kicks, I tried uploading the same XPI to production AMO again. This time I can't get past the Validation Results page ("Now validating your add-on ... " with infinite spinner/throbber). See attached screenshot.

If this is related to validation, is there any way you can selectively and temporarily disable validation so I can publish this? Alternatively, can you publish it manually on my behalf?
Comment 9 Eric H. Jung 2010-01-21 14:29:09 PST
Created attachment 422833 [details]
infinite throbber
Comment 10 Wil Clouser [:clouserw] 2010-01-21 16:46:32 PST
Please try this again
Comment 11 Wil Clouser [:clouserw] 2010-01-21 18:54:59 PST
I mean, try uploading on the live site again.  I think this is fixed.  I tried validating the attached add-on on the live site and it's working now.
Comment 12 Eric H. Jung 2010-01-21 19:54:57 PST
(In reply to comment #11)
> I mean, try uploading on the live site again.  I think this is fixed.  I tried
> validating the attached add-on on the live site and it's working now.

Worked, thanks. Did you change anything on the site or are we just being lucky? IOW, should this bug be closed?
Comment 13 Wil Clouser [:clouserw] 2010-01-21 19:56:48 PST
Live site is changed and the reason things were broken was stupid. :(
Comment 14 Jorge Villalobos [:jorgev] 2010-01-25 07:55:27 PST
*** Bug 541951 has been marked as a duplicate of this bug. ***
Comment 15 Stephen Donner [:stephend] 2010-01-26 11:15:45 PST
(In reply to comment #13)
> Live site is changed and the reason things were broken was stupid. :(

What was the fix?  I take it that it was something on IT's side?
Comment 16 Jorge Villalobos [:jorgev] 2010-01-26 15:42:27 PST
*** Bug 527250 has been marked as a duplicate of this bug. ***
Comment 17 Jorge Villalobos [:jorgev] 2010-02-10 10:44:00 PST
We're still receiving messages from authors saying that their uploads are being created with an In Sandbox status, rather than pending review.
Comment 18 Bee 2010-02-10 12:50:36 PST
Hi!!!!!!!!!!!!!!!!!!

Yeah!!!! I think that something of similar is happening also to Bee Free!!!!!!!
Maybe, this could help!!!! I also noticed that in the "Manage Versions and Files" page ( https://addons.mozilla.org/en-US/developers/versions/57131 ) all the values under the "Validation" column turned into "No Test Results"!!!!!!!!!!!!!!!
I remember there were written things about all the passed tests, and the `warnings' received!!!!!!!!!!!!! (and, also, the warnings themselves were shot without real reasons against unreferenced files!!!!!!!!!)

bye!!!!!!!!!!!!!!!!!!!!
Bee!!!!!!!!!!!!
Comment 19 Dave Garrett 2010-02-10 12:55:38 PST
(In reply to comment #18)
> the values under the "Validation" column turned into "No Test
> Results"!!!!!!!!!!!!!!!
> I remember there were written things about all the passed tests, and the
> `warnings' received!!!!!!!!!!!!!

First of all, the exclamations were clearly excessive...

What you're describing is correct. It only saves the results for a day. This is intentional to avoid caching obsolete results after test changes. (bug 513350)
Comment 20 Bee 2010-02-10 13:11:28 PST
Well, OKAY!!!!!!! So the two things aren't related!!!!
(though what you said is written nowhere, so i didn't knew it!!!!!!!!!!)!!!
THANK YOU DAVE!!!!!!!!!!!!!!!!

However, there is for sure something of strange in that page; i don't know why, but i feel it!!!!!!!!!!!!!!!!! maybe, it is right the same bug reported here!!!!but im almost sure that page doesn't work as it should!!!!!!

bye!!!!!!!!!!!!
~bee!!!!!!!!!
Comment 21 Wil Clouser [:clouserw] 2010-02-10 13:49:03 PST
Someone please attach an add-on that is failing
Comment 22 Wil Clouser [:clouserw] 2010-02-10 14:02:41 PST
Alright, I can't reproduce this on dev, stage, or prod.  Can someone try it on preview.addons.mozilla.org?  If it fails, please copy the entire error message and paste it here (or screenshot).  Thanks.
Comment 23 Michael Kraft [:morac] 2010-02-10 14:23:07 PST
I've found it's intermittent.  See bug 544037 for screen shots.
Comment 24 Wil Clouser [:clouserw] 2010-02-10 14:45:28 PST
Thanks Michael.  I got the feeling that wasn't the same bug but I replied to your bug with my idea.

Can anyone reproduce this bug right now on preview?
Comment 25 Jorge Villalobos [:jorgev] 2010-04-09 17:46:49 PDT
Created attachment 438224 [details]
List of add-ons and versions affected by this bug
Comment 26 Jorge Villalobos [:jorgev] 2010-04-09 17:50:46 PDT
(Hit submit too soon)

Since the problem kept coming up from developers, I took the time today to query the DB for add-ons that potentially could be presenting this issue. I came with this list of over 100 add-on updates that had an incorrect status.

A handful of these are false positives because I didn't cover the multiple file per version case, but there are still over a hundred cases here, the majority of which are fairly recent.
Comment 27 Wil Clouser [:clouserw] 2010-04-09 20:55:55 PDT
What query did you use to create that.  I'm looking at the first one on your list, 12, and I'm not sure what the problem is.
Comment 28 Jorge Villalobos [:jorgev] 2010-04-09 23:22:23 PDT
You'll have to run it in preview or some previous snapshot of the DB. I fixed the status for all of those before I posted that comment. The all had the latest file with an In Sandbox status and no editor review.
Comment 29 Jorge Villalobos [:jorgev] 2010-04-09 23:23:34 PDT
Created attachment 438240 [details]
SQL to get add-ons with incorrect status

And FWIW, here's the query.
Comment 30 Jorge Villalobos [:jorgev] 2010-04-23 10:38:17 PDT
Created attachment 441071 [details]
New list, generated 2010-04-23

Here's a new list I generated today. These are the updates that have had an incorrect status assigned to them in the past 2 weeks.

There are 26 add-ons on this list, and I already fixed their status in production.
Comment 31 Jorge Villalobos [:jorgev] 2010-04-30 10:31:50 PDT
Created attachment 442734 [details]
New list, generated 2010-04-30

More data.
Comment 32 Jorge Villalobos [:jorgev] 2010-05-07 14:40:12 PDT
Created attachment 444180 [details]
New list, generated 2010-05-07

11 add-on updates with incorrect status this week.
Comment 33 Chuck Baker 2010-05-08 09:09:34 PDT
Yesterday (05/07/2010) I tried uploading FEBE 6.3.3 (addon_id 2109) to AMO.  At the end of the process I got a message "Oops ... cannot move febe-6.3.3.xpi [3]" (or something very similar).

An email to fligtar got the status changed correctly to "1 In Sandbox; Pending Review file".

Could this problem be related to file size?  FEBE is a relatively large addon (~874k).
Comment 34 Wil Clouser [:clouserw] 2010-05-12 13:16:24 PDT
(In reply to comment #33)
> Yesterday (05/07/2010) I tried uploading FEBE 6.3.3 (addon_id 2109) to AMO.  At
> the end of the process I got a message "Oops ... cannot move febe-6.3.3.xpi
> [3]" (or something very similar).

Please copy the exact error next time.  Ending with a [3] is very different than ending with a [2] or [1].
Comment 35 Chuck Baker 2010-05-12 13:46:55 PDT
Well, the "[3]" I am certain about.  It's the "Oops" part that may be paraphrased.
Comment 36 Wil Clouser [:clouserw] 2010-05-12 16:06:28 PDT
Created attachment 445004 [details] [diff] [review]
lame sauce

I'm grasping at straws here since I can't reproduce this.  My latest idea is that it's a race condition.  While processing an update we put all pending files in the sandbox (so the new one will be the only pending one).  If that update query takes a long time, the new file might already be in the db by the time it finishes, and in that case it will mark the latest one as a sandbox add-on too.  -> r?fwenzel for crazy idea check
Comment 37 Fred Wenzel [:wenzel] 2010-05-14 09:48:58 PDT
Comment on attachment 445004 [details] [diff] [review]
lame sauce

That's fine, but I really don't think this is it. ACID dictates write queries are executed in order of arrival.

What does the [3] indicate? I wasn't able to find what part of the code that points at.
Comment 38 Wil Clouser [:clouserw] 2010-05-17 08:58:17 PDT
(In reply to comment #37)
> (From update of attachment 445004 [details] [diff] [review])
> That's fine, but I really don't think this is it. ACID dictates write queries
> are executed in order of arrival.
Maybe they get there out of order?  I dunno.  Thanks for the vote of confidence. ;) r67298

> What does the [3] indicate? I wasn't able to find what part of the code that
> points at.
We have the same (crappy) generic error message in 3 places, so I tacked that onto the end so I could see what was actually failing.

Alright, I'm closing this again.  Let's see what happens.
Comment 39 Chuck Baker 2010-05-22 16:43:35 PDT
(In reply to comment #38)
> Alright, I'm closing this again.  Let's see what happens.

Here's what happens:

Oops! There seems to be a problem with this file...

An error occurred moving febe-6.3.3.2.xpi [3].

Please correct this problem and upload your file again.

I just tried to upload a new version and got this message.  It is copied/pasted and therefore the exact message.  This time I did not try to re-upload, I just left it as is so that status can be viewed.
Comment 40 Chuck Baker 2010-05-22 22:07:01 PDT
I just checked the "Recent Activity for FEBE" and it shows that the same version  was uploaded five times within two minutes.  The first entry has the status of "1 In Sandbox file" and the other four state "No Files".  As I mentioned before, I only attempted to upload once and just walked away after getting the above error message.
Comment 41 Wil Clouser [:clouserw] 2010-05-24 08:37:26 PDT
The latest patch is only on preview.  It will be pushed to the live site next week.
Comment 42 Jorge Villalobos [:jorgev] 2010-06-11 09:23:06 PDT
Created attachment 450672 [details]
Incorrect status list 2010-06-11

It looks like this isn't fixed yet. Here's the list of add-on updates that I had to fix today. There are a number of updates there that were submitted days after the push.
Comment 43 Jorge Villalobos [:jorgev] 2010-06-11 09:23:51 PDT
Reopening.
Comment 44 Michael Kraft [:morac] 2010-06-16 21:47:54 PDT
Happened to me again.  

An error occurred moving 18086500 1276749623.xpi [3].


I notice a pattern though.  It seems to depend on how fast I hit the continue button when I upload.  If I take the time to look at the warnings that show up after the upload (or I switch tabs and don't come back for a minute or two), there's a high chance that I'll get an error when I press "continue".

If I press "continue" immediately after the warnings page displays, I usually don't get an error.

There's a work around to fix this when you hit the error.  Go to the versions and files and remove all the duplicated empty versions of the version you just uploaded.  In the version that contains a file, delete the file and re-upload it.
Comment 45 Jorge Villalobos [:jorgev] 2010-06-25 12:58:34 PDT
Created attachment 454110 [details]
Incorrect status list 2010-06-25
Comment 46 Jorge Villalobos [:jorgev] 2010-08-09 14:43:06 PDT
Created attachment 464174 [details]
Incorrect status list 2010-08-09

New list.
Comment 47 Jorge Villalobos [:jorgev] 2010-08-09 15:06:33 PDT
While reviewing the corrected add-ons today, I noticed that every single one of them was a very simple add-on with a rather simple update. This may have some relation with comment 44, that refers to the time the validator takes to review the file and when the Continue button is clicked.
Comment 48 Jorge Villalobos [:jorgev] 2010-10-29 16:53:23 PDT
This might be fixed now, maybe due to some other fix. Today I checked the DB for the first time in about a month, and I only had to restore one status, which was caused by the beta detection algorithm (so not really this bug).

I'll continue checking this weekly for some time and reopen if necessary, but it looks like it's fixed.

Note You need to log in before you can comment on or make changes to this bug.