1184979 - "Unexpected server error while validating." every time when uploading new extension

Status: RESOLVED INCOMPLETE

(addons.mozilla.org Graveyard :: Add-on Validation, defect)

Description

[Andrew Zitnay]

Attachment: lppre.xpi

User Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:39.0) Gecko/20100101 Firefox/39.0
Build ID: 20150630154324

Steps to reproduce:

1) Go to https://addons.mozilla.org/en-US/developers/addon/submit/2

2) Check "Do not list my add-on on this site (beta)" and "This add-on will be side-loaded via application installers."

3) Select the attached lppre.xpi and wait for it to finish uploading.


Actual results:

The upload failed with:

Your add-on failed validation with 1 error.

    Unexpected server error while validating.


Expected results:

The upload shouldn't fail.  This happened 2 out of 2 times yesterday, and 1 out of 1 time today.

Comment 1

[Kris Maglione [:kmag]]

This upload is failing due to a validation timeout. If you'd like me to, I can submit it for you and attach the validation results from my local instance to the review.

Comment 2

[Andrew Zitnay]

(In reply to Kris Maglione [:kmag] from comment #1)
> This upload is failing due to a validation timeout. If you'd like me to, I
> can submit it for you and attach the validation results from my local
> instance to the review.

I appreciate that, but it doesn't really fix the issue.  I'm mainly submitting this to get a bearing on how long unlisted reviews are taking.  I don't see how Firefox can go live with mandatory extension signing if the tools used to sign aren't working.

Comment 3

[Andrew Zitnay]

Any movement on this?  I tried again this morning, and it never actually timed out, it's just hanging forever while validating (close to an hour now).

Comment 4

[Kris Maglione [:kmag]]

It does time out, the error just isn't reported.

The timeout that's currently happening on production is in a particularly slow piece of code that should no longer exist as of the push this Thursday. Next week's push should have a set of changes to the validator that cut the validation time of your add-on in half.

Comment 5

[Andrew Zitnay]

Ok, thanks for the info.  About what time on Thursday should I try again?(In reply to Kris Maglione [:kmag] from comment #4)
> It does time out, the error just isn't reported.
> 
> The timeout that's currently happening on production is in a particularly
> slow piece of code that should no longer exist as of the push this Thursday.
> Next week's push should have a set of changes to the validator that cut the
> validation time of your add-on in half.

Ok, thanks for the info.  About what time on Thursday should I try again?

Comment 6

[Kris Maglione [:kmag]]

We generally push in the early afternoon PDT.

Comment 7

[Andrew Zitnay]

Welp, this specific problem appears to be fixed, but the AMO upload problems continue.  I finished uploading the XPI, selected the source zip to attach, and continued.  I ended up on a page that said:

-----
Oops! We had an error.

We'll get to fixing that soon.

You can try refreshing the page, or head back to the Add-ons homepage. 
-----

I tried again, and am now getting "Duplicate UUID found."  I assume this is referring to the extension ID ([e-mail]).  However, "Manage My Submissions" still only lists a single extension (ID [e-mail]).  Is the [e-mail] ID somehow in limbo now?

Comment 8

[Andrew Zitnay]

Attachment: lp_e10s.xpi

Comment 9

[Andrew Zitnay]

The problem unfortunately appears to remain when I try to upload our e10s extension (attached as lp_e10s.xpi).  It's been hanging on "Validating lp.xpi" for quite a while now.

Comment 10

[Kris Maglione [:kmag]]

There was an internal server error in the middle of your submission that left your listing without an owner. I'm looking into it now, but in the mean time I fixed the ownership of the listing.

Comment 11

[Andrew Zitnay]

Hmm, now I'm getting "Unexpected server error while validating." every time again, even when uploading the production version of LastPass that was working last week.

Comment 12

[Jorge Villalobos [:jorgev] (he/him)]

Please attach or link the file here.

Comment 13

[Andrew Zitnay]

Attachment: lp.xpi

This is the attachment from earlier, where validating was timing out.

Comment 14

[Kris Maglione [:kmag]]

Validation for your add-on is just on the borderline for timing out with the code currently on production, so it succeeds or fails mostly depending on server load. The code running on dev, which sill go live this week, is much faster, and doesn't  result in a server error if it times out.

Comment 15

[Andrew Zitnay]

Things do seem to be working better at the moment.  I'm seeing a weird issue where, after uploading, waiting for validation, and submitting, it says "Could not parse install.rdf" and the dialog doesn't go away.  However, the upload appears to succeed anyway.  I'll upload the xpi.

Comment 16

[Andrew Zitnay]

Attachment: lp.xpi

I get "Could not parse install.rdf" after uploading and validating this xpi.

Comment 17

[Kris Maglione [:kmag]]

It looks like that version was uploaded successfully: https://addons.mozilla.org/en-US/developers/addon/lastpass-password-manager/versions/1734032

When/where did you see the install.rdf error?

As for your other issues, it looks like we've been running into a transaction management bug in the version of Django we currently use. I plan to have it fixed by this Thursday's push.

Comment 18

[Andrew Zitnay]

From:

https://addons.mozilla.org/en-US/developers/addon/lastpass-dev/edit

1) Click "Upload New Version"

2) Click "Select a file" and choose the file I attached

3) Wait for validation to complete

4) Click "Add Version"

Instead of the dialog going away as expected, it scrolled to the top of the dialog, and the install.rdf error appeared in red, somewhere near the "Select a file" button I believe.

Comment 19

[Andrew Zitnay]

While attempting to debug the install.rdf issue, I just got the following:

Your add-on failed validation with 1 error.

    An unexpected error has occurred.

I'll upload the xpi.

Comment 20

[Andrew Zitnay]

Attachment: lp.xpi

Received the following while attempting to upload this xpi:

Your add-on failed validation with 1 error.

    An unexpected error has occurred.

Comment 21

[Andrew Zitnay]

We're unfortunately seeing AMO hang when uploading new versions again.  It's been hanging at "Validating lp.xpi" for almost an hour now.  Are there any known issues at the moment?

Comment 22

[Andy McKay]

You are hitting a time limit that is imposed on parsing add-ons.

Oct 26 18:02:49 cel2.mktweb.services.phx1.mozilla.com: [][] celery.worker.job:WARNING Soft time limit (180s) exceeded for devhub.tasks.validate_file_path[9ea5bd67-5ddb-463d-b873-6b87917f70ed]

I'm not sure why that is, it could be something in your add-on that is causing the problem. I'd be willing to give it a try locally, is one of the previous attachments similar?

Comment 23

[Andrew Zitnay]

Attachment: lp_e10s.xpi

hangs while validating

Comment 24

[Andrew Zitnay]

Attachment: lp.xpi

hangs while validating

Comment 25

[Andrew Zitnay]

I'd imagine the previous attachments would exhibit the problem, but I've gone ahead and attached the two I'm currently trying.

Comment 26

[Andy McKay]

Confirmed, I can reproduce locally in my puny local Docker instance. Not sure what the problem is though.

Comment 27

[Andy McKay]

jorgev, for this scenario, it needs to added with the admin override I think?

Comment 28

[Jorge Villalobos [:jorgev] (he/him)]

Yes. IIRC, Andrew managed to upload their update after several more tries, and I had to upload their beta version because that one still timed out.