957158 - error on add-on upload: server rejects multipacked files like theme + extension

Status: VERIFIED FIXED

(addons.mozilla.org Graveyard :: Add-on Validation, defect, P1)

Description

[Aris]

Attachment: 1.png

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:27.0) Gecko/20100101 Firefox/27.0 (Beta/Release)
Build ID: 20140106141415

Steps to reproduce:

Tried to upload a new version of Noia 4.

How to reproduce:
- download Noia 4 v1.8.8 ( https://addons.mozilla.org/en-us/firefox/addon/noia-4 )
- open add-ons validator ( https://addons.mozilla.org/en-US/developers/addon/validate )
- upload file


Actual results:

Your add-on failed validation with 1 error.
There was a problem contacting the server.


Expected results:

Upload should work fine like always.

Comment 1

[Aris]

Attachment: noia4v189.xpi - I try to upload this file

Comment 2

[Aris]

Uploadig single files from main xpi package works fine too.

Comment 3

[Aris]

Tricked the validator by renaming main file to "something.jar" before uploading.

There is still something majorly wrong with add-ons validator.

Comment 4

[Aris]

This bug is still not fixed. The renaming trick does not work anymore.

Comment 5

[Jorge Villalobos [:jorgev] (he/him)]

Can you attach the file that is failing to validate?

Comment 6

[Aris]

Attachment: noia4v200pre4.xpi

Comment 7

[Aris]

Hey Jorge,

at the momemt add-ons validator rejects every multi packed theme I tried to upload.
Even already fully reviewed files ( noia4v189.xpi ) won't pass anymore.

Comment 8

[Mark Striemer [:mstriemer]]

I confirmed the uploads are not working, I will look into it.

Comment 10

[Alexander Brandl]

Hi,

I have the same problem for all of my themes, which are also multi packed:
https://addons.mozilla.org/en-US/firefox/addon/utopia-ffse-white/
https://addons.mozilla.org/en-US/firefox/addon/oldfactory/
https://addons.mozilla.org/en-US/firefox/addon/afterglow-1/

Comment 11

[Aris]

This bug is still present. Could somebody please fix add-ons validator or disable it?

Comment 12

[Mark Striemer [:mstriemer]]

I "successfully" validated the noia4v200pre4.xpi file (and several other of the linked files) and got "Your add-on passed validation with no errors and 23 messages." Clicking through to the full validation report sporadically gives me an error though. Seems like it's probably a server configuration problem. I will talk to IT about how we can debug this.

Comment 13

[Mark Striemer [:mstriemer]]

I no longer get sporadic failures, validation *should* be working properly right now. Who knows if that will continue to be the case. I've brought the issue up with IT and they are looking into some errors we were seeing.

I will leave this open - please comment if you encounter any errors or successes.

Comment 14

[Aris]

Still getting same connection error.
Is it possible there are some regional restrictions or errors?
I'm uploading from Europe (Germany).

http://i57.tinypic.com/rhkavb.png

Comment 15

[Mark Striemer [:mstriemer]]

So I uploaded again and got the error. It looks like the server drops the connection and the JS doesn't handle it very gracefully. I grabbed the URL that the JS was hitting and tried it manually, got connection terminated then tried again and got a JSON dump. Looks like validation is working fine, it is just returning the response that is the problem.

Comment 17

[Red_Fat_Lazy_Cat]

Got the same error today trying to upload a non-multipacked theme. So things are getting worse from the looks of it.

Comment 18

[Aris]

Sorry for posting again, but is anybody trying to fix this? It has been now almost four month since this one got reported.

If it is not possible to fix, please disable Add-ons validator for themes!

Comment 19

[Jorge Villalobos [:jorgev] (he/him)]

Mark, are you working on this bug?

Comment 20

[Jorge Villalobos [:jorgev] (he/him)]

Fixing needinfo.

Comment 21

[Aris]

Attachment: Newer file I'm trying to upload.

It would be great, if someone coud force upload this file for me.

Comment 22

[Mark Striemer [:mstriemer]]

This always seems to work on staging so I'm getting IT to look into if we're killing these connections prematurely. I reproduced this on production a couple times and now I can't reproduce it. It isn't ideal but maybe uploading it a few times in a row will warm-up the servers enough to complete the request in time.

You can eventually get a good validation by resending the status request but the page is still stuck in its invalid state. If IT can't resolve this then I'll try and fix the JS or find someone who can.

The relevant JS [1] can probably try again in this case instead of flagging the error.

[1] https://github.com/mozilla/olympia/blob/master/media/js/common/upload-addon.js#L272

Comment 23

[Alexander Brandl]

Sorry for my ignorance, I'm not really much of a techie, but until december 2013 validation worked without a problem, then it suddenly stopped. So there must have been some change in the server configuration in about this stretch of time. Is it really so difficult to find out what was changed and who changed it in that time, and to roll it back to the state before?

For me it's not relevant anymore though, because with "Australis" my theming days are probably over anyway, so time has kinda outrun the bug for me. But I feel for people like Aris, who still puts a lot of work and energy in updating his Noia4, and can't upload it to the server now since months.

(And if I may add this small note here: Many thanks Aris also for your "Classic Theme Restorer", it will probably be the only way for me to keep using Firefox!)

Comment 24

[Aris]

(In reply to Mark Striemer [:mstriemer] from comment #22)
> ... It isn't ideal but maybe
> uploading it a few times in a row will warm-up the servers enough to
> complete the request in time.
> 

Tried this a couple of times on different days, still no luck.
Is there any chance some countries are more often affected than others? I'm uploading from Germany.

Comment 25

[Aris]

Afer a few more tests theme upload passes validator, if a small theme file is used.

Randomly removing files from theme package causes the theme not to work, but to pass add-ons validator. Does this make sense?

The main problem seems to be a faulty timeout inside the validator.

Can someone check this?

Comment 26

[Frank Lion]

(In reply to Aris from comment #25)
> Afer a few more tests theme upload passes validator, if a small theme file
> is used.
> 
> Randomly removing files from theme package causes the theme not to work, but
> to pass add-ons validator. Does this make sense?
Yes, it makes sense. My Type 32 (multi package) themes have always uploaded and validated OK, but their filesize is very small. 

However, it is not filesize alone doing this bug - as a test, I uploaded a test version of a trivial extension of mine that uses a massive 4MB .wav file for an amusing effect...and it uploaded without problem and I then cancelled out. 

That would suggest it's a timeout bug hitting large filesize Type 32 only?

Comment 27

[mo78]

I can confirm that bug.
Mozo

Comment 28

[Olin_Broadway]

Please do what you can to alleviate the pressure on the extension developers. They are a large part of  what makes Firefox so usable and many of them do so in their spare time. When tools no longer work or politics interfere with the process they will simply quit supporting their extensions, some of which are indispensable.

Thank you,

Olin

Comment 29

[mo78]

Yeah, and many quiting Firefox. Firefox is almost nithing without themes and extension developers.

Comment 30

[Mark Striemer [:mstriemer]]

I've got a PR in that should suffice as a workaround in these cases. It isn't perfect but hopefully it will work until a real fix can be done.

https://github.com/mozilla/olympia/pull/90

Comment 31

[Mark Striemer [:mstriemer]]

https://github.com/mozilla/olympia/commit/8b78c7172ea2472c86ae4e6dc642dc5c2963b457

Comment 33

[Madalin Cotetiu]

I'm still able to reproduce this issue using the xpi file attached in comment 21.
The validation takes a long time and at the end there is this error message: "Your add-on failed validation with 1 error.    There was a problem contacting the server."
Please view this screencast: http://screencast.com/t/wUbOIoiTHj
Reopening.

Comment 34

[Mark Striemer [:mstriemer]]

PR to move some processing to celery: https://github.com/mozilla/olympia/pull/127.

Comment 35

[Mark Striemer [:mstriemer]]

https://github.com/mozilla/olympia/commit/b2ab6efa30b7fa1662be5dfede081c852ad4e528

QA please verify that uploading addons still works everywhere that can be done (submission, standalone validation, new version).

Comment 36

[krupa raj[:krupa]]

test add-on from comment 1 completes validation without any errors. See https://addons.allizom.org/en-US/developers/upload/b46b2a951967484790d3b93c11a8f04f

This will be in production tomorrow.