358450 - AMO Additem not functioning - no uploads working

Status: VERIFIED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Justin Scott [:fligtar]]

Upon uploading any valid .xpi or .jar, step 2 of additem is a blank page.

After receiving 2 reports of this today and not being able to find problems with the install.rdf (the usual culprit in blank pages), I tried uploading them on my local install and it DOES work. I then tried on staging and it does NOT work. I tried 2 valid extensions on production that should work, and they do not.

Is this related to the netscaler DB changes?

Comment 1

[Michael Morgan [:morgamic]]

Is it possible that the config was changed so that the additem scripts point to the read-only db? That is my first guess.

Comment 2

[Justin Scott [:fligtar]]

Moving to mozilla.org::Server Ops as I don't think this is something AMO can fix.

Comment 3

[Jeremy Orem [:oremj]]

Config looks good.  There are SHADOW entries, but I'm pretty sure we have always had that.

Comment 4

[Jeremy Orem [:oremj]]

No errors in the logs.

Comment 5

[Justin Scott [:fligtar]]

I'm able to get incorrect maxapp version errors, which means the file is being moved successfully and parsed successfully. Everything seems to work fine up until displaying the step 2 form fields and such. And this occurs on both production and staging, but not on local installations.

Doing more testing.

Comment 6

[Dave Miller [:justdave]]

Based on the conversation on IRC it sounds like it's still using the zziplib stuff and the app servers its erroring on aren't running our recompiled php that has zziplib in it.

IIRC the stuff in the cluster never had the recompiled PHP, and that's why the developers directory was proxying to iguana instead of running directly on the webheads.

Comment 7

[Dave Miller [:justdave]]

But if that's the case, it should have started failing when we moved it to MPT, and it's been a while since then...

Comment 8

[Dave Miller [:justdave]]

[root@mradm01 ~]# /data/bin/issue-dynamic-command rpm -q php
===mrapp01===
php-4.3.9-3.18
===mrapp02===
php-4.3.9-3.18
===mrapp03===
php-4.3.9-3.18
===mrapp04===
php-4.3.9-3.18
===mrapp05===
php-4.3.9-3.18
===mrapp06===
php-4.3.9-3.18
===mrapp07===
php-4.3.9-3.18
===mrapp08===
php-4.3.9-3.18
===mrapp09===
php-4.3.9-3.18
===mrapp10===
php-4.3.9-3.18
[root@mradm01 ~]# ssh chameleon rpm -q php
php-4.3.9-3.12.moz
[root@mradm01 ~]# ssh iguana rpm -q php
php-4.3.2-30.ent.moz+zip.0

Comment 9

[Michael Morgan [:morgamic]]

(In reply to comment #6)
> IIRC the stuff in the cluster never had the recompiled PHP, and that's why the
> developers directory was proxying to iguana instead of running directly on the
> webheads.
So are you sure that addons/developers isn't proxied to a single box?  I didn't know that v1 code was running on the MPT cluster at all because of the custom compile.

Comment 10

[Michael Morgan [:morgamic]]

Can you guys verify whether or not v1 is running on the cluster?  Can you also add mradm01 to the list of checked PHP packages?

Comment 11

[Dave Miller [:justdave]]

mradm01 doesn't have a functioning web server, except as an RHN proxy internally, so it doesn't matter.  We couldn't run it there if we wanted to.

Based on the apache configuration on the cluster, it does indeed look like the developer section is running from the cluster.  The only way it wouldn't be is if the netscaler was sending it somewhere else, but I wasn't involved in the move when they moved it to MPT so I don't know for sure.

Comment 12

[Dave Miller [:justdave]]

wasn't someone working on removing the zziplib requirement?  Or does it actually still need that?

Comment 13

[Michael Morgan [:morgamic]]

(In reply to comment #12)
> wasn't someone working on removing the zziplib requirement?  Or does it
> actually still need that?
There isn't a way around it.  Any PEAR libraries or PECL extensions we've seen still require it indirectly...

Comment 14

[Jeremy Orem [:oremj]]

This problem looks like it is solved.  2/10 machines were not loading zip.so. Fixed now I just get Addons cannot have an external updateURL value. Please remove this from your install.rdf and try again.  Which I'm guessing is not a server issue.

Comment 15

[Justin Scott [:fligtar]]

Hooray! Thanks everyone!

Comment 16

[Giorgio Maone [:ma1]]

(In reply to comment #13)
> (In reply to comment #12)
> > wasn't someone working on removing the zziplib requirement?  Or does it
> > actually still need that?
> There isn't a way around it.  Any PEAR libraries or PECL extensions we've seen
> still require it indirectly...
> 

This is probably true, but in the dark ages of AMO 1 I remember myself coding a work-around which used the zip command line utility... don't know how much useful now, but there *was* a work-around: attachment 172285 [details] [diff] [review] from bug 270996

Comment 17

[Mike Shaver (:shaver emeritus)]

(In reply to comment #4)
> No errors in the logs.

(In reply to comment #14)
> This problem looks like it is solved.  2/10 machines were not loading zip.so.

Does it not generate an error if it can't load the shared library?  I've always seen an error either at Apache start time or when the php zip functions are called if there's a problem with the library or php config.

Comment 18

[Jeremy Orem [:oremj]]

No it doesn't generate an error, because the error is blocked with an @ in the developer code.  Otherwise, it would.

Comment 19

[Michael Morgan [:morgamic]]

Might consider doing a function_exists('zip_open') then throw an error using the logAndDie() func if it returns false.