686617 - getLogExcerpt.php?type=annotated times out for leaks

Status: VERIFIED FIXED

(Tree Management Graveyard :: TBPL, defect)

Description

[Phil Ringnalda (:philor)]

Two convenient leaks (that conveniently probably won't get starred): the OS X64 debug Moth in https://tbpl.mozilla.org/?tree=Mozilla-Inbound&onlyunstarred=1&rev=db34d544a36b refused to load the summary until I directly loaded the getLogExcerpt.php URL to get a longer timeout, and the WinXP debug Moth in https://tbpl.mozilla.org/?tree=Mozilla-Inbound&onlyunstarred=1&rev=d771ae3d3021 I have yet to persuade to load the summary, even directly loading https://tbpl.mozilla.org/php/getLogExcerpt.php?id=6390004&tree=Mozilla-Inbound&type=annotated

Comment 1

[Robert Helmer [:rhelmer]]

Having trouble reproducing on my dev instance, seems to load fairly quickly.

Working through a minor merge conflict in a config file on tbpl-dev (which is auto-pushed), going to try there next (since it's same hardware etc. as prod).

One thing I notice is that the PHP max execute time is 10 minutes in the .htaccess file, not sure if that really overrides in dev, should test that.

Comment 2

[Robert Helmer [:rhelmer]]

(In reply to Robert Helmer [:rhelmer] from comment #1)

> One thing I notice is that the PHP max execute time is 10 minutes in the
> .htaccess file, not sure if that really overrides in dev, should test that.

(imts "in dev and prod", aka "the generic cluster")

Comment 3

[Robert Helmer [:rhelmer]]

The generic cluster is shared, I'll see if we have any kind of perf graphs (ganglia etc) that I could look at.

Comment 4

[Robert Helmer [:rhelmer]]

(In reply to Robert Helmer [:rhelmer] from comment #3)
> The generic cluster is shared, I'll see if we have any kind of perf graphs
> (ganglia etc) that I could look at.

By this I mean to imply the server might have been extraordinarily busy, or something, since I am not seeing this locally or on other prod URLs ATM.

Comment 5

[Robert Helmer [:rhelmer]]

(In reply to Phil Ringnalda (:philor) from comment #0)
> https://tbpl.mozilla.org/php/getLogExcerpt.php?id=6390004&tree=Mozilla-
> Inbound&type=annotated

I take what I said before back, I can repro this.. seems to work ok on -dev ATM though, odd.

https://tbpl-dev.allizom.org/php/getLogExcerpt.php?id=6390004&tree=Mozilla-Inbound&type=annotated

Comment 6

[Robert Helmer [:rhelmer]]

(In reply to Robert Helmer [:rhelmer] from comment #5)
> (In reply to Phil Ringnalda (:philor) from comment #0)
> > https://tbpl.mozilla.org/php/getLogExcerpt.php?id=6390004&tree=Mozilla-
> > Inbound&type=annotated
> 
> I take what I said before back, I can repro this.. seems to work ok on -dev
> ATM though, odd.
> 
> https://tbpl-dev.allizom.org/php/getLogExcerpt.php?id=6390004&tree=Mozilla-
> Inbound&type=annotated

OK this is odd, I get an empty response after 42 seconds:

---
time curl 'https://tbpl.mozilla.org/php/getLogExcerpt.php?id=6390004&tree=Mozilla-Inbound&type=annotated'
curl: (52) Empty reply from server

real	0m42.513s
user	0m0.006s
sys	0m0.012s
---

It really shouldn't be timing out this fast. I suspect it's hitting an error.

I really wish we could reproduce this in dev but it's working fine in my local VM and on tbpl-dev.

Comment 7

[Robert Helmer [:rhelmer]]

OK we should get logs enabled once bug 686785 goes out to pick up bug 686702.

I suspect that this script is erroring out not timing out, I'll change the summary once/if we can confirm.

Comment 8

[Robert Helmer [:rhelmer]]

After enabling logs, IT was able to diagnose and fix (will provide more details later).

philor would you mind verifying? lgtm

Comment 9

[Robert Helmer [:rhelmer]]

(In reply to Robert Helmer [:rhelmer] from comment #8)
> After enabling logs, IT was able to diagnose and fix (will provide more
> details later).

jason says "php was segfaulting because of eaccelerator"
 
It's disabled on prod now.

Comment 10

[Phil Ringnalda (:philor)]

Does seem to have been it. Now I'll have to find another excuse to ignore leaks :)