781992 - crash-stats dev is almost twice as slow as staging

Status: RESOLVED WORKSFORME

(Infrastructure & Operations Graveyard :: WebOps: Other, task, P4)

Description

[Stephen Donner [:stephend] Not actively reading bugmail]

Summary: Crash-stats-new.allizom is nearly twice as slow as staging

Our http://qa-selenium.mv.mozilla.com:8080/view/Socorro/job/socorro.dev/ tests take, now, on average, 11+ minutes to run, whereas its stage counterpart takes about 6 minutes: http://qa-selenium.mv.mozilla.com:8080/view/Socorro/job/socorro.stage/

:jberkus says that -dev has been very slow, in general, and we need to get to the bottom of this.

https://crash-stats-dev.allizom.org/

Comment 1

[[:jberkus] Josh Berkus]

The UI is about 200% slower than standard.  I have no idea why, but it's not Postgres-related; load on the Dev DB server is negligable, and there's no long-running queries I can see.

Comment 2

[[:jberkus] Josh Berkus]

Decreasing priority to "normal" because this is a Dev server.

Comment 3

[Jake Maul [:jakem]]

Sounds like we need to dig down into which piece of this is slow.

Comment 4

[Brandon Burton [:solarce]]

:stephend , can you highlight some of the specific URLs / interactions that are the slowest?

Thanks

Comment 5

[[:jberkus] Josh Berkus]

OK, note that we verified and fixed some performance issues on Staging.  So the original issue may be resolved.

One problem here is that CSD is expected to be slower than prod or staging.  So it's hard to know exactly what kind of slowness is significant.  So yeah, a breakdown would be great.

Comment 6

[Stephen Donner [:stephend] Not actively reading bugmail]

Going to hold on this for now: different code is on dev (mobeta is master) vs. staging (release branch) -- the Selenium timing numbers aren't even internally consistent, likely due to load + # of concurrently running tests on dev vs. stage.

Let's revisit when dev and stage match, and we'll strip our Selenium tests of their concurrency, then (no -n X via py.test) and compare apples to apples, instead of, as Lonnen put it, empires to fujiis

Comment 7

[Jake Maul [:jakem]]

Renormalizing priority levels... P4 is "normal" now.

Comment 8

[Daniel Maher [:phrawzty]]

Hello everybody,

I'm curious to know if this bug is still valid or not.  If so, what are the specific, actionable items we should undertake in order to resolve this bug ?  If not, may we close this bug and move forward, ever forward, to the future ?

Thanks.

Comment 9

[Stephen Donner [:stephend] Not actively reading bugmail]

(In reply to Daniel Maher [:phrawzty] from comment #8)
> Hello everybody,
> 
> I'm curious to know if this bug is still valid or not.  If so, what are the
> specific, actionable items we should undertake in order to resolve this bug
> ?  If not, may we close this bug and move forward, ever forward, to the
> future ?
> 
> Thanks.

Hey Daniel -

From my end, it's not; both staging and dev oscillate between 3 min 1 sec and 4 min 1 sec; not a notable difference, and not consistent.

When we were experiencing this bug, it was a huge difference (comment 0); I don't see anything actionable here, at least not in this context.

Comment 10

[Daniel Maher [:phrawzty]]

Ok, thanks for the update.