Summary: Crash-stats-new.allizom is nearly twice as slow as staging Our http://qa-selenium.mv.mozilla.com:8080/view/Socorro/job/socorro.dev/ tests take, now, on average, 11+ minutes to run, whereas its stage counterpart takes about 6 minutes: http://qa-selenium.mv.mozilla.com:8080/view/Socorro/job/socorro.stage/ :jberkus says that -dev has been very slow, in general, and we need to get to the bottom of this. https://crash-stats-dev.allizom.org/
The UI is about 200% slower than standard. I have no idea why, but it's not Postgres-related; load on the Dev DB server is negligable, and there's no long-running queries I can see.
Decreasing priority to "normal" because this is a Dev server.
Sounds like we need to dig down into which piece of this is slow.
:stephend , can you highlight some of the specific URLs / interactions that are the slowest? Thanks
OK, note that we verified and fixed some performance issues on Staging. So the original issue may be resolved. One problem here is that CSD is expected to be slower than prod or staging. So it's hard to know exactly what kind of slowness is significant. So yeah, a breakdown would be great.
Going to hold on this for now: different code is on dev (mobeta is master) vs. staging (release branch) -- the Selenium timing numbers aren't even internally consistent, likely due to load + # of concurrently running tests on dev vs. stage. Let's revisit when dev and stage match, and we'll strip our Selenium tests of their concurrency, then (no -n X via py.test) and compare apples to apples, instead of, as Lonnen put it, empires to fujiis
Renormalizing priority levels... P4 is "normal" now.
Hello everybody, I'm curious to know if this bug is still valid or not. If so, what are the specific, actionable items we should undertake in order to resolve this bug ? If not, may we close this bug and move forward, ever forward, to the future ? Thanks.
(In reply to Daniel Maher [:phrawzty] from comment #8) > Hello everybody, > > I'm curious to know if this bug is still valid or not. If so, what are the > specific, actionable items we should undertake in order to resolve this bug > ? If not, may we close this bug and move forward, ever forward, to the > future ? > > Thanks. Hey Daniel - From my end, it's not; both staging and dev oscillate between 3 min 1 sec and 4 min 1 sec; not a notable difference, and not consistent. When we were experiencing this bug, it was a huge difference (comment 0); I don't see anything actionable here, at least not in this context.
Ok, thanks for the update.