If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

crash-stats dev is almost twice as slow as staging

RESOLVED WORKSFORME

Status

Infrastructure & Operations
WebOps: Other
P4
normal
RESOLVED WORKSFORME
5 years ago
4 years ago

People

(Reporter: stephend, Assigned: phrawzty)

Tracking

Details

(Whiteboard: [triaged 20120824], URL)

(Reporter)

Description

5 years ago
Summary: Crash-stats-new.allizom is nearly twice as slow as staging

Our http://qa-selenium.mv.mozilla.com:8080/view/Socorro/job/socorro.dev/ tests take, now, on average, 11+ minutes to run, whereas its stage counterpart takes about 6 minutes: http://qa-selenium.mv.mozilla.com:8080/view/Socorro/job/socorro.stage/

:jberkus says that -dev has been very slow, in general, and we need to get to the bottom of this.

https://crash-stats-dev.allizom.org/
The UI is about 200% slower than standard.  I have no idea why, but it's not Postgres-related; load on the Dev DB server is negligable, and there's no long-running queries I can see.
Decreasing priority to "normal" because this is a Dev server.
Severity: major → normal

Updated

5 years ago
Whiteboard: [pending triage]

Comment 3

5 years ago
Sounds like we need to dig down into which piece of this is slow.
Priority: -- → P3
Summary: -dev is almost twice as slow as staging → crash-stats dev is almost twice as slow as staging
Whiteboard: [pending triage] → [triaged 20120824]
:stephend , can you highlight some of the specific URLs / interactions that are the slowest?

Thanks
OK, note that we verified and fixed some performance issues on Staging.  So the original issue may be resolved.

One problem here is that CSD is expected to be slower than prod or staging.  So it's hard to know exactly what kind of slowness is significant.  So yeah, a breakdown would be great.
(Reporter)

Comment 6

5 years ago
Going to hold on this for now: different code is on dev (mobeta is master) vs. staging (release branch) -- the Selenium timing numbers aren't even internally consistent, likely due to load + # of concurrently running tests on dev vs. stage.

Let's revisit when dev and stage match, and we'll strip our Selenium tests of their concurrency, then (no -n X via py.test) and compare apples to apples, instead of, as Lonnen put it, empires to fujiis

Comment 7

5 years ago
Renormalizing priority levels... P4 is "normal" now.
Priority: P3 → P4
(Assignee)

Comment 8

5 years ago
Hello everybody,

I'm curious to know if this bug is still valid or not.  If so, what are the specific, actionable items we should undertake in order to resolve this bug ?  If not, may we close this bug and move forward, ever forward, to the future ?

Thanks.
Flags: needinfo?
(Reporter)

Comment 9

5 years ago
(In reply to Daniel Maher [:phrawzty] from comment #8)
> Hello everybody,
> 
> I'm curious to know if this bug is still valid or not.  If so, what are the
> specific, actionable items we should undertake in order to resolve this bug
> ?  If not, may we close this bug and move forward, ever forward, to the
> future ?
> 
> Thanks.

Hey Daniel -

From my end, it's not; both staging and dev oscillate between 3 min 1 sec and 4 min 1 sec; not a notable difference, and not consistent.

When we were experiencing this bug, it was a huge difference (comment 0); I don't see anything actionable here, at least not in this context.
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Flags: needinfo?
Resolution: --- → WORKSFORME
(Assignee)

Comment 10

5 years ago
Ok, thanks for the update.
Assignee: server-ops-webops → dmaher
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.