Closed Bug 1683671 Opened 4 years ago Closed 3 years ago

Several localized downloadpages perform really bad

Categories

(www.mozilla.org :: General, defect, P1)

Production

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: fryskefirefox, Unassigned)

Details

Thanks for reportiong - I can reproduce the same issues.

Priority: -- → P1

This seems to be a problem with our Frankfurt deployment, as far as I can tell: https://bedrock-prod.frankfurt.moz.works/en-US/firefox/all/

Flags: needinfo?(aalexander)

Current theory is that pages not cached by the cdn (localized pages that generally get less traffic would likely fit into this category), that are fairly large, like the /all page are currently slow.

Cause would likely be a change in infrastructure, in our case, an interaction between the load balancers and the networking setup in the new cluster. We'll work around by changing the infrastructure. Going to do this with a different service (nucleus) first, and then if that works port it over to bedrock.

Will update this issue when that work is complete.

Should now be resolved. Changes to the infra seem to have improved performance based on some light testing of all pages listed above.

Status: NEW → RESOLVED
Closed: 4 years ago
Flags: needinfo?(aalexander)
Resolution: --- → FIXED

I can still reproduce the issue using the following URL:

https://www.mozilla.org/en-US/firefox/all/?foo=bar (adding any random query param reproduces the issue for me)

Status: RESOLVED → REOPENED
Resolution: FIXED → ---

Hitting this URL directly without any query params also fails for me: https://bedrock-prod.frankfurt.moz.works/en-US/firefox/all/

Flags: needinfo?(aalexander)

Indeed, still bad performance.

Should be good now, discovered a similar problem with the scale of one of the components, after scaling that up everything seems to be more performant.

Status: REOPENED → RESOLVED
Closed: 4 years ago4 years ago
Flags: needinfo?(aalexander)
Resolution: --- → FIXED

Alan, sorry I'm still seeing the same issue when I visit: https://bedrock-prod.frankfurt.moz.works/en-US/firefox/all/

Status: RESOLVED → REOPENED
Flags: needinfo?(aalexander)
Resolution: FIXED → ---

We've been working on this for the past few weeks. Seems like an issue with meinheld workers in bedrock. Did an investigation to check the infra setup but no easy wins, so currently trying different worker types to try and solve.

Flags: needinfo?(aalexander)

Hi Alan, just checking in on this bug:

I still see the same issues unfortunately when visiting: https://bedrock-prod.frankfurt.moz.works/en-US/firefox/all/

GCP seems fine however: https://bedrock-prod.gcp.moz.works/en-US/firefox/all/

Any updates on your investigation? I'm curious on the impact this might be having on users in Europe.

Flags: needinfo?(aalexander)

I think we're mixing two mostly unrelated issues. The endpoints you shared are from an older setup where we noticed data being truncated. In that other issue we switched to a new infra setup using some different amazon objects. https://bedrock.prod.fr.moz.works/en-US/firefox/all behaves more as we expect, and is what real users will hit.

The less popular translations still seem to be fairly slow, https://bedrock.prod.fr.moz.works/fy-NL/firefox/all/#product-desktop-release. We stepped through the stack and found that was true at the individual application layer, so at the moment we're trying to find the time to really dig into bedrock and find that source of slowness. Attempts to just change worker types didn't help, and a reasonable number of experiments didn't return any helpful results.

Flags: needinfo?(aalexander)

I think we can close this one now, thanks all!

Status: REOPENED → RESOLVED
Closed: 4 years ago3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.