880740 - Access problems with try critical again; impacting developers

Status: RESOLVED FIXED

(Developer Services :: General, task)

Description

[Shyam Mani [:fox2mike]]

We have close to 8000 heads on try now and is causing issues when people are pushing. We need to fix this.

Comment 1

[Armen [:armenzg]]

hgweb is currently broken: https://hg.mozilla.org/try (can't load)
It also seems that individual tbpl revisions cannot be loaded up:
https://tbpl.mozilla.org/?tree=Try&rev=50e24e570269

Comment 2

[Hal Wine [:hwine] use NI!]

Thanks to :bkero & :fox2mike -- done.

Restarted apache on hgweb2 after draining in zeus. That unblocked try access on all webheads.

Comment 3

[Hal Wine [:hwine] use NI!]

I should note that we still need to evaluate doing a try reset, but the hangs in viewing http://hg.mozilla.org/try that were impacting developers have been resolved for now.

Comment 4

[Hal Wine [:hwine] use NI!]

updating summary - with comment #1, this bug focused on an acute instance of try not working for devs TODAY. That acute instance has been addressed.

The need for doing a reset is unclear at this moment, and will be more invasive to developer workflow. (While we've had problems with 3K heads, we've also no needed to reset until 14K heads.)

Comment 5

[Hal Wine [:hwine] use NI!]

Attachment: ganglia load during event