RFO needed for: Trees closed for 503 Internal Server error from hg.mozilla.org

RESOLVED WONTFIX

Status

Developer Services
Mercurial: hg.mozilla.org
RESOLVED WONTFIX
3 years ago
2 years ago

People

(Reporter: Tomcat, Unassigned)

Tracking

Details

(Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/1389] [RFO needed])

(Reporter)

Description

3 years ago
Ryan noticed and i confirmed that we get ISE Error 503 from hg.mozilla.org - could someone take a look at this, thanks!

Trees closed for this error
fubar alerted
Component: Buildduty → Mercurial: hg.mozilla.org
Product: Release Engineering → Developer Services
QA Contact: bugspam.Callek → hwine
(Reporter)

Comment 2

3 years ago
i don;t get the error 503 now but somehow hg operations are now super slow here. Keeping the trees closed for investigation
Severity: normal → blocker

Updated

3 years ago
Whiteboard: [kanban:engops:https://kanbanize.com/ctrl_board/6/532]
Nagios just recovered. 
hgweb8.dmz.scl3.mozilla.com:http - hgweb Port 80 is OK: HTTP OK: HTTP/1.1 200 Script output follows - 14339 bytes in 2.797 second response time

Fubar is looking at this, hopefully he has more info.
Flags: needinfo?(klibby)
(Reporter)

Comment 4

3 years ago
reopen the trees 7:41 but leaving the tree open
(Reporter)

Comment 5

3 years ago
(In reply to Carsten Book [:Tomcat] from comment #4)
> reopen the trees 7:41 but leaving the tree open

err that should be leaving the bug open for investigation :)
http://nigelbabu.github.io/hgstats/#hours/2 shows a spike in http requests, but not very far outside other spikes. the only errors logged are "Timeout when reading response headers from daemon process" from hgweb.wsgi, and resulting tracebacks for dropped connections. A bunch of build/tools and mozharness clones were taking a really long time - ~280 seconds, and it looks like a bunch of the above were some that took even longer. it looks like most of the build/ reqs were from AWS

I don't see any obvious smoking gun.
Flags: needinfo?(klibby)
updating to show this is RFO needed
Severity: blocker → normal
Summary: Trees closed for 503 Internal Server error from hg.mozilla.org → RFO needed for: Trees closed for 503 Internal Server error from hg.mozilla.org
Whiteboard: [kanban:engops:https://kanbanize.com/ctrl_board/6/532] → [kanban:engops:https://kanbanize.com/ctrl_board/6/532] [RFO needed]

Updated

3 years ago
Whiteboard: [kanban:engops:https://kanbanize.com/ctrl_board/6/532] [RFO needed] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/1379] [kanban:engops:https://kanbanize.com/ctrl_board/6/532] [RFO needed]

Updated

3 years ago
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/1379] [kanban:engops:https://kanbanize.com/ctrl_board/6/532] [RFO needed] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/1380] [kanban:engops:https://kanbanize.com/ctrl_board/6/532] [RFO needed]

Updated

3 years ago
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/1380] [kanban:engops:https://kanbanize.com/ctrl_board/6/532] [RFO needed] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/1382] [kanban:engops:https://kanbanize.com/ctrl_board/6/532] [RFO needed]

Updated

3 years ago
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/1382] [kanban:engops:https://kanbanize.com/ctrl_board/6/532] [RFO needed] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/1389] [kanban:engops:https://kanbanize.com/ctrl_board/6/532] [RFO needed]
(Assignee)

Updated

3 years ago
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/1389] [kanban:engops:https://kanbanize.com/ctrl_board/6/532] [RFO needed] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/1389] [RFO needed]
If we haven't done it in a year, we're not doing it.
Status: NEW → RESOLVED
Last Resolved: 2 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.