Closed Bug 1086564 Opened 10 years ago Closed 9 years ago

RFO needed for: Trees closed for 503 Internal Server error from hg.mozilla.org

Categories

(Developer Services :: Mercurial: hg.mozilla.org, defect)

Product:
Developer Services
Component:
Mercurial: hg.mozilla.org
Platform:
x86
All
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: cbook, Unassigned)

Details

(Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/1389] [RFO needed]) 

Ryan noticed and i confirmed that we get ISE Error 503 from hg.mozilla.org - could someone take a look at this, thanks!

Trees closed for this error
fubar alerted
Component: Buildduty → Mercurial: hg.mozilla.org
Product: Release Engineering → Developer Services
QA Contact: bugspam.Callek → hwine
i don;t get the error 503 now but somehow hg operations are now super slow here. Keeping the trees closed for investigation
Severity: normal → blocker
Whiteboard: [kanban:engops:https://kanbanize.com/ctrl_board/6/532]
Nagios just recovered. 
hgweb8.dmz.scl3.mozilla.com:http - hgweb Port 80 is OK: HTTP OK: HTTP/1.1 200 Script output follows - 14339 bytes in 2.797 second response time

Fubar is looking at this, hopefully he has more info.
Flags: needinfo?(klibby)
reopen the trees 7:41 but leaving the tree open
(In reply to Carsten Book [:Tomcat] from comment #4)
> reopen the trees 7:41 but leaving the tree open

err that should be leaving the bug open for investigation :)
http://nigelbabu.github.io/hgstats/#hours/2 shows a spike in http requests, but not very far outside other spikes. the only errors logged are "Timeout when reading response headers from daemon process" from hgweb.wsgi, and resulting tracebacks for dropped connections. A bunch of build/tools and mozharness clones were taking a really long time - ~280 seconds, and it looks like a bunch of the above were some that took even longer. it looks like most of the build/ reqs were from AWS

I don't see any obvious smoking gun.
Flags: needinfo?(klibby)
updating to show this is RFO needed
Severity: blocker → normal
Summary: Trees closed for 503 Internal Server error from hg.mozilla.org → RFO needed for: Trees closed for 503 Internal Server error from hg.mozilla.org
Whiteboard: [kanban:engops:https://kanbanize.com/ctrl_board/6/532] → [kanban:engops:https://kanbanize.com/ctrl_board/6/532] [RFO needed]
Whiteboard: [kanban:engops:https://kanbanize.com/ctrl_board/6/532] [RFO needed] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/1379] [kanban:engops:https://kanbanize.com/ctrl_board/6/532] [RFO needed]
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/1379] [kanban:engops:https://kanbanize.com/ctrl_board/6/532] [RFO needed] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/1380] [kanban:engops:https://kanbanize.com/ctrl_board/6/532] [RFO needed]
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/1380] [kanban:engops:https://kanbanize.com/ctrl_board/6/532] [RFO needed] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/1382] [kanban:engops:https://kanbanize.com/ctrl_board/6/532] [RFO needed]
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/1382] [kanban:engops:https://kanbanize.com/ctrl_board/6/532] [RFO needed] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/1389] [kanban:engops:https://kanbanize.com/ctrl_board/6/532] [RFO needed]
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/1389] [kanban:engops:https://kanbanize.com/ctrl_board/6/532] [RFO needed] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/1389] [RFO needed]
If we haven't done it in a year, we're not doing it.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.