Closed Bug 870472 Opened 13 years ago Closed 13 years ago

500 Internal Server Error when loading TryServer hg revisions

Categories

(Developer Services :: General, task)

Product:
Developer Services
Component:
General
Platform:
x86_64
Linux
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED INCOMPLETE

People

(Reporter: dholbert, Assigned: bkero)

Details

STR: Visit one of these URLs: https://hg.mozilla.org/try/rev/11661097c8f1 https://hg.mozilla.org/try/rev/b0786de7d698 ACTUAL RESULTS: A lot of waiting, and then finally a 500 Internal Server Error page loads, with these contents: { Internal Server Error The server encountered an internal error or misconfiguration and was unable to complete your request. Please contact the server administrator, root@localhost and inform them of the time the error occurred, and anything you might have done that may have caused the error. More information about this error may be available in the server error log. Apache/2.2.15 (Red Hat) Server at hg.mozilla.org Port 80 } (Note: I'm having trouble even loading just https://hg.mozilla.org/try/ -- seems to be loading forever.)
Assignee: server-ops → server-ops-devservices
Component: Server Operations → Server Operations: Developer Services
Output from trying to access with "curl" { $ time curl -vvv https://hg.mozilla.org/try/rev/11661097c8f1 * About to connect() to hg.mozilla.org port 443 (#0) * Trying 63.245.215.25... * Connected to hg.mozilla.org (63.245.215.25) port 443 (#0) * successfully set certificate verify locations: * CAfile: none CApath: /etc/ssl/certs * SSLv3, TLS handshake, Client hello (1): * SSLv3, TLS handshake, Server hello (2): * SSLv3, TLS handshake, CERT (11): * SSLv3, TLS handshake, Server finished (14): * SSLv3, TLS handshake, Client key exchange (16): * SSLv3, TLS change cipher, Client hello (1): * SSLv3, TLS handshake, Finished (20): * SSLv3, TLS change cipher, Client hello (1): * SSLv3, TLS handshake, Finished (20): * SSL connection using RC4-SHA * Server certificate: * subject: serialNumber=q0fphJMA8aMebDO07ghuPLGrws-4iZxr; C=US; ST=California; L=Mountain View; O=Mozilla Corporation; OU=IT; CN=hg.mozilla.org * start date: 2011-11-22 06:44:15 GMT * expire date: 2013-11-24 07:50:10 GMT * subjectAltName: hg.mozilla.org matched * issuer: C=US; O=GeoTrust, Inc.; CN=GeoTrust SSL CA * SSL certificate verify ok. > GET /try/rev/11661097c8f1 HTTP/1.1 > User-Agent: curl/7.29.0 > Host: hg.mozilla.org > Accept: */* > < HTTP/1.1 500 Internal Server Error < Date: Thu, 09 May 2013 18:48:58 GMT < Server: Apache/2.2.15 (Red Hat) < X-Backend-Server: hgweb2.dmz.scl3.mozilla.com < Vary: Accept-Encoding < Content-Length: 615 < Connection: close < Content-Type: text/html; charset=iso-8859-1 < <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>500 Internal Server Error</title> </head><body> <h1>Internal Server Error</h1> <p>The server encountered an internal error or misconfiguration and was unable to complete your request.</p> <p>Please contact the server administrator, root@localhost and inform them of the time the error occurred, and anything you might have done that may have caused the error.</p> <p>More information about this error may be available in the server error log.</p> <hr> <address>Apache/2.2.15 (Red Hat) Server at hg.mozilla.org Port 80</address> </body></html> * Closing connection 0 * SSLv3, TLS alert, Client hello (1): real 5m5.049s user 0m0.008s sys 0m0.004s }
[root@hgweb3.dmz.scl3 ~]# curl -H 'Host: hg.mozilla.org' http://localhost/try/rev/11661097c8f1 & [root@hgweb3.dmz.scl3 ~]# strace -f -p 10950 Process 10950 attached - interrupt to quit read(16, ^C <unfinished ...> Process 10950 detached [root@hgweb3.dmz.scl3 ~]# ls -hl /proc/10950/fd/16 lrwx------ 1 root root 64 May 9 11:56 16 -> socket:[1774707202] Eventually this process dies with the httpd error: [Thu May 09 11:58:51 2013] [error] [client ::1] Premature end of script headers: hgweb.wsgi
In #gfx, snorp and gw280 are reporting issues pushing to try at the moment, too. (in one case, with "remote: waiting for lock on repository /repo/hg/mozilla/try held by 'hgssh1.dmz.scl3.mozilla.com:16946'") Seems like that might be related.
This sometimes happens when folks try to commit to try at the same time. I can confirm process 16946 has ended and there is no lock file associated with try anymore. This seems unrelated to me. Additionally both revs are now cached and unreproducable. The author of Mercurial seems to think this issue was the same one I brought up a year ago in Copenhagen and was fixed in version 2.2.2 with relevant commit available here. http://www.selenic.com/hg/rev/7bf48bc7de23
I'm hoping this issue will get resolved when we upgrade later this week. Unfortunately since this is unreproducible now we'll have to resolve it.
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → INCOMPLETE
Assignee: server-ops-devservices → bkero
Component: Server Operations: Developer Services → General
Product: mozilla.org → Developer Services
You need to log in before you can comment on or make changes to this bug.