Closed Bug 1272480 Opened 9 years ago Closed 9 years ago

OrangeFactor bug commenter broken since nginx changes on 2016-05-03

Categories

(Tree Management Graveyard :: OrangeFactor, defect)

Product:
Tree Management Graveyard
Component:
OrangeFactor
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jwalker, Assigned: emorley)

References

Details

Did I interpret that right Ed?
Flags: needinfo?(emorley)
The data that appears in bugzilla is aggregated periodically (weekly?) and so it's expected that the most recent occurrences may not appear there.
It also only comments if you reach a daily or weekly minimum threshold of failures.
The thresholds and periodic nature are true (for more details see: ), however in this instance the bug commenter has actually started failing. The last successful run was at 2016-05-03 02:00 UTC+1.. Getting bug data... -> Fetching JSON from http://localhost/orangefactor/api/bybug?tree=all&startday=2016-05-02&endday=2016-05-02 Submitting comment to bug 1268163 (15 occurrences) ... Complete! Getting bug data... -> Fetching JSON from http://localhost/orangefactor/api/bybug?tree=all&startday=2016-05-03&endday=2016-05-03 HTTPError 404 fetching http://localhost/orangefactor/api/bybug?tree=all&startday=2016-05-03&endday=2016-05-03: <html> <head><title>404 Not Found</title></head> <body bgcolor="white"> <center><h1>404 Not Found</h1></center> <hr><center>nginx/1.2.8</center> </body> </html> ...which was just prior to :atoll changing the nginx config (and recompiling nginx) to fix the certificate issues with the custom version of nginx that was running on there previously: 428 2016-04-28 19:06:26 vmware-config-tools.pl 429 2016-05-03 17:13:31 cd /etc/nginx/ 430 2016-05-03 17:13:32 ls -al 431 2016-05-03 17:13:33 cd conf.d 432 2016-05-03 17:13:34 ls -al 433 2016-05-03 17:13:35 vim orangefactor.conf 434 2016-05-03 17:13:43 cd /usr/local/nginx/ssl 435 2016-05-03 17:13:43 ls -al 436 2016-05-03 17:13:46 less brasstacks.mozilla.org.pem 437 2016-05-03 17:13:51 cp brasstacks.mozilla.org.pem ~/ 438 2016-05-03 17:13:52 cd ~/ 439 2016-05-03 17:13:53 vim brasstacks.mozilla.org.pem 440 2016-05-03 17:14:01 cd - 441 2016-05-03 17:14:02 ls -al 442 2016-05-03 17:14:04 mv brasstacks.mozilla.org.pem{,.orig} 443 2016-05-03 17:14:06 mv ~/brasstacks.mozilla.org.pem . 444 2016-05-03 17:14:07 ls -al 445 2016-05-03 17:14:10 service nginx graceful 446 2016-05-03 17:14:14 service nginx restart 447 2016-05-03 17:20:29 ls -al 448 2016-05-03 17:20:31 cd /etc/httpd 449 2016-05-03 17:20:32 ls -al 450 2016-05-03 17:20:33 cd /etc/nginx 451 2016-05-03 17:20:35 ls -al 452 2016-05-03 17:20:40 vim nginx.conf 453 2016-05-03 17:20:44 cd conf. 454 2016-05-03 17:20:46 cd conf.d 455 2016-05-03 17:20:46 ls -al 456 2016-05-03 17:20:47 vim orangefactor.conf 457 2016-05-03 17:21:34 service nginx restart 458 2016-05-03 17:22:34 fg 459 2016-05-03 17:23:57 service nginx restart 460 2016-05-03 17:24:03 fg 461 2016-05-03 17:24:49 service nginx restart 462 2016-05-03 17:25:23 ls -al 463 2016-05-03 17:25:25 fg 464 2016-05-03 17:26:25 cd /var/log/nginx 465 2016-05-03 17:26:26 ls -al 466 2016-05-03 17:26:27 tail error.log 467 2016-05-03 17:26:39 date 468 2016-05-03 17:28:26 dpkg -L | grep openssl 469 2016-05-03 17:35:37 vim orangefactor.conf 470 2016-05-03 17:35:39 cd - 471 2016-05-03 17:35:40 vim orangefactor.conf 472 2016-05-03 17:35:48 service nginx restart 473 2016-05-03 17:35:58 fg 474 2016-05-03 17:38:03 apt-get install openssl 475 2016-05-03 17:38:05 yum 476 2016-05-03 17:38:08 yum install openssl 477 2016-05-03 17:38:57 openssl ec 478 2016-05-03 17:39:07 rpm -qi nginx 479 2016-05-03 17:39:09 yum install nginx 480 2016-05-03 17:53:03 cd ~/ 481 2016-05-03 17:53:06 yum upgradeonly 482 2016-05-03 17:53:10 yum install --upgradeony 483 2016-05-03 17:53:12 yum install --upgradeonly 484 2016-05-03 17:53:15 yum install --help 485 2016-05-03 17:53:30 yum upgrade --help 486 2016-05-03 17:53:32 ls 487 2016-05-03 17:53:36 rpm -qa | grep nginx 488 2016-05-03 17:53:38 cd ~rsoderberg/ 489 2016-05-03 17:53:39 ls 490 2016-05-03 17:54:22 yum upgrade nginx-1.2.8-1.el6.ngx.x86_64.rpm 491 2016-05-03 17:54:33 service nginx 492 2016-05-03 17:54:35 service nginx configtest 493 2016-05-03 17:54:40 service nginx restart 494 2016-05-03 17:55:13 cd /etc/nginx 495 2016-05-03 17:55:14 ls 496 2016-05-03 17:55:16 cd conf.d 497 2016-05-03 17:55:16 ls 498 2016-05-03 17:55:18 vim orangefactor.conf 499 2016-05-03 17:55:47 service nginx restart 500 2016-05-03 18:25:48 which nginx 501 2016-05-16 12:19:07 history | less 502 2016-05-16 12:19:28 su - webtools 503 2016-05-16 12:30:38 history | tail -n 100 IRC log: 18:09 <atoll> is httpd intentionally off? 18:09 <emorley> we're using nginx 18:09 <atoll> the site is down 18:09 <emorley> wfm? https://brasstacks.mozilla.com/orangefactor/ 18:09 <atoll> so.. if that's cool, wfm 18:09 <atoll> huh 18:10 <atoll> whaaat 18:10 — atoll pokes at something 18:10 <atoll> http://brasstacks.mozilla.com 18:10 <atoll> doesn't work for me. 18:10 <atoll> https, either 18:10 <atoll> https://brasstacks.mozilla.com 18:10 <atoll> oh, wow 18:10 <atoll> i bet i have SHA1 disabled or something 18:11 <atoll> nope. trippy. 18:11 <atoll> oh, yeah, this is the one with the damaged SSL config 18:11 <emorley> I have us using the more aggressive config from https://wiki.mozilla.org/Security/Server_Side_TLS#Modern_compatibility 18:11 <atoll> it's apparently incompatible with OS X as deployed, which is interesting 18:12 — atoll pokes around a bit 18:12 <atoll> your intermediate cert is deployed incorrectlyt 18:12 <atoll> let me get a correct file,s ec 18:12 <emorley> thank you 18:13 — atoll general purpose diagnostic machine 18:13 <emorley> someone kindly set it up post-renewal recently 18:14 <emorley> config in /etc/nginx/conf.d/orangefactor.conf 18:14 <atoll> okay, i fixed the intermediate on the server 18:14 <emorley> great :-) 18:14 <atoll> i'll keep looking about the osx thing, but will report back without making changes on that part 18:14 <atoll> more later! 18:15 <emorley> I'll have another look at what SSL labs says 18:15 <atoll> it says A+ now 18:15 <atoll> intermediate is repaired 18:15 <emorley> I'd checked before and it said A+ then, strange 18:16 <atoll> if you look in the output down by IE 11 and Safari, you can see the various handshake_failures, which is interesting to me 18:16 <atoll> yeah, the root in intermediate thing isn't a - ... 18:22 <atoll> i tried removing the single quotes from the cipherstring first, since they might be screwing it up 18:23 <atoll> nope, no change ... 18:25 <atoll> i have ulfr on the other side trying to understand what in our config is breaking this 18:25 <atoll> since he authors the config generator ... 18:37 <atoll> okay, so 18:37 <atoll> your server doesn't support ECDHE 18:37 <atoll> so you aren't capable of running the high security config 18:38 <atoll> and that's why it's collapsing 18:40 <emorley> strange since http://nginx.org/en/CHANGES says nginx 1.1.0 supports ECDHE 18:41 <atoll> yeah, but redhat didn't at the time 18:41 <atoll> and we're running some stupid self compiled nginx 18:41 <atoll> fffff ... 18:52 <atoll> okay, i have nginx 1.2.8 available now. 18:52 <atoll> recompiled with ec support, presumably 18:52 <atoll> shall i found out? 18:52 <atoll> ^find out? 18:52 <emorley> yeah might as well! 18:54 <atoll> there you go 18:54 <atoll> works in safari now 18:55 <atoll> bumping up your ssl config to the latest High one 18:58 <emorley> amazing, thank you :-)
Assignee: nobody → emorley
Component: Treeherder → OrangeFactor
Flags: needinfo?(emorley)
Summary: Filtering OrangeFactor by mochitest-devtools-chrome-8 gives results inconsistent with bugzilla → OrangeFactor bug commenter broken since nginx changes on 2016-05-03
(In reply to Ed Morley [:emorley] from comment #4) > The thresholds and periodic nature are true (for more details see: ) https://groups.google.com/d/msg/mozilla.dev.tree-management/az643p0u4hs/3el7fqIDBwAJ -- So the nginx served static files are fine being served from localhost: [emorley@brasstacks1.dmz.scl3 ~]$ curl -is http://localhost/orangefactor/ | head -n 20 HTTP/1.1 200 OK Server: nginx/1.2.8 Date: Mon, 16 May 2016 12:44:21 GMT Content-Type: text/html Content-Length: 10757 Last-Modified: Tue, 15 Mar 2016 16:28:41 GMT Connection: keep-alive Accept-Ranges: bytes <!DOCTYPE html> <html> <!-- This Source Code is subject to the terms of the Mozilla Public License - version 2.0 (the "License"). You can obtain a copy of the License at - http://mozilla.org/MPL/2.0/. --> <head> <title>OrangeFactor</title> <link rel="icon" type="image/vnd.microsoft.icon" href="favicon.ico" /> <link rel="stylesheet" href="style/main.css" type="text/css"/> <link rel="stylesheet" href="vendor/jquery.datepick.css" type="text/css"/> <link rel="stylesheet" href="vendor/datatables.css" type="text/css"/> ...just not the fastcgi passthrough to the OrangeFactor API: [emorley@brasstacks1.dmz.scl3 ~]$ curl -is http://localhost/orangefactor/api/bybug | head -n 20 HTTP/1.1 404 Not Found Server: nginx/1.2.8 Date: Mon, 16 May 2016 12:45:11 GMT Content-Type: text/html Content-Length: 168 Connection: keep-alive <html> <head><title>404 Not Found</title></head> <body bgcolor="white"> <center><h1>404 Not Found</h1></center> <hr><center>nginx/1.2.8</center> </body> </html> The nginx config: [emorley@brasstacks1.dmz.scl3 ~]$ cat /etc/nginx/conf.d/orangefactor.conf server { listen 80; server_name localhost; include /etc/nginx/default_locations/*.conf; } server { listen 443; server_name brasstacks.mozilla.com; include /etc/nginx/default_locations/*.conf; ssl on; ssl_certificate SNIP; ssl_certificate_key SNIP; # New config from https://mozilla.github.io/server-side-tls/ssl-config-generator/ ... ssl_session_timeout 1d; ssl_session_cache shared:SSL:50m; # Diffie-Hellman parameter for DHE ciphersuites, recommended 2048 bits ssl_dhparam SNIP; # Updated modern. 20160503 atoll ssl_protocols TLSv1.2; ssl_ciphers SNIP; ssl_prefer_server_ciphers on; # HSTS (ngx_http_headers_module is required) (15768000 seconds = 6 months) add_header Strict-Transport-Security max-age=15768000; } The locations config: [emorley@brasstacks1.dmz.scl3 ~]$ cat /etc/nginx/default_locations/orangefactor.conf location / { root /usr/share/nginx/html; index index.html index.htm; rewrite ^/$ https://brasstacks.mozilla.com/orangefactor/ permanent; rewrite ^/index.html$ https://brasstacks.mozilla.com/orangefactor/ permanent; } location /orangefactor/api/ { fastcgi_ignore_client_abort on; fastcgi_read_timeout 300; fastcgi_param REQUEST_METHOD $request_method; fastcgi_param QUERY_STRING $query_string; fastcgi_param CONTENT_TYPE $content_type; fastcgi_param CONTENT_LENGTH $content_length; fastcgi_param GATEWAY_INTERFACE CGI/1.1; fastcgi_param SERVER_SOFTWARE nginx/$nginx_version; fastcgi_param REMOTE_ADDR $remote_addr; fastcgi_param REMOTE_PORT $remote_port; fastcgi_param SERVER_ADDR $server_addr; fastcgi_param SERVER_PORT $server_port; fastcgi_param SERVER_NAME $server_name; fastcgi_param SERVER_PROTOCOL $server_protocol; fastcgi_param SCRIPT_FILENAME $fastcgi_script_name; fastcgi_split_path_info ^(/orangefactor)(.*)$; fastcgi_param PATH_INFO $fastcgi_path_info; fastcgi_pass 127.0.0.1:9500; } The root account history doesn't show any changes being made to the locations config, so guessing there must have been a breaking change as part of updating from nginx 1.2.0 to 1.2.8 ? Looking at the changelog there only seem to have been 'fixes' to bugs in the fastcgi handling however: http://nginx.org/en/CHANGES-1.2
I've just worked around this - and updated woo_cron.conf such that the cron jobs use: local_server_url = https://brasstacks.mozilla.com/orangefactor/api ...rather than: local_server_url = http://localhost/orangefactor/api I've re-run the daily and weekly bug comment jobs and also the weekly email job now (though no easy way to re-run for prior days, but I think that's fine - the original data is still available on OrangeFactor).
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Product: Tree Management → Tree Management Graveyard
You need to log in before you can comment on or make changes to this bug.