965323 - Make http://esfrontline1.bugs.scl3.mozilla.com:9292 publicly accessible

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: Infrastructure, defect)

Description

[Kyle Lahnakoski [:ekyle]]

Private bugs are not showing in the public cluster anymore.  There is still an outstanding legal bug.

Is this just an ACL thing, or is more involved?

Comment 1

[Kyle Lahnakoski [:ekyle]]

Added mark and curtis, in case there are any last questions.

Comment 2

[Kendall Libby [:fubar] (he/him)]

As I recall, the sec review stuff is all set, so...

1) what URL do we want people to access this through? esfrontline.mozilla.org? esfrontline.bugzilla.m.o? something else?
2) does it need a cert for https?
3) do we need to keep it on port 9292, or can we move it to 80? If not, it'll likely require an ACL change for zeus to access it.

I think those are the big ones off the top of my head.

Comment 3

[Kyle Lahnakoski [:ekyle]]

1) esfrontline.bugzilla.m.o  seems good.  The "bugzilla" name hints as to what it's for/
2) No cert required.  This is read only public data.
3) Any port will be fine

Comment 4

[Kendall Libby [:fubar] (he/him)]

http://esfrontline.bugzilla.mozilla.org/ is all set; will go add monitoring.

Comment 5

[Kyle Lahnakoski [:ekyle]]

I am unable to connect to http://esfrontline.bugzilla.mozilla.org:80/

Comment 6

[Kendall Libby [:fubar] (he/him)]

<title>400 Bad Request</title>
<h1>Bad Request</h1>
<p>The browser (or proxy) sent a request that this server could not understand.</p>

and logged on the server:

2014-02-04 07:23:56,598 - esFrontLine - WARNING - request must be of form: {index_name} "/" {type_name} "/_search" 
2014-02-04 07:23:56,598 - esFrontLine - WARNING - Not allowed: :

Using the POST check from bug 963883, I get:
$curl -d @foo -XPOST http://esfrontline.bugzilla.mozilla.org/public_bugs/bug_version/_search
{"took":2800,"timed_out":false,"_shards":{"total":3,"successful":3,"failed":0},"hits":{"total":113958,"max_score":0.0,"hits":[]},"facets":{"0":{"_type":"statistical","count":113958,"total":1.58415154826988992E17,"min":1.388536127E12,"max":1.391527192E12,"mean":1.3901187703100176E12,"sum_of_squares":2.2021595154604802E29,"variance":6.2584007279647014E17,"std_deviation":7.911005453142288E8}}}

And in the logs:
2014-02-04 07:29:04,981 - esFrontLine - DEBUG - path: public_bugs/bug_version/_search, request bytes=288, response bytes=393
2014-02-04 07:29:06,920 - esFrontLine - WARNING - request must be of form: {index_name} "/" {type_name} "/_search" 
2014-02-04 07:29:07,027 - esFrontLine - WARNING - Not allowed: :


So, it's getting the traffic, but something's not right and I'm not sure what it is from the error. The ES cluster appears to be happy, though.

Comment 7

[Kyle Lahnakoski [:ekyle]]

Your analysis looks good, the esfrontline is responding like it should (it will respond with 400 on everything but reasonable search).  

I still can not get any response.  I tried both when connected to MozillaVPN and when not.  Do you have a machine on the public internet to test this with?

Comment 8

[Kendall Libby [:fubar] (he/him)]

I get the same results as above from my offsite VM. Check your DNS? 

planet10$ host esfrontline.bugzilla.mozilla.org
esfrontline.bugzilla.mozilla.org is an alias for bugzilla-esfrontline.vips.scl3.mozilla.com.
bugzilla-esfrontline.vips.scl3.mozilla.com has address 63.245.215.67

Comment 9

[Mark Côté [:mcote]]

FYI this works for me (taken from comment 19 of bug 931066):

$ curl 'http://esfrontline.bugzilla.mozilla.org/public_bugs/bug_version/_search' -X POST -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H 'Accept-Encoding: gzip, deflate' -H 'Accept-Language: en-US,en;q=0.5' -H 'Cache-Control: no-cache' -H 'Connection: keep-alive' -H 'Content-Type: text/plain; charset=UTF-8' -H 'Host: esfrontline1.bugs.scl3.mozilla.com:9292' -H 'Origin: null' -H 'Pragma: no-cache' -H 'User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:26.0) Gecko/20100101 Firefox/26.0' -d '{ "query":{"filtered":{ "query":{"match_all":{}}, "filter":{"and":[{"match_all":{}},{"exists":{"field":"bug_id"}}]} }}, "from":0, "size":0, "sort":[], "facets":{} }'

{"took":368,"timed_out":false,"_shards":{"total":3,"successful":3,"failed":0},"hits":{"total":8112699,"max_score":0.0,"hits":[]}}

Comment 10

[Kyle Lahnakoski [:ekyle]]

Discussed with fubar: it works in the browser, but only from http://bugzilla-esfrontline.vips.scl3.mozilla.com:80/ I am guessing the browser sees the DNS redirect and prevents the http request (save user from dns hijacking?)

Comment 11

[Kendall Libby [:fubar] (he/him)]

DNS for esfrontline.bugzilla.mozilla.org was changed from a CNAME to an A record; please let me know if it works now.

Comment 12

[Kyle Lahnakoski [:ekyle]]

YEA!!  \o/   IT WORKS!!

Here is a dashboard made using the public API:

http://claw.cs.uwaterloo.ca/~okononen/

Comment 13

[Kyle Lahnakoski [:ekyle]]

http://esfrontline.bugzilla.mozilla.org:80/public_bugs/bug_version/_search