Closed Bug 879822 Opened 9 years ago Closed 8 years ago

Store Bugzilla data in Elastic Search

Categories

(Testing :: General, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: mcote, Unassigned)

References

Details

Attachments

(1 file, 2 obsolete files)

This is a tracking bug for the steps required to set up an ElasticSearch database containing the same kind of data as metrics' internal cluster, but publicly available and thus restricted to public bugs only.

Background copied from bug 872363:

metrics has an ES cluster that contains not only the current state of Bugzilla, but past history as well.  ElasticSearch provides a better method of performing certain types of analyses, and use of it doesn't impact the main Bugzilla database.  However, it contains *some* information on confidential bugs, so it is behind LDAP, which means that any tools that use it generally must be behind LDAP as well, which limits their usefulness.

It would thus be really great to have a publicly accessible ES cluster which contains *only* data on public bugs (or to word it another way, *no* data on confidential bugs of any type).

mcoates has already signed off on this idea from a security standpoint, provided that bugs with any limitations on visibility are excluded, and that all data concerning bugs that are made confidential some time after creation (i.e. were at one point public, but no longer) is removed in a timely fashion.
Depends on: 879833
Depends on: 879835
Depends on: 922477
Redefining this bug a little to reflect expanded scope.

Since metrics would like to decommission their Bugzilla ES cluster entirely, we will need a replacement.  This will be a FULL mirror of Bugzilla data, including confidential bugs, and hence any applications using it MUST be behind LDAP.  This is how the current metrics-run cluster works.

This is in addition to the cluster containing only public bugs.  We can use the same ETL software for both; I'm not sure if we will need two separate machines for adequate processing power, though.  Filing request for machine separately.
Summary: ElasticSearch cluster for public bugs → Store Bugzilla data in Elastic Search
Depends on: 922877
What will it mean to be behind LDAP? What LDAP accounts will have access to it?
Josh,

There are now two parts.  The first is public with confidential bugs removed on ES cluster.   

The second ES cluster includes confidential bugs, but with summary and comments removed.   This second one will remain behind LDAP.  I believe anyone with LDAP access should be allowed access to this data, just like the current Metrics setup.  Security may have another opinion.

Both clusters will be fed by one common piece of ETL code (https://github.com/klahnakoski/Bugzilla-ETL)
Yeah, as Kyle said, right now the metrics cluster is accessible by anyone with an LDAP account.  We can revisit this if necessary.  At least Kyle and I will need access to it for starters.
Blocks: 885464
Attached file Architecture Drawing (v1) (obsolete) —
Please add suggestions
Attachment #828667 - Flags: feedback?(mcote)
Comment on attachment 828667 [details]
Architecture Drawing (v1)

I think this is good--however I would much prefer the diagram to be in a standard picture format like PNG.
Attachment #828667 - Flags: feedback?(mcote) → feedback+
Just found out that we *can* create Gliffy diagrams in mana; you just have to have the correct permissions for the space.  I suggest creating your own space (in the drop-down, top right) and using Gliffy to publish the diagram there.
Attached file Architecture Drawing (v2) (obsolete) —
Added port numbers, and client/server relationships.  Showed LDAP machines as cloud, and Kyle's ES Cluster specifically.
Attachment #828667 - Attachment is obsolete: true
This looks a little off in LibreOffice.  Can you export to a PNG?
Attached image Architecture v3
png is easier to view
Attachment #8337726 - Attachment is obsolete: true
Depends on: 965329
\o/

An example of a public dashboard:  http://claw.cs.uwaterloo.ca/~okononen/
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Depends on: 968318
Depends on: 968848
Depends on: 970309
Depends on: 971421
Depends on: 976128
Depends on: 976408
Depends on: 977041
Depends on: 977042
Depends on: 977059
Depends on: 982249
Depends on: 982509
Depends on: 982511
Depends on: 982512
Depends on: 982604
Depends on: 982617
You need to log in before you can comment on or make changes to this bug.