1506144 - (bmo-stmo) Access bugzilla security bug metadata via STMO

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: General, enhancement, P1)

Description

[Simon Bennetts [:psiinon]]

We (the Firefox security teams) would like to be able to access security bug metadata via STMO so that we can create live dashboards similar to the static info Dan is now emailing.

The plan is to use the Secops metrics pipeline - that already gathers a growing set of security metrics from a variety of sources and makes them accessible via STMO.

The current pipleline is:
* Scripts run in the Ops Jenkins server and collect raw metrics
* They write the raw data into folders under s3://foxsec-metrics 
* Post processing is applied if we need to reformat it (for Athena)
* Athena tables are applied to the relevant folders and added to the foxsec_metrics db
* The foxsec_metrics db is a data source in STMO

The foxsec_metrics data source in STMO is currently restricted to the Secops team, but the plan is to expand that to all of the Firefox security teams.
We also plan to make some STMO visualisations on this data accessible to everyone in Mozilla.

We would like a subset of the available bugzilla fields so that we don't inadvertently expose details of unfixed security bugs.
The bugs that we are interested in and the fields we would like are currently being collated in this gdoc: https://docs.google.com/document/d/1v5HZlH77PHcPSup0gDJTKUKzuCBE3ltGVwgz0d_DeU0/edit#
Note that right now this is work in progress - I'll update this bug with that info once we have finalized it.

We will need to be able to combine bugzilla data with data from the foxsec_metrics db. I'm not sure if we can combine multiple data sources in STMO.

Comment 1

[Simon Bennetts [:psiinon]]

:robotblake - can you confirm if we can combine data from multiple data sources in the same query in STMO?

Comment 2

[Hal Wine [:hwine] use NI!]

:robotblake - we also would want to query both data sources in redash, so if that's a limitation, please let us know

Comment 3

[Dylan Hardison [:dylan] (he/him)]

Is this really security sensitive?

Comment 4

[Simon Bennetts [:psiinon]]

:dylan, problably not ;) Just internal then?

Comment 5

[Simon Bennetts [:psiinon]]

A meeting was held on Nov 14th and the above doc updated with notes.
:dylan - you were down to define the schema that BMO will use to provide access to the data. Any eta on this?

Comment 6

[Dylan Hardison [:dylan] (he/him)]

I plan on hashing out a gdoc today. Sorry for the delay.
Also can we drop the groups from this bug? security bugs in this product in my inbox give me anxiety.

Comment 7

[Hal Wine [:hwine] use NI!]

decisions & approach on BMO access in meeting notes:
  https://docs.google.com/document/d/1v5HZlH77PHcPSup0gDJTKUKzuCBE3ltGVwgz0d_DeU0/edit#heading=h.6526yk3oj16u

Comment 8

[Simon Bennetts [:psiinon]]

Any progress on the schema Dylan?

Comment 9

[Dylan Hardison [:dylan] (he/him)]

I hope to hop back on this soon.

Comment 10

[Dylan Hardison [:dylan] (he/him)]

Attachment: Framework for sending data to STMO

Comment 11

[Dylan Hardison [:dylan] (he/him)]

Attachment: Database model helpers

Comment 12

[Dylan Hardison [:dylan] (he/him)]

Attachment: Add DBIx::Class

Comment 13

[Simon Bennetts [:psiinon]]

:dylan I'm afraid I dont understand the BMO schema / API definition enough to understand if the code in the above attachment includes all of fields we highlighted in the gdoc: https://docs.google.com/document/d/1v5HZlH77PHcPSup0gDJTKUKzuCBE3ltGVwgz0d_DeU0/edit#heading=h.9zk2sxea4fs
Can you confirm if all of the fields we highlighted will be included in the schema or if not which fields will be missing?
I'm not concerned about the structure of the data, we can manipulate that in STMO, its just whether the raw info will be there.

Comment 14

[Dylan Hardison [:dylan] (he/him)]

we chatted about this, and the PR has a revised list.

Comment 15

[Simon Bennetts [:psiinon]]

The fields we need are:

blocks
component
creation_time
creator
depends_on
dupe_of
duplicates
flags
groups
id
is_open
keywords
last_change_time
last_resolved
priority
product
resolution
severity
bug_status
target_milestone
whiteboard

Ideally we'd also like:

platform

:dylan does that help?

Comment 16

[Dylan Hardison [:dylan] (he/him)]

Attachment: GitHub Pull Request

Comment 17

[Dylan Hardison [:dylan] (he/him)]

(In reply to Simon Bennetts [:psiinon] from comment #15)

The first version omits the following fields, which will be added in a future update.

The fields we need are:
creation_time

Missing due to dates being difficult.

is_open

Missing because it's non-trivial to expose.

last_change_time

Because it's a date.

last_resolved

Because it's added by an extension

whiteboard

Because it's added by an extension.

platform

Missing because it was overlooked.